- If you work with the raw bytes in a non-hierarchical (isotropic) model, such as a byte transformer, you get better transfer to downstream tasks, but at a huge computational cost.
- if you use a tokenizer (pretrained or regexp based), you gain a lot of computational efficiency, but loose semantics and the transfer to downstream tasks may degrade
- finally, with H-Nets you get the best of both worlds. The model learns how to chunk the data in a better way than a hardcoded/separately trained tokenizer
Authors also demonstrate that this approach works in other modalities, such as DNA sequences and images, where we don't normally have lots of ready to use tokenizers.
- If you work with the raw bytes in a non-hierarchical (isotropic) model, such as a byte transformer, you get better transfer to downstream tasks, but at a huge computational cost.
- if you use a tokenizer (pretrained or regexp based), you gain a lot of computational efficiency, but loose semantics and the transfer to downstream tasks may degrade
- finally, with H-Nets you get the best of both worlds. The model learns how to chunk the data in a better way than a hardcoded/separately trained tokenizer
Authors also demonstrate that this approach works in other modalities, such as DNA sequences and images, where we don't normally have lots of ready to use tokenizers.