My notes:
It's a 3 component model.
- Encoder: Takes byte groupings and outputs a hidden state/encoding called patches
- Transformer: Takes these encodings of patches in autoregressive fashion
- Decoder: Takes processed encodings by transformers and outputs bytes
Loss is on byte to byte crossentropy (Next byte prediction)
How they group bytes.
- Use entropy thresholds: If a sequence of bytes have entropy lower than a threshold, group them
- This is a learned model (from data)
Why this helps over current byte-pair tokenization in LLMs.
- Encoder/decoder essentially act as “learnable” tokenization scheme
- Better efficiency tradeoffs (as for highly predictable sequence of bytes, encoder can “offload” computation effort from the main transformer)
- History teaches us that end to end learned system beats human designed mechanisms
It's a 3 component model.
- Encoder: Takes byte groupings and outputs a hidden state/encoding called patches
- Transformer: Takes these encodings of patches in autoregressive fashion
- Decoder: Takes processed encodings by transformers and outputs bytes
Loss is on byte to byte crossentropy (Next byte prediction)
How they group bytes.
- Use entropy thresholds: If a sequence of bytes have entropy lower than a threshold, group them
- This is a learned model (from data)
Why this helps over current byte-pair tokenization in LLMs.
- Encoder/decoder essentially act as “learnable” tokenization scheme
- Better efficiency tradeoffs (as for highly predictable sequence of bytes, encoder can “offload” computation effort from the main transformer)
- History teaches us that end to end learned system beats human designed mechanisms