Comment by p1esk

There’s has a ton of work on character or byte level encodings for llms. The problem is you expand your input tokens by 3-4x. Expensive.

Also, you still need token embeddings (I think you might be confused how that works).