Comment by amelius

Could be! I have not (yet) spent much time learning about how llms work, just the occasional blog here and there. My main question is why we _need_ a bit of additional code to massage the input into tokens and especially why the neural network cannot do it, i.e., let the embedding be a latent space that forms naturally when training the network. If that makes sense.