Comment Comment by Chu4eeno No, that's not how anything works.You can finetune an LLM to a new tokenizer by nudging just a few layers (even wildly different kinds of tokens), and there's nothing stopping a lab from using someone else's tokenizer. parent Chu4eeno 2 months ago
You can finetune an LLM to a new tokenizer by nudging just a few layers (even wildly different kinds of tokens), and there's nothing stopping a lab from using someone else's tokenizer.