Comment by znnajdla

This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.