> The tokenization speed issue is already addressed by the language server protocol [...]
It's not really addressed. The semantic tokens API is intended for semantic highlighting:
> Semantic tokenization allows language servers to provide additional token information based on the language server's knowledge on how to resolve symbols in the context of a project.
Abusing the semantic tokens API for syntactic highlighting is slow, unnecessarily complex (why do I need to implement a language server just to do syntactic highlighting?), and only a partial solution (still need a TM grammar, still don't get correct code folding, etc.).
It's not really addressed. The semantic tokens API is intended for semantic highlighting:
> Semantic tokenization allows language servers to provide additional token information based on the language server's knowledge on how to resolve symbols in the context of a project.
Abusing the semantic tokens API for syntactic highlighting is slow, unnecessarily complex (why do I need to implement a language server just to do syntactic highlighting?), and only a partial solution (still need a TM grammar, still don't get correct code folding, etc.).