The tokenization speed issue is already addressed by the language server protocol, which delegates syntax highlighting to language servers via the "semantic tokens" API. Language servers can choose to implement this however they want, and the API allows for incremental additions.
I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specific grammars) to add syntax highlighting. That way it can be applied to any editor that supports LSP integration. Really the world would be a better place for IDEs and text editors if we could just get LSP more standardized, and VS Code's dominance is a decent place to start with it. I'm tired of relying on editor and IDE authors for language support.
I think this is a much better approach that baking tree sitter into VS Code and continuing to use TextMate grammars (or tree sitter specific grammars) to add syntax highlighting. That way it can be applied to any editor that supports LSP integration. Really the world would be a better place for IDEs and text editors if we could just get LSP more standardized, and VS Code's dominance is a decent place to start with it. I'm tired of relying on editor and IDE authors for language support.