Comment by jshobrook

You can use the AST to chunk the code: https://docs.sweep.dev/blogs/chunking-2m-files