I tried out the Assistants API and noticed that similarly bad performance, but with a catch. Apparently if you combine all the files into one single text file, then the performance is amazing. But if it's spread across multiple files the performance is pretty bad.
Analysis here if anyone is curious https://news.ycombinator.com/item?id=38280718