Comment by HanClinto

Can you give me an example of the sort of thing you're talking about? I've been using Defog's sql-eval a little bit, but I'd be interested in knowing more about its shortcomings when evaluating these systems.

https://github.com/defog-ai/sql-eval