Comment by gergelycsegzi

100% the really hard challenge is that the intermediate representation (ie the parquet equivalent) will be dependent on the given use case. So what we do with the platform is have the users configure the intermediate layer that serves most of their queries, and if they need to extend it we will suggest it for them. For example for the demo on the grounded reasoning benchmark I referred to, here is what the intermediate layer looks like on top of which the agents can more efficiently query: https://demo.parsewise.ai/projects/39bee9d8-d722-4b23-8894-e...