Comment Comment by dweinus If I understand the paper right...At indexing time:- run LLM over every data point multiple times ("gleanings") for entity extraction and constructing a graph index- run an LLM over the graph multiple times to create clusters ("communities")At query time:- Run the LLM across all clusters, creating an answer from each and score them- Run the LLM across all but the lowest scoring answers to produce a "global answer"...aren't the compute requirements here untenable for any decent sized dataset? parent dweinus 2 years ago
At indexing time:
- run LLM over every data point multiple times ("gleanings") for entity extraction and constructing a graph index
- run an LLM over the graph multiple times to create clusters ("communities")
At query time:
- Run the LLM across all clusters, creating an answer from each and score them
- Run the LLM across all but the lowest scoring answers to produce a "global answer"
...aren't the compute requirements here untenable for any decent sized dataset?