A question to the author. Can you perform an ablation study with respect to the chunks? In other words, if you put in the context irrelevant/random chunks from the document would the quality of answers decrease/stay similar?
Potential issue might be that chunks just serve to activate massive knowledge of GPT4 and not actually used as a basis for an answer. For example, GPT4 has surely seen Dune in its training corpus and could be answering from memory.
Potential issue might be that chunks just serve to activate massive knowledge of GPT4 and not actually used as a basis for an answer. For example, GPT4 has surely seen Dune in its training corpus and could be answering from memory.