Chunks can only be as large as the embedding model’s token limit, about 512-1024 tokens usually. Anything longer gets truncated.
Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.
Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.