chiccomagnus

IMHO, your article is missing an important point: 90% of implementations today flatten documents to plain text before chunking them. Why not consider the visual appearance that the human gave to the document? Using layout information combined with semantics, you can increase rag performances by +160% (tested via benchmarks), so why do most of us only use text?

Note: multimodal ≠ layout

How does credits works?

If you don't want to reinvent the wheel, we have built exactly that, goggle "Preprocess"

Have you compared this solution with tools like Preprocess, Reducto, etc.. ? I'm curious about the performance gain you can achieve with your approach

It seems it tries to always extract tables even if the content is just text. Is not working and at a similar price you can get high performing solutions like preprocess.co and similars.

I'm curious to know how you handle the chunking of complex documents

If for "documents" you mean PDF, Word, Powerpoints, Excel you should try preprocess.co

If that’s not what you meant, can you please clarify?

Do you use naive chunking? Have you tried something else?

By chance, have you tried preprocess.co for text extraction + chucking?

I see a huge missing point here: real world files are PDF, Word, PowerPoint and Excel, not only plain text.

There are more sophisticated chunking strategies, you will lose lots of context like that

Well-written article, missing key considerations:

- Titles matter, a lot: if you add the title of the section at the start of each chunk you will get 10x better embeddings and so more accurate results.

- The size doesn't matter: It depends on the combination of the layout and semantics of the content.

- Avoid garbage in / out: increased context windows don't mean you can put trash inside them. The more good you are at putting relevant information the more precise answers you get. Especially for enterprise-grade solutions, this is so important.

There are good emerging API solutions that implement semantic + layout-based chunking, which in my opinion is the best chunking strategy for PDF / Office files (the widest use case scenario for enterprises).

The real point that has been completely ignored is that data for real applications come from Office and PDF files, and making them plain text throws all the visual information.