What RAG document chunking strategies do you all use?
I'm using RAG for internal document search, and the answer quality varies way too much depending on how I set the chunking unit.
Right now I've fixed it at around 500 tokens with 100 overlap, but it feels like that doesn't cover documents with tables or code mixed in.
Is there anyone who takes a different chunking approach by document type? Or I'm curious whether you use a method where you first split by headers and then split again from there.
3 answers
Docs with tables really can't be covered with 500 tokens, fr. I went through hell because of that too.
I do a first pass split based on Markdown/HTML header levels, and if a section exceeds 800 tokens, I split it again by paragraph. I only apply about 10–15% overlap during the second split, and I exclude tables and code blocks from splitting entirely, storing them as single chunks. For tables, if they lose their surrounding context, even if they get retrieved, the answer won't match. If you put the document name, section path, and type (table/code/body) into the chunk metadata, it makes filtering and prompt construction much easier. Since switching from a fixed 500 to this approach, accuracy on table-related questions has jumped a lot. That said, if document formats are all over the place, the preprocessing pipeline becomes hell, so be prepared for that...
Well, if you keep fixating on the chunking strategy, there’s no end in sight. A fixed 500 tokens isn’t the worst option, and it’s actually more likely a retrieval-stage issue. Adding a reranker and increasing top-k will probably feel like a bigger win than tearing apart your chunking. Did you actually measure quantitatively that changing the chunking made things better? Judging only by feel often leads to wasted effort.