Three Weeks of Building Internal Document Search with an LLM
There were too many internal wiki pages and PDF manuals, so I started with, 'Let’s just bolt on vector search,' and ended up doing much more trial and error than I expected. I’m writing this up for anyone who tries it later.
1. Chunking is 80% of the work
Splitting by paragraph was the most reliable approach. When I split into fixed 500-character chunks, tables and procedural descriptions got cut off midway and answer quality dropped sharply. I switched to splitting by heading, then splitting again if a section was too long.
2. A reranker is almost mandatory
Vector search alone looks plausible, but it often brings back the wrong documents. Adding a single reranker noticeably improved perceived accuracy.
3. Always include source citations
This was the most important part. If there are no sources, no one trusts it. Once I added document names and page numbers, usage patterns changed entirely.
4. Build an evaluation set
You can’t improve based on 'it feels better.' Even just 50 questions makes A/B comparisons possible.
In the end, pipeline design mattered far more than the embedding model. Swapping models was the last step.
7 answers
Citing sources really changes the whole reaction—100% agree. When they even include the document name and page number, the credibility hits different.
Saying chunking accounts for 80% of it feels like a stretch. It varies by domain, and the perceived impact depends a lot on the embedding model. The real issue was cutting everything at a fixed 500 characters—not chunking itself being 80%.
Creating a 50-question evaluation set is absolutely key. We also wasted two weeks without one, just saying "it seems a little better"... lol
Which reranker are you using? I'm currently using bge-reranker, but it's a bit iffy with Korean documents, so I'm thinking about switching to something else.
Oh, I didn't know that.
I can relate. I spent almost a month wrestling with PDF tables, and in the end, maintaining the layout during parsing was the most important part. Chunking was the next issue. I'm curious—which library did you use for PDFs?
Well, isn't that old news? With the long-context models coming out these days, I doubt you even need a reranker. Have you run the cost calculations?