The Story of How I Wasted 3 Weeks Building an Internal Document Search RAG
We have about 8,000 internal wiki documents, and the search was so bad that I started this thinking I’d build a smart search tool with RAG. To give the result upfront, I’m about halfway there now. Here’s a log of my trial and error.
First, crawling. The wiki is on the internal network, so I got blocked by access permissions right away, and in the end it took a week just to get API access working and pull the data.
Then came the real problem. I just chopped the documents into small pieces and embedded them, and the chunks lost all their context. When I asked about the “vacation policy,” it only pulled up documents about requesting permissions lol. In the end, I reindexed at the document level, attaching titles and headings as metadata.
After fixing it this way, answer quality clearly improved. The conclusion: preprocessing is 80% of the game, more than embeddings.
Right now I’m working on adding a reranker, and I have no idea what kind of hell this will unleash. If anyone here has built an internal RAG, please give me advice on how you set chunk size and prompts.
7 answers
Oh, I didn’t know that. Preprocessing makes up 80% compared to embeddings—I should note this down.
Chunk size varies depending on the document type, but I set it to 512 tokens with 20% overlap and embedded the heading path as text at the front of each chunk. For example, '[HR Policy > Leave > Annual Leave] body...'. Just this alone significantly improved the match rate for ambiguous queries like "leave policy." For the reranker, starting with the bge-reranker family is a safe bet.
Well, that's a bit... When you say 80% is preprocessing, isn't that ultimately just saying they attached metadata well? I'm curious whether there are numbers comparing different embedding models. After all, perceived experience and actual metrics can differ.