Lessons Learned Building an Internal Document Search Chatbot (A RAG Misadventure)
I built an internal RAG-based chatbot to solve the problem that our company wiki is too massive to search effectively. About 20 people are using it now, and I figured I’d write down the mistakes I made in case they help others in the same situation.
First, your chunking strategy accounts for 80% of performance. At first, I split documents into page-sized chunks and inserted them, but the answers often missed the context. When I split by titles and subheadings and attached the document title before and after each chunk, it improved dramatically.
Second, if you change the embedding model, you have to rebuild the index. I glossed over this and spent three days debugging strange answers.
Third, always show the retrieved documents. It makes a huge difference in user trust. Even if the answer is wrong, if users can see the evidence, it becomes a usable tool.
Fourth, I should have created evaluation data. I got fooled by plausible answers and wasted a lot of time. Now I’ve prepared about 100 questions and run them before deployment.
The remaining challenge is permissions. HR documents and general documents are mixed in one index, so in the end I realized I need to split them into separate collections. If anyone else has built an internal RAG system, I’d love to hear your experience.
9 answers
The third one is really important. If you don’t show sources, no one will trust a chatbot’s answers.
Do you think 100 evaluation questions are enough?
But isn't it a bit risky to leave permission handling as a "remaining task" and already roll it out to 20 people? You mentioned that HR documents are mixed into a single index. Under company policy, even just having documents you're not authorized to access show up in search results can be a problem — so in my view, this isn't a feature, it's a must-have before launch.
I'd like to hear more about chunking. If you split by subheadings, how much overlap did you set between chunks? I'm also curious how you handled documents with tables or images mixed in.
Oh, I didn’t know this. I should start by creating an evaluation dataset.
I really agree with the saying that chunking is 80% of the work lol. We used to split by page too, but once we preserved the document structure, the answer quality improved dramatically.
I also ran into weird answers because I changed the embedding model and didn’t regenerate the index—tracking down the real cause is absolute hell. We started embedding the model name and version hash as metadata in the index creation script and checking it at deploy time, and we haven’t had that issue since. Might be worth a look.
Honestly, isn't this all stuff anyone who's built a RAG even once already knows? Points 1 and 2 especially are the absolute basics... It's a bit disappointing that they just toss out the genuinely hard part, like permission handling, and leave it at that.
For permissions, pre-filtering is the answer, not post-filtering. You need to apply user ACLs as metadata filters at the search stage. If you filter after receiving the results, you might retrieve 5 chunks, lose 2, and then build an answer from the remaining ones—and quality swings wildly. We mapped permission groups to top-level document folders and attached them there.