Three Weeks of Building an Internal Document Search RAG: A Retrospective (with Failure Stories)

I spent three weeks building a chatbot to search our internal wiki + meeting notes + Slack archive. To give the conclusion first: one week to get to 'it works,' and the remaining two weeks to get to 'it's actually usable.'

At first I thought I could just push everything into a vector DB and add embedding search, but once people actually used it, it was a completely different story.

• Chunking: it has to be done differently for each document type. Meeting notes were much better split by utterance, and the wiki by heading.
• Hybrid search: only after switching to a BM25 + vector combo did internal abbreviations (e.g., 'XX Project') start being searchable.
• Reranker: I added one small model, and it gave the biggest improvement in perceived quality.
• Prompting: even after emphasizing 'if you don't know, say you don't know' three times, it still wouldn't comply, so I applied a relevance score cutoff at the code level.

The thing that burned me the most was permissions handling. I found on the last day of QA that documents the user didn't have permission to see were mixed into search results, and I pulled an all-nighter. You have to filter at the indexing stage; if you filter after retrieval, it's bound to cause an incident eventually.

Right now about 30 people internally use it, and satisfaction is around 7/10. Next time, I plan to start by building an evaluation set first. If anyone needs even just a record of my fumbling, feel free to ask questions.