Notes on My Experience Building RAG for Internal Document Search

Our internal wiki search was so bad that I tried hooking up RAG myself. To get straight to the point, the biggest problem was “search works, but it isn’t trustworthy.”

  • Chunking: Cutting documents into 500-token chunks broke the context, so the answers came out strange. Attaching titles and headers as metadata made it somewhat better.
  • Embeddings: Since there were many Korean documents, switching to a multilingual model made a huge difference in practice.
  • Reranker: The setup that retrieved top-k 20 and then reduced it to 5 with a reranker was the most stable.
  • Evaluation: This was the hardest part. I manually built a 100-item answer set, and after that the pace of improvement really picked up.

Now I’ve made it so every answer must include a source link. That alone raised user trust quite a bit. I’m curious how you all automate evaluation.

by 알고리즘고수504

0 answers