Trying Internal Document Search with Claude: A Half-Day RAG Misadventure

I hooked up internal wiki and meeting-notes search using Claude instead of GPT. Spoiler: I'm happy with it.

I was already using OpenAI text-embedding-3-small for embeddings, but I switched only the generation model to Claude Sonnet. Even with the same chunks, the answers feel much less half-hearted. I like its habit of grounding responses with citations like "According to the document..."

Here's what I banged my head against:

• Setting chunk size to 800 tokens with 150 overlap dramatically improved answer quality. At first I used 1500, and the context got mixed up and it was a mess.
• Adding document creation date and department to metadata and applying filters made the biggest difference in practice.
• top_k of 5 was too few; around 8 was a good balance.
• Putting one line in the system prompt — "If you don't know, say you don't know" — is really important.

Cost comes out to about $20/month. A team of 12 uses it, and at that level it's basically coffee money.

That said, the Claude API rate limits are a bit tight, so queue management is essential when running batches. That cost me a whole day T_T

If anyone has gone with LlamaIndex instead of LangChain, I'd love to hear your experience. I'm weighing it these days.