RAG Pipeline: A Trial-and-Error Log from Start to Finish
I was asked to build an internal document search chatbot and spent six weeks on it. The result isn't bad, but the process was so rough that I'm writing it down.
Step 1 was just scraping PDFs, embedding them, and putting them into a vector DB. The demo went well. The problems started in real-world use.
- Scanned PDFs were mixed in, so 30% of files had no text extracted at all
- For documents with many tables, chunking completely broke the context
- The same content existed in multiple versions, making it unclear which one to use for an answer
In Step 2, I changed direction. Instead of spending time improving embedding quality, I added a process for people to clean up documents once. Each department assigned an owner and uploaded only the latest version. It was an organizational problem more than a technical one.
Step 3 was evaluation. The hardest part was figuring out how to know whether an answer was correct, so I mapped 100 questions to their correct documents and measured only retrieval accuracy. I treated generation quality as the next problem. If retrieval is wrong, the answer is always wrong.
As a result, I raised retrieval accuracy from 62% to 89%. I never changed the model even once. It was all down to three things: the chunking strategy, metadata filters, and the document cleanup process.
Looking back, the first two weeks were a complete waste of effort. I shouldn't have spent time swapping models; I should have looked at the data. But since the model was what I could see, I kept going back to it.
9 answers
Agreed, 30% scanned copies is literal hell... the moment you add OCR, maintenance hell begins.
The post got cut off lol, I'm curious what % search accuracy it got up to
I really agree with the point that it's an organizational problem, not a technical one. At our company, just creating a rule for managing the latest versions of documents took three weeks. In the end, the department heads couldn't agree on 'who's responsible for the latest version,' so the project got delayed by six months. Changing the model takes a day, but persuading people is measured in quarters.
Well, isn't "document cleanup comes before embedding tuning" kind of an outdated take? These days, parsing and chunking cover most of that. I'm curious how much actual embedding model comparison you did over those 6 weeks.
For the problem of chunking documents with lots of tables, going through Markdown conversion was the relatively better option. I extracted the documents into HTML tables, then created chunks by attaching headers to each row. If you just split it as plain text, you can't tell which item a value like "Q3 revenue" belongs to. I wasted two weeks building this, but after that, search quality went way up.
Isn't a 100-item eval set a bit small? For us, it wasn't until we got to around 300 that we started seeing variation by query type. And if you're only looking at retrieval accuracy, try measuring recall@k and MRR as well. With just one metric, you can't really judge whether adding a reranker helped.
You didn't mention what the search accuracy percentage is. And what the criteria for "not bad" are. Without that, it's just a diary entry.
For versioning, the cleanest approach was to put the effective date and deprecation date in the metadata and filter on those. If you delete the documents themselves, you can't answer questions about past points in time. Just something to keep in mind.
Please write the conclusion too ㅠㅠ As someone who got stuck at step 3, I'm really curious.