Curious about the selection criteria for LLM fine-tuning vs RAG
Hello. We're looking to build an in-house document-based Q&A chatbot, and I'd like advice on whether it's better to adopt fine-tuning or RAG first.
The domain is specialized, so general LLMs fall short, and we have about 10,000 pages of documents. We're currently using the GPT-4o API.
Is the standard approach to start with RAG and fine-tune only the areas where it falls short? Or, given the large volume of documents, would it be better to fine-tune from the beginning? Cost and infrastructure aren't a concern, so I'd like to hear from those with actual experience.
4 answers
RAG first is the answer lol
I've done it at a similar scale, and starting with RAG is the right call. Fine-tuning can wait unless your domain terminology keeps changing. Nailing down search accuracy first turned out to be far more effective.
I, on the other hand, spent three months struggling with RAG before eventually switching to fine-tuning. When the document structure is complex and there are lots of tables and charts, the retrieval kept pulling from the wrong places. Fine-tuning was also frustrating at first, but after training on summarized versions of the documents, it turned out to be more stable. However, if your document update cycle is short, the retraining cost can be a burden, so be sure to keep that in mind.
Well, if you're dealing with 10,000 pages, starting with RAG makes sense. But if your internal docs are mostly tables and charts, fine-tuning won't help much either. The real answer might not be combining RAG with fine-tuning—organizing the documents and designing the metadata could be the priority. If the retrieval step breaks down, neither method will work.