LangChain Struggles: Lessons Learned from Implementing RAG
I used LangChain to build an RAG system, but I hit a wall right from document chunking.
1. Using RecursiveCharacterTextSplitter with 500-character chunks broke the context, making search quality terrible.
2. For the embedding model, I debated between bge-large-ko and Korean RoBERTa, and ended up going with bge.
3. Between FAISS and Chroma for the retriever, I chose FAISS because it was faster.
Conclusion: LangChain itself is convenient, but the chunking strategy is what really matters. I found that splitting by Markdown headers worked much better.
Anyone using a different approach?
9 answers
Agreed, chunk is the hardest part.
Oh, I hadn't thought about using Markdown headers as a criterion. I've tried semantic chunking and it worked pretty well. It feels much more natural than cutting by sentence units.
Has anyone compared bge-large-ko vs Korean RoBERTa? I heard RoBERTa is better for Korean.
This is right lol. I also used a recursive splitter at first and it was a total mess. After switching to header-based splitting, the search accuracy went way up. A chunk size of 1000 with an overlap of 200 turned out to be just right.
Well, I don't use LangChain; I just use llama_index. LangChain has too much abstraction, making debugging tough.
Do you have a source? I'm curious if FAISS is actually faster than Chroma.
I use the nltk sentence tokenizer for chunking, splitting by sentence before running embeddings, and it maintains context well. The 500-character standard is indeed too short.
Disagree. Recursive splitters depend on how you use them. With proper overlap and metadata attached per section, they're quite usable. But I'll also try header-based splitting.
Wow, I totally agree. I gave up on LangChain entirely due to chunk stress and built it myself using HuggingFace pipelines. But in the end, I ended up switching to LlamaIndex. LangChain is convenient at first, but it's tough when you need to fine-tune things later.