Three Months After Introducing an LLM at Work: An Honest Review
Since late last year, we've been running an LLM for internal document search and code review assistance. To get straight to the point: it works better than expected in some ways, and not as well as expected in others.
The part that works well is search. It used to take 10 minutes to search the wiki and open five links, but now the answer comes up right away. The new hires felt the impact most during onboarding.
The part that doesn't work well is accuracy. Sometimes it confidently gives plausible-sounding but wrong answers, so we ended up forcing a "Verify against the original source" banner at the bottom of answers for questions about internal policies or contracts. Without it, we almost had an incident.
Cost isn't as high as expected. We run embeddings in a batch once, and after adding caching to calls, it costs about 300,000 KRW per month. That said, the first two months involved a lot of wasted effort tuning it.
Conclusion: definitely adopt search assistance. For work that requires judgment, it's still too early.
4 answers
Search assistance is an absolute must. We used to waste 10 minutes digging through the wiki and Notion, but after we added a chatbot, new-hire onboarding speed was completely different. That said, accuracy ultimately comes down to how well your data chunks and metadata are organized, so teams that haven't tagged their documents probably won't have great search either.
300,000 won a month seems a bit suspicious. How many users is that based on? Even with caching, if the document volume is large and call volume grows, it can spike quickly—so it's hard to take that number at face value. And if you convert the first two months of trial and error into labor costs, isn't that the real cost?
Well, that's a bit much. It's hard to flat-out say that 'judgment tasks are premature'—doesn't it depend on the team? We're using it well for draft generation and review checklists. Of course, assuming a human does the final review.
Forcing that 'verify original text' banner was a really good choice. We didn't add it and ended up giving wrong info on the contract terms, which caused a huge mess;; Even if you can't catch hallucinations, you've got to prevent accidents.