Three Months After Introducing an LLM at Work: An Honest Review

Since late last year, we've been running an LLM for internal document search and code review assistance. To get straight to the point: it works better than expected in some ways, and not as well as expected in others.

The part that works well is search. It used to take 10 minutes to search the wiki and open five links, but now the answer comes up right away. The new hires felt the impact most during onboarding.

The part that doesn't work well is accuracy. Sometimes it confidently gives plausible-sounding but wrong answers, so we ended up forcing a "Verify against the original source" banner at the bottom of answers for questions about internal policies or contracts. Without it, we almost had an incident.

Cost isn't as high as expected. We run embeddings in a batch once, and after adding caching to calls, it costs about 300,000 KRW per month. That said, the first two months involved a lot of wasted effort tuning it.

Conclusion: definitely adopt search assistance. For work that requires judgment, it's still too early.