LLM-Based Customer Support Automation: Our Experience + Action Plan
Over the past three months, I built an LLM-based customer response automation system for our internal CS team, and the results were satisfying enough to share in detail.
Originally, whenever a customer inquiry came in, agents would search for similar questions with Ctrl+F and reply with copy-paste answers—a typical manual workflow. The number of inquiries kept growing while headcount stayed the same, so the average response time soared to 4 hours, which was the core problem.
Here's the order in which we built it:
1. Extracted 30,000 customer inquiries from the past year into CSV and labeled them by type.
2. Used a small fine-tuned model for the first-pass classification of question type (shipping, returns, refunds, errors, other).
3. Then had the LLM retrieve past inquiry logs and relevant policy documents through RAG to generate draft responses.
4. Finally, changed the workflow so agents only reviewed the draft and clicked 'Send.'
The key point was preventing the LLM from answering freely. So we strongly embedded the rule 'If you cannot provide a clear answer, escalate to an agent' into the prompt, and made the system route the conversation to a human whenever a speculative response would be required.
As a result, average response time for simple inquiries dropped to around 20 minutes, and agent throughput increased by about 3.2x. Of course, the LLM still misses nuances hidden in responses, so we're monitoring it continuously. This project made me realize that adopting AI isn't for show—organizing the actual workflow from data onward matters far more.