Notes from a Third-Year Backend Engineer on Adding AI
Since last year, I've been adding various LLM features to internal services, and I'm jotting down some things I've learned. To get straight to the conclusion, "input design" mattered much more than "model performance."
At first I thought the answer was polishing prompts beautifully, but the places where incidents actually happened and CS tickets came in were almost all on the data side. I didn't know what would end up in the text users submitted, and I was just blaming the model.
These are the principles I use now.
- Assume the model can be swapped out at any time and keep the interface thin
- Separate prompts into their own files, not code, and version-control them
- Only save output if it passes schema validation
- Log every failure case and periodically feed them back into the prompts
The last item was especially effective. When I gathered about 300 failure logs from the first two months, the patterns converged into exactly three. Fixing those made the perceived quality jump.
I can't leave out cost either. I once ran it without caching and got a monthly bill of 2 million won. Now I cache based on identical input hashes and route simple classification to a smaller model, bringing it down to around 150,000 won.
Sometimes I envy that there's a separate role called AI engineer, and sometimes I wonder if we're in an era where backend engineers just do it all.
9 answers
Agreed—input design really is 80% of the work. Cleaning the incoming data takes way longer than swapping out the model.
Could you elaborate on the part about converging on three patterns from 300 failure logs? I’ve accumulated something similar, but I can’t figure out how to classify them.
Well, I'm not so sure about that. It's true that input design deserves emphasis, but I wonder if it's fair to assert that it matters “more than model performance.” Doesn't it depend on task difficulty?
On our team, even when we got both the prompts and inputs right, we kept running into cases we couldn't get past because of the model's own reasoning limitations. That wasn't a data problem.
I totally agree with pulling prompts out into a separate file. Once you've experienced the diff hell of hardcoding them as strings in your code, everyone ends up doing the same thing lol
Is the cost really 2 million won? lol Just adding caching would cut it in half.
Is there a source?
Schema validation is a real must—there was a time we saved data without validating it, and later bad values ended up stuck in the DB, so we had to write a separate data-cleaning script. Since then, whether it's pydantic or whatever, we always make sure it passes validation before inserting. Also, if you extract the failure rate as a metric, it's easier to catch regressions when you change prompts. What format do you use for logging? We log inputs/outputs and even the model version, but the logs get so large that we're worried about retention period and cost.
I disagree. Models these days handle most inputs well on their own, so I don't see why you'd need to strip the input down that much. In fact, over-refining it often kills the nuance the user intended, and in many cases the result ends up even blander.
"Assume models are replaceable and keep the interface thin"—this really seems like the key point. For us, our logic was initially all tangled up with a specific vendor SDK, so when we later tried to switch to another model, we basically had to rip everything apart and redo it. Since then, we've put in an adapter layer and changed things so that internally we only work with our own format. Like in your post, if you also run a loop that periodically reviews failure cases and feeds them back into the prompt, it really makes a difference.