My trial-and-error notes from running local LLMs at home (on a single GPU)
I’ve been running open-source models at home for fun, and I made enough messes that I figured I’d write them down. The conclusion is: “It runs, but making it actually usable is the hard part.”
At first I just installed Ollama and ran a 7B model, and the speed wasn’t bad. The problem was quality. When I asked in Korean, the answers were subtly awkward, so it was hard to use for real work.
Then I wrestled with GPU memory. Quantizing and loading 14B or 32B models definitely brought the quality back, but when VRAM ran short and some layers got offloaded to the CPU, the speed dropped off a cliff. That’s the point where it stopped being fun lol.
Still, what I learned:
- Model size is almost everything for quality (quantization seems to hit its limit around 4-bit)
- For Korean, you need to pick a model that was trained well on Korean data from the start
- Speed is determined by memory bandwidth, not core count
- Running a server and using it through an API is just the most convenient
In the end, I currently run one small model and use it only for summarization. I’m curious what models people with better hardware are using.
3 answers
I really agree that memory bandwidth is what determines speed. I also bought a good CPU trusting core count alone and wasted a bunch of time lol. In the end, VRAM bandwidth was the answer.
Well, isn't "Korean works best with models that have been well-fed Korean data" kind of outdated? The large models coming out these days produce Korean pretty well even when they're mostly trained on English data. I suspect that's only true for small models around 7B.