Three Weeks of Running LLMs Locally (A Model-by-Model Comparison)
Because of company security, I couldn't use external APIs, so I tried running all sorts of local models. My setup is a single 4090 GPU.
1. 7B-class models
- The speed is genuinely fast. It was sufficient for code autocomplete.
- However, when I feed it long context, it keeps forgetting the earlier parts.
2. 13B-class
- The quality improves noticeably, but the speed feels less than half.
- For static tasks like translation or summarization, this tier offered good value.
3. 30B and above
- Honestly, with a single 4090, it's frustrating. Even when quantized, the token speed is questionable for real-world use.
- In the end, I felt that using multiple cards or a small model + fine-tuning combo is more realistic.
Overall: Local LLMs seem better suited for 'specific tasks' than as an 'API replacement.' The moment you try to run one model that's good at everything, costs jump sharply.
7 answers
I'm running a similar setup, and I agree that 13B is the sweet spot for value. When running translation/summarization, the difference from 7B was striking. But with long contexts, 13B forgets just as much—I'm curious whether that's a model issue or some kind of setup issue.
I really agree with the concluding section. If you set 'API replacement' as your goal, there's no end to it, but narrowing it down to 'this one task, this model' is less stressful and far more cost-effective. Thanks for the great post.
I wish there were some speed numbers. It doesn't say how many t/s it got, so I can't compare. The claim that 7B and 13B feel half as fast is also subjective...
How many tokens/sec do you get on a single 4090 with a 30B 4-bit quantized model? I figure if it's around 10–15 t/s, that should be usable for code autocomplete.
Well, saying 7B code autocomplete was "sufficient" feels a bit case-by-case, doesn't it? It depends on the language or framework, and especially when legacy code is mixed in, 7B starts spitting out nonsense pretty quickly. I'm curious what language it was tested on.
Agreed. Once you go past 30B, with a single 4090 it’s really just a vibes thing lol.
If the situation is that you're running locally because of company security, then a small model + RAG + a bit of fine-tuning is the most realistic combo. We also ran 30B-class models on two 4090s, but eventually settled on a single 7B model specialized for internal documents. The moment you try to make it good at everything in general, GPU costs spiral out of control... In the end, it's pretty much the same as the OP's overall take.