Three Weeks of Running LLMs Locally (A Model-by-Model Comparison)
Because of company security, I couldn't use external APIs, so I tried running all sorts of local models. My setup is a single 4090 GPU.
1. 7B-class models
- The speed is genuinely fast. It was sufficient for code autocomplete.
- However, when I feed it long context, it keeps forgetting the earlier parts.
2. 13B-class
- The quality improves noticeably, but the speed feels less than half.
- For static tasks like translation or summarization, this tier offered good value.
3. 30B and above
- Honestly, with a single 4090, it's frustrating. Even when quantized, the token speed is questionable for real-world use.
- In the end, I felt that using multiple cards or a small model + fine-tuning combo is more realistic.
Overall: Local LLMs seem better suited for 'specific tasks' than as an 'API replacement.' The moment you try to run one model that's good at everything, costs jump sharply.
7 answers
Agreed. Once you go past 30B, with a single 4090 it’s really just a vibes thing lol.
I wish there were some speed numbers. It doesn't say how many t/s it got, so I can't compare. The claim that 7B and 13B feel half as fast is also subjective...
I'm running a similar setup, and I agree that 13B is the sweet spot for value. When running translation/summarization, the difference from 7B was striking. But with long contexts, 13B forgets just as much—I'm curious whether that's a model issue or some kind of setup issue.