Honest Review of Running an 8B Local LLM
I tried running a few local LLMs on an RTX 3060 12GB that was just sitting around at home.
An 8B Llama-family model runs better than I expected. With quantization it only uses about 6–7GB of VRAM, so 12GB is plenty of headroom. Speed was around 30 tokens per second, which didn't feel sluggish at all.
But the moment I tried to go up to 34B or larger, I hit a wall immediately. It reminded me once again that VRAM is money. Unless you're working with something privacy-sensitive, I think just using an API is the better call.
7 answers
The RTX 3060 12GB is still relevant for a reason — agreed. For value, nothing comes close.
Isn't 30 tokens per second a bit of a stretch? For an 8B Q4_K_M model with a 4k context, around 20 tokens per second on a 3060 is realistic. You also need to include the model, quantization type, and prompt length for the comparison to be meaningful, don't you think?
I started with the same setup last year too, but eventually moved to a used 3090. Up to 8B is genuinely usable, but the moment you go above that, the difference hits you hard... Then after buying a 3090 24GB, now I’ve got my eye on 70B. It’s an endless money pit, seriously lol
34B is a bit much. You'd be better off looking into MoE models instead. Models with fewer active parameters can run reasonably well even on 12GB. Since I switched over to Qwen, my satisfaction has gone up a lot.
Got a source? The token speed numbers are way too cleanly rounded lol
Since we're on the topic of personal data—there are a ton of teams at companies that use APIs instead of local models to summarize internal documents. It's just more convenient. But many don't realize that all of it gets uploaded to external servers. Even if local is slow and annoying, for sensitive data that's the right choice. In the end, it seems the answer is that it depends on the use case.
I don't really agree with the conclusion that API is better. If you're only looking at price and speed, sure, but local models run offline too, and there's the fun of swapping models. Leave that out and you're only seeing half the picture.