My Experience Running a Local LLM at Home (7B-Class)
I tried running a quantized 7B model on a 12GB graphics card. To get straight to the point: "It's more usable than I expected, but not quite good enough to be my main setup."
Speed was around 20 tokens per second, which was tolerable, and it did okay with simple summarization or translation. But once you get into reasoning or long-context processing, the difference from API models is clearly big.
Still, the biggest advantage seems to be that it runs without an internet connection and data doesn't leave the machine. I'm considering using it when handling sensitive documents at work.
6 answers
Agreed, the fact that it doesn’t use data is huge. That alone is why I installed it too.
With 12GB, 7B q4 is just right lol. If you get greedy and go above that, it starts swapping and freezes up.
It really does a great job at summarizing and translating. But sometimes I wonder if you really need to run that locally lol. When you think about the electricity bill...
Hold on, 20 tokens per second is considered readable? I get 15 tokens/sec on a 12B-class q4, and it feels painfully sluggish... They didn't mention the model or GPU, so I don't trust that speed. Depending on whether it's a 4090 or a 3060, isn't that a completely different story?
A big part of the weakness in reasoning and long contexts is also due to quantization. If you bump it up from q4 to q5 or q6, you'll probably notice a big difference. The tradeoff is that speed drops a bit. And with long contexts, the KV cache VRAM tends to blow up before the model does, so you often can't even fill the full 32k context. In my experience, the answer was to add RAG and only insert the necessary chunks.
If you're using this for sensitive documents at work, the bottom line is that the issue isn't model performance—it's everything upstream. Running it locally doesn't automatically make your data safe; you need to verify where prompt logs, caches, and backups are stored, and under company policy, uploading documents marked 'no external transfer' to a personal GPU may well require separate approval. I just settled on deploying vLLM on an air-gapped network and attaching audit logs. Running it on a personal PC is only recommended for testing.