My Trial-and-Error Notes on Running a Local LLM at Home
I’m leaving behind a record of a month spent fumbling with running local models on a single 12GB graphics card. I’ll start with the conclusion: it’s doable. It just takes more work than you’d expect.
First, you need to get a feel for model size. A 7B model runs comfortably on 12GB with 4-bit quantization, and a 13B model barely runs if you shorten the context length. I just gave up on anything 30B or larger.
Speed comes out around 20–30 tokens per second, which is enough for chat. But if you feed it a long document, it runs out of memory while processing the context and you have to reload the model.
Things I learned while fumbling:
- GGUF is the most hassle-free quantization format
- Start with a 4K context length and increase it from there
- If the driver version is wrong, it quietly gets slower
Now I run simple summarization tasks locally and use a cloud API for complex ones. I started out trying to save money, and somehow ended up with a hybrid setup.
10 answers
Agreed, all the conclusions are spot on lol. Especially the driver thing—it quietly getting slower is the scariest part.
Is there a source? I'm curious which model the token speed of 20–30 is based on.
Well, it’s a bit disappointing that they gave up on the 30B. With offloading, it does run even if it’s slow—if they gave up without even trying it once, I feel like that’s only half a review.
GGUF is really solid, but I recently switched to exl2. The speed difference is immediately noticeable, making it usable even on 12GB.
This is so true lol — tried to save money and ended up wasting more on electricity and time.
I can really relate to slowdowns caused by driver versions. After updating, I couldn't figure out why it got slower and wasted hours ㅠㅠ
Isn't 12GB tight even for a 13B? I ran a 13B Q4 on a 3060 12GB and it was barely enough even at 4K context—can you share your settings?
I started similarly too, but eventually settled on a hybrid setup. Splitting it so local handles one-off tasks like summarization and translation, while cloud handles long-context reasoning, cut my stress a lot. The cost isn’t as high as I expected either.
I disagree. These days, a 7B model at 4-bit is enough even with 8GB, and with 12GB you can even run a 14B Q5. I’m not sure when this was written, but the information seems a bit outdated.
Good advice to start with a 4K context. I loaded 32K from the start and wasted a whole day on OOM... I should have listened to my senior.