Sharing my setup for running a 7B model at home (12GB GPU)

Since I set up an environment that runs entirely locally without external APIs, I figured I'd write up my setup. This is based on a 3060 12GB.

I started out downloading and using the default model through Ollama, but it definitely got slower as the context grew. So after switching to a quantized version, the perceived speed improved quite a bit. It's good enough for code autocomplete.

To be honest, though, complex refactoring is still too much for it. I only use it for asking about short function-sized chunks, and for bigger tasks I just use a paid API. The electricity bill adds up too.

The reason I run it locally is sensitive code, and I've definitely achieved that goal. If you're interested, I'd suggest starting by just installing it.

by 클라우드러버828

5 answers

Got a source? I’m curious about the electricity bill figures.

by 취준생김씨262 · ▲0

Agreed — I'm running the exact same setup on a 3060, and after I changed the quantization, the difference felt huge.

by 주말개발자501 · ▲0

I really relate to running it locally because it's sensitive code. I also feel uneasy pasting internal company code, so I run it at home—but the catch is that for bigger tasks I end up using a paid service lol. For asking about short, function-sized snippets, it seems absolutely perfect.

by 스타트업러206 · ▲0

Honestly, I'm not so sure... The claim that a 7B can't refactor is just a model limitation, and it feels like blaming quantization for it. Have you compared Q4 and Q8? For me, on 12GB, Q4_K_M had the best speed/quality balance, and once context goes past 8k, the KV cache eats VRAM anyway, so the perceived speed drop is the same. Also, "electricity costs add up" is a bit vague; even if a 3060 pulls 130W at full load, two or three hours a day comes to only a few thousand won a month, so that doesn't seem like a reason to give up on local. If the goal is sensitive code, you should also set up network blocking and log storage paths together to actually achieve that goal. FYI, I run ollama via systemd and set OLLAMA_KEEP_ALIVE long to reduce model reloads.

by 지나가던행인703 · ▲0

They didn’t mention what quantization version it is. Whether it’s Q4 or Q5, and which model it is, makes a huge difference in the actual experience—but that key detail is missing, so it’s a bit disappointing as information.

by 월급루팡910 · ▲0