Review: Running a Local LLM on a Spare GPU at Home

I had a 3060 12GB left over from gaming a while back, so I tried running a local model just for fun. I'd only used paid online services before, but running one myself has its own appeal.

First off, installation wasn't as hard as I expected. I installed Ollama, downloaded one model, and was chatting right away. It probably took less than 30 minutes.

Here's roughly what I thought while using it.

  • Privacy is definitely a plus. I can put in internal company documents or personal notes without worrying
  • 7B–8B models are a bit awkward at Korean. English is passable, but honorifics often break down
  • Speed feels like it's printing one character at a time, so it can be frustrating. Once you go past 14B, it gets noticeably slower
  • In the end, I concluded that cloud APIs are better for actual use

Still, it's amazing that we can now do this much offline. Before long, we'll probably be running 20B-class models at home too.

by 디지털노마드238

10 answers

The "done in 30 minutes" thing seems like a bit of an exaggeration lol. When you think about having to download tens of gigabytes just for the model at first... Still, it's true that the barrier to entry has dropped a lot.

by 문과출신개발자178 · ▲0

Starting with Ollama was a good choice. But the awkward Korean from 7B–8B models isn’t because they’re local—it’s just how models at that scale are. If you try EXAONE or Qwen 14B or larger, honorifics are much more stable. With 12GB VRAM, even a 14B model can barely run with 4-bit quantization.

by 클라우드러버299 · ▲0

Well, that's a bit... The conclusion that 'cloud APIs are better for real-world use' is based on personal use cases, and when it comes to running internal company documents, local is often the answer because of privacy. It feels like the conclusion was drawn too narrowly.

by 카페인중독850 · ▲0

You can fix broken Korean honorifics pretty well with prompting, too. Put 'always answer in polite formal Korean' in the system prompt and add just 2-3 few-shot examples, and it feels noticeably better. Try that before swapping out the model.

by 뉴비탈출218 · ▲0

Same here lol. I run it locally because of privacy, and it puts my mind at ease that I can even put in documents that are banned from being brought into the company.

by 뉴비탈출767 · ▲0

Oh, I didn’t know this. Honestly, I’ve only used the cloud, so my only experience with local is from messing around with it back in the day—but I should try setting it up again sometime soon.

by 호기심천국712 · ▲0

For local use, the 3060 12GB has insane value, agreed. I'm also thinking about buying another used one.

by 뉴비탈출355 · ▲0

I really agree with the privacy benefits. But you can't ignore the power consumption. Leave it on 24/7 and the electricity bill ends up costing more than a cloud subscription... Of course, that's case by case.

by 카페인중독319 · ▲0

I tried running a 14B int4 model on a 3090 24GB and got about 30 tokens per second. That's just slightly faster than human reading speed, so it didn't feel sluggish. On a 3060, 14B would be under 10 tokens per second, so the slowdown would be very noticeable. If you use GGUF, llama.cpp tends to be a bit faster, so give it a try.

by 지나가던행인3 · ▲0

Isn't that old news? Maybe if it were a year ago, but now that both Metal and ROCm have landed, it runs well on Mac and AMD too. 'This level of performance offline' is already years out of date.

by 월급루팡102 · ▲0