My Experience Running a Local LLM at Home (RTX 4070)
I was using APIs at work, then suddenly got worried about privacy, so I tried running one locally at home. The conclusion: "It works better than expected, but it's also more underwhelming than expected."
Installation takes 10 minutes with Ollama. Models in the 7B–14B range were realistic. No matter how much you quantize a 32B model, it's painful on 12GB VRAM.
Speed is around 30 tokens per second for a 7B model. In practice, it feels usable. But the problem is quality. For coding or logical reasoning, it falls far short of GPT. Simple summarization or translation is fine, but complex tasks are honestly too much.
So now I use it like this: sensitive content locally, everything else in the cloud. Splitting privacy and performance depending on the situation works well for me. If you have a GPU, it's worth trying once for fun.
9 answers
Agreed, I also ran it on a 4070 and got the same conclusion lol
But isn't forcing a 32B model onto 12GB the real problem? That class of model should be 4090 territory to begin with lol
What quantization is the 30 tokens/sec speed based on? It feels quite different depending on whether it's Q4_K_M or Q5.
If you're worried about privacy, isn't it already game over the moment you use a cloud API? I don't really see what running it locally solves. It's just a hobby—doesn't seem like a security measure.
Running 14B Q5 on a 4060ti 16GB for in-house document masking. It's really good at summarization and structured extraction. But when you ask it to code, it's a hallucination zone, so I just use the cloud for that. Seems like hybrid really is the answer.
Oh, I didn't know that. So you just need to install Ollama.
Ollama is convenient, but if you build llama.cpp yourself, even the same model runs a bit faster. Also, for Korean, Qwen2.5 14B or EXAONE felt better in my experience. Just take it as a reference.
Sensitive stuff locally, everything else in the cloud — this is genuinely the right answer. My stress dropped a ton once I started splitting it up this way too.