My Experience Running a Local LLM at Home (RTX 4070)
These days I’ve been mindful of cloud API costs, so I tried running one locally. The conclusion: “More usable than I expected, but more of a hassle than I expected.”
First, on a 4070 12GB, 7B–8B models run quite comfortably when quantized. 13B works, but the speed is a bit frustrating. Honestly, 32B was not feasible for real use.
Things I installed:
- Ollama: Easiest to install, best for beginners
- LM Studio: Convenient GUI, easy to swap models
- llama.cpp: Well optimized if you build it yourself
Honestly, for coding assistance, cloud models are still overwhelmingly better. I think local models are best for summarizing sensitive documents or offline work.
Electricity costs add up, too. Considering that, I’d say only do it if you really need it...
10 answers
I agree that 7–8B is the practical range—on a 4070, that's exactly how it is.
I'm curious about the electricity bill—do you have any rough measurements? I heard it draws over 200W under load, and running it all night might actually start to add up.
I'm on a 4070 Ti, and even 13B models are usable when quantized, but once the context gets longer, it slows down dramatically, so in the end I only use it for document summarization. I really can't trust it with code.