A Month of Running a 70B Model at Home: A Realistic Review
I got into local LLMs late last year with a single 4090, and this month I finally expanded to two. Here's a record of the various things I tried over the past month.
The biggest takeaway was that "running it locally" shouldn't become the goal in itself. At first, I forced a quantized 70B onto my setup and ran it, but the answer quality was clearly worse than cloud models, and all I got was a higher electricity bill. Even with 4-bit quantization to fit VRAM, speed drops sharply once the context length gets long.
So now I split things by use case.
- Summarizing internal code snippets and handling sensitive data: local 13B model (good for use without external transmission)
- Complex reasoning and long documents: cloud API
- Experimental prompt tuning: local 8B
In conclusion, local LLMs seem best used when you "have no choice because of security" or when "API costs have grown too high." There's no need to push yourself too hard on personal projects. If anyone has tuning tips for vLLM or llama.cpp, please share them.
7 answers
Well, isn't it a bit outdated to say quality drops sharply at 4-bit? These days Q4_K_M or AWQ are practically indistinguishable in practice. I'd be curious what model and what metrics you used for the comparison.
I totally get the side that’s forced to use it for security reasons. In environments with rules restricting what can be brought onto the internal network, local is the only option.
How much would the electricity bill be per month for two 4090s? Once you calculate that, the comparison with cloud gets a bit murky.
Agreed, this is so true. I also wasted two months struggling and eventually went back to the cloud lol
If you're on llama.cpp, just offloading all layers up through the last layer with -ngl, enabling --flash-attn, and quantizing the KV cache to q8_0 makes a pretty big difference in long-context speed. I run a 70B at Q4_K_M across 2 GPUs, but once the context goes past 8k it still stutters, so in the end I only run short tasks locally.
The conclusion is a bit generic. It would be much more convincing if you included numbers showing the actual time or cost savings achieved locally during that month. As it stands, the examples in the post are mostly personal impressions.
With two 4090s, vLLM gets almost no benefit from tensor parallelism. There’s no NVLink and P2P is blocked, so the cards just communicate over PCIe. Instead, saving up the cost of a second card and buying a used A6000 would be better—48GB of memory makes it much easier to run 70B quantized models. If it were me, I’d look into that before adding a second card.