The Time I Tried to Run a 70B Model at Home and Wasted Two 4090s

I run local LLM setups at home as a hobby, and this time I got ambitious and tried to run a 70B-class model. Here's the post-mortem: I got hit with a triple whammy of power draw, heat, and speed.

1. First, a 70B FP16 model won't fit on two 4090 24GB cards. The problem was that I tried it even though I knew that.

2. I barely got it running with 4-bit quantization, but as soon as I increased the context length, it OOM'd.

3. I messed with the vLLM settings incorrectly and the driver crashed once; after rebooting, CUDA wasn't detected and I lost an hour.

4. In the end, I dropped down to a 32B model, and the perceived quality difference wasn't as big as I expected, which left me feeling deflated.

The lesson is: "Bigger isn't always better for home use." The electricity bill isn't trivial either, and the noise means I can't run it at night. Right now I'm satisfied with a 32B quantized model + RAG setup. If anyone out there runs a 70B at home, I'm curious how much power it draws.