My Experience Running Local LLMs at Home (7B / 13B / 32B)
I picked up a used 3090 and tried running all sorts of local models. To get straight to the point: if your use case is clear, it’s plenty usable.
Models tested
- 7B class: Response speed was insane. 60 tokens per second. But reasoning is a bit weak.
- 13B class: Feels like the best balance. Good enough for summarization and translation.
- 32B class: Clearly smarter, but 12 tokens per second. If you watch it stream, it’s fairly tolerable.
Pros
1. No internet required. It even runs on a plane.
2. Even if you feed it company code or similar, it never leaves your machine.
3. The psychological comfort of zero token cost is huge.
Cons
1. Electricity bill... leaving it on for a month is only a few thousand won, but in summer the room gets hot.
2. Compared with the latest commercial models, the limitations are clear.
3. In the end, quantization and serving setup take quite a bit of time.
Right now, I use local for summarizing sensitive documents and mix in APIs for everything else. If you have questions, leave a comment.
3 answers
Is 60 tokens per second from running a 7B at q4? Since the model and quantization level aren't specified, the numbers are a bit ambiguous.
I've also been running one on a 3090 since last year, and I relate to a lot of this. Especially #3 — the quantization setup is real grind work lol. At first I was just kind of running it with llama.cpp, but it only became usable once I finally got vLLM set up. I personally agree that 13B is enough for summarization and translation. That said, 32B feels too sluggish for me, so I just don't use it. If you queue up long documents to run overnight and go to sleep, even 12 tokens/sec doesn't really matter. I also like how the closing point about mixing in APIs is realistic.
Well, isn't 'zero token cost' a bit of an exaggeration? Once you factor in the used price of a 3090, electricity, and all the time you pour into setup, there are plenty of cases where just using an API is cheaper. Unless it's sensitive data, I don't really see a reason to insist on local.