My Results Running Local LLMs on 12GB of VRAM
I put a 3080 12GB in my home server and ran a few local models on it. To cut to the chase: for coding assistance, cloud is still the better option.
- 7B 4-bit quantized: good speed, but awkward Korean
- 14B: usable, but it slows down sharply past 8k context
- 32B: it does run, but 40 seconds per response... a test of patience
On the other hand, it was genuinely useful for summarizing sensitive documents or searching personal notes. Nothing leaves the house, so it's peace of mind. I'm running llama.cpp with a decent frontend attached, but once you factor in electricity costs, just using an API is cheaper. If you're doing it for fun, I'd recommend it.
5 answers
32B at 40 seconds is a bit brutal lol. Even with my 4090 24GB, I gave up on 32B because once the context gets long, it starts swapping. In the end, 14B seems to be the practical limit.
What frontend did you attach to llama.cpp? I'm using open-webui, but something breaks every time I update, so I'm thinking about switching.
Well, isn't saying "cloud is better for coding assistance" a bit of a hasty generalization? A 14B model is more than usable for code autocomplete, and there are many places that can't use the cloud at all because they're on internal networks. If you included benchmark numbers as well, it would be more convincing.
If you're summarizing sensitive documents, truly local is the way to go. No matter how much you trust the cloud provider's terms, it still feels uneasy.
How did you calculate the electricity cost? You can get a rough estimate by multiplying the RTX 3080's idle power draw by its average power during actual inference and then plugging in the kWh rate, but I ran those numbers and got such a reality check that I shut my server down lol