For those of you running local LLMs, what are you using?
Because of company data, I can't use external APIs, so I'm running local models at home for testing.
Here's a summary of what I've tried:
1. 7B class — speed is satisfying, but long document summarization feels a bit lacking
2. 13B class — quality is better, but it's barely fitting on my GPU (12GB)
3. Quantized versions — memory definitely drops, but the answers feel a bit duller
I keep thinking VRAM might be the answer, but then I also wonder if for that money I'd be better off just using an API. If you could share the model and hardware combo you're using, I'd appreciate it.
10 answers
Agreed, with 12GB, a 13B is a bit tough lol
For summarization tasks, adding a few few-shot examples is often much more effective than scaling up the model. Even a 7B model can produce good enough results.
Same here ㅠ I want to get a 3090, but then I think about the electricity bill...
Got a source?
Running Qwen2.5-14B Q4_K_M on a 3060 12GB, and dropping the context to 8192 just barely fits. At first I was using 32k and wondering why it kept dying... The KV cache really eats up a lot of VRAM. In llama.cpp, if you adjust -ngl and offload a few layers to CPU, it'll run even if it's a bit slower. FYI, for long document summarization, the context window and chunking strategy turn out to be more important than model size. Even with a 7B, if you split it up and feed it in, summary quality goes up quite a bit.
I somewhat disagree with the conclusion that you're better off using an API for that kind of money. If you compare only token costs, the API looks cheaper, but once you factor in data exfiltration risk and the staff time spent on reviews and approvals, local is often much cheaper. And if you're saying you can't use it in the first place because it's internal data, that comparison doesn't quite add up.
Well, that's a bit... In practice, the feeling that quantization makes a model duller is often a misperception. If you compare Q4 and Q8 blind, you can hardly tell them apart on short questions. I'd say there are far more cases where answers get weird because the prompt template or chat template was applied incorrectly, and people blame quantization instead. Try an A/B test under the same conditions.
If you have 12GB, take a look at the MoE side. With something like Qwen3-30B-A3B, the active parameter count is around 3B, so the speed is decent and the quality is clearly better than 13B. If you're short on VRAM, even offloading some layers, it still runs at only a few tokens per second, but it's usable. I settled on this combo on a 4070 Ti 12GB. I regret the time I spent clinging to dense 13B.
Ollama is convenient once you install it, but if you want to mess with fine-grained options, you eventually end up on llama.cpp or vLLM lol
A lot of the time, the reason you can't use external APIs because of internal data isn't a real technical problem—it's a security-team/contract issue. Some organizations are covered by clauses saying data won't be used for training if you run something like Azure OpenAI or Bedrock inside a VPC, so checking that first is the way to cut down on wasted effort. Of course, if you're in a regulated industry, that's a non-starter.