For those of you running local LLMs, what are you using?
Because of company data, I can't use external APIs, so I'm running local models at home for testing.
Here's a summary of what I've tried:
1. 7B class — speed is satisfying, but long document summarization feels a bit lacking
2. 13B class — quality is better, but it's barely fitting on my GPU (12GB)
3. Quantized versions — memory definitely drops, but the answers feel a bit duller
I keep thinking VRAM might be the answer, but then I also wonder if for that money I'd be better off just using an API. If you could share the model and hardware combo you're using, I'd appreciate it.
10 answers
Agreed, with 12GB, a 13B is a bit tough lol
Running Qwen2.5-14B Q4_K_M on a 3060 12GB, and dropping the context to 8192 just barely fits. At first I was using 32k and wondering why it kept dying... The KV cache really eats up a lot of VRAM. In llama.cpp, if you adjust -ngl and offload a few layers to CPU, it'll run even if it's a bit slower. FYI, for long document summarization, the context window and chunking strategy turn out to be more important than model size. Even with a 7B, if you split it up and feed it in, summary quality goes up quite a bit.
Well, that's a bit... In practice, the feeling that quantization makes a model duller is often a misperception. If you compare Q4 and Q8 blind, you can hardly tell them apart on short questions. I'd say there are far more cases where answers get weird because the prompt template or chat template was applied incorrectly, and people blame quantization instead. Try an A/B test under the same conditions.
Got a source?