I tried running a 7B model at home, and it's surprisingly usable
I ran a local LLM on a 12GB graphics card. At first I thought it would be toy-level, but for summarization tasks I couldn't feel much difference from cloud APIs. Of course, it clearly falls behind when it comes to long documents or complex reasoning, but for simple tasks this seems sufficient.
The best part is that it runs without the internet. Summarizing internal company documents is no burden with this. However, when I first set it up, I messed with the quantization options incorrectly and wasted two hours lol.
I'm curious—for those of you actually using local models, what size do you use and what tasks do you mainly use them for?
10 answers
I agree—summarization really seems good enough to do locally now.
I feel you on wasting two hours on the quantization options lol. At first I kept getting OOM errors because I was exceeding VRAM, and I had no idea what the problem was, so I wandered around for a while trying to figure it out.
Do you have a source? Which models were compared, and on which benchmarks?
I like the idea of running internal company documents locally. We can't use the cloud either because of security, but this seems sufficient for that kind of use.
A 14B model barely runs on 12GB. It’s slower, but the quality difference is pretty big, so I went with 14B Q3. Just a heads-up.
I'm using it alongside an embedding model for internal document search RAG. Even with 7B, the retrieval quality is pretty good. The only hassle is that it ties up the GPU every time I need to re-index, haha.
I also run a 7B Q4_K_M quantized model on 12GB, and for summarization and simple classification tasks I can’t really tell the difference from the cloud. But when I use it for coding, it’s clearly lacking, so I end up switching back and forth with a larger model. As for setup, once you get it configured with llama.cpp, it’s smooth sailing after that.
When you say it falls behind on long documents, what's the rough threshold? I'm curious whether it degrades sharply past 8k.
Well, that's a bit... It feels too early to say there's "no big difference from the cloud" after trying a few summaries. Have you checked for hallucinations? The scariest part of local models is how they can quietly slip incorrect sentences in.
Honestly, at this point I wonder whether local 7B models have any meaning beyond a hobby. Cloud API prices have gotten so cheap that when you factor in electricity and time, it hardly seems worth it... Of course, I do acknowledge the advantage of being offline.