My Experience Running a 7B Model at Home (feat. 3080 10GB)
Last night I got inspired and set up a local LLM environment. To get straight to the results: "It's more usable than I expected, but the cloud is more convenient."
Running a 7B model with 4-bit quantization gave me token speeds of about 20–30 per second. Personally, that felt plenty smooth. However, once I increased the context beyond 8k, VRAM started blowing up, so it's unusable for summarizing long documents.
What was interesting was that it handles Korean reasonably well even without fine-tuning. Of course, API models are much better at subtle nuances. In the end, I think I'll only use local models for "sensitive data" or "offline" situations.
7 answers
Agreed, the last paragraph is the key. In the end, it seems like a debate over what the purpose actually is.
20–30 tokens seems a bit low for a 7B 4-bit model, doesn't it? You didn't mention the model or backend, so I can't judge, but at that speed it seems like something is misconfigured. Running it with exllama gets you at least twice that on the same GPU.
Isn't that old news? These days, even 7B-class models run much faster at Q4 lol
Using local models because of sensitive data is the right call, but the maintenance cost is surprisingly high. Model updates, rebuilding quantized versions, aligning prompt templates... in the end, one or two people have to stay on it constantly just to keep things running. So our team eventually settled on using an API behind an internal network proxy.
Is there a source? The post doesn't say what model was run.
I also did some tinkering with a 3080 10GB, and with GGUF Q4_K_M, depending on the model and settings, you can get up to 40–50 tokens/s. 20–30 is a bit underwhelming, so check your llama.cpp build options or batch size. And when going past 8k, the KV cache is the main culprit behind VRAM blowing up; options like --cache-type-k q8_0 make a pretty noticeable difference. After I found that out, I somehow managed to push it to 12k.
I really relate to what you said about Korean nuances. I only use it as an assistant for translation and summarization from the start, and I do the final polishing by hand—for that purpose, local is enough. Honestly, though, I think long documents are too much. The moment you increase the context, speed drops too, so I end up going back to the cloud.