What's a realistic local LLM to run at home with a 3060 12GB?
I'm in a situation where I can't upload company data to external APIs, so I'm trying to run one locally. I only have one GPU—a 3060 12GB—and 32GB of RAM.
I've seen a lot of posts saying 7B–8B quantized models are usable, but if anyone has actually tried them for Korean summarization, I'd appreciate it if you could share which model combination worked well.
5 answers
With a 3060 12GB, an 8B model at Q4 quantization just runs. Agreed. With 32GB of RAM, it can handle a bit more context too.
From someone who’s run local models at work for a similar reason: for summarization, the exaone 3.5 7.8B or gemma2 9B Q4_K_M combo was good enough. Running it with llama.cpp, it was practically usable up to around 8k context on 12GB VRAM; push it further and offloading kicks in, making the perceived speed drop sharply. That said, Korean summarization quality is heavily prompt-dependent, so be sure to include constraints like “Do not add anything not in the original text.” Otherwise, it quite often makes up things that aren’t there.
Well, that’s a bit... Most of the posts saying 7–8B models are 'good enough' are based on English benchmarks, so in real-world Korean summarization use, they don’t perform as well as expected. When you feed them long documents, they often lose the beginning and get lost by the end. For company data, given the risk of false positives/hallucinations, don’t trust the results as-is; you should use it on the premise that a human will review them. I think expecting 'work automation' from a single 12GB card is a bit too much.
Got a source? I'm curious which benchmark this comparison is based on.
The person above is being pessimistic, but I disagree. For summarization, the pipeline matters more than the model size. If you split the document, summarize each chunk, then merge them back, you can get good results even with an 8B model. I alternate between qwen2.5 7B instr and exaone; exaone is more natural for Korean style, while qwen follows instructions better. I recommend quickly testing with ollama, and if it’s okay, moving on to vllm. If you only have one 3060, run it with batch size 1 only.