Has anyone tried running a local LLM at home?
Due to privacy concerns, I can't send company documents to external APIs, so I'm planning to run a local model on my home desktop (RTX 4070, 32GB RAM).
From what I've looked up, around 7B–13B seems realistic—how is the Korean-language performance? My main goal is document summarization rather than coding assistance.
If you've tried this, I'd appreciate model recommendations or setup tips.
8 answers
With a 4070, a 7B quantized model runs like a champ, agreed. Even a 13B is enough at 4-bit, and
To be honest about Korean-language performance, 7B-class models are a bit disappointing. They can summarize, but the sentences often come out sounding like translationese. EXAONE and the Qwen2.5 family were relatively better. In particular, Qwen2.5-14B 4-bit runs on a 4070 12GB if you reduce the context a bit. I recommend starting with Ollama, and if it feels too limited, try changing the quantization yourself with llama.cpp.
Well, shouldn't you first check whether running it locally violates company security policy? Even if the model weights are local, things like output management and random access logs are all subject to policy. It bothers me a bit that they say it's because of personal information but didn't check the regulations.
Got a source? Where did you see that 7B is realistic?
If you're only doing document summarization, there's no need for a big model. Gemma2-9B or Qwen2.5-7B is enough, and instead, giving it ample context (32k or more) matters much more for summary quality. For Korean documents, also check how finely the tokenizer splits Korean. Even with the same 7B, perceived performance varies a lot depending on Korean token efficiency. Ollama is convenient, but it doesn't give you much freedom in parameter settings, so you end up moving to vLLM or llama.cpp lol
Same here. I tried to do at home what I couldn't use at work, but eventually I just settled on local.
Honestly, when you use a 7B to generate summaries, there are many cases where the hallucinations are unmanageable lol. Seeing it sneakily change numbers or proper nouns breaks your trust. If the goal is document summarization, go with 13B or higher, or even if you use 7B, you absolutely must include a verification step that cross-checks against the source text. If you just trust that 'it's safe because it's local' and use the output as-is, it can be even more dangerous.
If you're on Windows, I recommend just dual-booting Linux instead of using WSL2. With 12GB VRAM, 13B 4-bit is pretty much the absolute limit, but on Windows, drivers + background processes eat up VRAM, so you'll hit OOM often. Also, from the start, try running different quantization types (Q4_K_M, Q5_K_M) and compare summarization quality. Quantization makes a much bigger real-world difference than the model name.