My Experience Running a Local LLM at Home (12GB VRAM)
I tried running a bunch of models on an RTX 3060 12GB I had lying around at home. Korean processing showed bigger differences than English, so I decided to write this up.
The 7B–8B class ran better than expected. 14B would run with 4-bit quantization, but it took over 30 seconds just to generate a response, so interactive use was tough. Honestly, 30B and above was a no-go.
Things I ran into during setup:
- I wasted 3 hours because the CUDA version and build version didn't match. Be sure to follow an installation guide that specifies versions from the start.
- Just because it's a Korean model doesn't mean it's all good. You need to pick one with a tokenizer suited to Korean.
- Increasing context length eats VRAM terribly. Going over 4096 causes OOM.
- For your sanity, start with a GUI tool.
What was useful: summarizing English documents and generating regexes were comparable to GPT, while coding assistance was still a bit lacking. Anyway, the fact that it runs without internet made a bigger impression than I expected. It's definitely usable for company intranet environments or organizing personal documents.
8 answers
Agreed—especially the tokenizer point really resonates. I installed it just based on the model name, then saw it butcher Korean and had a reality check.
I really feel you on the CUDA pain... I also got confused between cu121 and cu118, stayed up all night, and eventually escaped to containers. Like you said, following a guide with explicit versions is the fastest way. And with 12GB, 8B at Q4 seems like the sweet spot.
Isn't 30 seconds for a 14B 4-bit model a bit of an exaggeration? On my 3060Ti 8GB, I was getting 12–15 tok/s with Q4_K_M. With ctx 4096, it didn't even take 30 seconds. I'm curious what runtime you used.
Well, saying "English document summarization is no less impressive than GPT" is a bit... maybe for short documents, but if you feed it long ones, hallucinations go way up. If you have the prompts or samples you compared, I'd be convinced.
You're totally right about GUI tools. I started out using only the CLI and spent three whole days just struggling with loading GGUF files lol. It wasn't until I installed LM Studio that I thought, "Ah, so that's what it was." Beginners should definitely start with a GUI.