My Experience Running Local LLMs at Home (7B: 32B)

Since I work on AI-related things at work, I wanted to run some experiments at home too, so I played around with it for about a month. TL;DR: "More doable than expected, and also less workable than expected."

My home PC specs are a 4070Ti 12GB and 64GB of RAM.

• 7B–8B class is genuinely smooth. At this level, it can handle document summarization, translation, and simple coding questions well enough
• 13B runs if quantized, but the speed drops sharply. I can't use it for time-sensitive work
• 32B was too much. As soon as I loaded it into 12GB VRAM, the speed dropped to a crawl, token by token, so I eventually gave up
• Ollama was the most convenient to use, and LM Studio had a nice UI, so it was good for beginners

What surprised me is that even 8B models had terrible Korean writing quality as recently as last year, but now they're pretty decent. For documents involving personal information, not sending them to the cloud and running locally definitely felt more reassuring.

Next, I'm thinking of upgrading to 128GB of RAM and trying an MoE model.