Sharing My Experience Running Local LLMs Without a GPU
I tried running several quantized models on a desktop I had lying around at home: a Ryzen 5600 with 32GB of RAM. To cut to the chase: "It's usable, but if you want to use it comfortably, a GPU is the way to go."
A 7B-class 4-bit quantized model can answer short questions in about 3–5 seconds. But once the context gets long, the perceived speed drops sharply. Beyond 30B, it's a test of patience.
I don't recommend running locally just to save money; for those who need it for privacy or offline environments, I suggest starting with a small model. With Ollama, installation itself takes only about 10 minutes.
9 answers
Oh, I didn't know that.
For those who use local models for privacy reasons, this article gives exactly the right advice. You can't use the cloud when running sensitive documents. I also agree with starting with small models.
Ollama is really convenient lol. I also struggled with llama.cpp at first and then switched over, and the installation difficulty is on a whole different level. FYI, if you tweak OLLAMA_NUM_PARALLEL or the context settings, the perceived speed changes a bit. Look into it.
30B is a real patience test lol. I waited 20 minutes for an answer, got a reality check, and just shut it off. Honestly, running 30B on 32GB of RAM is a stretch. Once it starts swapping, it’s not even about speed—your hard drive starts crying.
Do you have a source? I’m curious what model and prompt the 3–5 second figure is based on.
For 7B at 4-bit, 3–5 seconds is decent. But the slowdown when context gets longer is due to KV cache and RAM bandwidth, so that's unavoidable—agreed. I'm happy running 14B q4 on a 5600X with 64GB. I only use short contexts anyway, for code autocomplete or summarization.
Well, I think the conclusion that "running locally to save money isn't recommended" is a bit hasty. For someone paying tens of thousands of won a month in API costs, buying a used 3060 12GB would pay for itself in two or three months. Even after factoring in electricity. I feel like lumping it all under "not recommended" is an overgeneralization.
Same here. In the end, local is just a hobby, and for real use I keep coming back to the cloud.
It would have been nice to have the benchmark results organized in a table by quantization model type—too bad. Still, thanks for sharing your experience.