Notes from Running a Local LLM for 3 Weeks
There was a project at work where we couldn't use external APIs, so I ran local models for about three weeks. The conclusion: usable, but not a silver bullet.
I started with a 7B-class model, but the Korean quality was disappointing, so I moved to 13B, and eventually up to the 30B range. The bigger the model, the better the answers, but GPU memory and electricity costs rose along with it. When I ran it quantized, memory dropped sharply, but it felt a bit fuzzier on long contexts.
What I noticed most was inference speed. If only a few tokens per second come out, users drop off quickly. Even if it meant sacrificing a bit of accuracy, prioritizing speed worked better in real use.
Now I split things: simple tasks like summarization and classification run locally, while complex generation goes to the cloud. A hybrid setup seems like the realistic answer.
6 answers
I'm curious about quantization—what method did you run it with? When I dropped it to 4-bit, it kept losing the earlier parts of long contexts, so I eventually went back up to 8-bit. It saves memory, but missing the earlier context when continuing a conversation is quietly stressful. I was wondering if you compensated with something like prompt compression or summary chaining.
Electricity bills lol. You really can't ignore this. Even if you run it on the company's electricity, once you've got the AC on too in summer... you end up learning firsthand why the cloud is cheap.
Agreed, hybrid turned out to be the realistic answer. I settled on the same thing.
I totally relate to the point that users bail if they don't see even a few tokens come out... Perceived speed matters way more than accuracy.
Well, that's a bit... You can already get decent summarization and classification out of a 7B model, so shouldn't we first figure out whether bumping it up to 30B was because of the model or the prompt? Refining prompts is often cheaper than adding more GPUs. It's a bit disappointing to conclude that 'bigger is better' without evidence.
Without batching, the speed is hopeless, but once I turned on continuation batching with vLLM or llama.cpp server mode, the difference was night and day. Have you tried this? If not, give it a shot—on the same hardware, throughput can be several times higher.