A Month of Running a 7B Model at Home: My Honest Review
At work I use Claude and GPT, but at home I run a local model to save money. I'm running a quantized model on a 3060 12GB, and it's more usable than I expected.
For one thing, I don't need the internet, and internal documents or personal notes don't leave my machine, so I feel at ease. Speed is around one second for short questions.
That said, the downsides are clear too.
- Summarizing long documents is noticeably worse
- Coding works for simple functions, but structural design is too much
- Having to fiddle with settings again every time I switch models is surprisingly stressful
Conclusion: For work where confidentiality matters or in offline environments, it's definitely worth it; otherwise, you're better off just using an API. Last month's electricity bill was about 8,000 won.
9 answers
Agreed, that's a clean conclusion.
How did you calculate the 8,000 won electricity bill? Running a 3060 for a month would come out higher than that. I’m curious whether it’s based on idle state or actual usage time.
With a 3060 12GB, up to 8B at Q4 runs fine. I use a similar setup, but 14B Q4 was a bit sluggish, so I ended up dropping to 8B. Still, once you get it set up, it's definitely convenient after that.
The local ecosystem has improved a lot lately. With Ollama, switching models is just a one-line command, so it takes some of the setup stress away. LM Studio is also convenient thanks to its GUI.
I really relate to using local because of security. Uploading internal documents to the cloud is a risk in itself.
Oh, I didn't know that.
If the conclusion is "otherwise, you're better off just using the API," what was the point of writing the post? lol Isn't this just something each person can decide for themselves?
I have a similar setup, but honestly, I just can't do coding locally. Even for simple functions, it spouts nonsense every now and then, so I end up turning the API back on. And forget about discussing architecture design.
Well, that’s a bit... These days even 7–8B models summarize quite well, so if performance is dropping, isn’t that more of a model choice or prompt issue? Try reworking your prompt.