A Month of Running a 7B Model at Home: My Honest Review

At work I use Claude and GPT, but at home I run a local model to save money. I'm running a quantized model on a 3060 12GB, and it's more usable than I expected.

For one thing, I don't need the internet, and internal documents or personal notes don't leave my machine, so I feel at ease. Speed is around one second for short questions.

That said, the downsides are clear too.

  • Summarizing long documents is noticeably worse
  • Coding works for simple functions, but structural design is too much
  • Having to fiddle with settings again every time I switch models is surprisingly stressful

Conclusion: For work where confidentiality matters or in offline environments, it's definitely worth it; otherwise, you're better off just using an API. Last month's electricity bill was about 8,000 won.

by AI덕후621

9 answers

Agreed, that's a clean conclusion.

by 프롬프트장인4 · ▲0

How did you calculate the 8,000 won electricity bill? Running a 3060 for a month would come out higher than that. I’m curious whether it’s based on idle state or actual usage time.

by 디지털노마드897 · ▲0

With a 3060 12GB, up to 8B at Q4 runs fine. I use a similar setup, but 14B Q4 was a bit sluggish, so I ended up dropping to 8B. Still, once you get it set up, it's definitely convenient after that.

by 뉴비탈출571 · ▲0

The local ecosystem has improved a lot lately. With Ollama, switching models is just a one-line command, so it takes some of the setup stress away. LM Studio is also convenient thanks to its GUI.

by 디지털노마드211 · ▲0

I really relate to using local because of security. Uploading internal documents to the cloud is a risk in itself.

by 밤샘코더381 · ▲0

Oh, I didn't know that.

by 알고리즘고수494 · ▲0

If the conclusion is "otherwise, you're better off just using the API," what was the point of writing the post? lol Isn't this just something each person can decide for themselves?

by 주말개발자169 · ▲0

I have a similar setup, but honestly, I just can't do coding locally. Even for simple functions, it spouts nonsense every now and then, so I end up turning the API back on. And forget about discussing architecture design.

by 지나가던행인692 · ▲0

Well, that’s a bit... These days even 7–8B models summarize quite well, so if performance is dropping, isn’t that more of a model choice or prompt issue? Try reworking your prompt.

by 월급루팡315 · ▲0