Curious about experiences from people who've run local LLMs (Mac vs RTX)
We have a project at work where we can't use external APIs, so I'm looking into local models. I'm curious whether something in the 7B–14B range is actually usable in production, or whether it's better to just pay more and use the cloud.
Based on my research, it roughly comes down to this, but I'd like to hear from people who have actually used them.
- Mac Studio 64GB: Low power and quiet. However, people say prompt processing speed is disappointing.
- RTX 4090: Generation speed is definitely fast, but with only 24GB, going beyond a quantized 14B model is tough.
- Two 3090s: Many say the price-to-performance is good, but can you handle the heat/noise?
Honestly, for the price of a GPU you could cover three years of API costs, but if regulatory or security issues blow up, there's no good answer, so I'm torn. If anyone here has done an on-prem setup, I'd appreciate some realistic advice.
5 answers
Honestly, using 7B–14B for production work is a bit iffy. For narrow tasks like summarization, classification, or extraction, maybe, but for conversational or reasoning-heavy work, even 14B feels noticeably different from cloud models. Don't judge based only on benchmark scores—take about 200 of your actual prompts and run them yourself. That's the fastest way.
I've been running two 3090s for over a year now, and you really have to brace yourself for the heat. Put them in a case and your room turns into a sauna in the summer lol. Noise depends on the preset too, but with the default fan curve you can hear it all the way from the living room. Performance-wise, 24GB×2 means 14B Q5 runs comfortably and batch jobs are easy, so I'm happy. That said, if you can't stomach the power bill and the noise, just get a Mac, seriously.
There are a lot of complaints that Macs are disappointing when it comes to prompt processing speed, but is it really that noticeable in practice? I use an M2 Ultra with 128GB, and generation is slow, but if I just wait a bit for the first token, it's usable after that.
Any source?
If you're going on-prem because of security issues, that's not something to judge by cost. In regulated industries, requirements come as a package—audit logs, access controls, and model version management. Once you do all that, GPU cost is only part of the total cost. I'd recommend starting small with a single 4090 running a quantized 14B model for a PoC, then scaling up based on the response.