Review: Running a Local LLM on a Mac Mini for a Week
I ran a 7B-class model on an M2 Mac mini with 16GB using Ollama. To get to the conclusion first, it's good enough for document summarization, but coding is still out of reach.
What I like most is being able to feed in internal documents or personal notes without security concerns. It's really convenient for summarizing materials that are awkward to upload to a cloud API by running them locally. At around 7B, the speed didn't feel frustrating.
The downside is quality. With long context, it forgets earlier parts, and Korean responses sometimes come out awkward. To run 32B or larger, memory is insufficient, so I ended up going back to the cloud.
Still, having an "AI that runs without internet" is reassuring. If you run a home server, it's worth trying.
3 answers
Hmm... I'm not so sure. You said "without security issues," but just because it's local doesn't mean it's completely safe. Have you checked whether the model weights or Ollama itself send anything externally? And if you're putting internal company documents into it, there's also the issue of data leaving the company under company policy. "Runs without the internet" also only applies when you first download the model. It's a good intention, but emphasizing "security" seems a bit excessive.
With 16GB, 7B is honestly about the limit lol. 32B isn’t even a dream. I have an M2 Pro with 16GB too, and in the end I barely use it for anything other than summarization. Still, having something that runs offline gives me some peace of mind.