Review: Two Weeks of Running a Local LLM on My Home PC

I'm running a quantized 14B-class model on a 4070 Ti Super with 32GB of RAM. To get straight to the point, it's definitely usable for "tasks where you can't upload company confidential data," but for everything else, the cloud is better.

Code autocomplete and summarization work well enough locally. In particular, a big advantage is being able to freely throw in internal documents or personal notes for summarization without worrying about tokens.

On the other hand, complex reasoning and long context are still clearly lacking. When I run the same prompt on GPT, it gets it done in one shot, but locally I have to ask three times. It also draws a fair amount of power.

So now I split usage by purpose. Sensitive stuff locally, everything else in the cloud. This combination is the most satisfying.

by 무한도전러124

5 answers

Agreed — for sensitive documents, local is the way to go. I'm jealous; at my company, documents are locked down so tightly I can't even touch them.

by 코딩하는곰166 · ▲0