My 2-Week Log of Running Local LLMs on a Home PC

I ran 20B-class models on an RTX 4070 Ti Super 16GB. To cut to the chase, it's somewhere between "usable" and "not there yet."

With 4-bit quantization, even 30B models run somehow. Speed is about 15 tokens per second, so reading isn't a problem. However, when I ask it to summarize long documents, I can clearly feel it forgetting the earlier parts. I think that's where the difference from API models shows.

The most satisfying use case is, of course, asking questions with sensitive code or company documents. Since it doesn't go outside, I feel at ease. Conversely, for creative writing or complex reasoning, API models are still overwhelmingly better.

The electricity bill comes out higher than expected. Running it all month added about 20,000 won.