A Log of My Failed Attempts to Run a 7B Model at Home

Since I can't use the company GPU, I decided to try running a local LLM at home and poured two days of my weekend into it. To start with the result: it does run, but it killed any desire I had to use it. lol

My setup is an RTX 3060 12GB with 32GB of RAM. I started with Ollama, and installation itself took only about 10 minutes. The problem was what came after.

1. A 7B 4-bit quantized model runs quite comfortably. Token speed is also decent.

2. But once you go past 13B, it feels noticeably slower. It's at the level where you can go grab a coffee while waiting for a single response.

3. If you set a long context, speed drops further and memory gets eaten up in an instant.

4. When I loaded an embedding model too in order to add RAG, the GPU started screaming.

In the end, I concluded that using an API is better. Still, learning firsthand how this stuff works by running it locally was a valuable experience in its own way. In particular, seeing directly how quality degrades at each quantization level was a big takeaway.

If anyone is actually using a 3060 12GB day to day, please share your model and settings. I still can't let it go.