Even with an RTX 3060 12GB, it's barely enough even with quantization of a 7B model. Running foundation models locally really costs a lot of money...
Do you have a source?
Well, with a 3060 12GB, full precision 7B isn't possible, but it runs well enough with q4. What were you originally trying to run?
I also have a 3060 12GB, and when I run 7B q4_k_m, aside from the RAM usage, it's usable. But 13B is tough.
Yeah, costs a lot lol
Have you tried llama.cpp? Or I'd recommend Ollama or LM Studio. Since system RAM is shared too, a 12GB card can handle most things. VRAM shortage also depends on token length and prompt size. I use a 7B model with 4-bit quantization and 2048 context without issues.
If you use 4-bit GGUF, it should be fine.
Yeah, I know... local LLMs ultimately come down to money... and servers are even more expensive. Running them at home definitely has its limits.
But is there really a reason to run it locally? Cloud APIs are quite cheap these days.