Trying to run LLaMA locally but VRAM is insufficient lol
Even with an RTX 3060 12GB, it's barely enough even with quantization of a 7B model. Running foundation models locally really costs a lot of money...
8 answers
Yeah, costs a lot lol
If you use 4-bit GGUF, it should be fine.
Do you have a source?
I also have a 3060 12GB, and when I run 7B q4_k_m, aside from the RAM usage, it's usable. But 13B is tough.
Well, with a 3060 12GB, full precision 7B isn't possible, but it runs well enough with q4. What were you originally trying to run?
Have you tried llama.cpp? Or I'd recommend Ollama or LM Studio. Since system RAM is shared too, a 12GB card can handle most things. VRAM shortage also depends on token length and prompt size. I use a 7B model with 4-bit quantization and 2048 context without issues.
Yeah, I know... local LLMs ultimately come down to money... and servers are even more expensive. Running them at home definitely has its limits.
But is there really a reason to run it locally? Cloud APIs are quite cheap these days.