Why is everyone so hooked on local LLMs these days?

There's talk at work about using only local models because of security, but what decent models are there for 24GB VRAM? llama3 70B is out of the question—is around 8B the limit?

If anyone has actually used them in practice, please let me know.

by 뉴비탈출668

7 answers

With 24GB, you can run an 8B model quantized just fine. Speed isn't bad either.

by 무한도전러337 · ▲0