I tried fine-tuning a 7B model on internal company data, and QLoRA is truly innovative. It runs fine on a single 24GB GPU.
However, training was too slow, so I needed to optimize. Reducing batch size and using gradient accumulation helped a bit. The results exceeded expectations, but inference latency is a bit of an issue, so I'm considering switching to a lighter model.