LLM Fine-Tuning Misadventures (with QLoRA)
I tried fine-tuning a 7B model on internal company data, and QLoRA is truly innovative. It runs fine on a single 24GB GPU.
However, training was too slow, so I needed to optimize. Reducing batch size and using gradient accumulation helped a bit. The results exceeded expectations, but inference latency is a bit of an issue, so I'm considering switching to a lighter model.
9 answers
Agreed, QLoRA is amazing.
I also tried it with 24GB and it worked better than expected.
But isn't the learning speed really slow?
Thank you for the tips on adjusting batch size and gradient accumulation. I used 8-bit Adam and found I could reduce memory further. For inference latency, consider lowering the quantization bit or using frameworks like vLLM.
I'm not sure QLoRA is that innovative... It doesn't seem much different from LoRA? In fact, I noticed performance degradation due to quantization.
Haha, I'm also in the middle of a rabbit hole.
Inference latency issues seem to be more affected by prompt length than model size; would caching strategies help improve it?
I also went through a similar struggle, but eventually I put a 7B model with 4-bit quantization on an edge device. I trained it with QLoRA and served it with vLLM, and the latency came out to around 200ms. But data quality turned out to be more important...
Do you have a source? According to the QLoRA paper, it says 65B is possible with 24GB.