GiVA: Gradient-Informed Bases for Vector-Based Adaptation
Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky, Hancao Li, Vivek Mittal + 2 more
TLDR
GiVA introduces a gradient-informed initialization for vector-based adaptation, achieving LoRA-comparable performance with 8x lower rank and extreme parameter efficiency.
Key contributions
- GiVA is a gradient-based initialization strategy for vector-based adaptation (VBA).
- Matches LoRA's performance while maintaining VBA's extreme parameter efficiency.
- Reduces rank requirements for vector-based adaptation by a factor of eight (8x).
- Achieves training times comparable to LoRA across diverse benchmarks.
Why it matters
Vector-based adaptation offers extreme parameter efficiency but often requires high ranks, increasing training costs. GiVA addresses this by significantly reducing rank requirements while maintaining performance. This advancement makes highly efficient fine-tuning more practical and accessible for large models.
Original Abstract
As model sizes continue to grow, parameter-efficient fine-tuning has emerged as a powerful alternative to full fine-tuning. While LoRA is widely adopted among these methods, recent research has explored vector-based adaptation methods due to their extreme parameter efficiency. However, these methods typically require substantially higher ranks than LoRA to match its performance, leading to increased training costs. This work introduces GiVA, a gradient-based initialization strategy for vector-based adaptation. It achieves training times comparable to LoRA and maintains the extreme parameter efficiency of vector-based adaptation. We evaluate GiVA across diverse benchmarks, including natural language understanding, natural language generation, and image classification. Experiments show that our approach consistently outperforms or achieves performance competitive with existing vector-based adaptation methods and LoRA while reducing rank requirements by a factor of eight ($8\times$).
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.