UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo + 5 more
TLDR
UniSD is a unified self-distillation framework that systematically improves LLM adaptation through mechanisms for reliable supervision, alignment, and stability.
Key contributions
- Proposes UniSD, a unified framework to systematically study and improve self-distillation in autoregressive LLMs.
- Integrates mechanisms for reliable supervision, representation alignment, and training stability.
- Reveals when self-distillation works, which components drive gains, and how they interact across tasks.
- UniSDfull improves LLM performance by +5.4 points over base and +2.8 over baselines, enabling efficient adaptation.
Why it matters
UniSD provides a unified framework to overcome challenges in LLM self-distillation. It systematically investigates design choices, revealing key components for effective adaptation. This makes self-distillation a practical and steerable approach for efficiently adapting LLMs without external teachers.
Original Abstract
Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because self-generated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear. In this paper, we propose UniSD, a unified framework to systematically study self-distillation. UniSD integrates complementary mechanisms that address supervision reliability, representation alignment, and training stability, including multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. Across six benchmarks and six models from three model families, UniSD reveals when self-distillation improves over static imitation, which components drive the gains, and how these components interact across tasks. Guided by these insights, we construct UniSDfull, an integrated pipeline that combines complementary components and achieves the strongest overall performance, improving over the base model by +5.4 points and the strongest baseline by +2.8 points. Extensive evaluation highlights self-distillation as a practical and steerable approach for efficient LLM adaptation without stronger external teachers.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.