Diverse Image Priors for Black-box Data-free Knowledge Distillation
Tri-Nhan Vo, Dang Nguyen, Trung Le, Kien Do, Sunil Gupta
TLDR
DIP-KD improves black-box data-free knowledge distillation by synthesizing diverse image priors and using a novel primer student for state-of-the-art results.
Key contributions
- Synthesizes diverse image priors to capture varied visual patterns and semantics.
- Employs contrastive learning to enhance distinction among synthetic data samples.
- Utilizes a novel primer student for effective soft-probability knowledge distillation.
- Achieves state-of-the-art performance in black-box data-free KD across 12 benchmarks.
Why it matters
This paper addresses a critical challenge in secure and decentralized AI: knowledge distillation without access to original data or teacher models. By generating diverse synthetic data, DIP-KD enables efficient model transfer in privacy-sensitive environments. This advances practical applications of KD where data access is restricted.
Original Abstract
Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher's interface and original datasets. These constraints define a challenging black-box data-free KD scenario where only top-1 predictions and no training data are available. While recent approaches utilize synthetic data, they still face limitations in data diversity and distillation signals. We propose Diverse Image Priors Knowledge Distillation (DIP-KD), a framework that addresses these challenges through a three-phase collaborative pipeline: (1) Synthesis of image priors to capture diverse visual patterns and semantics; (2) Contrast to enhance the collective distinction between synthetic samples via contrastive learning; and (3) Distillation via a novel primer student that enables soft-probability KD. Our evaluation across 12 benchmarks shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.