Differentially Private Model Merging
Qichuan Yin, Manzil Zaheer, Tian Li
TLDR
Proposes post-processing techniques to merge existing differentially private models, enabling adaptation to any privacy requirement without retraining.
Key contributions
- Proposes two post-processing techniques (random selection, linear combination) to merge differentially private models.
- Enables generating models for any target DP requirement without costly retraining steps.
- Provides privacy accounting using R'enyi DP and privacy loss distributions.
- Theoretically establishes linear combination's superiority over random selection in private mean estimation.
Why it matters
Privacy requirements are dynamic, making it costly to adapt models to new regulations. This paper offers a flexible and efficient post-training solution to meet any target differential privacy level without expensive retraining, significantly reducing operational overhead.
Original Abstract
In machine learning applications, privacy requirements during inference or deployment time could change constantly due to varying policies, regulations, or user experience. In this work, we aim to generate a magnitude of models to satisfy any target differential privacy (DP) requirement without additional training steps, given a set of existing models trained on the same dataset with different privacy/utility tradeoffs. We propose two post processing techniques, namely random selection and linear combination, to output a final private model for any target privacy parameter. We provide privacy accounting of these approaches from the lens of R'enyi DP and privacy loss distributions for general problems. In a case study on private mean estimation, we fully characterize the privacy/utility results and theoretically establish the superiority of linear combination over random selection. Empirically, we validate our approach and analyses on several models and both synthetic and real-world datasets.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.