Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle
Filippo Giovagnini, Sotirios Kotitsas, Marco Romito
TLDR
This paper proves quantitative bounds on the 2-Wasserstein distance between infinite-width deep neural networks and their Gaussian limit.
Key contributions
- Establishes quantitative bounds on the 2-Wasserstein distance for infinite-width DNNs.
- Compares fully connected deep neural networks with general weights to their Gaussian limit.
- Introduces a novel Lindeberg principle tailored for Deep Neural Networks.
Why it matters
This work provides crucial theoretical insights into the universality of deep neural networks in their infinite-width limit. It quantifies the distance to a Gaussian limit, explaining why large networks often behave like Gaussian processes. The new Lindeberg principle offers a powerful analytical tool.
Original Abstract
We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the $2$-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.