Phase Transitions in the Fluctuations of Functionals of Random Neural Networks
Simmaco Di Lillo, Leonardo Maini, Domenico Marinucci
TLDR
This paper reveals phase transitions in random neural network fluctuations, showing asymptotic behavior depends on covariance fixed points, leading to three distinct limit regimes.
Key contributions
- Establishes central/non-central limit theorems for functionals of wide random neural networks.
- Identifies three distinct limiting regimes based on the covariance function's fixed points.
- Regimes include convergence to a Gaussian field functional, Gaussian distribution, or Qth Wiener chaos.
- Introduces novel use of the covariance's iterative operator fixed-point structure to explain regimes.
Why it matters
This research uncovers fundamental "phase transitions" in large random neural networks, linking their asymptotic fluctuations to covariance fixed points. Understanding these distinct limiting regimes is crucial for predicting and controlling deep learning model statistical properties. It offers new theoretical tools for analyzing network dynamics.
Original Abstract
We establish central and non-central limit theorems for sequences of functionals of the Gaussian output of an infinitely-wide random neural network on the d-dimensional sphere . We show that the asymptotic behaviour of these functionals as the depth of the network increases depends crucially on the fixed points of the covariance function, resulting in three distinct limiting regimes: convergence to the same functional of a limiting Gaussian field, convergence to a Gaussian distribution, convergence to a distribution in the Qth Wiener chaos. Our proofs exploit tools that are now classical (Hermite expansions, Diagram Formula, Stein-Malliavin techniques), but also ideas which have never been used in similar contexts: in particular, the asymptotic behaviour is determined by the fixed-point structure of the iterative operator associated with the covariance, whose nature and stability governs the different limiting regimes.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.