A Complete Symmetry Classification of Shallow ReLU Networks
TLDR
This paper provides the first complete symmetry classification for shallow ReLU neural networks, leveraging ReLU's non-differentiability.
Key contributions
- Provides the first complete symmetry classification for shallow ReLU networks.
- Exploits ReLU's non-differentiability to enable this classification.
- Overcomes limitations of previous methods requiring analytic activation functions.
- Addresses parameter identifiability, a key factor in neural network optimization.
Why it matters
Understanding parameter symmetries is crucial as distinct parameters can realize the same function, forming a "neuromanifold" that impacts optimization. This work provides a foundational classification for shallow ReLU networks, a widely used architecture. It opens new avenues for studying neural network geometry and improving training dynamics.
Original Abstract
Parameter space is not function space for neural network architectures. This fact, investigated as early as the 1990s under terms such as ``reverse engineering," or ``parameter identifiability", has led to the natural question of parameter space symmetries\textemdash the study of distinct parameters in neural architectures which realize the same function. Indeed, the quotient space obtained by identifying parameters giving rise to the same function, called the \textit{neuromanifold}, has been shown in some cases to have rich geometric properties, impacting optimization dynamics. Thus far, techniques towards complete classifications have required the analyticity of the activation function, notably excising the important case of ReLU. Here, in contrast, we exploit the non-differentiability of the ReLU activation to provide a complete classification of the symmetries in the shallow case.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.