Hyper Input Convex Neural Networks for Shape Constrained Learning and Optimal Transport
Shayan Hundrieser, Insung Kong, Johannes Schmidt-Hieber
TLDR
Introducing HyCNNs, a novel neural network architecture for learning convex functions, combining Maxout and ICNNs for better efficiency and performance.
Key contributions
- HyCNNs combine Maxout networks with ICNNs to create always input-convex neural networks.
- Proven to require exponentially fewer parameters than ICNNs for approximating quadratic functions.
- Outperforms ICNNs and MLPs in convex regression, interpolation, and optimal transport tasks.
- Demonstrates reliable performance at scale, addressing limitations of existing ICNNs.
Why it matters
This paper introduces HyCNNs, a significant advancement in learning convex functions. By improving parameter efficiency and predictive performance over existing ICNNs, it offers a more scalable and accurate tool for shape-constrained learning and optimal transport, crucial for various scientific applications.
Original Abstract
We introduce Hyper Input Convex Neural Networks (HyCNNs), a novel neural network architecture designed for learning convex functions. HyCNNs combine the principles of Maxout networks with input convex neural networks (ICNNs) to create a neural network that is always convex in the input, theoretically capable of leveraging depth, and performs reliable when trained at scale compared to ICNNs. Concretely, we prove that HyCNNs require exponentially fewer parameters than ICNNs to approximate quadratic functions up to a given precision. Throughout a series of synthetic experiments, we demonstrate that HyCNNs outperform existing ICNNs and MLPs in terms of predictive performance for convex regression and interpolation tasks. We further apply HyCNNs to learn high-dimensional optimal transport maps for synthetic examples and for single-cell RNA sequencing data, where they oftentimes outperform ICNN-based neural optimal transport methods and other baselines across a wide range of settings.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.