A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity
Damek Davis, Dmitriy Drusvyatskiy
TLDR
This paper offers a simpler, direct proof for near-linear convergence of adaptive gradient descent under convexity and fourth-order growth.
Key contributions
- Provides a direct Lyapunov-based proof for adaptive gradient descent convergence.
- Simplifies prior intricate arguments that relied on monitoring "ravine" manifolds.
- Applies to convex functions with unique minimizers and fourth-order growth.
- Introduces a more adaptive variant of the algorithm with better numerical results.
Why it matters
This paper significantly simplifies the proof for adaptive gradient descent's near-linear convergence, making it more accessible. By using a direct Lyapunov argument, it bypasses prior complexities and offers a more robust theoretical foundation. The proposed adaptive variant also shows practical performance benefits.
Original Abstract
Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth -- called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.