Stochastic Gradient Optimization Techniques

Latest papers

• Multi-Task Optimization over Networks of Tasks

  Julian Hatzky, Thomas Bartz-Beielstein, A. E. Eiben, Anil Yaman · 27 April 2026

  Multi-task optimization is a powerful approach for solving a large number of tasks in parallel. However, existing algorithms face distinct limitations: Population-based methods scale poorly and remain underexplored for large task sets. Approaches that do scale beyond a thousand tasks are mostly MAP-…

• Relaxation-Informed Training of Neural Network Surrogate Models

  Calvin Tsay · 27 April 2026

  ReLU neural networks trained as surrogate models can be embedded exactly in mixed-integer linear programs (MILPs), enabling global optimization over the learned function. The tractability of the resulting MILP depends on structural properties of the network, i.e., the number of binary variables in a…

• SOC-ICNN: From Polyhedral to Conic Geometry for Learning Convex Surrogate Functions

  Kang Liu, Jianchen Hu · 27 April 2026

  Classical ReLU-based Input Convex Neural Networks (ICNNs) are equivalent to the optimal value functions of Linear Programming (LP). This intrinsic structural equivalence restricts their representational capacity to piecewise-linear polyhedral functions. To overcome this representational bottleneck, …

• Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction

  Ziyang Wei, Wanrong Zhu, Jingyang Lyu, Wei Biao Wu · 24 April 2026

  We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence.…

• SGD at the Edge of Stability: The Stochastic Sharpness Gap

  Fangshuo Liao, Afroditi Kolomvaki, Anastasios Kyrillidis · 24 April 2026

  When training neural networks with full-batch gradient descent (GD) and step size $\eta$, the largest eigenvalue of the Hessian -- the sharpness $S(\boldsymbol{\theta})$ -- rises to $2/\eta$ and hovers there, a phenomenon termed the Edge of Stability (EoS). \citet{damian2023selfstab} showed that thi…

• Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

  Eylon E. Krause · 24 April 2026

  The choice of activation function plays a crucial role in the optimization and performance of deep neural networks. While the Rectified Linear Unit (ReLU) remains the dominant choice due to its simplicity and effectiveness, its lack of smoothness may hinder gradient-based optimization in deep archit…

• ILDR: Geometric Early Detection of Grokking

  Shreel Golwala · 24 April 2026

  Grokking describes a delayed generalization phenomenon in which a neural network achieves perfect training accuracy long before validation accuracy improves, followed by an abrupt transition to strong generalization. Existing detection signals are indirect: weight norm reflects parameter-space regul…

• The Origin of Edge of Stability

  Elon Litman · 23 April 2026

  Full-batch gradient descent on neural networks drives the largest Hessian eigenvalue to the threshold $2/\eta$, where $\eta$ is the learning rate. This phenomenon, the Edge of Stability, has resisted a unified explanation: existing accounts establish self-regulation near the edge but do not explain …

• On the Stability and Generalization of First-order Bilevel Minimax Optimization

  Xuelin Zhang, Peipei Yuan · 23 April 2026

  Bilevel optimization and bilevel minimax optimization have recently emerged as unifying frameworks for a range of machine-learning tasks, including hyperparameter optimization and reinforcement learning. The existing literature focuses on empirical efficiency and convergence guarantees, leaving a cr…

• Properties and limitations of geometric tempering for gradient flow dynamics

  Francesca Romana Crucinio, Sahani Pathiraja · 23 April 2026

  We consider the problem of sampling from a probability distribution $\pi$. It is well known that this can be written as an optimisation problem over the space of probability distributions in which we aim to minimise the Kullback--Leibler divergence from $\pi$. We consider the effect of replacing $…

• Too Sharp, Too Sure: When Calibration Follows Curvature

  Alessandro Morosini, Matea Gjika, Tomaso Poggio, Pierfrancesco Beneventano · 23 April 2026

  Modern neural networks can achieve high accuracy while remaining poorly calibrated, producing confidence estimates that do not match empirical correctness. Yet calibration is often treated as a post-hoc attribute. We take a different perspective: we study calibration as a training-time phenomenon on…

• Geometric Layer-wise Approximation Rates for Deep Networks

  Shijun Zhang, Zuowei Shen, Yuesheng Xu · 23 April 2026

  Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantit…

• Phase Transitions in the Fluctuations of Functionals of Random Neural Networks

  Simmaco Di Lillo, Leonardo Maini, Domenico Marinucci · 22 April 2026

  We establish central and non-central limit theorems for sequences of functionals of the Gaussian output of an infinitely-wide random neural network on the d-dimensional sphere . We show that the asymptotic behaviour of these functionals as the depth of the network increases depends crucially on the …

• Generalization at the Edge of Stability

  Mario Tuci, Caner Korkmaz, Umut \c{S}im\c{s}ekli, Tolga Birdal · 22 April 2026

  Training modern neural networks often relies on large learning rates, operating at the edge of stability, where the optimization dynamics exhibit oscillatory and chaotic behavior. Empirically, this regime often yields improved generalization performance, yet the underlying mechanism remains poorly u…

• Curiosity-Critic: Cumulative Prediction Error Improvement as a Tractable Intrinsic Reward for World Model Training

  Vin Bhaskara, Haicheng Wang · 22 April 2026

  Local prediction-error-based curiosity rewards focus on the current transition without considering the world model's cumulative prediction error across all visited transitions. We introduce Curiosity-Critic, which grounds its intrinsic reward in the improvement of this cumulative objective, and show…

• Local Updates in Distributed Optimization: Provable Acceleration and Topology Effects

  Zuang Wang, Yongqiang Wang · 22 April 2026

  Inspired by the success of performing multiple local optimization steps between communication rounds in federated learning, incorporating such local updates into distributed optimization has recently attracted growing interest. However, unlike federated learning, where local updates can accelerate t…

• PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

  Chenyang Wang, Yun Yang · 21 April 2026

  We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers,…

• Trajectory-Restricted Optimization Conditions and Geometry-Aware Linear Convergence

  Faris Chaudhry, Anthea Monod, Keisuke Yano · 21 April 2026

  Linear convergence of first-order methods is typically characterized by global optimization conditions whose constants reflect worst-case geometry of the ambient space. In high-dimensional or structured problems, these global constants can be arbitrarily conservative and fail to capture the geometry…

• Negative Momentum for Convex-Concave Optimization

  Henry Shugart, Shuyi Wang, Jason M. Altschuler · 21 April 2026

  This paper revisits momentum in the context of min-max optimization. Momentum is a celebrated mechanism for accelerating gradient dynamics in settings like convex minimization, but its direct use in min-max optimization makes gradient dynamics diverge. Surprisingly, Gidel et al. 2019 showed that neg…

• Duality for the Adversarial Total Variation

  Leon Bungert, Lucas Schmitt · 21 April 2026

  Adversarial training of binary classifiers can be reformulated as regularized risk minimization involving a nonlocal total variation. Building on this perspective, we establish a characterization of the subdifferential of this total variation using duality techniques. To achieve this, we derive a du…

• BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation"

  Vladimer Khasia · 21 April 2026

  The activation memory required for exact backpropagation scales linearly with network depth, context length, and feature dimensionality, forming an O(L * BN ) spatial bottleneck (where B is the sequence-batch cardinality and N is the feature dimension). This constraint historically throttles the sca…

• A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad, Shampoo and Muo

  S. Gratton, Ph. L. Toint · 21 April 2026

  A unified framework for first-order optimization algorithms fornonconvex unconstrained optimization is proposed that uses adaptivelypreconditioned gradients and includes popular methods such as full anddiagonal AdaGrad, AdaNorm, as well as adpative variants of Shampoo andMuon. This framework also al…

• Continuous Limits of Coupled Flows in Representation Learning

  Zilin Li, Weiwei Xu, Xuchun Tong, Xuanbo Lu, Xuanqi Zhao · 21 April 2026

  While modern representation learning relies heavily on global error signals, decentralized algorithms driven by local interactions offer a fundamental distributed alternative. However, the macroscopic convergence properties of these discrete dynamics on continuous data manifolds remain theoretically…

• Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis

  Yuheng Zhao, Yu-Hu Yan, Amit Attia, Tomer Koren, Lijun Zhang, Peng Zhao · 21 April 2026

  Parameter-free stochastic optimization aims to design algorithms that are agnostic to the underlying problem parameters while still achieving convergence rates competitive with optimally tuned methods. While some parameter-free methods do not require the specific values of the problem parameters, th…

• Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussion

  Ou Wu · 21 April 2026

  Large language model optimization has historically bifurcated into isolated data-centric and model-centric paradigms: the former manipulates involved samples through selection, augmentation, or poisoning, while the latter tunes model weights via masking, quantization, or low-rank adaptation. This pa…