Stochastic Gradient Optimization Techniques

Latest papers

• Convergence of gradient descent for deep neural networks

  Sourav Chatterjee · 23 February 2026

  We give a simple local Polyak-Lojasiewicz (PL) criterion that guarantees linear (exponential) convergence of gradient flow and gradient descent to a zero-loss solution of a nonnegative objective. We then verify this criterion for the squared training loss of a feedforward neural network with smooth,…

• Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors

  Jingquan Yan, Yuwei Miao, Peiran Yu, Junzhou Huang · 23 February 2026

  Attention-based regression models are often trained by jointly optimizing Mean Squared Error (MSE) loss and Pearson correlation coefficient (PCC) loss, emphasizing the magnitude of errors and the order or shape of targets, respectively. A common but poorly understood phenomenon during training is th…

• Interactive Learning of Single-Index Models via Stochastic Gradient Descent

  Nived Rajaraman, Yanjun Han · 23 February 2026

  Stochastic gradient descent (SGD) is a cornerstone algorithm for high-dimensional optimization, renowned for its empirical successes. Recent theoretical advances have provided a deep understanding of how SGD enables feature learning in high-dimensional nonlinear models, most notably the \textit{sing…

• Asynchronous Heavy-Tailed Optimization

  Junfei Sun, Dixi Yao, Xuchen Gong, Tahseen Rabbani, Manzil Zaheer, Tian Li · 23 February 2026

  Heavy-tailed stochastic gradient noise, commonly observed in transformer models, can destabilize the optimization process. Recent works mainly focus on developing and understanding approaches to address heavy-tailed noise in the centralized or distributed, synchronous setting, leaving the interactio…

• Understanding the Generalization of Bilevel Programming in Hyperparameter Optimization: A Tale of Bias-Variance Decomposition

  Yubo Zhou, Jun Shu, Junmin Liu, Deyu Meng · 23 February 2026

  Gradient-based hyperparameter optimization (HPO) have emerged recently, leveraging bilevel programming techniques to optimize hyperparameter by estimating hypergradient w.r.t. validation loss. Nevertheless, previous theoretical works mainly focus on reducing the gap between the estimation and ground…

• Asymptotic Smoothing of the Lipschitz Loss Landscape in Overparameterized One-Hidden-Layer ReLU Networks

  Saveliy Baturin · 20 February 2026

  We study the topology of the loss landscape of one-hidden-layer ReLU networks under overparameterization. On the theory side, we (i) prove that for convex $L$-Lipschitz losses with an $\ell_1$-regularized second layer, every pair of models at the same loss level can be connected by a continuous path…

• Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking

  Afroditi Kolomvaki, Fangshuo Liao, Evan Dramko, Ziyun Guang, Anastasios Kyrillidis · 20 February 2026

  We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each use…

• Powering Up Zeroth-Order Training via Subspace Gradient Orthogonalization

  Yicheng Lang, Changsheng Wang, Yihua Zhang, Mingyi Hong, Zheng Zhang, Wotao Yin, Sijia Liu · 20 February 2026

  Zeroth-order (ZO) optimization provides a gradient-free alternative to first-order (FO) methods by estimating gradients via finite differences of function evaluations, and has recently emerged as a memory-efficient paradigm for fine-tuning large-scale models by avoiding backpropagation. However, ZO …

• Gradient Testing and Estimation by Comparisons

  Xiwen Tao, Chenyi Zhang, Helin Wang, Yexin Zhang, Tongyang Li · 20 February 2026

  We study gradient testing and gradient estimation of smooth functions using only a comparison oracle that, given two points, indicates which one has the larger function value. For any smooth $f\colon\mathbb R^n\to\mathbb R$, $\mathbf{x}\in\mathbb R^n$, and $\varepsilon>0$, we design a gradient testi…

• Malliavin Calculus as Stochastic Backpropogation

  Kevin D. Oden · 20 February 2026

  We establish a rigorous connection between pathwise (reparameterization) and score-function (Malliavin) gradient estimators by showing that both arise from the Malliavin integration-by-parts identity. Building on this equivalence, we introduce a unified and variance-aware hybrid estimator that adapt…

• Sign Lock-In: Randomly Initialized Weight Signs Persist and Bottleneck Sub-Bit Model Compression

  Akira Sakai, Yuma Ichikawa · 20 February 2026

  Sub-bit model compression seeks storage below one bit per weight; as magnitudes are aggressively compressed, the sign bit becomes a fixed-cost bottleneck. Across Transformers, CNNs, and MLPs, learned sign matrices resist low-rank approximation and are spectrally indistinguishable from an i.i.d. Rade…

• Low-Dimensional and Transversely Curved Optimization Dynamics in Grokking

  Yongzhong Xu · 20 February 2026

  Grokking -- the delayed transition from memorization to generalization in small algorithmic tasks -- remains poorly understood. We present a geometric analysis of optimization dynamics in transformers trained on modular arithmetic. PCA of attention weight trajectories reveals that training evolves p…

• Differentially Private Non-convex Distributionally Robust Optimization

  Difei Xu, Meng Ding, Zebin Ma, Huanyi Xie, Youming Tao, Aicha Slaitane, Di Wang · 19 February 2026

  Real-world deployments routinely face distribution shifts, group imbalances, and adversarial perturbations, under which the traditional Empirical Risk Minimization (ERM) framework can degrade severely. Distributionally Robust Optimization (DRO) addresses this issue by optimizing the worst-case exp…

• Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC

  Abdulla Jasem Almansoori, Maria Ivanova, Andrey Veprikov, Aleksandr Beznosikov, Samuel Horv\'ath, Martin Tak\'a\v{c} · 19 February 2026

  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning. We propose LoRS…

• High entropy leads to symmetry equivariant policies in Dec-POMDPs

  Johannes Forkel, Constantin Ruhdorfer, Andreas Bulling, Jakob Foerster · 19 February 2026

  We prove that in any Dec-POMDP, sufficiently high entropy regularization ensures that policy gradient ascent with tabular softmax parametrization always converges, for any initialization, to the same joint policy, and that this joint policy is equivariant w.r.t. all symmetries of the Dec-POMDP. In p…

• Error Propagation and Model Collapse in Diffusion Models: A Theoretical Study

  Nail B. Khelifa, Richard E. Turner, Ramji Venkataramanan · 19 February 2026

  Machine learning models are increasingly trained or fine-tuned on synthetic data. Recursively training on such data has been observed to significantly degrade performance in a wide range of tasks, often characterized by a progressive drift away from the target distribution. In this work, we theoreti…

• Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks

  Jayadev Billa · 19 February 2026

  Capability emergence during neural network training remains mechanistically opaque. We track five geometric measures across five model scales (405K-85M parameters), 120+ emergence events in eight algorithmic tasks, and three Pythia language models (160M-2.8B). We find: (1) training begins with a uni…

• Benchmarking Stochastic Approximation Algorithms for Fairness-Constrained Training of Deep Neural Networks

  Andrii Kliachkin, Jana Lep\v{s}ov\'a, Gilles Bareilles, Jakub Mare\v{c}ek · 19 February 2026

  The ability to train Deep Neural Networks (DNNs) with constraints is instrumental in improving the fairness of modern machine-learning models. Many algorithms have been analysed in recent years, and yet there is no standard, widely accepted method for the constrained training of DNNs. In this paper,…

• The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks

  Eitan Gronich, Gal Vardi · 19 February 2026

  We study the implicit bias of momentum-based optimizers on homogeneous models. We first extend existing results on the implicit bias of steepest descent in homogeneous models to normalized steepest descent with an optional learning rate schedule. We then show that for smooth homogeneous models, mome…

• Optimizer choice matters for the emergence of Neural Collapse

  Jim Zhao, Tin Sum Cheng, Wojciech Masarczyk, Aurelien Lucchi · 19 February 2026

  Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains limited. Existing analyses largely ignore the role of the op…

• Conjugate Learning Theory: Uncovering the Mechanisms of Trainability and Generalization in Deep Neural Networks

  Binchuan Qi · 19 February 2026

  In this work, we propose a notion of practical learnability grounded in finite sample settings, and develop a conjugate learning theoretical framework based on convex conjugate duality to characterize this learnability property. Building on this foundation, we demonstrate that training deep neural n…

• On Surprising Effectiveness of Masking Updates in Adaptive Optimizers

  Taejong Joo, Wenhan Xia, Cheolmin Kim, Ming Zhang, Eugene Ie · 18 February 2026

  Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We challenge this by showing that randomly masking parameter updates can be highly effective, with a masked variant of RMSProp consistently outperforming rece…

• ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

  Ziyu Zhao, Tong Zhu, Zhi Zhang, Tiantian Fan, Jinluan Yang, Kun Kuang, Zhongyu Wei, Fei Wu, Yu Cheng · 18 February 2026

  Mixture-of-Experts (MoE) effectively scales model capacity while preserving computational efficiency through sparse expert activation. However, training high-quality MoEs from scratch is prohibitively expensive. A promising alternative is to convert pretrained dense models into sparse MoEs. Existing…

• POP: Prior-fitted Optimizer Policies

  Jan Kobiolka, Christian Frey, Gresa Shala, Arlind Kadra, Erind Bedalli, Josif Grabocka · 18 February 2026

  Optimization refers to the task of finding extrema of an objective function. Classical gradient-based optimizers are highly sensitive to hyperparameter choices. In highly non-convex settings their performance relies on carefully tuned learning rates, momentum, and gradient accumulation. To address t…

• Functional Central Limit Theorem for Stochastic Gradient Descent

  Kessang Flamand, Victor-Emmanuel Brunel · 18 February 2026

  We study the asymptotic shape of the trajectory of the stochastic gradient descent algorithm applied to a convex objective function. Under mild regularity assumptions, we prove a functional central limit theorem for the properly rescaled trajectory. Our result characterizes the long-term fluctuation…