Stochastic Gradient Optimization Techniques

Neueste Paper

• DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuning

  Mihaela Hudi\c{s}teanu, Edwige Cyffers, Nikita P. Kalinin · 26. November 2025

  Adaptive optimizers are the de facto standard in non-private training as they often enable faster convergence and improved performance. In contrast, differentially private (DP) training is still predominantly performed with DP-SGD, typically requiring extensive compute and hyperparameter tuning. We …

• ROOT: Robust Orthogonalized Optimizer for Neural Network Training

  Wei He, Kai Han, Hang Zhou, Hanting Chen, Zhicheng Liu, Xinghao Chen, Yunhe Wang · 26. November 2025

  The optimization of large language models (LLMs) remains a critical challenge, particularly as model scaling exacerbates sensitivity to algorithmic imprecision and training instability. Recent advances in optimizers have improved convergence efficiency through momentum orthogonalization, but suffer …

• A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descent

  Shuo Xie, Tianhao Wang, Beining Wu, Zhiyuan Li · 26. November 2025

  Adaptive optimizers can reduce to normalized steepest descent (NSD) when only adapting to the current gradient, suggesting a close connection between the two algorithmic families. A key distinction between their analyses, however, lies in the geometries, e.g., smoothness notions, they rely on. In th…

• HVAdam: A Full-Dimension Adaptive Optimizer

  Yiheng Zhang, Shaowu Wu, Yuanzhuo Xu, Jiajun Wu, Shang Xu, Steve Drew, Xiaoguang Niu · 26. November 2025

  Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse than non-adaptive methods, such as SGD on classical architectures like CNNs. We identify a key cause of this performance g…

• Row-stochastic matrices can provably outperform doubly stochastic matrices in decentralized learning

  Bing Liu, Boao Kong, Limin Lu, Kun Yuan, Chengcheng Zhao · 26. November 2025

  Decentralized learning often involves a weighted global loss with heterogeneous node weights $\lambda$. We revisit two natural strategies for incorporating these weights: (i) embedding them into the local losses to retain a uniform weight (and thus a doubly stochastic matrix), and (ii) keeping the o…

• Adam Simplified: Bias Correction Simplified

  Sam Laing, Antonio Orvieto · 26. November 2025

  The Adam optimizer is a cornerstone of modern deep learning, yet the empirical necessity of each of its individual components is often taken for granted. This paper presents a focused investigation into the role of bias-correction, a feature whose contribution remains poorly understood. Through a se…

• Designing Preconditioners for SGD: Local Conditioning, Noise Floors, and Basin Stability

  Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi · 26. November 2025

  Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise. We analyze preconditioned SGD in the geometry induced by a symmetric positive definite matrix $\mathbf{M}$, deriving bounds in which both the convergence rate and the stochast…

• AdaCap: An Adaptive Contrastive Approach for Small-Data Neural Networks

  Bruno Belucci, Karim Lounici, Katia Meziani · 26. November 2025

  Neural networks struggle on small tabular datasets, where tree-based models remain dominant. We introduce Adaptive Contrastive Approach (AdaCap), a training scheme that combines a permutation-based contrastive loss with a Tikhonov-based closed-form output mapping. Across 85 real-world regression dat…

• On the Limits of Momentum in Decentralized and Federated Optimization

  Riccardo Zaccone, Sai Praneeth Karimireddy, Carlo Masone · 26. November 2025

  Recent works have explored the use of momentum in local methods to enhance distributed SGD. This is particularly appealing in Federated Learning (FL), where momentum intuitively appears as a solution to mitigate the effects of statistical heterogeneity. Despite recent progress in this direction, it …

• Lower Complexity Bounds for Nonconvex-Strongly-Convex Bilevel Optimization with First-Order Oracles

  Kaiyi Ji · 26. November 2025

  Although upper bound guarantees for bilevel optimization have been widely studied, progress on lower bounds has been limited due to the complexity of the bilevel structure. In this work, we focus on the smooth nonconvex-strongly-convex setting and develop new hard instances that yield nontrivial low…

• When, Where and Why to Average Weights?

  Niccol\`o Ajroldi, Antonio Orvieto, Jonas Geiping · 25. November 2025

  Averaging checkpoints along the training trajectory is a simple yet powerful approach to improve the generalization performance of Machine Learning models and reduce training time. Motivated by these potential gains, and in an effort to fairly and thoroughly benchmark this technique, we present an e…

• Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers

  Nischal Mainali, Lucas Teixeira · 25. November 2025

  Transformer models exhibit remarkable in-context learning (ICL), adapting to novel tasks from examples within their context, yet the underlying mechanisms remain largely mysterious. Here, we provide an exact analytical characterization of ICL emergence by deriving the closed-form stochastic gradient…

• On the Stability of the Jacobian Matrix in Deep Neural Networks

  Benjamin Dadoun, Soufiane Hayou, Hanan Salam, Mohamed El Amine Seddik, Pierre Youssef · 25. November 2025

  Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes that ensure Jacobian stability, but these analyses are typical…

• An operator splitting analysis of Wasserstein--Fisher--Rao gradient flows

  Francesca Romana Crucinio, Sahani Pathiraja · 25. November 2025

  Wasserstein-Fisher-Rao (WFR) gradient flows have been recently proposed as a powerful sampling tool that combines the advantages of pure Wasserstein (W) and pure Fisher-Rao (FR) gradient flows. Existing algorithmic developments implicitly make use of operator splitting techniques to numerically appr…

• Optimal Rates for Generalization of Gradient Descent for Deep ReLU Classification

  Yuanfan Li, Yunwen Lei, Zheng-Chu Guo, Yiming Ying · 25. November 2025

  Recent advances have significantly improved our understanding of the generalization performance of gradient descent (GD) methods in deep neural networks. A natural and fundamental question is whether GD can achieve generalization rates comparable to the minimax optimal rates established in the kerne…

• Low-Rank GEMM: Efficient Matrix Multiplication via Low-Rank Approximation with FP8 Acceleration

  Alfredo Metere · 25. November 2025

  Large matrix multiplication is a cornerstone of modern machine learning workloads, yet traditional approaches suffer from cubic computational complexity (e.g., $\mathcal{O}(n^3)$ for a matrix of size $n\times n$). We present Low-Rank GEMM, a novel approach that leverages low-rank matrix approximatio…

• Solving a Research Problem in Mathematical Statistics with AI Assistance

  Edgar Dobriban · 25. November 2025

  Over the last few months, AI models including large language models have improved greatly. There are now several documented examples where they have helped professional mathematical scientists prove new results, sometimes even helping resolve known open problems. In this short note, we add another e…

• Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?

  Vincent-Daniel Yun · 25. November 2025

  Low-precision training has become crucial for reducing the computational and memory costs of large-scale deep learning. However, quantizing gradients introduces magnitude shrinkage, which can change how stochastic gradient descent (SGD) converges. In this study, we explore SGD convergence under a gr…

• Frugality in second-order optimization: floating-point approximations for Newton's method

  Giuseppe Carrino, Elena Loli Piccolomini, Elisa Riccietti, Theo Mary · 25. November 2025

  Minimizing loss functions is central to machine-learning training. Although first-order methods dominate practical applications, higher-order techniques such as Newton's method can deliver greater accuracy and faster convergence, yet are often avoided due to their computational cost. This work analy…

• Improved Sample Complexity for Full Coverage in Compact and Continuous Spaces

  Lyu Yuhuan · 25. November 2025

  Verifying uniform conditions over continuous spaces through random sampling is fundamental in machine learning and control theory, yet classical coverage analyses often yield conservative bounds, particularly at small failure probabilities. We study uniform random sampling on the $d$-dimensional uni…

• Learning Rate Scheduling with Matrix Factorization for Private Training

  Nikita P. Kalinin, Joel Daniel Andersson · 25. November 2025

  We study differentially private model training with stochastic gradient descent under learning rate scheduling and correlated noise. Although correlated noise, in particular via matrix factorizations, has been shown to improve accuracy, prior theoretical work focused primarily on the prefix-sum work…

• Deterministic Continuous Replacement: Fast and Stable Module Replacement in Pretrained Transformers

  Rowan Bradbury, Aniket Srinivasan Ashok, Sai Ram Kasanagottu, Gunmay Jhingran, Shuai Meng · 25. November 2025

  Replacing modules in pretrained models, especially swapping quadratic self-attention for efficient attention alternatives, poses a hard optimization problem: cold-start reinitialization destabilizes frozen backbones. We isolate this core stability challenge in a controlled study. Deterministic Conti…

• Doubly Wild Refitting: Model-Free Evaluation of High Dimensional Black-Box Predictions under Convex Losses

  Haichen Hu, David Simchi-Levi · 25. November 2025

  We study the problem of excess risk evaluation for empirical risk minimization (ERM) under general convex loss functions. Our contribution is an efficient refitting procedure that computes the excess risk and provides high-probability upper bounds under the fixed-design setting. Assuming only black-…

• Frugality in second-order optimization: floating-point approximations for Newton's method

  Giuseppe Carrino, Elena Loli Piccolomini, Elisa Riccietti, Theo Mary · 25. November 2025

  Minimizing loss functions is central to machine-learning training. Although first-order methods dominate practical applications, higher-order techniques such as Newton's method can deliver greater accuracy and faster convergence, yet are often avoided due to their computational cost. This work analy…

• Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes

  Wei-Kai Chang, Rajiv Khanna · 24. November 2025

  As deep learning models continue to scale, the growing computational demands have amplified the need for effective coreset selection techniques. Coreset selection aims to accelerate training by identifying small, representative subsets of data that approximate the performance of the full dataset. Am…