Stochastic Gradient Optimization Techniques

Últimos artículos

• Convergence of continuous-time stochastic gradient descent with applications to deep neural networks

  Gabor Lugosi, Eulalia Nualart · 3 de noviembre de 2025

  We study a continuous-time approximation of the stochastic gradient descent process for minimizing the population expected loss in learning problems. The main results establish general sufficient conditions for the convergence, extending the results of Chatterjee (2022) established for (nonstochasti…

• Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence

  Matin Ansaripour, Shayan Talaei, Giorgi Nadiradze, Dan Alistarh · 3 de noviembre de 2025

  Distributed optimization is the standard way of speeding up machine learning training, and most of the research in the area focuses on distributed first-order, gradient-based methods. Yet, there are settings where some computationally-bounded nodes may not be able to implement first-order, gradient-…

• Exploring Landscapes for Better Minima along Valleys

  Tong Zhao, Jiacheng Li, Yuanchang Zhou, Guangming Tan, Weile Jia · 3 de noviembre de 2025

  Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest …

• A geometric framework for momentum-based optimizers for low-rank training

  Steffen Schotth\"ofer, Timon Klein, Jonas Kusch · 31 de octubre de 2025

  Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we…

• Data-driven Projection Generation for Efficiently Solving Heterogeneous Quadratic Programming Problems

  Tomoharu Iwata, Futoshi Futami · 31 de octubre de 2025

  We propose a data-driven framework for efficiently solving quadratic programming (QP) problems by reducing the number of variables in high-dimensional QPs using instance-specific projection. A graph neural network-based model is designed to generate projections tailored to each QP instance, enabling…

• Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime

  Beomhan Baek, Minhak Song, Chulhee Yun · 31 de octubre de 2025

  Adam [Kingma and Ba, 2015] is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with $\ell_\infty$-geometry, but these results are restricted to the full-batch regime. In this work, we study the implicit…

• Accumulative SGD Influence Estimation for Data Attribution

  Yunxiao Shi, Shuo Yang, Yixin Su, Rui Zhang, Min Xu · 31 de octubre de 2025

  Modern data-centric AI needs precise per-sample influence. Standard SGD-IE approximates leave-one-out effects by summing per-epoch surrogates and ignores cross-epoch compounding, which misranks critical examples. We propose ACC-SGD-IE, a trajectory-aware estimator that propagates the leave-one-out p…

• Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks

  Andrea Montanari, Pierfrancesco Urbani · 30 de octubre de 2025

  Understanding the inductive bias and generalization properties of large overparametrized machine learning models requires to characterize the dynamics of the training algorithm. We study the learning dynamics of large two-layer neural networks via dynamical mean field theory, a well established tech…

• The Price equation reveals a universal force-metric-bias law of algorithmic learning and natural selection

  Steven A. Frank · 30 de octubre de 2025

  Diverse learning algorithms, optimization methods, and natural selection share a common mathematical structure, despite their apparent differences. Here I show that a simple notational partitioning of change by the Price equation reveals a universal force-metric-bias (FMB) law: $\Delta\mathbf{\theta…

• Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner

  Runa Eschenhagen, Aaron Defazio, Tsung-Hsien Lee, Richard E. Turner, Hao-Jun Michael Shi · 30 de octubre de 2025

  The recent success of Shampoo in the AlgoPerf contest has sparked renewed interest in Kronecker-factorization-based optimization algorithms for training neural networks. Despite its success, Shampoo relies heavily on several heuristics such as learning rate grafting and stale preconditioning to achi…

• Stochastic Momentum Methods for Non-smooth Non-Convex Finite-Sum Coupled Compositional Optimization

  Xingyu Chen, Bokun Wang, Ming Yang, Qihang Lin, Tianbao Yang · 30 de octubre de 2025

  Finite-sum Coupled Compositional Optimization (FCCO), characterized by its coupled compositional objective structure, emerges as an important optimization paradigm for addressing a wide range of machine learning problems. In this paper, we focus on a challenging class of non-convex non-smooth FCCO, …

• Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking

  Yuatyong Chaichana, Thanapat Trachu, Peerat Limkonchotiwat, Konpat Preechakul, Tirasan Khandhawit, Ekapol Chuangsuwanich · 30 de octubre de 2025

  In the era of large-scale training, model merging has evolved into a tool for creating multitasking models efficiently. It enables the knowledge of models to be fused, without the need for heavy computation as required in traditional multitask learning. Existing merging methods often assume that ent…

• ASGO: Adaptive Structured Gradient Optimization

  Kang An, Yuxing Liu, Rui Pan, Yi Ren, Shiqian Ma, Donald Goldfarb, Tong Zhang · 30 de octubre de 2025

  Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this structural representation, it has been widely observed that gradients are low-rank and Hessians are approximately block diag…

• Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLU

  Jingzhou Liu · 30 de octubre de 2025

  In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gaussian inputs and equal layer width k, (1) we establish, based on the equivariant gradient degree, a theoretical framework, applicable to any number of neurons …

• A Convexity-dependent Two-Phase Training Algorithm for Deep Neural Networks

  Tomas Hrycej, Bernhard Bermeitinger, Massimo Pavone, G\"otz-Henrik Wiegand, Siegfried Handschuh · 30 de octubre de 2025

  The key task of machine learning is to minimize the loss function that measures the model fit to the training data. The numerical methods to do this efficiently depend on the properties of the loss function. The most decisive among these properties is the convexity or non-convexity of the loss funct…

• Dynamically Weighted Momentum with Adaptive Step Sizes for Efficient Deep Network Training

  Zhifeng Wang, Longlong Li, Chunyan Zeng · 30 de octubre de 2025

  Within the current sphere of deep learning research, despite the extensive application of optimization algorithms such as Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), there remains a pronounced inadequacy in their capability to address fluctuations in learning efficiency,…

• What Really Matters in Matrix-Whitening Optimizers?

  Kevin Frans, Pieter Abbeel, Sergey Levine · 30 de octubre de 2025

  A range of recent optimizers have emerged that approximate the same "matrix-whitening" transformation in various ways. In this work, we systematically deconstruct such optimizers, aiming to disentangle the key components that explain performance. Across tuned hyperparameters across the board, all fl…

• Uni-LoRA: One Vector is All You Need

  Kaiyang Li, Shaobo Han, Qing Su, Wei Li, Zhipeng Cai, Shihao Ji · 29 de octubre de 2025

  Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models (LLMs) by constraining weight updates to low-rank matrices. Recent works such as Tied-LoRA, VeRA, and VB-LoRA push efficiency further by introducing additional constraints to re…

• Mirror Descent and Novel Exponentiated Gradient Algorithms Using Trace-Form Entropies and Deformed Logarithms

  Andrzej Cichocki, Toshihisa Tanaka, Frank Nielsen, Sergio Cruces · 29 de octubre de 2025

  This paper introduces a broad class of Mirror Descent (MD) and Generalized Exponentiated Gradient (GEG) algorithms derived from trace-form entropies defined via deformed logarithms. Leveraging these generalized entropies yields MD \& GEG algorithms with improved convergence behavior, robustness to v…

• Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations

  Alexandru Cr\u{a}ciun, Debarghya Ghoshdastidar · 29 de octubre de 2025

  The theory of training deep networks has become a central question of modern machine learning and has inspired many practical advancements. In particular, the gradient descent (GD) optimization algorithm has been extensively studied in recent years. A key assumption about GD has appeared in several …

• How do simple rotations affect the implicit bias of Adam?

  Adela DePavia, Vasileios Charisopoulos, Rebecca Willett · 29 de octubre de 2025

  Adaptive gradient methods such as Adam and Adagrad are widely used in machine learning, yet their effect on the generalization of learned models -- relative to methods like gradient descent -- remains poorly understood. Prior work on binary classification suggests that Adam exhibits a ``richness bia…

• Accelerated Gradient Methods for Nonconvex Optimization: Escape Trajectories From Strict Saddle Points and Convergence to Local Minima

  Rishabh Dixit, Mert Gurbuzbalaban, Waheed U. Bajwa · 28 de octubre de 2025

  This paper considers the problem of understanding the behavior of a general class of accelerated gradient methods on smooth nonconvex functions. Motivated by some recent works that have proposed effective algorithms, based on Polyak's heavy ball method and the Nesterov accelerated gradient method, t…

• Convergence Analysis of SGD under Expected Smoothness

  Yuta Kawamoto, Hideaki Iiduka · 28 de octubre de 2025

  Stochastic gradient descent (SGD) is the workhorse of large-scale learning, yet classical analyses rely on assumptions that can be either too strong (bounded variance) or too coarse (uniform noise). The expected smoothness (ES) condition has emerged as a flexible alternative that ties the second mom…

• Accelerated Evolving Set Processes for Local PageRank Computation

  Binbin Huang, Luo Luo, Yanghua Xiao, Deqing Yang, Baojian Zhou · 28 de octubre de 2025

  This work proposes a novel framework based on nested evolving set processes to accelerate Personalized PageRank (PPR) computation. At each stage of the process, we employ a localized inexact proximal point iteration to solve a simplified linear system. We show that the time complexity of such locali…

• Variational Regularized Unbalanced Optimal Transport: Single Network, Least Action

  Yuhao Sun, Zhenyi Zhang, Zihan Wang, Tiejun Li, Peijie Zhou · 28 de octubre de 2025

  Recovering the dynamics from a few snapshots of a high-dimensional system is a challenging task in statistical physics and machine learning, with important applications in computational biology. Many algorithms have been developed to tackle this problem, based on frameworks such as optimal transport…