Stochastic Gradient Optimization Techniques

Derniers papiers

• Natural gradient descent with momentum

  Anthony Nouy, Agust\'in Somacal · 20 avril 2026

  We consider the problem of approximating a function by an element of a nonlinear manifold which admits a differentiable parametrization, typical examples being neural networks with differentiable activation functions or tensor networks. Natural gradient descent (NGD) for the optimization of a loss f…

• Stein Variational Black-Box Combinatorial Optimization

  Thomas Landais, Olivier Goudet, Adrien Go\"effon, Fr\'ed\'eric Saubion, Sylvain Lamprier · 20 avril 2026

  Combinatorial black-box optimization in high-dimensional settings demands a careful trade-off between exploiting promising regions of the search space and preserving sufficient exploration to identify multiple optima. Although Estimation-of-Distribution Algorithms (EDAs) provide a powerful model-bas…

• Collective Kernel EFT for Pre-activation ResNets

  Hidetoshi Kawase, Toshihiro Ota · 20 avril 2026

  In finite-width deep neural networks, the empirical kernel $G$ evolves stochastically across layers. We develop a collective kernel effective field theory (EFT) for pre-activation ResNets based on a $G$-only closure hierarchy and diagnose its finite validity window. Exploiting the exact conditional …

• Plateaus, Optima, and Overfitting in Multi-Layer Perceptrons: A Saddle-Saddle-Attractor Scenario

  Alex Al\`i Maleknia, Yuzuru Sato · 20 avril 2026

  Vanishing gradients and overfitting are central problems in machine learning, yet are typically analyzed in asymptotic regimes that obscure their dynamical origins. Here we provide a dynamical description of learning in multi-layer perceptrons (MLPs) via a minimal model inspired by Fukumizu and Amar…

• StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models

  Dingzhi Yu, Rui Pan, Yuxing Liu, Tong Zhang · 20 avril 2026

  Sign-based optimization algorithms, such as SignSGD, have garnered significant attention for their remarkable performance in distributed learning and training large foundation models. Despite their empirical superiority, SignSGD is known to diverge on non-smooth objectives, which are ubiquitous in m…

• Scalable Posterior Uncertainty for Flexible Density-Based Clustering

  Nicola Bariletto, Stephen G. Walker · 20 avril 2026

  We introduce a novel framework for uncertainty quantification in clustering that combines martingale posterior distributions with density-based clustering. Unlike classical model-based approaches, which define clusters at the latent level of a mixture model, we treat clusters as explicit functionals…

• Zeroth-Order Optimization at the Edge of Stability

  Minhak Song, Liang Zhang, Bingcong Li, Niao He, Michael Muehlebach, Sewoong Oh · 17 avril 2026

  Zeroth-order (ZO) methods are widely used when gradients are unavailable or prohibitively expensive, including black-box learning and memory-efficient fine-tuning of large models, yet their optimization dynamics in deep learning remain underexplored. In this work, we provide an explicit step size co…

• SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization

  Wooin Lee, Hyun-Tae Kim · 17 avril 2026

  The AdamW optimizer, while standard for LLM pretraining, is a critical memory bottleneck, consuming optimizer states equivalent to twice the model's size. Although light-state optimizers like SinkGD attempt to address this issue, we identify the embedding layer dilemma: these methods fail to handle …

• Path-Sampled Integrated Gradients

  Firuz Kamalov, Fadi Thabtah, R. Sivaraj, Neda Abdelhamid · 17 avril 2026

  We introduce path-sampled integrated gradients (PS-IG), a framework that generalizes feature attribution by computing the expected value over baselines sampled along the linear interpolation path. We prove that PS-IG is mathematically equivalent to path-weighted integrated gradients, provided the we…

• High Probability Guarantees for Random Reshuffling

  Hengxu Yu, Xiao Li · 17 avril 2026

  We consider the stochastic gradient method with random reshuffling ($\mathsf{RR}$) for tackling smooth nonconvex optimization problems. $\mathsf{RR}$ finds broad applications in practice, notably in training neural networks. In this work, we provide high probability complexity guarantees for this me…

• Beyond Importance Sampling: Rejection-Gated Policy Optimization

  Ziwu Sun, Zhen Gao, Jiyong Zhang, Jiaheng Li · 17 avril 2026

  We propose a new perspective on policy optimization: rather than reweighting all samples by their importance ratios, an optimizer should select which samples are trustworthy enough to drive a policy update. Building on this view, we introduce Rejection-Gated Policy Optimization (RGPO), which replace…

• Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization

  Mathias Dus (IRMA) · 17 avril 2026

  We present a geometric framework for Reinforcement Learning (RL) that views policies as maps into the Wasserstein space of action probabilities. First, we define a Riemannian structure induced by stationary distributions, proving its existence in a general context. We then define the tangent space o…

• Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

  Wentao Hu, Yanbo Zhai, Xiaohui Hu, Mingkuan Zhao, Shanhong yu, Xue Liu, Kaidong Yu, Shuangyong Song, Xuelong Li · 17 avril 2026

  Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare fact…

• CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization

  Feihu Huang, Guanyi Zhang, Songcan Chen · 17 avril 2026

  Lion optimizer is a popular learning-based optimization algorithm in machine learning, which shows impressive performance in training many deep learning models. Although convergence property of the Lion optimizer has been studied, its generalization analysis is still missing. To fill this gap, we st…

• Gradient Descent's Last Iterate is Often (slightly) Suboptimal

  Guy Kornowski, Ohad Shamir · 16 avril 2026

  We consider the well-studied setting of minimizing a convex Lipschitz function using either gradient descent (GD) or its stochastic variant (SGD), and examine the last iterate convergence. By now, it is known that standard stepsize choices lead to a last iterate convergence rate of $\log T/\sqrt{T}$…

• A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

  Damek Davis, Dmitriy Drusvyatskiy · 16 avril 2026

  Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative t…

• Stochastic Trust-Region Methods for Over-parameterized Models

  Aike Yang, Hao Wang · 16 avril 2026

  Under interpolation-type assumptions such as the strong growth condition, stochastic optimization methods can attain convergence rates comparable to full-batch methods, but their performance, particularly for SGD, remains highly sensitive to step-size selection. To address this issue, we propose a u…

• Mini-Batch Covariance, Diffusion Limits, and Oracle Complexity in Stochastic Gradient Descent: A Sampling-Design Perspective

  Daniel Zantedeschi, Kumar Muthuraman · 16 avril 2026

  Stochastic gradient descent (SGD) is central to simulation optimization, stochastic programming, and online M-estimation, where sampling effort is a decision variable. We study the mini-batch gradient noise as a sampling-design object. Under exchangeable fresh-sampling mini-batches, the conditional …

• HUANet: Hard-Constrained Unrolled ADMM for Constrained Convex Optimization

  Trinh Tran, Binh Nguyen, Truong X. Nghiem · 16 avril 2026

  This paper presents HUANet, a constrained deep neural network architecture that unrolls the iterations of the Alternating Direction Method of Multipliers (ADMM) into a trainable neural network for solving constrained convex optimization problems. Existing end-to-end learning methods operate as black…

• Universality of Gaussian-Mixture Reverse Kernels in Conditional Diffusion

  Nafiz Ishtiaque, Syed Arefinul Haque, Kazi Ashraful Alam, Fatima Jahara · 16 avril 2026

  We prove that conditional diffusion models whose reverse kernels are finite Gaussian mixtures with ReLU-network logits can approximate suitably regular target distributions arbitrarily well in context-averaged conditional KL divergence, up to an irreducible terminal mismatch that typically vanishes …

• Neural Mean-Field Games: Extending Mean-Field Game Theory with Neural Stochastic Differential Equations

  Anna C. M. Th\"oni, Yoram Bachrach, Tal Kachman · 16 avril 2026

  Mean-field game theory relies on approximating games that are intractable to model due to a very large to infinite population of players. While these kinds of games can be solved analytically via the associated system of partial derivatives, this approach is not model-free, can lead to the loss of t…

• Momentum Further Constrains Sharpness at the Edge of Stochastic Stability

  Arseniy Andreyev, Advikar Ananthkumar, Marc Walden, Tomaso Poggio, Pierfrancesco Beneventano · 16 avril 2026

  Recent work suggests that (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found. Momentum and mini-batch gradients are widely used in practical deep learning optimization, but it remains unclear whether they operate in a comparab…

• Parameter-Free Non-Ergodic Extragradient Algorithms for Solving Monotone Variational Inequalities

  Lingqing Shen, Fatma K{\i}l{\i}n\c{c}-Karzan · 16 avril 2026

  Monotone variational inequalities (VIs) provide a unifying framework for convex minimization, equilibrium computation, and convex-concave saddle-point problems. Extragradient-type methods are among the most effective first-order algorithms for such problems, but their performance hinges critically o…

• Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates

  Saumya Goyal, Rohith Rongali, Ritabrata Ray, Barnab\'as P\'oczos · 16 avril 2026

  We study learning to learn for regression problems through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a convex regression task. We prove the exi…

• On the continuum limit of t-SNE for data visualization

  Jeff Calder, Zhonggan Huang, Ryan Murray, Adam Pickarski · 15 avril 2026

  This work is concerned with the continuum limit of a graph-based data visualization technique called the t-Distributed Stochastic Neighbor Embedding (t-SNE), which is widely used for visualizing data in a variety of applications, but is still poorly understood from a theoretical standpoint. The t-SN…