Stochastic Gradient Optimization Techniques

Últimos artículos

• UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models

  Van-Tuan Tran, Hong-Hanh Nguyen-Le, Marco Ruffini, Merim Dzaferagic · 19 de mayo de 2026

  Heterogeneous LoRA-rank methods address system heterogeneity in federated fine-tuning of foundation models by assigning client-specific ranks based on computational capabilities. However, these methods achieve only marginal computational savings, as dense feed-forward computations dominate. Sparse M…

• Pointwise Generalization in Deep Neural Networks

  Shaojie Li, Yunbei Xu · 19 de mayo de 2026

  We address the fundamental question of why deep neural networks generalize by establishing a pointwise generalization theory for fully connected networks. This framework resolves long-standing barriers to characterizing the rich nonlinear feature-learning regime and builds a new statistical foundati…

• Flow-Direct: Feedback-Efficient and Reusable Guidance for Flow Models via Non-Parametric Guidance Field

  Kim Yong Tan, Yueming Lyu, Ivor Tsang, Yew-Soon Ong · 19 de mayo de 2026

  Training-free guidance enables pre-trained diffusion and flow models to optimize application-specific objectives using feedback from external black-box reward functions. However, existing methods are feedback-inefficient because reward feedback is used only transiently to inform a localized gradient…

• Does Weight Decay Enhance Training Stability?

  Marius Saether, Amir Kolic, Tomaso Poggio, Pierfrancesco Beneventano · 19 de mayo de 2026

  In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: *does weight decay stabilize training dynamics, and if so, through which mechanism?* Indeed, trai…

• The Geometry of Projection Heads: Conditioning, Invariance, and Collapse

  Faris Chaudhry · 19 de mayo de 2026

  We develop a geometric theory of projection heads in self-supervised learning by modeling the head as a trainable Riemannian metric on the backbone representation manifold. We show that linear heads perform implicit subspace whitening, while nonlinear heads adapt local metrics to satisfy the specifi…

• Decoupled Conformal Optimisation: Efficient Prediction Sets via Independent Tuning and Calibration

  Fanyi Wu, Lihua Niu, Samuel Kaski, Michele Caprio · 19 de mayo de 2026

  Bayesian conformal optimisation methods often use the same held-out data both to search for efficient prediction sets and to certify coverage or risk. This coupling is natural for high-probability risk-control guarantees, but it is not necessary when the target is standard finite-sample marginal con…

• SignMuon: Communication-Efficient Distributed Muon Optimization

  Neel Mishra, Kushagara Trivedi, Pawan Kumar · 19 de mayo de 2026

  Distributed training of large neural networks is bottlenecked by full-precision gradient communication and by coordinatewise optimizers that ignore the matrix structure of weight tensors. We propose Sign-Muon, a 1-bit, matrix-aware optimizer that combines majority-vote sign aggregation from signSGD …

• Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

  Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri, Ansh Tiwari, Hideyuki Kawashima, Ioannis Mitliagkas, Guillaume Rabusseau, Hiroki Naganuma · 19 de mayo de 2026

  Low-rank gradient compression reduces communication in distributed training by representing updates with rank-$r$ factors. Dion is a recent method that approximates Muon, a spectral optimizer that orthogonalizes momentum, using one step of power iteration followed by column normalization (rescaling …

• Global Convergence of Sampling-Based Nonconvex Optimization through Diffusion-Style Smoothing

  Zeji Yi, Chaoyi Pan, Guanya Shi, Guannan Qu · 19 de mayo de 2026

  Sampling-based optimization (SBO), like cross-entropy method and evolutionary algorithms, has achieved many successes in solving non-convex problems without gradients, yet its convergence is poorly understood. In this paper, we establish a non-asymptotic convergence analysis for SBO through the lens…

• High-dimensional Limit of SGD for Diagonal Linear Networks

  Bego\~na Garc\'ia Malaxechebarr\'ia, Courtney Paquette, Maryam Fazel, Dmitriy Drusvyatskiy · 19 de mayo de 2026

  Understanding the behavior of stochastic gradient methods is a central problem in modern machine learning. Recent work has highlighted diagonal linear networks as a simplified yet expressive setting for analyzing the optimization and generalization properties of neural models. In this work, we show …

• Propagation of Chaos in Contextual Flow Maps

  Shi Chen, Zhengjiang Lin, Kaizhao Liu, Philippe Rigollet · 19 de mayo de 2026

  We develop a quantitative statistical theory of transformers in the large-context regime by adopting the abstraction of contextual flow maps (CFMs): dynamical systems that evolve a distinguished token in the presence of a contextual measure across a stack of attention blocks. Within this framework, …

• Proximal basin hopping: global optimization with guarantees

  Guillaume Lauga (LJAD), Cesare Molinari (MaLGA), Samuel Vaiter (CNRS,LJAD) · 19 de mayo de 2026

  Global optimization is a challenging problem, with plenty of algorithms displaying empirical success, but scarce theoretical backing. In this work, we propose a new theoretical framework called Proximal Basin Hopping (PBH), carefully tailored to combine proximal optimization and local minimization. …

• RL4RLA: Teaching ML to Discover Randomized Linear Algebra Algorithms Through Curriculum Design and Graph-Based Search

  Jinglong Xiong, Xiaotian Liu, Ruoxin Wang, Zihang Liu, Yefan Zhou, Yujun Yan, Yaoqing Yang · 19 de mayo de 2026

  Randomized linear algebra (RLA) algorithms are a modern class of numerical linear algebra techniques that play an essential role in scientific computing and machine learning, with broad and growing adoption. However, their discovery remains mostly a manual process that requires deep expert knowledge…

• New Insight of Variance reduce in Zero-Order Hard-Thresholding: Mitigating Gradient Error and Expansivity Contradictions

  Xinzhe Yuan (Harbin Institute of Technology), William de Vazelhes (Mohamed bin Zayed University of Artificial Intelligence), Bin Gu (Mohamed bin Zayed University of Artificial Intelligence, Jilin University), Huan Xiong (Harbin Institute of Technology, Mohamed bin Zayed University of Artificial Intelligence) · 19 de mayo de 2026

  Hard-thresholding is an important type of algorithm in machine learning that is used to solve $\ell_0$ constrained optimization problems. However, the true gradient of the objective function can be difficult to access in certain scenarios, which normally can be approximated by zeroth-order (ZO) meth…

• Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting

  Karthik Prakhya, Alp Yurtsever · 19 de mayo de 2026

  We show that the training problem of a deep linear neural network under the squared loss admits an exact convex reformulation in a lifted space over a generalized completely positive cone. The reformulation has the same optimal value as the original nonconvex problem and is linear in the lifted vari…

• Mirror Descent-Type Algorithms for the Variational Inequality Problem with Functional Constraints

  Mohammad S. Alkousa, Fedor S. Stonyakin, Belal A. Alashqar, Seydamet S. Ablaev · 19 de mayo de 2026

  Variational inequalities play a key role in machine learning research, such as generative adversarial networks, reinforcement learning, adversarial training, and generative models. This paper is devoted to the constrained variational inequality problems with functional constraints (inequality-type c…

• The Neural Tangent Kernel for Classification

  Jonathan Plenk, Sergio Calvo-Ordonez, Alvaro Cartea, Yarin Gal, Mark van der Wilk, Kamil Ciosek · 19 de mayo de 2026

  In wide neural networks, the Neural Tangent Kernel (NTK) remains approximately constant during training, providing a powerful theoretical tool for studying training dynamics, generalization, and connections to kernel methods. However, this theory is largely restricted to regression losses. It was pr…

• Training Infinitely Deep and Wide Transformers

  Rapha\"el Barboni, Maarten V. de Hoop, Takashi Furuya, Gabriel Peyr\'e · 19 de mayo de 2026

  Transformers have become the dominant architecture in modern machine learning, yet the theoretical understanding of their training dynamics remains limited. This paper develops a rigorous mathematical framework for analyzing gradient-based training of transformers in the mean-field regime, where bot…

• Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method

  Abdurakhmon Sadiev, Artavazd Maranjyan, Ivan Ilin, Peter Richt\'arik · 19 de mayo de 2026

  Muon has recently emerged as a strong alternative to AdamW for training neural networks, with encouraging large-scale pretraining results and growing evidence that matrix-structured updates can be faster in practice. Yet Muon, and more generally Linear Minimization Oracle (LMO) based methods, are ty…

• Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates

  Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Mingyi Hong · 19 de mayo de 2026

  It is widely believed that stochastic gradient descent (SGD) performs significantly worse than adaptive optimizers such as Adam in pre-training Large Language Models (LLMs). Yet the underlying reason for this gap remains unclear. In this work, we attribute a large part of the discrepancy to SGD's in…

• Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration

  Sachin Garg, Micha{\l} Derezi\'nski · 19 de mayo de 2026

  Accelerating stochastic gradient methods with classical momentum schemes, such as Polyak's heavy ball, has proven highly successful in training large-scale machine learning models, particularly when combined with the hardware acceleration of large mini-batch computations. Yet, the effect of classica…

• Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation

  Mingfei Sun · 19 de mayo de 2026

  Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhonov-regularized natural…

• FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

  Ramakrishnan Sathyavageeswaran · 19 de mayo de 2026

  Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calib…

• Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

  Radu Lecoiu, Debarghya Mukherjee, Pragya Sur · 19 de mayo de 2026

  Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage esti…

• Stochastic Penalty-Barrier Methods for Constrained Machine Learning

  Adam Bos\'ak, Andrii Kliachkin, Jana Lep\v{s}ov\'a, Gilles Bareilles, Jakub Mare\v{c}ek · 19 de mayo de 2026

  Constrained machine learning enables fairness-aware training, physics-informed neural networks, and integration of symbolic domain knowledge into statistical models. Despite its practical importance, no general method exists for the non-convex, non-smooth, stochastic setting that arises naturally in…