Stochastic Gradient Optimization Techniques

Últimos artículos

• PACED: Distillation at the Frontier of Student Competence

  Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang · 13 de marzo de 2026

  Standard LLM distillation wastes compute on two fronts: problems the student has already mastered (near-zero gradients) and problems far beyond its reach (incoherent gradients that erode existing capabilities). We show that this waste is not merely intuitive but structurally inevitable: the gradient…

• Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

  Yulu Gan, Phillip Isola · 13 de marzo de 2026

  Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in sma…

• On-Average Stability of Multipass Preconditioned SGD and Effective Dimension

  Simon Vary, Tyler Farghly, Ilja Kuzborskij, Patrick Rebeschini · 13 de marzo de 2026

  We study trade-offs between the population risk curvature, geometry of the noise, and preconditioning on the generalisation ability of the multipass Preconditioned Stochastic Gradient Descent (PSGD). Many practical optimisation heuristics implicitly navigate this trade-off in different ways -- for i…

• Deep Randomized Distributed Function Computation (DeepRDFC): Neural Distributed Channel Simulation

  Didrik Bergstr\"om, Onur G\"unl\"u · 12 de marzo de 2026

  The randomized distributed function computation (RDFC) framework, which unifies many cutting-edge distributed computation and learning applications, is considered. An autoencoder (AE) architecture is proposed to minimize the total variation distance between the probability distribution simulated by …

• Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime

  Reza Ghane, Danil Akhtiamov, Babak Hassibi · 12 de marzo de 2026

  In this work we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent, Gradient Clipping and Adam. We consider preconditioners of the form $\nabla K$, where $K: \mathbb{R}^p \to \mathbb{R}$ is convex and assume…

• Marginals Before Conditionals

  Mihir Sahasrabudhe · 12 de marzo de 2026

  We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring t…

• Breaking the Stochasticity Barrier: An Adaptive Variance-Reduced Method for Variational Inequalities

  Yungi Jeong, Takumi Otsuka · 12 de marzo de 2026

  Stochastic non-convex non-concave optimization, formally characterized as Stochastic Variational Inequalities (SVIs), presents unique challenges due to rotational dynamics and the absence of a global merit function. While adaptive step-size methods (like Armijo line-search) have revolutionized conve…

• Brenier Isotonic Regression

  Han Bao, Amirreza Eshraghi, Yutong Wang · 12 de marzo de 2026

  Isotonic regression (IR) is shape-constrained regression to maintain a univariate fitting curve non-decreasing, which has numerous applications including single-index models and probability calibration. When it comes to multi-output regression, the classical IR is no longer applicable because the mo…

• Memorization capacity of deep ReLU neural networks characterized by width and depth

  Xin Yang, Yunfei Yang · 11 de marzo de 2026

  This paper studies the memorization capacity of deep neural networks with ReLU activation. Specifically, we investigate the minimal size of such networks to memorize any $N$ data points in the unit ball with pairwise separation distance $\delta$ and discrete labels. Most prior studies characterize t…

• OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality

  Ganzhao Yuan · 11 de marzo de 2026

  The Exponential Moving Average (EMA) is a cornerstone of widely used optimizers such as Adam. However, existing theoretical analyses of Adam-style methods have notable limitations: their guarantees can remain suboptimal in the zero-noise regime, rely on restrictive boundedness conditions (e.g., boun…

• The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference

  Vignesh Adhinarayanan, Nuwan Jayasena · 11 de marzo de 2026

  Mixture-of-Experts (MoE) models deliver high quality at low training FLOPs, but this efficiency often vanishes at inference. We identify a double penalty that structurally disadvantages MoE architectures during decoding: first, expert routing fragments microbatches and reduces weight reuse; second, …

• Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets

  Arthur da Cunha, Francesco d'Amore, Emanuele Natale · 11 de marzo de 2026

  The Strong Lottery Ticket Hypothesis (SLTH) states that randomly-initialised neural networks likely contain subnetworks that perform well without any training. Although unstructured pruning has been extensively studied in this context, its structured counterpart, which can deliver significant comput…

• Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Betting

  Abhinaba Basu · 11 de marzo de 2026

  We present a comprehensive ablation of nine finite-sample bound families for selective prediction with risk control, combining concentration inequalities (Hoeffding, Empirical Bernstein, Clopper-Pearson, Wasserstein DRO, CVaR) with multiple-testing corrections (union bound, Learn Then Test fixed-seq…

• An Optimal Control Approach To Transformer Training

  Ka\u{g}an Akman, Naci Sald{\i}, Serdar Y\"uksel · 11 de marzo de 2026

  In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the problem, and (iii) positional dependence. We model the Transforme…

• Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning

  Yechen Zhang, Shuhao Xing, Junhao Huang, Kai Lv, Yunhua Zhou, Xipeng Qiu, Qipeng Guo, Kai Chen · 11 de marzo de 2026

  Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improve generalization. However, Muon implicitly assumes an isotropic optimization landscape, enforcing a uniform spectral update…

• Towards Understanding Adam Convergence on Highly Degenerate Polynomials

  Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou, Zhi-Qin John Xu, Yaoyu Zhang · 11 de marzo de 2026

  Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $\beta_2$ near 1 for convergence, this work investigates the "natural" auto…

• Convergence Rate for the Last Iterate of Stochastic Gradient Descent Schemes

  Marcel Hudiani · 11 de marzo de 2026

  We study the convergence rate for the last iterate of stochastic gradient descent (SGD) and stochastic heavy ball (SHB) in the parametric setting when the objective function $F$ is globally convex or non-convex whose gradient is $\gamma$-H\"{o}lder. Using only discrete Gronwall's inequality without …

• On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer

  Ruihan Xu, Jiajin Li, Yiping Lu · 11 de marzo de 2026

  A central question in modern deep learning is how to design optimizers whose behavior remains stable as the network width $w$ increases. We address this question by interpreting several widely used neural-network optimizers, including \textrm{AdamW} and \textrm{Muon}, as instances of steepest descen…

• Step-Size Decay and Structural Stagnation in Greedy Sparse Learning

  Pablo M. Bern\'a · 10 de marzo de 2026

  Greedy algorithms are central to sparse approximation and stage-wise learning methods such as matching pursuit and boosting. It is known that the Power-Relaxed Greedy Algorithm with step sizes $m^{-\alpha}$ may fail to converge when $\alpha>1$ in general Hilbert spaces. In this work, we revisit this…

• Mini-batch Estimation for Deep Cox Models: Statistical Foundations and Practical Guidance

  Lang Zeng, Weijing Tang, Zhao Ren, Ying Ding · 10 de marzo de 2026

  The stochastic gradient descent (SGD) algorithm has been widely used to optimize deep Cox neural network (Cox-NN) by updating model parameters using mini-batches of data. We show that SGD aims to optimize the average of mini-batch partial-likelihood, which is different from the standard partial-like…

• Probabilistic Inference and Learning with Stein's Method

  Qiang Liu, Lester Mackey, Chris Oates · 10 de marzo de 2026

  This monograph provides a rigorous overview of theoretical and methodological aspects of probabilistic inference and learning with Stein's method. Recipes are provided for constructing Stein discrepancies from Stein operators and Stein sets, and properties of these discrepancies such as computabilit…

• Faster Gradient Methods for Highly-Smooth Stochastic Bilevel Optimization

  Lesi Chen, Junru Li, El Mahdi Chayti, Jingzhao Zhang · 10 de marzo de 2026

  This paper studies the complexity of finding an $\epsilon$-stationary point for stochastic bilevel optimization when the upper-level problem is nonconvex and the lower-level problem is strongly convex. Recent work proposed the first-order method, F${}^2$SA, achieving the $\tilde{\mathcal{O}}(\epsilo…

• Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts

  Truong Xuan Khanh, Truong Quynh Hoa · 10 de marzo de 2026

  Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions…

• Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers

  Tao Shi, Liangming Chen, Long Jin, Mengchu Zhou · 10 de marzo de 2026

  In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance. A widely accepted explanation for its defect in generalization is that it often tends to converge to sharp minima. To enhance its ability to find flat mi…

• A Unified View of Drifting and Score-Based Models

  Chieh-Hsin Lai, Bac Nguyen, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, Stefano Ermon, Molei Tao · 10 de marzo de 2026

  Drifting models train one-step generators by optimizing a mean-shift discrepancy induced by a kernel between the data and model distributions, with Laplace kernels used by default in practice. At each point, this discrepancy compares the kernel-weighted displacement toward nearby data samples with t…