Stochastic Gradient Optimization Techniques

Últimos artículos

• ButterflyMoE: Compression-Scalable Ternary Experts via Structured Butterfly Orbits

  Aryan Karmore · 9 de julio de 2026

  In current Mixture of Experts (MoE) architectures, linear memory scaling is present, the memory grows as the number of experts increases. $N$ independent expert weight matrices require $\mathcal{O}(N \cdot d^2)$ memory which exceeds the memory budget of edge devices. Current compression methods like…

• Finding a stationary point of a stochastic convex problem

  Felipe Areces, John Duchi, Malo Sommers · 9 de julio de 2026

  We consider the problem of finding stationary points for stochastic convex optimization problems. Rather than surrogates to stationarity, such as a proximity-to-stationarity guarantee or small gradient of the Moreau envelope, we ask for a stronger notion: that the subdifferential of the objective ac…

• Hybrid Least Squares/Gradient Descent Methods for MIONets

  Jun Choi, Chang-Ock Lee, Minam Moon · 9 de julio de 2026

  In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets. Since MIONet is the sum of the entrywise product of multiple branch networks and a trunk network, it can be viewed as …

• Optimization Geometrodynamics: A Framework for Dynamic Geometric Optimization

  Zavier Li · 9 de julio de 2026

  Most gradient-based optimization methods move parameters through a fixed background geometry, even when their internal states implicitly define changing notions of length, curvature, and preconditioning. We introduce optimization geometrodynamics, a benchmark language in which optimization is a coup…

• GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

  Jieying Wang, Shuyuan Fan, Mingkai Zheng, Zhao Zhang · 9 de julio de 2026

  Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can significantly reduce the communication volume. Existing methods quantize gradients via linear or nonlinear mappings in Euclid…

• Deep Neural Variation Spaces: A Unifying Perspective on Depth and Complexity

  Julia Nakhleh, Robert D. Nowak · 8 de julio de 2026

  We develop a unified function space theory of deep fully connected neural networks. Functions in our spaces are defined recursively as $\ell^1$-bounded linear combinations of activated functions from preceding layers, with a dictionary of affine functions at the first layer. Unlike existing theories…

• Quantitative Gaussian-Process limits of Tensor Programs

  Andrea Agazzi, Eloy Mosig Garc\'ia, Dario Trevisan · 8 de julio de 2026

  We study the infinite-width Gaussian-process limit of random neural networks through the lens of tensor programs, and we provide a quantitative convergence theory in Wasserstein distance. Our main result gives explicit finite-width error bounds, of order inverse square-root of the widths bet…

• Width-Robust Learnability in Mean-Field Bayesian Neural Networks

  Dmitry Vaintrob, Kaarel H\"anni · 8 de julio de 2026

  Infinite-width limits are a standard way to reason about neural networks, but it is not automatic that the limiting learner has the same complexity-theoretic inductive bias as large finite networks. We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, …

• Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

  Xiangwu Wang, Chengwei Cao, Yicheng Song, Ran Bi, Peilin Yu · 8 de julio de 2026

  Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For …

• Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

  Yao Fu, Chunxia Zhang, Junmin Liu, Yihang Jin, Haishan Ye, Yuanao Yang · 8 de julio de 2026

  Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima asso…

• A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

  Arkaprabha Ganguli, Emil Constantinescu · 8 de julio de 2026

  A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of $\textbf{when}$ and $\textbf{by how much}$ has been lacking. Working on the unit circle, we give such an account …

• On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

  Don Li · 8 de julio de 2026

  The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs. However, in ill-conditioned or non-convex landscapes, the implicit inverse Hessian approximation can suffer from an exploding condition number, leading to numeric…

• Kernel-based Operator Learning: Error Analysis, Budget Allocation, and a Physics-Informed Extension

  R\"udiger Kempf · 8 de julio de 2026

  We study kernel-based operator learning in a two-stage sampling framework, where an offline kernel regression operator learns a discretized representation of the target operator from input-output pairs and an online kernel reconstruction operator recovers the output function from predicted observati…

• No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

  Noel Thomas · 8 de julio de 2026

  Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimate…

• A Gradient Flow Perspective on Minimum MMD Estimation

  Sophia Seulkee Kang, Louis Sharrock, Xiaoyuan Cheng, Fran\c{c}ois-Xavier Briol, Zonghao Chen · 7 de julio de 2026

  Minimum maximum mean discrepancy (MMD) estimation has emerged as a robust and likelihood-free alternative to maximum likelihood estimation for parameter estimation. Yet, despite its practical success, the associated optimization problem remains poorly understood, with theoretical guarantees for exis…

• A Unified Framework for Quantized and Continuous Strong Lottery Tickets

  Aakash Kumar, Emanuele Natale · 7 de julio de 2026

  The Strong Lottery Ticket Hypothesis (SLTH) asserts that sufficiently overparameterized, randomly initialized neural networks contain sparse subnetworks that, even without any training, can match the performance of a small trained network on a given dataset. A key mathematical tool in the theoretica…

• SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

  Omer Tariq, Syed Muhammad Raza, Jeongbae Son · 7 de julio de 2026

  Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \textbf{SAD-LoRA} (\textbf{S}pectral \textbf{A}lignment \textbf{D}isti…

• Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

  Thibaut Boissin (IRIT-MISFIT), Thomas Massena (DTIPG - SNCF, IRIT-MISFIT), Franck Mamalet (IRIT-MISFIT), Mathieu Serrurier (IRIT-MISFIT) · 7 de julio de 2026

  Orthogonality-based optimizers, such as Muon, have recently shown strong performance across large-scale training and community-driven efficiency challenges. However, these methods rely on a costly gradient orthogonalization step. Even efficient iterative approximations such as Newton-Schulz remain e…

• What Does a Discrete Diffusion Model Learn?

  Rodrigo Casado Noguerales, Bernhard Sch\"olkopf, Thomas Hofmann, Aran Raoufi · 7 de julio de 2026

  What does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural network in the wrong coordinate changes the process being trained and sampled. Starting with a rigorou…

• Implicit Bias of SGD in Multivariate ReLU Networks: Effective Width Collapse

  Shuang Liang, Tom Jacobs, Guido Mont\'ufar · 7 de julio de 2026

  We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression. In a mean-field regime, the training dynamics are approximated by a Wasserstein gradient flow that converges to a unique stationary measure. We characterize the struc…

• Compressed Computation under $L^4$ Loss is likely Computation in Superposition

  Francisco Ferreira da Silva, Stefan Heimersheim · 7 de julio de 2026

  Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition.…

• msPCA: An R Package for Sparse PCA with Multiple Components

  Ryan Cory-Wright, Jean Pauphilet · 7 de julio de 2026

  We present msPCA: an open-source R package for sparse principal component analysis with multiple components. It implements an alternating maximization algorithm to generate a set of sparse loading vectors that collectively explain a large fraction of the variance in a dataset, while remaining non-re…

• Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI

  Bing Cheng, Yi-Shuai Niu, Howell Tong, Shing-Tung Yau · 7 de julio de 2026

  Conventional uniform convergence bounds and empirical risk minimization break down in massive over-parameterized models, such as large language transformers and biological sequence networks. With near-infinite unconstrained internal degrees of freedom, their optimization landscapes develop flat vert…

• The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

  Thomas Hofmann · 7 de julio de 2026

  Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them. Edge-of-stability behavior, sharpness oscillations, catapult phases, balancing, and movement toward flatter representations are effects of the tra…

• Fast, Parallel, Query-Efficient Binary Classification

  Ishani Karmarkar, Liam O'Carroll, Aaron Sidford · 7 de julio de 2026

  We study the fundamental classification problem of computing a separating hyperplane for a binary-labeled dataset of size $n$ with normalized $d$-dimensional features. Letting $\Phi \in \mathbb{R}^{n \times d}$ denote the feature matrix and $\gamma$ the margin of the maximum-margin separating hyperp…