Stochastic Gradient Optimization Techniques

Neueste Paper

• Spectral DPPs via NEPv: A Scalable Continuous Relaxation of Determinantal MAP for Diversity-Aware Data Selection

  Richard Yi Da Xu · 19. Juni 2026

  Selecting a small, diverse, high-quality subset from a massive pool of candidates is a recurring primitive in modern machine learning -- data curation and coreset selection for training and fine-tuning large models, active-learning batch acquisition, prompt and exemplar selection for in-context lear…

• On the Oracle Complexity of Interpolation-Based Gradient Descent

  Dongmin Lee, William Lu, Anuran Makur · 19. Juni 2026

  Recent work on first-order optimizers for empirical risk minimization (ERM) has suggested that smoothness of ERM loss functions in the training data, rather than in the optimization parameters, can be leveraged to improve the oracle complexity of gradient descent (GD) methods. In this paper, we prop…

• Recurrent neural networks approximate continuous functions

  Valentin Abadie, Clemens Hutter, Helmut B\"olcskei · 19. Juni 2026

  Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuo…

• Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds

  Dat H. Do, Rushi Shah, Duc V. Le, Dianbo Liu · 19. Juni 2026

  Compositionality is believed to be the foundation for generalization, enabling models to reuse meaningful primitives in novel combinations. Yet, models trained with standard gradient-based optimization rarely, and often only weakly, exhibit compositional internal structure, and it remains unclear ho…

• Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima

  Md Sakir Ahmed, Kumaresh Sarmah, Hemen Dutta · 19. Juni 2026

  A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations …

• Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence

  Alexander Tyurin · 19. Juni 2026

  The score matching problem is a central training objective in modern generative modeling, diffusion models, fitting unnormalized statistical models, and inverse problems. A standard approach is to minimize the forward Fisher divergence, where the expectation is taken with respect to the teacher dist…

• Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods

  Depen Morwani, Alexandru Meterez, Pranav Nair, Sham Kakade · 18. Juni 2026

  Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al., 2018] are widely used in modern training, but their stochastic benefits depend on two distinct quantities: serial runtime, the number of iterations needed to reach a target…

• Stochastic Adaptive Gradient Descent Without Descent

  Jean-Fran\c{c}ois Aujol, J\'er\'emie Bigot, Camille Castera · 18. Juni 2026

  We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning. The method comes from a theoretically-grounded adaptatio…

• Exponentially many initializations to avoid barren plateaus

  Ankit Kulshrestha, Ricard Puig, Diego Garc\'ia-Mart\'in, Lukasz Cincio, Ilya Safro, Zo\"e Holmes, M. Cerezo · 18. Juni 2026

  Barren plateaus are stated as an average-case phenomenon: pick an ansatz, initialize it naively, and concentration follows. This has led to the common view that a potential cure for barren plateaus is simply to initialize the parameters more carefully. Here we show that the situation is subtler. We …

• Mixed-Precision Communication-Avoiding SGD for Generalized Linear Models on GPUs

  Aditya Devarakonda, Irene Sim\'o Mu\~noz, Giulia Guidi · 18. Juni 2026

  Distributed stochastic gradient descent (SGD) is limited by communication rather than computation, since each iteration requires an AllReduce across processes. Communication-avoiding SGD (CA-SGD) amortizes communication over $s$ iterations by replacing $s$ consecutive AllReduces with a single AllRed…

• KANLib -- An Modular, Extensible and Fast Kolmogorov-Arnold Network Implementation

  Julian Hoever, Gregor Schiele · 17. Juni 2026

  Kolmogorov-Arnold Networks (KANs) have recently emerged as a promising alternative to traditional multilayer perceptrons by replacing linear weights with learnable univariate functions. Despite their theoretical advantages in interpretability and expressiveness, practical research of KANs remains di…

• Sum-of-Squares Degree Barriers for the Reweighted-Hinge Method in Robust Halfspace Learning: A Christoffel-Function Characterization

  Xiaoyu Li · 17. Juni 2026

  A certificate that removes outliers sees the data only through its low-degree moments, and an adversary exploits exactly this, hiding corruption where the clean data already looks typical, in the blind spot no bounded-degree test resolves. That blind spot turns out to have an exact size: the Christo…

• MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

  Da Chang, Ganzhao Yuan · 17. Juni 2026

  Efficient optimization is essential for training large language models. Although intra-layer selective updates have been explored, a general mechanism that enables fine-grained control while ensuring convergence guarantees is still lacking. To bridge this gap, we propose \textbf{MGUP}, a novel mecha…

• Beyond IGO-Flow: Toward Convergence Analysis of IGO in Continuous Spaces

  Ryosuke Kimura, Youhei Akimoto · 17. Juni 2026

  Information-Geometric Optimization (IGO) provides a unified framework for black-box optimization by interpreting the adaptation of a search distribution as a natural gradient update. Despite its conceptual importance, the convergence theory of IGO remains limited: most existing results concern conti…

• Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descent

  Ning Yang, Yikuan Zhang, Qi Ouyang, Chao Tang, Yuhai Tu · 17. Juni 2026

  Stochastic gradient descent (SGD) is central to deep learning, yet the dynamical origin of its preference for flatter, more generalizable solutions remains unclear. Here, by analyzing SGD learning dynamics, we identify a nonequilibrium mechanism that governs solution selection during training. Numer…

• Monotonic Kolmogorov-Arnold Networks: A Theoretical and Empirical Study of Monotonicity as an Inductive Bias

  Mikhail Krasnov, Carolina Fortuna, Bla\v{z} Bertalani\v{c} · 17. Juni 2026

  Monotonicity has been a long-running architectural inductive bias for neural networks, motivated by tabular, scientific, and economic settings where outputs are known to respond monotonically to certain inputs. Existing approaches are MLP- or flow-based and lack per-edge functional transparency; the…

• Edge Flow: A Tractable and Predictive Continuous-Time Model for Gradient Descent at the Edge of Stability

  Pierre Marion · 17. Juni 2026

  Gradient descent in deep learning may operate at the edge of stability (EoS), a regime in which the largest eigenvalue of the loss Hessian hovers near the stability threshold $2/\eta$, where $\eta$ is the learning rate. Classical analysis tools such as gradient flow and the descent lemma do not appl…

• Accelerated Convex Optimization via Hamiltonian Dynamics with Deterministic Integration Time

  Xiuyuan Wang, Vishwak Srinivasan, Qiang Fu, Siddharth Mitra, Ashia Wilson, Andre Wibisono · 17. Juni 2026

  We develop Hamiltonian dynamics-based algorithms for smooth convex optimization that achieve accelerated rates of convergence. By exploiting contraction of averaged Hamiltonian flow trajectories rather than requiring contraction at trajectory endpoints, we show that Hamiltonian dynamics-based optimi…

• Functional Gradient Descent with Adaptive Representations

  Daniel Csillag, Rodrigo Schuller, Pedro Dall'Antonia, Leonidas Guibas, Luiz Velho, Tiago Novello · 16. Juni 2026

  Functional optimization problems are typically solved by optimizing the parameters of a fixed representation, such as a neural network, resulting in highly nonconvex losses that complicate both training and theoretical analysis. An interesting alternative is functional gradient descent (FGD), that i…

• Exploding and vanishing gradients in deep neural networks: the effect of residual connections

  Vivek S Borkar · 16. Juni 2026

  The well known phenomenon of exploding and vanishing gradients in deep neural networks is analyzed using multiplicative ergodic theory. The effect of adding a residual connection is explained in this context. Specifically, a characterization of Liapunov exponents due to Furstenberg and Kifer is expl…

• Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

  Greg Ongie, Rahul Parhi · 16. Juni 2026

  We develop a general framework for analyzing representation costs of parametric data-fitting methods through their parameter-space regularizers. From this abstract perspective, we define representation costs for arbitrary parametric models and reveal their induced (native) function spaces. This unif…

• Wasserstein Convergence of ODE-Based Samplers in Decentralized Diffusion Model via Velocity Field Decomposition

  Chencheng Tang, Xuanyu Xue, Fangyikang Wang, Chao Zhang, Hubery Yin · 16. Juni 2026

  Diffusion models have achieved impressive empirical success in generative tasks, and their convergence theory is now relatively well understood. Motivated by privacy and scalability, recent decentralized diffusion architectures replace a single global velocity field with multiple local experts and a…

• Variance Reduction for Non-Log-Concave Sampling with Applications to Inverse Problems

  M. Berk Sahin, Ahmet Ege Tanriverdi, Behzad Sharif, Abolfazl Hashemi · 16. Juni 2026

  Sampling from high-dimensional, non-log-concave distributions with unnormalized densities is a fundamental challenge in machine learning, particularly when the exact gradient of the potential is unavailable and must be approximated via stochastic gradients that exhibit high variance under a fixed bu…

• Schattor: Schatten-family methods for deep learning optimization

  Bohao Ma, Junyu Zhang, Chuan He · 16. Juni 2026

  Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis. Motivated by the limitations of SGD and the success of adaptive optimizers, we propose {\…

• Brownian Kernel Ladders

  Mahdi Mohammadigohari, Giuseppe Di Fatta, Giuseppe Nicosia, Panos M Pardalos · 16. Juni 2026

  Constructing mathematically tractable function spaces that capture hierarchical compositional representations remains a central challenge in statistical learning theory. We introduce Brownian kernel ladders (BKLs), a recursively defined hierarchy of integral reproducing kernel Hilbert spaces generat…