Stochastic Gradient Optimization Techniques

Derniers papiers

• Learning reveals invisible structure in low-rank RNNs

  Yoav Ger, Omri Barak · 7 mai 2026

  Learning in neural systems arises from synaptic changes that reshape the representations underlying behavior. While low-rank recurrent neural networks (RNNs) have emerged as a powerful framework for linking connectivity to function, a theoretical understanding of their learning process remains elusi…

• Layerwise LQR for Geometry-Aware Optimization of Deep Networks

  Simon Dufort-Labb\'e, Pierre-Luc Bacon, Razvan Pascanu, Simon Lacoste-Julien, Aristide Baratin · 7 mai 2026

  Geometry-aware optimizers such as Newton and natural gradient can improve conditioning in deep learning, but scalable variants such as K-FAC, Shampoo, and related preconditioners usually impose structural approximations early, often discarding cross-layer interactions induced by the network computat…

• Estimating the expected output of wide random MLPs more efficiently than sampling

  Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, Paul Christiano · 7 mai 2026

  By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, we show how to estimate its expected output over Gaussian inputs…

• Sharp Capacity Thresholds in Linear Associative Memory: From Winner-Take-All to Listwise Retrieval

  Nicholas Barnfield, Juno Kim, Eshaan Nichani, Jason D. Lee, Yue M. Lu · 7 mai 2026

  How many key-value associations can a $d\times d$ linear memory store? We show that the answer depends not only on the $d^2$ degrees of freedom in the memory matrix, but also on the retrieval criterion. In an isotropic Gaussian model for the stored pairs, we show that top-1 retrieval, where every si…

• BOOOM: Loss-Function-Agnostic Black-Box Optimization over Orthonormal Manifolds for Machine Learning and Statistical Inference

  Beomchang Kim, Subhrajyoty Roy, Priyam Das · 7 mai 2026

  Optimization over the Stiefel manifold $\mathrm{St}(p,d)$, the set of $p \times d$ column-orthonormal matrices, is fundamental in statistics, machine learning, and scientific computing, yet remains challenging in the presence of non-convex, non-smooth, or black-box objectives. Existing methods large…

• Exact Dual Geometry of SOC-ICNN Value Functions

  Kang Liu, Jianchen Hu, Wei Peng · 7 mai 2026

  Input Convex Neural Networks (ICNNs) are commonly used in a two-stage manner: one first trains a convex network and then minimizes it over its input in a downstream inference problem. Recent second-order-cone ICNNs (SOC-ICNNs) enrich ReLU-based ICNNs with quadratic and conic modules and admit an exa…

• Anon: Extrapolating Adaptivity Beyond SGD and Adam

  Yiheng Zhang, Kaiyan Zhao, Shaowu Wu, Yiming Wang, Jiajun Wu, Leong Hou U, Steve Drew, Xiaoguang Niu · 7 mai 2026

  Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse than non-adaptive methods, such as SGD on classical architectures like CNNs. We identify a key cause of this performance g…

• Training-Time Batch Normalization Reshapes Local Partition Geometry in Piecewise-Affine Networks

  Xuan Qi, Yi Wei, Fanqi Yu, Furao shen, Vittorio Murino, Cigdem Beyan · 7 mai 2026

  Batch normalization (BN) is central to modern deep networks, but its effect on the realized function during training remains less understood than its optimization benefits. We study training-time BN in continuous piecewise-affine (CPA) networks through the geometry of switching hyperplanes and the i…

• Scale-Parameter Selection in Gaussian Kolmogorov-Arnold Networks

  Amir Noorizadegan, Sifan Wang · 7 mai 2026

  Kolmogorov--Arnold Networks (KANs) have recently attracted attention as edge-based neural architectures in which learnable univariate functions replace conventional fixed activation functions. A key source of flexibility in KANs is the choice of basis functions used to parameterize the learnable edg…

• On Adaptivity in Zeroth-Order Optimization

  Hassan Dbouk, Nidham Gazagnadou, Matthias Reisser, Christos Louizos · 6 mai 2026

  We investigate the effectiveness of adaptive zeroth-order (ZO) optimization for memory-constrained fine-tuning of large language models (LLMs). Contrary to prior claims, we show that adaptive ZO methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD, while incurring significan…

• A second-order method landing on the Stiefel manifold via Newton$\unicode{x2013}$Schulz iteration

  Xinhui Xiong, Bin Gao, P. -A. Absil · 6 mai 2026

  Retraction-free approaches offer attractive low-cost alternatives to Riemannian methods on the Stiefel manifold, but they are often first-order, which may limit the efficiency under high-accuracy requirements. To this end, we propose a second-order method landing on the Stiefel manifold without invo…

• Provable Accuracy Collapse in Embedding-Based Representations under Dimensionality Mismatch

  Dionysis Arvanitakis, Vaggos Chatziafratis, Yiyuan Luo · 6 mai 2026

  Embedding-based representations in Euclidean space $\mathbb{R}^d$ are a cornerstone of modern machine learning, where a major goal is to use the \emph{smallest dimension} that faithfully captures data relations. In this work, we prove sharp dimension--accuracy tradeoffs and identify a fundamental in…

• Optimal Rates for Pure $\varepsilon$-Differentially Private Stochastic Convex Optimization with Heavy Tails

  Andrew Lowy · 6 mai 2026

  We study stochastic convex optimization (SCO) with heavy-tailed gradients under pure $\varepsilon$-differential privacy (DP). Instead of assuming a bound on the worst-case Lipschitz parameter of the loss, we assume only a bounded $k$-th moment. This assumption allows for unbounded, heavy-tailed stoc…

• Rethinking the Rank Threshold for LoRA Fine-Tuning

  Juneyoung Park · 6 mai 2026

  A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition $r(r+1)/2 > KN$ on the LoRA rank $r$ for the absence of spurious local minima under squared-error loss, prescribing $r \geq 12$ on canonical few-shot RoBERTa setups. The condition i…

• Deciphering Shortcut Learning from an Evolutionary Game Theory Perspective

  Xiayang Li, Kuo Gai, Shihua Zhang · 6 mai 2026

  Shortcut learning causes deep learning models to rely on non-essential features within the data. However, its formation in deep neural network training still lacks theoretical understanding. In this paper, we provide a formal definition of core and shortcut features and employ evolutionary game theo…

• A Universal Reproducing Kernel Hilbert Space from Polynomial Alignment and IMQ Distance

  Taha Bouhsine · 6 mai 2026

  We introduce the Yat kernel $$k_{b,\varepsilon}(\mathbf{w},\mathbf{x})=\frac{(\mathbf{w}^\top\mathbf{x}+b)^2}{\|\mathbf{x}-\mathbf{w}\|^2+\varepsilon},\qquad b\ge 0,\ \varepsilon>0,$$ a rational hidden-unit primitive whose units are Mercer sections over a shared input/weight space. For $b\ge 0$ the …

• Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

  Zhe Li, Bicheng Ying, Zidong Liu, Haibo Yang · 6 mai 2026

  Classical optimization theory establishes that zeroth-order (ZO) algorithms suffer from a dimension-dependent slowdown, with convergence rates typically scaling with the model dimension compared to first-order methods. However, in contrast to these theoretical expectations, a growing body of recent …

• Information Plane Analysis of Binary Neural Networks

  Maximilian Nothnagel, Bernhard C. Geiger · 6 mai 2026

  Information plane (IP) analysis has been suggested to study the training dynamics of deep neural networks through mutual information (MI) between inputs, representations, and targets. However, its statistical validity is often compromised by the difficulty of estimating MI from samples of high-dimen…

• Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer

  Jinghui Yuan, Jiaxuan Zou, Shuo Wang, Yong Liu, Feiping Nie · 6 mai 2026

  Matrix-based optimizers have demonstrated immense potential in training Large Language Models (LLMs), however, designing an ideal optimizer remains a formidable challenge. A superior optimizer must satisfy three core desiderata: efficiency, achieving Muon-like preconditioning to accelerate optimizat…

• Fast Log-Domain Sinkhorn Optimal Transport with Warp-Level GPU Reductions

  Hao Xiao · 5 mai 2026

  Entropic regularized optimal transport (OT) via the Sinkhorn algorithm has become a fundamental tool in machine learning, yet existing implementations either suffer from numerical instability for small regularization parameters or incur significant overhead from deep learning frameworks. We present …

• Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models

  Daniel Agyei Asante, Ernie Chang, Yang Li · 5 mai 2026

  Low-rank decomposition is a compelling approach for compressing large language models, but its effectiveness hinges on selecting which singular-vector bases to retain for a target task. Existing methods such as Basel adapt singular-value coefficients on downstream data and prune bases with small re-…

• A Parameter-Free First-Order Algorithm for Non-Convex Optimization with $\tilde{\mkern1mu O}(\epsilon^{-5/3})$ Global Rate

  Sichao Xiong, Sadok Jerad, Coralia Cartis · 5 mai 2026

  We introduce PF-AGD, the first parameter-free, deterministic, accelerated first-order method to achieve $O(\epsilon^{-5/3}\log(1/\epsilon))$ oracle complexity bound when minimizing sufficiently smooth, non-convex functions; this is the best-known bound for first-order methods on smooth non-convex ob…

• Model Merging: Foundations and Algorithms

  Donato Crisostomi · 5 mai 2026

  Modern deep learning usually treats models as separate artifacts: trained independently, specialized for particular purposes, and replaced when improved versions appear. This thesis studies model merging as an alternative paradigm: combining independently trained neural networks directly in weight s…

• A second-order method on the Stiefel manifold via Newton$\unicode{x2013}$Schulz

  Xinhui Xiong, Bin Gao, P. -A. Absil · 5 mai 2026

  Retraction-free approaches offer attractive low-cost alternatives to Riemannian methods on the Stiefel manifold, but they are often first-order, which may limit the efficiency under high-accuracy requirements. To this end, we propose a second-order method landing on the Stiefel manifold without invo…

• LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment

  Banseok Lee, Youngmin Kim · 5 mai 2026

  We identify the Spectral Energy Gain in extreme model compression, where low-rank binary approximations outperform tiny-rank floating-point baselines for heavy-tailed spectra. However, prior attempts fail to realize this potential, trailing state-of-the-art 1-bit methods. We attribute this degradati…