Stochastic Gradient Optimization Techniques

Derniers papiers

• Merging of Kolmogorov-Arnold networks trained on disjoint datasets

  Andrew Polar, Michael Poluektov · 23 décembre 2025

  Training on disjoint datasets can serve two primary goals: accelerating data processing and enabling federated learning. It has already been established that Kolmogorov-Arnold networks (KANs) are particularly well suited for federated learning and can be merged through simple parameter averaging. Wh…

• On the Convergence Rate of LoRA Gradient Descent

  Siqiao Mu, Diego Klabjan · 23 décembre 2025

  The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing th…

• Secret mixtures of experts inside your LLM

  Enric Boix-Adsera · 23 décembre 2025

  Despite being one of the earliest neural network layers, the Multilayer Perceptron (MLP) is arguably one of the least understood parts of the transformer architecture due to its dense computation and lack of easy visualization. This paper seeks to understand the MLP layers in dense LLM models by hyp…

• Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method

  Jiaming Liang · 23 décembre 2025

  High-probability guarantees in stochastic optimization are often obtained only under strong noise assumptions such as sub-Gaussian tails. We show that such guarantees can also be achieved under the weaker assumption of bounded variance by developing a stochastic proximal point method. This method co…

• Towards Guided Descent: Optimization Algorithms for Training Neural Networks At Scale

  Ansh Nagwekar · 23 décembre 2025

  Neural network optimization remains one of the most consequential yet poorly understood challenges in modern AI research, where improvements in training algorithms can lead to enhanced feature learning in foundation models, order-of-magnitude reductions in training time, and improved interpretabilit…

• Finite-sample guarantees for data-driven forward-backward operator methods

  Filippo Fabiani, Barbara Franci · 23 décembre 2025

  We establish finite sample certificates on the quality of solutions produced by data-based forward-backward (FB) operator splitting schemes. As frequently happens in stochastic regimes, we consider the problem of finding a zero of the sum of two operators, where one is either unavailable in closed f…

• Theoretical Convergence Guarantees for Variational Autoencoders

  Sobihan Surendran (LPSM), Antoine Godichon-Baggioni (LPSM), Sylvain Le Corff (LPSM) · 23 décembre 2025

  Variational Autoencoders (VAE) are popular generative models used to sample from complex data distributions. Despite their empirical success in various machine learning tasks, significant gaps remain in understanding their theoretical properties, particularly regarding convergence guarantees. This p…

• Deep Legendre Transform

  Aleksey Minabutdinov, Patrick Cheridito · 23 décembre 2025

  We introduce a novel deep learning algorithm for computing convex conjugates of differentiable convex functions, a fundamental operation in convex analysis with various applications in different fields such as optimization, control theory, physics and economics. While traditional numerical methods s…

• Optimizer Dynamics at the Edge of Stability with Differential Privacy

  Ayana Hussain, Ricky Fang · 23 décembre 2025

  Deep learning models can reveal sensitive information about individual training examples, and while differential privacy (DP) provides guarantees restricting such leakage, it also alters optimization dynamics in poorly understood ways. We study the training dynamics of neural networks under DP by co…

• Operator-Based Generalization Bound for Deep Learning: Insights on Multi-Task Learning

  Mahdi Mohammadigohari, Giuseppe Di Fatta, Giuseppe Nicosia, Panos M. Pardalos · 23 décembre 2025

  This paper presents novel generalization bounds for vector-valued neural networks and deep kernel methods, focusing on multi-task learning through an operator-theoretic framework. Our key development lies in strategically combining a Koopman based approach with existing techniques, achieving tighter…

• Bridging Training and Merging Through Momentum-Aware Optimization

  Alireza Moayedikia, Alicia Troncoso · 22 décembre 2025

  Training large neural networks and merging task-specific models both exploit low-rank structure and require parameter importance estimation, yet these challenges have been pursued in isolation. Current workflows compute curvature information during training, discard it, then recompute similar inform…

• Polyharmonic Cascade

  Yuriy N. Bakhvalov · 22 décembre 2025

  This paper presents a deep machine learning architecture, the "polyharmonic cascade" -- a sequence of packages of polyharmonic splines, where each layer is rigorously derived from the theory of random functions and the principles of indifference. This makes it possible to approximate nonlinear funct…

• Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs

  Aaron Defazio, Konstantin Mishchenko, Parameswaran Raman, Hao-Jun Michael Shi, Lin Xiao · 22 décembre 2025

  We propose Generalized Primal Averaging (GPA), an extension of Nesterov's method in its primal averaging formulation that addresses key limitations of recent averaging-based optimizers such as single-worker DiLoCo and Schedule-Free (SF) in the non-distributed setting. These two recent algorithmic ap…

• Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation

  Zhenyu Liu, Yunzhen Liu, Zehao Fan, Garrett Gagnon, Yayue Hou, Nan Wu, Yangwook Kang, Liu Liu · 22 décembre 2025

  Mixture-of-Experts (MoE) models scale capacity via sparse activation but stress memory and bandwidth. Offloading alleviates GPU memory by fetching experts on demand, yet token-level routing causes irregular transfers that make inference I/O-bound. Static uniform quantization reduces traffic but degr…

• BOLT: Block-Orthonormal Lanczos for Trace estimation of matrix functions

  Kingsley Yeon, Promit Ghosal, Mihai Anitescu · 22 décembre 2025

  Efficient matrix trace estimation is essential for scalable computation of log-determinants, matrix norms, and distributional divergences. In many large-scale applications, the matrices involved are too large to store or access in full, making even a single matrix-vector (mat-vec) product infeasible…

• A Systems-Theoretic View on the Convergence of Algorithms under Disturbances

  Guner Dilsad Er, Sebastian Trimpe, Michael Muehlebach · 22 décembre 2025

  Algorithms increasingly operate within complex physical, social, and engineering systems where they are exposed to disturbances, noise, and interconnections with other dynamical systems. This article extends known convergence guarantees of an algorithm operating in isolation (i.e., without disturban…

• A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel

  Abiel Kim · 19 décembre 2025

  It has been demonstrated both theoretically and empirically that the ReLU MLP tends to extrapolate linearly for an out-of-distribution evaluation point. The machine learning literature provides ample analysis with respect to the mechanisms to which linearity is induced. However, the analysis of extr…

• Explicit and Non-asymptotic Query Complexities of Rank-Based Zeroth-order Algorithms on Smooth Functions

  Haishan Ye · 19 décembre 2025

  Rank-based zeroth-order (ZO) optimization -- which relies only on the ordering of function evaluations -- offers strong robustness to noise and monotone transformations, and underlies many successful algorithms such as CMA-ES, natural evolution strategies, and rank-based genetic algorithms. Despite …

• Muon is Provably Faster with Momentum Variance Reduction

  Xun Qian, Hussein Rammal, Dmitry Kovalev, Peter Richt\'arik · 19 décembre 2025

  Recent empirical research has demonstrated that deep learning optimizers based on the linear minimization oracle (LMO) over specifically chosen Non-Euclidean norm balls, such as Muon and Scion, outperform Adam-type methods in the training of large language models. In this work, we show that such opt…

• Learning Confidence Ellipsoids and Applications to Robust Subspace Recovery

  Chao Gao, Liren Shan, Vaidehi Srinivas, Aravindan Vijayaraghavan · 19 décembre 2025

  We study the problem of finding confidence ellipsoids for an arbitrary distribution in high dimensions. Given samples from a distribution $D$ and a confidence parameter $\alpha$, the goal is to find the smallest volume ellipsoid $E$ which has probability mass $\Pr_{D}[E] \ge 1-\alpha$. Ellipsoids ar…

• Optimization with Access to Auxiliary Information

  El Mahdi Chayti, Sai Praneeth Karimireddy · 19 décembre 2025

  We investigate the fundamental optimization question of minimizing a target function $f$, whose gradients are expensive to compute or have limited availability, given access to some auxiliary side function $h$ whose gradients are cheap or more available. This formulation captures many settings of pr…

• Polyharmonic Spline Packages: Composition, Efficient Procedures for Computation and Differentiation

  Yuriy N. Bakhvalov · 19 décembre 2025

  In a previous paper it was shown that a machine learning regression problem can be solved within the framework of random function theory, with the optimal kernel analytically derived from symmetry and indifference principles and coinciding with a polyharmonic spline. However, a direct application of…

• Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study

  Yotam Alexander, Yonatan Slutzky, Yuval Ran-Milo, Nadav Cohen · 19 décembre 2025

  Conventional wisdom attributes the mysterious generalization abilities of overparameterized neural networks to gradient descent (and its variants). The recent volume hypothesis challenges this view: it posits that these generalization abilities persist even when gradient descent is replaced by Guess…

• Stochastic Optimization with Random Search

  El Mahdi Chayti, Taha El Bakkali El Kadi, Omar Saadi, Martin Jaggi · 19 décembre 2025

  We revisit random search for stochastic optimization, where only noisy function evaluations are available. We show that the method works under weaker smoothness assumptions than previously considered, and that stronger assumptions enable improved guarantees. In the finite-sum setting, we design a va…

• A Split-Client Approach to Second-Order Optimization

  El Mahdi Chayti, Martin Jaggi · 19 décembre 2025

  Second-order methods promise faster convergence but are rarely used in practice because Hessian computations and decompositions are far more expensive than gradients. We propose a \emph{split-client} framework where gradients and curvature are computed asynchronously by separate clients. This abstra…