Stochastic Gradient Optimization Techniques

Derniers papiers

• Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics

  Connall Garrod, Jonathan P. Keating, Christos Thrampoulidis · 4 décembre 2025

  Cross-entropy (CE) training loss dominates deep learning practice, yet existing theory often relies on simplifications, either replacing it with squared loss or restricting to convex models, that miss essential behavior. CE and squared loss generate fundamentally different dynamics, and convex linea…

• Convergence of a class of gradient-free optimisation schemes when the objective function is noisy, irregular, or both

  Christophe Andrieu, Nicolas Chopin, Ettore Fincato, Mathieu Gerber · 4 décembre 2025

  We investigate the convergence properties of a class of iterative algorithms designed to minimize a potentially non-smooth and noisy objective function, which may be algebraically intractable and whose values may be obtained as the output of a black box. The algorithms considered can be cast under t…

• Optimal Transportation and Alignment Between Gaussian Measures

  Sanjit Dandapanthula, Aleksandr Podkopaev, Shiva Prasad Kasiviswanathan, Aaditya Ramdas, Ziv Goldfeld · 4 décembre 2025

  Optimal transport (OT) and Gromov-Wasserstein (GW) alignment provide interpretable geometric frameworks for comparing, transforming, and aggregating heterogeneous datasets -- tasks ubiquitous in data science and machine learning. Because these frameworks are computationally expensive, large-scale ap…

• Convergence for Discrete Parameter Updates

  Paul Wilson, Fabio Zanasi, George Constantinides · 4 décembre 2025

  Modern deep learning models require immense computational resources, motivating research into low-precision training. Quantised training addresses this by representing training components in low-bit integers, but typically relies on discretising real-valued updates. We introduce an alternative appro…

• When does Gaussian equivalence fail and how to fix it: Non-universal behavior of random features with quadratic scaling

  Garrett G. Wen, Hong Hu, Yue M. Lu, Zhou Fan, Theodor Misiakiewicz · 4 décembre 2025

  A major effort in modern high-dimensional statistics has been devoted to the analysis of linear predictors trained on nonlinear feature embeddings via empirical risk minimization (ERM). Gaussian equivalence theory (GET) has emerged as a powerful universality principle in this context: it states that…

• A Fully First-Order Layer for Differentiable Optimization

  Zihao Zhao, Kai-Chia Mo, Shing-Hei Ho, Brandon Amos, Kai Wang · 3 décembre 2025

  Differentiable optimization layers enable learning systems to make decisions by solving embedded optimization problems. However, computing gradients via implicit differentiation requires solving a linear system with Hessian terms, which is both compute- and memory-intensive. To address this challeng…

• Safeguarded Stochastic Polyak Step Sizes for Non-smooth Optimization: Robust Performance Without Small (Sub)Gradients

  Dimitris Oikonomou, Nicolas Loizou · 3 décembre 2025

  The stochastic Polyak step size (SPS) has proven to be a promising choice for stochastic gradient descent (SGD), delivering competitive performance relative to state-of-the-art methods on smooth convex and non-convex optimization problems, including deep neural network training. However, extensions …

• Solving Neural Min-Max Games: The Role of Architecture, Initialization & Dynamics

  Deep Patel, Emmanouil-Vasileios Vlatakis-Gkaragkounis · 2 décembre 2025

  Many emerging applications - such as adversarial training, AI alignment, and robust optimization - can be framed as zero-sum games between neural nets, with von Neumann-Nash equilibria (NE) capturing the desirable system behavior. While such games often involve non-convex non-concave objectives, emp…

• Using physics-inspired Singular Learning Theory to understand grokking & other phase transitions in modern neural networks

  Anish Lakkapragada · 2 décembre 2025

  Classical statistical inference and learning theory often fail to explain the success of modern neural networks. A key reason is that these models are non-identifiable (singular), violating core assumptions behind PAC bounds and asymptotic normality. Singular learning theory (SLT), a physics-inspire…

• Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance

  Robin Yadav, Shuo Xie, Tianhao Wang, Zhiyuan Li · 2 décembre 2025

  Adaptive optimization methods (such as Adam) play a major role in LLM pretraining, significantly outperforming Gradient Descent (GD). Recent studies have proposed new smoothness assumptions on the loss function to explain the advantages of adaptive algorithms with structured preconditioners, e.g., c…

• The Spectral Dimension of NTKs is Constant: A Theory of Implicit Regularization, Finite-Width Stability, and Scalable Estimation

  Praveen Anilkumar Shukla · 2 décembre 2025

  Modern deep networks are heavily overparameterized yet often generalize well, suggesting a form of low intrinsic complexity not reflected by parameter counts. We study this complexity at initialization through the effective rank of the Neural Tangent Kernel (NTK) Gram matrix, $r_{\text{eff}}(K) = (\…

• Low-Rank Prehab: Preparing Neural Networks for SVD Compression

  Haoran Qin, Shansita Sharma, Ali Abbasi, Chayne Thrash, Soheil Kolouri · 2 décembre 2025

  Low-rank approximation methods such as singular value decomposition (SVD) and its variants (e.g., Fisher-weighted SVD, Activation SVD) have recently emerged as effective tools for neural network compression. In this setting, decomposition acts as a "surgical" intervention, followed by fine-tuning th…

• Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets

  Aaryan Gupta, Rishi Saket, Aravindan Raghuveer · 2 décembre 2025

  Given a training dataset, the goal of dataset distillation is to derive a synthetic dataset such that models trained on the latter perform as well as those trained on the training dataset. In this work, we develop and analyze an efficient dataset distillation algorithm for supervised learning, speci…

• Does Flatness imply Generalization for Logistic Loss in Univariate Two-Layer ReLU Network?

  Dan Qiao, Yu-Xiang Wang · 2 décembre 2025

  We consider the problem of generalization of arbitrarily overparameterized two-layer ReLU Neural Networks with univariate input. Recent work showed that under square loss, flat solutions (motivated by flat / stable minima and Edge of Stability phenomenon) provably cannot overfit, but it remains uncl…

• Achieving Linear Speedup with ProxSkip in Distributed Stochastic Optimization

  Luyao Guo, Sulaiman A. Alghunaim, Kun Yuan, Laurent Condat, Jinde Cao · 2 décembre 2025

  The ProxSkip algorithm for distributed optimization is gaining increasing attention due to its effectiveness in reducing communication. However, existing analyses of ProxSkip are limited to the strongly convex setting and fail to achieve linear speedup with respect to the number of nodes. Key questi…

• Tuning Universality in Deep Neural Networks

  Arsham Ghavasieh · 2 décembre 2025

  Deep neural networks (DNNs) exhibit crackling-like avalanches whose origin lacks a mechanistic explanation. Here, I derive a stochastic theory of deep information propagation (DIP) by incorporating Central Limit Theorem (CLT)-level fluctuations. Four effective couplings $(r, h, D_1, D_2)$ characteri…

• Sharpness of Minima in Deep Matrix Factorization: Exact Expressions

  Anil Kamber, Rahul Parhi · 2 décembre 2025

  Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eig…

• On the Condition Number Dependency in Bilevel Optimization

  Lesi Chen, Jingzhao Zhang · 1 décembre 2025

  Bilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an $\epsilon$-stationary point with first-order methods when the upper-level problem is nonconvex and the …

• Arithmetic-Mean $\mu$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets

  Haosong Zhang, Shenxi Wu, Yichi Zhang, Xi Chen, Wei Lin · 1 décembre 2025

  Choosing an appropriate learning rate remains a key challenge in scaling depth of modern deep networks. The classical maximal update parameterization ($\mu$P) enforces a fixed per-layer update magnitude, which is well suited to homogeneous multilayer perceptrons (MLPs) but becomes ill-posed in heter…

• Variational analysis of determinantal varieties

  Yan Yang, Bin Gao, Ya-xiang Yuan · 1 décembre 2025

  Determinantal varieties -- the sets of bounded-rank matrices or tensors -- have attracted growing interest in low-rank optimization. The tangent cone to low-rank sets is widely studied and underpins a range of geometric methods. The second-order geometry, which encodes curvature information, is more…

• Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian

  Yiran Zhang, Weihang Xu, Mo Zhou, Maryam Fazel, Simon Shaolei Du · 1 décembre 2025

  Score matching has become a central training objective in modern generative modeling, particularly in diffusion models, where it is used to learn high-dimensional data distributions through the estimation of score functions. Despite its empirical success, the theoretical understanding of the optimiz…

• List-Decodable Regression via Expander Sketching

  Herbod Pourali, Sajjad Hashemian, Ebrahim Ardeshir-Larijani · 1 décembre 2025

  We introduce an expander-sketching framework for list-decodable linear regression that achieves sample complexity $\tilde{O}((d+\log(1/\delta))/\alpha)$, list size $O(1/\alpha)$, and near input-sparsity running time $\tilde{O}(\mathrm{nnz}(X)+d^{3}/\alpha)$ under standard sub-Gaussian assumptions. O…

• Entropy is all you need for Inter-Seed Cross-Play in Hanabi

  Johannes Forkel, Jakob Foerster · 1 décembre 2025

  We find that in Hanabi, one of the most complex and popular benchmarks for zero-shot coordination and ad-hoc teamplay, a standard implementation of independent PPO with a slightly higher entropy coefficient 0.05 instead of the typically used 0.01, achieves a new state-of-the-art in cross-play betwee…

• Sparse Multiple Kernel Learning: Alternating Best Response and Semidefinite Relaxations

  Dimitris Bertsimas, Caio de Prospero Iglesias, Nicholas A. G. Johnson · 1 décembre 2025

  We study Sparse Multiple Kernel Learning (SMKL), which is the problem of selecting a sparse convex combination of prespecified kernels for support vector binary classification. Unlike prevailing l1 regularized approaches that approximate a sparsifying penalty, we formulate the problem by imposing an…

• Adam Simplified: Bias Correction Debunked

  Sam Laing, Antonio Orvieto · 27 novembre 2025

  The Adam optimizer is a cornerstone of modern deep learning, yet the empirical necessity of each of its individual components is often taken for granted. This paper presents a focused investigation into the role of bias-correction, a feature whose contribution remains poorly understood. Through a se…