Stochastic Gradient Optimization Techniques

Derniers papiers

• Revisiting Chebyshev Polynomial and Anisotropic RBF Models for Tabular Regression

  Luciano Gerber, Huw Lloyd · 27 février 2026

  Smooth-basis models such as Chebyshev polynomial regressors and radial basis function (RBF) networks are well established in numerical analysis. Their continuously differentiable prediction surfaces suit surrogate optimisation, sensitivity analysis, and other settings where the response varies gradu…

• NoRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion

  Hung-Hsuan Chen · 27 février 2026

  Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning (PEFT). However, it faces a critical ``linear ceiling'' in complex reasoning tasks: simply increasing the rank yields diminishing returns due to intrinsic linear constraints. We introduce NoRA (Non-linear Rank Adaptation), a weight…

• Statistical Advantage of Softmax Attention: Insights from Single-Location Regression

  O. Duranthon, P. Marion, C. Boyer, B. Loureiro, L. Zdeborov\'a · 27 février 2026

  Large language models rely on attention mechanisms with a softmax activation. Yet the dominance of softmax over alternatives (e.g., component-wise or linear) remains poorly understood, and many theoretical works have focused on the easier-to-analyze linearized attention. In this work, we address thi…

• DP-aware AdaLN-Zero: Taming Conditioning-Induced Heavy-Tailed Gradients in Differentially Private Diffusion

  Tao Huang, Jiayang Meng, Xu Yang, Chen Hou, Hong Chen · 27 février 2026

  Condition injection enables diffusion models to generate context-aware outputs, which is essential for many time-series tasks. However, heterogeneous conditional contexts (e.g., observed history, missingness patterns or outlier covariates) can induce heavy-tailed per-example gradients. Under Differe…

• Model Agreement via Anchoring

  Eric Eaton, Surbhi Goel, Marcel Hussing, Michael Kearns, Aaron Roth, Sikata Bela Sengupta, Jessica Sorrell · 27 février 2026

  Numerous lines of aim to control $\textit{model disagreement}$ -- the extent to which two machine learning models disagree in their predictions. We adopt a simple and standard notion of model disagreement in real-valued prediction problems, namely the expected squared difference in predictions betwe…

• Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks

  Wenquan Ma, Yang Sui, Jiaye Teng, Bohan Wang, Jing Xu, Jingqin Yang · 27 février 2026

  Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize $\eta_t = \mathcal{O}(1/t)$ under non-convex training regimes, where $t$ denotes iterations. This rigid decay of the stepsize potentially impedes optimization and…

• Differentiable Zero-One Loss via Hypersimplex Projections

  Camilo Gomez, Pengyang Wang, Liansheng Tang · 27 février 2026

  Recent advances in machine learning have emphasized the integration of structured optimization components into end-to-end differentiable models, enabling richer inductive biases and tighter alignment with task-specific objectives. In this work, we introduce a novel differentiable approximation to th…

• Low-degree Lower bounds for clustering in moderate dimension

  Alexandra Carpentier, Nicolas Verzelen · 27 février 2026

  We study the fundamental problem of clustering $n$ points into $K$ groups drawn from a mixture of isotropic Gaussians in $\mathbb{R}^d$. Specifically, we investigate the requisite minimal distance $\Delta$ between mean vectors to partially recover the underlying partition. While the minimax-optimal …

• Exploiting Low-Rank Structure in Max-K-Cut Problems

  Ria Stevens, Fangshuo Liao, Barbara Su, Jianqiang Li, Anastasios Kyrillidis · 26 février 2026

  We approach the Max-3-Cut problem through the lens of maximizing complex-valued quadratic forms and demonstrate that low-rank structure in the objective matrix can be exploited, leading to alternative algorithms to classical semidefinite programming (SDP) relaxations and heuristic techniques. We pro…

• Neural network optimization strategies and the topography of the loss landscape

  Jianneng Yu, Alexandre V. Morozov · 26 février 2026

  Neural networks are trained by optimizing multi-dimensional sets of fitting parameters on non-convex loss landscapes. Low-loss regions of the landscapes correspond to the parameter sets that perform well on the training data. A key issue in machine learning is the performance of trained neural netwo…

• Defensive Generation

  Gabriele Farina, Juan Carlos Perdomo · 26 février 2026

  We study the problem of efficiently producing, in an online fashion, generative models of scalar, multiclass, and vector-valued outcomes that cannot be falsified on the basis of the observed data and a pre-specified collection of computational tests. Our contributions are twofold. First, we expand o…

• On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective

  Jinshu Huang, Mingfei Sun, Chunlin Wu · 25 février 2026

  Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has recently emerged as a powerful framework, offering new mathematical insights into the structure and learning behavior of DN…

• Improving the Convergence of Private Shuffled Gradient Methods with Public Data

  Shuli Jiang, Pranay Sharma, Zhiwei Steven Wu, Gauri Joshi · 25 février 2026

  We consider the problem of differentially private (DP) convex empirical risk minimization (ERM). While the standard DP-SGD algorithm is theoretically well-established, practical implementations often rely on shuffled gradient methods that traverse the training data sequentially rather than sampling …

• Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^{s,\lambda}$ Targets

  Yanming Lai, Defeng Sun · 25 février 2026

  The tremendous success of Transformer models in fields such as large language models and computer vision necessitates a rigorous theoretical investigation. To the best of our knowledge, this paper is the first work proving that standard Transformers can approximate H\"older functions $ C^{s,\lambda}…

• On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes

  Boao Kong, Hengrui Zhang, Kun Yuan · 25 février 2026

  We study stochastic gradient descent (SGD) for composite optimization problems with $N$ sequential operators subject to perturbations in both the forward and backward passes. Unlike classical analyses that treat gradient noise as additive and localized, perturbations to intermediate outputs and grad…

• Does Order Matter : Connecting The Law of Robustness to Robust Generalization

  Himadri Mandal, Vishnu Varadarajan, Jaee Ponde, Aritra Das, Mihir More, Debayan Gupta · 25 février 2026

  Bubeck and Sellke (2021) pose as an open problem the connection between the law of robustness and robust generalization. The law of robustness states that overparameterization is necessary for models to interpolate robustly; in particular, robust interpolation requires the learned function to be Lip…

• A Long-Short Flow-Map Perspective for Drifting Models

  Zhiqi Li, Bo Zhu · 25 février 2026

  This paper provides a reinterpretation of the Drifting Model~\cite{deng2026generative} through a semigroup-consistent long-short flow-map factorization. We show that a global transport process can be decomposed into a long-horizon flow map followed by a short-time terminal flow map admitting a close…

• Complexity of Classical Acceleration for $\ell_1$-Regularized PageRank

  Kimon Fountoulakis, David Mart\'inez-Rubio · 25 février 2026

  We study the degree-weighted work required to compute $\ell_1$-regularized PageRank using the standard one-gradient-per-iteration accelerated proximal-gradient method (FISTA). For non-accelerated local methods, the best known worst-case work scales as $\widetilde{O} ((\alpha\rho)^{-1})$, where $\alp…

• In Defense of Cosine Similarity: Normalization Eliminates the Gauge Freedom

  Taha Bouhsine · 24 février 2026

  Steck, Ekanadham, and Kallus [arXiv:2403.05440] demonstrate that cosine similarity of learned embeddings from matrix factorization models can be rendered arbitrary by a diagonal ``gauge'' matrix $D$. Their result is correct and important for practitioners who compute cosine similarity on embeddings …

• The Hidden Power of Normalization Layers in Neural Networks: Exponential Capacity Control

  Khoat Than · 24 février 2026

  Normalization layers are critical components of modern AI systems, such as ChatGPT, Gemini, DeepSeek, etc. Empirically, they are known to stabilize training dynamics and improve generalization ability. However, the underlying theoretical mechanism by which normalization layers contribute to both opt…

• Robustness of Deep ReLU Networks to Misclassification of High-Dimensional Data

  V\v{e}ra K\r{u}rkov\'a · 24 février 2026

  We present a theoretical study of the robustness of parameterized networks to random input perturbations. Specifically, we analyze local robustness at a given network input by quantifying the probability that a small additive random perturbation of the input leads to misclassification. For deep netw…

• Generalized Random Direction Newton Algorithms for Stochastic Optimization

  Soumen Pachal, Prashanth L. A., Shalabh Bhatnagar, Avinash Achar · 24 février 2026

  We present a family of generalized Hessian estimators of the objective using random direction stochastic approximation (RDSA) by utilizing only noisy function measurements. The form of each estimator and the order of the bias depend on the number of function measurements. In particular, we demonstra…

• Optimal Local Convergence Rates of Stochastic First-Order Methods under Local $\alpha$-PL

  Saeed Masiha, Saber Salehkaleybar, Niao He, Negar Kiyavash, Patrick Thiran · 24 février 2026

  We study the local convergence rate of stochastic first-order methods under a local $\alpha$-Polyak-Lojasiewicz ($\alpha$-PL) condition in a neighborhood of a target connected component $\mathcal{M}$ of the local minimizer set. The parameter $\alpha \in [1,2]$ is the exponent of the gradient norm in…

• Virtual Parameter Sharpening: Dynamic Low-Rank Perturbations for Inference-Time Reasoning Enhancement

  Saba Kublashvili · 24 février 2026

  I introduce Virtual Parameter Sharpening (VPS), an inference-time technique that augments frozen transformer linear layers with dynamic, activation-conditioned low-rank perturbations. Unlike parameter-efficient fine-tuning methods such as LoRA, which learn static low-rank adapters, VPS constructs it…

• Implicit Bias and Convergence of Matrix Stochastic Mirror Descent

  Danil Akhtiamov, Reza Ghane, Babak Hassibi · 24 février 2026

  We investigate Stochastic Mirror Descent (SMD) with matrix parameters and vector-valued predictions, a framework relevant to multi-class classification and matrix completion problems. Focusing on the overparameterized regime, where the total number of parameters exceeds the number of training sample…