Stochastic Gradient Optimization Techniques

Últimos artículos

• Adaptive Lipschitz-Free Conditional Gradient Methods for Stochastic Composite Nonconvex Optimization

  Ganzhao Yuan · 9 de marzo de 2026

  We propose ALFCG (Adaptive Lipschitz-Free Conditional Gradient), the first \textit{adaptive} projection-free framework for stochastic composite nonconvex minimization that \textit{requires neither global smoothness constants nor line search}. Unlike prior conditional gradient methods that use openlo…

• First-Order Softmax Weighted Switching Gradient Method for Distributed Stochastic Minimax Optimization with Stochastic Constraints

  Zhankun Luo, Antesh Upadhyay, Sang Bin Moon, Abolfazl Hashemi · 9 de marzo de 2026

  This paper addresses the distributed stochastic minimax optimization problem subject to stochastic constraints. We propose a novel first-order Softmax-Weighted Switching Gradient method tailored for federated learning. Under full client participation, our algorithm achieves the standard $\mathcal{O}…

• Random Dot Product Graphs as Dynamical Systems: Limitations and Opportunities

  Giulio Valentino Dalla Riva · 9 de marzo de 2026

  Can we learn the differential equations governing the evolution of a temporal network? We investigate this within Random Dot Product Graphs (RDPGs), where each network snapshot is generated from latent positions evolving under unknown dynamics. We identify three fundamental obstructions: gauge freed…

• Improved Scaling Laws via Weak-to-Strong Generalization in Random Feature Ridge Regression

  Diyuan Wu, Lehan Chen, Theodor Misiakiewicz, Marco Mondelli · 9 de marzo de 2026

  It is increasingly common in machine learning to use learned models to label data and then employ such data to train more capable models. The phenomenon of weak-to-strong generalization exemplifies the advantage of this two-stage procedure: a strong student is trained on imperfect labels obtained fr…

• Dynamic Momentum Recalibration in Online Gradient Learning

  Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, Dazhou Li · 9 de marzo de 2026

  Stochastic Gradient Descent (SGD) and its momentum variants form the backbone of deep learning optimization, yet the underlying dynamics of their gradient behavior remain insufficiently understood. In this work, we reinterpret gradient updates through the lens of signal processing and reveal that fi…

• Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence

  Hui Yang, Tao Ren, Jinyang Jiang, Wan Tian, Yijie Peng · 9 de marzo de 2026

  Memory-efficient optimization methods have recently gained increasing attention for scaling full-parameter training of large language models under the GPU-memory bottleneck. Existing approaches either lack clear convergence guarantees, or only achieve the standard ${\mathcal{O}}(\epsilon^{-4})$ iter…

• Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation

  Yize Wu, Ke Gao, Ling Li, Yanjun Wu · 6 de marzo de 2026

  Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. It updates the weight matrix as $W=W_0+sBA$, where $W_0$ is the original frozen weight, $s$ is a scaling factor and $A$,$B$ are trainable low-rank matrices. Despite its robust empirical e…

• K-Means as a Radial Basis function Network: a Variational and Gradient-based Equivalence

  Felipe de Jesus Felix Arredondo, Alejandro Ucan-Puc, Carlos Astengo Noguez · 6 de marzo de 2026

  This work establishes a rigorous variational and gradient-based equivalence between the classical K-Means algorithm and differentiable Radial Basis Function (RBF) neural networks with smooth responsibilities. By reparameterizing the K-Means objective and embedding its distortion functional into a sm…

• Non-Euclidean Gradient Descent Operates at the Edge of Stability

  Rustem Islamov, Michael Crawshaw, Jeremy Cohen, Robert Gower · 6 de marzo de 2026

  The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian converges to $2/\eta$ during training with gradient descent (GD) with a step-size $\eta$. Despite (apparently) violating classical smoothness assumptions, EoS has been widely observed in deep learning…

• Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness

  Baekrok Shin, Chulhee Yun · 6 de marzo de 2026

  We study matrix completion via deep matrix factorization (a.k.a. deep linear neural networks) as a simplified testbed to examine how network depth influences training dynamics. Despite the simplicity and importance of the problem, prior theory largely focuses on shallow (depth-2) models and does not…

• Curse of Dimensionality in Neural Network Optimization

  Sanghoon Na, Haizhao Yang · 6 de marzo de 2026

  This paper demonstrates that when a shallow neural network with a Lipschitz continuous activation function is trained using either empirical or population risk to approximate a target function that is $r$ times continuously differentiable on $[0,1]^d$, the population risk may not decay at a rate fas…

• The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization

  Tongtong Liang, Esha Singh, Rahul Parhi, Alexander Cloninger, Yu-Xiang Wang · 6 de marzo de 2026

  We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent. Prior work has established that for fully connected networks, the strength of this regularization is governed solely by the global input geometry; consequen…

• How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?

  Kuo-Wei Lai, Guanghui Wang, Molei Tao, Vidya Muthukumar · 6 de marzo de 2026

  Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we char…

• Generalization Below the Edge of Stability: The Role of Data Geometry

  Tongtong Liang, Alexander Cloninger, Rahul Parhi, Yu-Xiang Wang · 6 de marzo de 2026

  Understanding generalization in overparameterized neural networks hinges on the interplay between the data geometry, neural architecture, and training dynamics. In this paper, we theoretically explore how data geometry controls this implicit bias. This paper presents theoretical results for overpara…

• Cheap Thrills: Effective Amortized Optimization Using Inexpensive Labels

  Khai Nguyen, Petros Ellinas, Anvita Bhagavathula, Priya Donti · 6 de marzo de 2026

  To scale the solution of optimization and simulation problems, prior work has explored machine-learning surrogates that inexpensively map problem parameters to corresponding solutions. Commonly used approaches, including supervised and self-supervised learning with either soft or hard feasibility en…

• Even Faster Kernel Matrix Linear Algebra via Density Estimation

  Rikhav Shah, Sandeep Silwal, Haike Xu · 5 de marzo de 2026

  This paper studies the use of kernel density estimation (KDE) for linear algebraic tasks involving the kernel matrix of a collection of $n$ data points in $\mathbb R^d$. In particular, we improve upon existing algorithms for computing the following up to $(1+\varepsilon)$ relative error: matrix-vect…

• Transport Clustering: Solving Low-Rank Optimal Transport via Clustering

  Henri Schmidt, Peter Halmos, Ben Raphael · 5 de marzo de 2026

  Optimal transport (OT) finds a least cost transport plan between two probability distributions using a cost matrix defined on pairs of points. Unlike standard OT, which infers unstructured pointwise mappings, low-rank optimal transport explicitly constrains the rank of the transport plan to infer la…

• A Stein Identity for q-Gaussians with Bounded Support

  Sophia Sklaviadis, Thomas Moellenhoff, Andre F. T. Martins, Mario A. T. Figueiredo, Mohammad Emtiyaz Khan · 5 de marzo de 2026

  Stein's identity is a fundamental tool in machine learning with applications in generative models, stochastic optimization, and other problems involving gradients of expectations under Gaussian distributions. Less attention has been paid to problems with non-Gaussian expectations. Here, we consider …

• Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents

  Xiaotong Liu, Yunwen Lei, Xiangyu Chang, Shao-Bo Lin · 5 de marzo de 2026

  This paper proposes a novel parameter selection strategy for kernel-based gradient descent (KGD) algorithms, integrating bias-variance analysis with the splitting method. We introduce the concept of empirical effective dimension to quantify iteration increments in KGD, deriving an adaptive parameter…

• Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

  Ruinan Jin, Yingbin Liang, Shaofeng Zou · 4 de marzo de 2026

  Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving the empirical performance gap insufficiently explained. In this paper, we uncover a key second-moment normalization in …

• CeRA: Breaking the Linear Ceiling of Low-Rank Adaptation via Manifold Expansion

  Hung-Hsuan Chen · 4 de marzo de 2026

  Low-Rank Adaptation (LoRA) dominates parameter-efficient fine-tuning (PEFT). However, it faces a critical ``linear ceiling'' in complex reasoning tasks: simply increasing the rank yields diminishing returns due to intrinsic linear constraints. We introduce CeRA (Capacity-enhanced Rank Adaptation), a…

• Combinatorial Sparse PCA Beyond the Spiked Identity Model

  Syamantak Kumar, Purnamrita Sarkar, Kevin Tian, Peiyuan Zhang · 4 de marzo de 2026

  Sparse PCA is one of the most well-studied problems in high-dimensional statistics. In this problem, we are given samples from a distribution with covariance $\Sigma$, whose top eigenvector $v \in R^d$ is $s$-sparse. Existing sparse PCA algorithms can be broadly categorized into (1) combinatorial al…

• Scalable Uncertainty Quantification for Black-Box Density-Based Clustering

  Nicola Bariletto, Stephen G. Walker · 4 de marzo de 2026

  We introduce a novel framework for uncertainty quantification in clustering. By combining the martingale posterior paradigm with density-based clustering, uncertainty in the estimated density is naturally propagated to the clustering structure. The approach scales effectively to high-dimensional and…

• HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization

  Feihu Huang, Guanyi Zhang, Songcan Chen · 4 de marzo de 2026

  Adam and AdamW are a class of default optimizers for training deep learning models in machine learning. These adaptive algorithms converge faster but generalize worse compared to SGD. In fact, their proved generalization error $O(\frac{1}{\sqrt{N}})$ also is larger than $O(\frac{1}{N})$ of SGD, wher…

• A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Properties

  Junwen Qiu, Li Jiang, Andre Milzarek · 4 de marzo de 2026

  The proximal stochastic gradient method (PSGD) is one of the state-of-the-art approaches for stochastic composite-type problems. In contrast to its deterministic counterpart, PSGD has been found to have difficulties with the correct identification of underlying substructures (such as supports, low r…