Stochastic Gradient Optimization Techniques

Neueste Paper

• Near-optimal and Efficient First-Order Algorithm for Multi-Task Learning with Shared Linear Representation

  Shihong Ding, Fangyu Du, Cong Fang · 4. Mai 2026

  Multi-task learning (MTL) has emerged as a pivotal paradigm in machine learning by leveraging shared structures across multiple related tasks. Despite its empirical success, the development of likelihood-based efficiently solvable algorithms--even for shared linear representations--remains largely u…

• A unified perspective on fine-tuning and sampling with diffusion and flow models

  Carles Domingo-Enrich, Yuanqi Du, Michael S. Albergo · 4. Mai 2026

  We study the problem of training diffusion and flow generative models to sample from target distributions defined by an exponential tilting of a base density; a formulation that subsumes both sampling from unnormalized densities and reward fine-tuning of pre-trained models. This problem can be appro…

• Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden States

  Eli Chien, Wei-Ning Chen, Pan Li · 4. Mai 2026

  Zeroth-order optimization has emerged as a promising approach for fine-tuning large language models under differential privacy (DP) and memory constraints. While privacy amplification by iteration (PABI) provides convergent DP bounds for first-order methods, establishing similar guarantees for zerot…

• Information-Theoretic Generalization Bounds for Stochastic Gradient Descent with Predictable Virtual Noise

  Mohammad Partohaghighi · 4. Mai 2026

  Information-theoretic generalization bounds analyze stochastic optimization by relating expected generalization error to the mutual information between learned parameters and training data. Virtual perturbation analyses of SGD add auxiliary Gaussian noise only in the proof, making mutual information…

• Budget Constraints as Riemannian Manifolds

  Michael Helcig, Dan Alistarh · 4. Mai 2026

  Assigning one of K options to each of N groups under a total cost budget is a recurring problem in machine learning, appearing in mixed-precision quantization, non-uniform pruning, and expert selection. The objective (model loss) depends jointly on all assignments and does not decompose across group…

• High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking

  Aleksandar Armacki, Haoyuan Cai, Ali H. Sayed · 4. Mai 2026

  We study high-probability (HP) convergence guarantees in decentralized stochastic optimization, where multiple agents collaborate to jointly train a model over a network. Existing HP results in decentralized settings almost exclusively focus on the Decentralized Stochastic Gradient Descent ($\mathtt…

• Gradient Regularized Newton Boosting Trees with Global Convergence

  Nikita Zozoulenko, Daniel Falkowski, Thomas Cass, Lukas Gonon · 4. Mai 2026

  Gradient Boosting Decision Trees (GBDTs) dominate tabular machine learning, with modern implementations like XGBoost, LightGBM, and CatBoost being based on Newton boosting: a second-order descent step in the space of decision trees. Despite its empirical success, the global convergence of Newton boo…

• Randomized Subspace Nesterov Accelerated Gradient

  Gaku Omiya, Pierre-Louis Poirion, Akiko Takeda · 4. Mai 2026

  Randomized-subspace methods reduce the cost of first-order optimization by using only low-dimensional projected-gradient information, a feature that is attractive in forward-mode automatic differentiation and communication-limited settings. While Nesterov acceleration is well understood for full-gra…

• Adaptive Norm-Based Regularization for Neural Networks

  Muhammad Qasim, Farrukh Javed · 4. Mai 2026

  In this paper, we study norm-based regularization methods for neural networks. We compare existing penalization approaches and introduce two regularization strategies that extend classical ridge- and lasso-type penalties to neural network models. The first strategy modifies weight decay by incorpora…

• Prediction-powered Inference by Mixture of Experts

  Yanwu Gu, Linglong Kong, Dong Xia · 1. Mai 2026

  The rapidly expanding artificial intelligence (AI) industry has produced diverse yet powerful prediction tools, each with its own network architecture, training strategy, data-processing pipeline, and domain-specific strengths. These tools create new opportunities for semi-supervised inference, in w…

• Exponential families from a single KL identity

  Marc Dymetman · 1. Mai 2026

  Exponential families encompass the distributions central to modern machine learning -- softmax, Gaussians, and Boltzmann distributions -- and underlie the theory of variational inference, entropy-regularized reinforcement learning, and RLHF. We isolate a simple identity for exponential families that…

• State-Dependent Lyapunov Method for Rank-1 Matrix Factorization

  Jaehong Moon · 1. Mai 2026

  We study gradient descent for rank-1 matrix factorization through a certificate-based viewpoint. The central object is a parameterized quadratic certificate $I(\delta;\,\cdot)$ whose level sets shrink along the dynamics, thereby inducing a monotone state parameter $\delta_t$. In the certified regime…

• Post-Optimization Adaptive Rank Allocation for LoRA

  Vishnuprasadh Kumaravelu, Sunil Gupta, P. K. Srijith · 1. Mai 2026

  Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform ran…

• Layer-wise Lipschitz-Product Control for Deep Kolmogorov--Arnold Network Representations of Compositionally Structured Functions

  Aleksander Tankman · 30. April 2026

  We prove that any continuous function f from [0,1]^n to R representable by a finite computation tree with N internal nodes and compositional sparsity s = O(1) admits a deep Kolmogorov-Arnold Network (KAN) representation. Each internal node is realised by a primitive KAN block with controlled block d…

• NeuroPlastic: A Plasticity-Modulated Optimizer for Biologically Inspired Learning Dynamics

  Douglas Jiang, Yuechen Wang, Jiayi Wang, Jiaying Geng, Qinglong Wang, Feng Tian · 30. April 2026

  Optimization algorithms are fundamental to modern deep learning, yet most widely used methods rely on update rules based primarily on local gradient statistics. We introduce NeuroPlastic, a plasticity-modulated optimizer that augments gradient-based updates with an adaptive multi-signal modulation m…

• Learning Over-Relaxation Policies for ADMM with Convergence Guarantees

  Junan Lin, Paul J. Goulart, Luca Furieri · 30. April 2026

  The Alternating Direction Method of Multipliers (ADMM) is a widely used method for structured convex optimization, and its practical performance depends strongly on the choice of penalty and relaxation parameters. Motivated by settings such as Model Predictive Control (MPC), where one repeatedly sol…

• A projection-based framework for gradient-free and parallel learning

  Andreas Bergmeister, Manish Krishan Lal, Stefanie Jegelka, Suvrit Sra · 30. April 2026

  We present a feasibility-seeking approach to neural network training. This mathematical optimization framework is distinct from conventional gradient-based loss minimization and uses projection operators and iterative projection algorithms. We reformulate training as a large-scale feasibility proble…

• The Role of Symmetry in Optimizing Overparameterized Networks

  Kusha Sareen, Mohammad Pedramfar, S\'ekou-Oumar Kaba, Mehran Shakerinava, Siamak Ravanbakhsh · 29. April 2026

  Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. We analyze weight-space symmetries in neural networks and show that overparameterization introduces additional symmetries that benefit optimization in…

• Transformer Approximations from ReLUs

  Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee, Han Liu · 29. April 2026

  We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yields target-specific, economic resource bounds beyond universal approximation statements. We showcase the recipe on multip…

• Elite-Driven Support Vector Machines for Classification

  Mohammad Jafari Jozani, Bahram Moeinianfar · 29. April 2026

  Support vector machines (SVMs) are a standard tool for binary classification, but their classical formulations are purely data-driven and offer no direct way to encode trusted benchmark models or structured preferences on selected subsets of the data. We propose Elite-Driven Support Vector Machines …

• Gradient-Direction Sensitivity Reveals Linear-Centroid Coupling Hidden by Optimizer Trajectories

  Yongzhong Xu · 29. April 2026

  We show that replacing the rolling SVD of AdamW updates with a rolling SVD of loss gradients changes the diagnostic by 1-2 orders of magnitude. Performing SVD on the loss gradient instead of the AdamW update increases the measured perturbative coupling between SED directions and Linear Centroid Hypo…

• Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticity

  Hugo Ninou, Jonathan Kadmon, N. Alex Cayco-Gajic · 29. April 2026

  Gradient-based algorithms are a cornerstone of artificial neural network training, yet it remains unclear whether biological neural networks use similar gradient-based strategies during learning. Experiments often discover a diversity of synaptic plasticity rules, but whether these amount to an appr…

• Sharp Risk Bounds for Early-Stopping in Gaussian Linear Regression

  Tobias Wegel, Gil Kur, Patrick Rebeschini · 29. April 2026

  We study early-stopped mirror descent (ESMD) for high-dimensional Gaussian linear regression over arbitrary convex bodies and design matrices, where the task is to minimize the in-sample mean squared error. Our main result shows that some of the sharpest risk bounds for the least squares estimator (…

• Nearly Optimal Subdata Selection

  Min Yang, Wei Zheng, John Stufken, Ming-Chung Chang, Ting Tian, Xueqin Wang · 28. April 2026

  When, in terms of the number of data points, the size of a dataset exceeds available computing resources, or when labeling is expensive, an attractive solution consists of selecting only some of the data points (subdata) for further consideration. A central question for selecting subdata of size $n$…

• Inference of Online Newton Methods with Nesterov's Accelerated Sketching

  Haoxuan Wang, Xinchen Du, Sen Na · 28. April 2026

  Reliable decision-making with streaming data requires principled uncertainty quantification of online methods. While first-order methods enable efficient iterate updates, their inference procedures still require updating proper (covariance) matrices, incurring $O(d^2)$ time and memory complexity, an…