Stochastic Gradient Optimization Techniques

Últimos artículos

• Reinforcement learning for adaptive interior point methods in convex quadratic programming

  Jeremy Bertoncini, Alberto De Marchi, Matthias Gerdts, Simon Gottschalk · 18 de mayo de 2026

  Quadratic programming is a workhorse of modern nonlinear optimization, control, and data science. Although regularized methods offer convergence guarantees under minimal assumptions on the problem data, they can exhibit the slow tail-convergence typical of first-order schemes, thus requiring many it…

• Rethinking Neural Network Learning Rates: A Stackelberg Perspective

  Sihan Zeng, Sujay Bhatt, Sumitra Ganesh · 18 de mayo de 2026

  Neural networks are typically trained with a single learning rate across all layers. While recent empirical evidence suggests that assigning layer-specific learning rates can accelerate training, a principled understanding of the conditions and mechanisms under which non-uniform learning rates are b…

• Embedding-perturbed Exploration Preference Optimization for Flow Models

  Sujie Hu, Chubin Chen, Jiashu Zhu, Jiahong Wu, Xiangxiang Chu, Xiu Li · 18 de mayo de 2026

  Recent advancements have established Reinforcement Learning (RL) as a pivotal paradigm for aligning generative models with human intent. However, group-based optimization frameworks (e.g., GRPO) face a critical limitation: the rapid decay of intra-group variance. As the distinctiveness among samples…

• Stochastic Non-Smooth Convex Optimization with Unbounded Gradients

  Dmitry Kovalev · 18 de mayo de 2026

  Much of the existing theory on first-order non-smooth optimization is built on a restrictive assumption that the gradients of the objective function are uniformly bounded. We introduce a much more realistic class of generalized Lipschitz functions, where the gradient norms are bounded by an affine f…

• Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

  Sijia Liu, Yicheng Lang, Soumyadeep Pal, Changsheng Wang, Yancheng Huang, Chongyu Fan, James Diffenderfer, Bhavya Kailkhura, Yihua Zhang · 18 de mayo de 2026

  Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unsc…

• Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise

  Antesh Upadhyay, Arda Fazla, Abolfazl Hashemi · 18 de mayo de 2026

  We study nonconvex stochastic optimization under the Blum-Gladyshev ($\mathsf{BG}$-0) noise model, where the stochastic gradient variance grows quadratically with the distance from the initialization. We consider this problem under both standard smoothness and the symmetric generalized-smoothness fr…

• Unified High-Probability Analysis of Stochastic Variance-Reduced Estimation

  Zhankun Luo, Antesh Upadhyay, M. Berk Sahin, Sang Bin Moon, Anuran Makur, Abolfazl Hashemi · 18 de mayo de 2026

  Stochastic estimators are fundamental to large-scale optimization, where population quantities must be inferred from noisy oracle observations. Although influential methods such as momentum, SPIDER, STORM, and PAGE have been highly successful, their analyses are largely estimator-specific and expect…

• Bounded-Rationality, Hedging, and Generalization

  Pedro A. Ortega · 18 de mayo de 2026

  A learner does not only fit data; it also determines how strongly the training sample may shape its output and how much distortion it can hedge. We study this relation as a bounded-rational decision problem whose primitive object is the induced channel from samples to outputs. The learner's response…

• From Layers to Networks: Comparing Neural Representations via Diffusion Geometry

  Atharva Khandait, Jan E. Gerken · 18 de mayo de 2026

  Diffusion geometry is a manifold learning framework that uses random walks defined by Markov transition matrices to characterize the geometry of a dataset at multiple scales. We use diffusion geometry for neural representations, incorporating tools from multi-view learning into this field for the fi…

• When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing

  Libo Sun, Po-wei Harn, Peixiong He, Xiao Qin · 18 de mayo de 2026

  Mixture-of-Experts (MoE) networks promise favorable accuracy-compute trade-offs, yet practical vision deployments are hindered by expert collapse and limited end-to-end efficiency gains. We study when sparse top-$k$ routing with hard capacity constraints helps in vision classification, evaluated und…

• Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfe

  El Mahdi Chayti · 18 de mayo de 2026

  Stochastic compositional optimization minimizes objectives of the form $\min_{\bm{x} \in \mathcal{X}} F(\bm{f}(\bm{x}), \bm{x})$, where $\bm{f}$ is accessible only through noisy stochastic queries. Existing methods for this problem assume that the outer function $F$ is continuously differentiable, w…

• On Kernel Eigen-alignments of KRR: Reconstruction and Generalization

  Yang Liu, Ernest Fokoue, Richard Lange, Daniel Krutz · 18 de mayo de 2026

  This paper investigates the critical role of eigenalignments between the kernel matrix and learning targets in achieving robust generalization in learning problems. We establish a direct connection between generalization performance in kernel methods and the estimation of eigenvectors and eigenvalue…

• Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration

  Jiahe Chen, Ziye Ma · 18 de mayo de 2026

  Zeroth-order (ZO) optimization has become increasingly popular and important in fine-tuning large language models (LLMs), especially on edge devices due to its ability to adjust the model to local data without the need for memory-intensive back-propagation. Recent works try to reduce ZO variance thr…

• Accelerated Gradient Descent for Faster Convergence with Minimal Overhead

  Manuel Graca, L. Miguel Silveira, Arlindo Oliveira, Frank Liu · 18 de mayo de 2026

  In this paper, we present CT-AGD (Curvature-Tuned Accelerated Gradient Descent), an optimization method for non-convex optimization problems in deep learning training tasks. CT-AGD is a general boosting procedure that accelerates first-order methods by explicitly capturing the local curvature using …

• A Unified View of Score-Based and Drifting Models

  Chieh-Hsin Lai, Bac Nguyen, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, Stefano Ermon, Molei Tao · 18 de mayo de 2026

  Drifting models train one-step generators by optimizing a kernel-induced mean-shift discrepancy between the data and model distributions, with Laplace kernels used by default in practice. At each point, this discrepancy compares the kernel-weighted displacement toward nearby data samples with the co…

• Large Dimensional Kernel Ridge Regression: Extending to Product Kernels

  Yang Zhou, Yicheng Li, Yuqian Cheng, Qian Lin · 15 de mayo de 2026

  Recent studies have reported $\textit{saturation effects}$ and $\textit{multiple descent behavior}$ in large dimensional kernel ridge regression (KRR). However, these findings are predominantly derived under restrictive settings, such as inner product kernels on sphere or strong eigenfunction assump…

• A Survey on Data-Dependent Worst-Case Generalization Bounds

  Hubert Leroux, Jean Marcus, Julien Roger · 15 de mayo de 2026

  Deep neural networks generalize well despite being heavily overparameterized, in apparent contradiction with classical learning theory based on uniform convergence over fixed hypothesis spaces. Uniform bounds over the entire parameter space are vacuous in this regime, and recent work has shown that …

• Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model

  Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi, Florent Krzakala, Bruno Loureiro · 15 de mayo de 2026

  We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination of latent compositional features whose weights decrease as a…

• Turning Stale Gradients into Stable Gradients: Coherent Coordinate Descent with Implicit Landscape Smoothing for Lightweight Zeroth-Order Optimization

  Chen Liang, Xiatao Sun, Qian Wang, Daniel Rakita · 15 de mayo de 2026

  Zeroth-Order (ZO) optimization is pivotal for scenarios where backpropagation is unavailable, such as memory-constrained on-device learning and black-box optimization. However, existing methods face a stark trade-off: they are either sample-inefficient (e.g., standard finite differences) or suffer f…

• LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning

  Changryeol Choi, Hyewon Park, Yujin Kwon, Gowun Jeong · 15 de mayo de 2026

  Recent tabular learning benchmarks increasingly show a tight performance cluster rather than a clear hierarchy among leading methods, spanning gradient boosted decision trees, attention-based architectures, and implicit ensembles such as TabM. As benchmark gains plateau, a complementary goal is to u…

• Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning

  Shin So, Kyelim Lee, Albert No · 15 de mayo de 2026

  Critical-data-size accounts of grokking suggest a natural post-threshold intuition: once training data is sufficient to identify the underlying rule, additional data should accelerate validation convergence. We show that this intuition can fail in a controlled structured-output task. In Needleman--W…

• A Non-Monotone Preconditioned Trust-Region Method for Neural Network Training

  Andrea Angino, Bindi \c{C}apriqi, Shega Likaj, Ken Trotti, Rolf Krause · 15 de mayo de 2026

  Training deep neural networks at scale can benefit from domain decomposition, where the network is split into subdomains trained in parallel and coupled by a global trust-region mechanism. Building on the Additively Preconditioned Trust-Region Strategy (APTS), we propose a non-monotone variant with …

• Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models

  Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy, Maryam Fazel · 15 de mayo de 2026

  We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction. Specifically, we consider the problem of recovering a multi-index polynomial $f^*(x)=h(Ux)$, with $U\in\mathbb{R}^{r\ti…

• Learning with Shallow Neural Networks on Cluster-Structured Features

  Elisabetta Cornacchia, Laurent Massouli\'e · 15 de mayo de 2026

  The success of deep learning in high-dimensional settings is often attributed to the presence of low-dimensional structure in real-world data. While standard theoretical models typically assume that this structure lies in the target function, projecting unstructured inputs onto a low-dimensional sub…

• Distribution Shift in Missing Data Imputation: A Risk-Based Perspective and Importance-Weighted Correction under MAR

  Luke Shannon, Song Liu, Katarzyna Reluga · 14 de mayo de 2026

  Missing data imputation, where a model is trained on observed data to estimate unobserved values, is a fundamental problem in machine learning. In this paper, we rigorously formulate imputation model learning as a mean-squared error risk minimisation problem. We show that when the probability of mis…