Stochastic Gradient Optimization Techniques

Derniers papiers

• Backbone-Equated Diffusion OOD via Sparse Internal Snapshots

  Yadang Alexis Rouzoumka, Jean Pinsolle, Eug\'enie Terreaux, Christ\`ele Morisseau, Jean-Philippe Ovarlez, Chengfang Ren · 13 mai 2026

  Fair comparison between diffusion-based OOD detectors is challenging, as conclusions can vary with backbone choice, corruption parameterization, and test-time budget. We address this issue through a Mutualized Backbone-Equated (MBE) protocol that aligns canonical corruption levels and logical test-t…

• Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives

  Konstantinos Oikonomidis, Jan Quan, Kimon Antonakopoulos, Antonio Silveti-Falls, Volkan Cevher, Panagiotis Patrinos · 13 mai 2026

  In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a wide variety of convex and nonconvex constraints and study its…

• Minimax Rates and Spectral Distillation for Tree Ensembles

  Binh Duc Vu, David S. Watson · 13 mai 2026

  Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive mi…

• Extending Kernel Trick to Influence Functions

  Zhenhuan Sun, Shahrokh Valaee · 13 mai 2026

  In this paper, we present a dual representation of the influence functions, whose computational complexity scales with dataset size rather than model size. Both analytically and experimentally, we show that this representation can be an efficient alternative to the original influence functions for e…

• SRG: Score-based Relaxation-guided Generation for Mixed Integer Linear Programming

  Ruobing Wang, Xin Li, Yujie Fang, Mingzhong Wang · 13 mai 2026

  We propose Score-based Relaxation-guided Generation (SRG), a generative framework based on an approximate formulation of relaxation-guided stochastic differential equations (SDEs) for mixed-integer linear programming. SRG employs a Transformer-based score network that incorporates feasibility and op…

• POP: Prior-Fitted First-Order Optimization Policies

  Jan Kobiolka, Christian Frey, Gresa Shala, Arlind Kadra, Erind Bedalli, Josif Grabocka · 13 mai 2026

  Gradient-based optimizers are highly sensitive to design choices in their adaptive learning rate mechanisms. To address this limitation, we introduce POP, a meta-learned Reinforcement Learning (RL) policy that predicts adaptive learning rates for gradient descent, conditioned on the contextual infor…

• DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

  Chenyang Song, Weilin Zhao, Xu Han, Chaojun Xiao, Yingfa Chen, Zhiyuan Liu · 13 mai 2026

  While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-access bottlenecks, which hinder efficient end-side deployment that simultaneously requires high performance, low computati…

• Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

  Arijit Sehanobish, Charles Lovering · 13 mai 2026

  We study the \textit{parameter placement problem}: given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models,…

• Efficient and provably convergent end-to-end training of deep neural networks with linear constraints

  Zonglin Yang, Zhexuan Gu, Yancheng Yuan · 13 mai 2026

  Training a deep neural network with the outputs of selected layers satisfying linear constraints is required in many contemporary data-driven applications. While this can be achieved by incorporating projection layers into the neural network, its end-to-end training remains challenging due to the la…

• Detecting overfitting in Neural Networks during long-horizon grokking using Random Matrix Theory

  Hari K. Prakash, Charles H Martin · 13 mai 2026

  Training Neural Networks (NNs) without overfitting is difficult; detecting that overfitting is difficult as well. We present a novel Random Matrix Theory method that detects the onset of overfitting in deep learning models without access to train or test data. For each model layer, we randomize each…

• Delightful Gradients Accelerate Corner Escape

  Jincheng Mei, Ian Osband · 13 mai 2026

  Softmax policy gradient converges at $O(1/t)$, but its transient behavior near sub-optimal corners of the simplex can be exponentially slow. The bottleneck is self-trapping: negative-advantage actions reinforce the corner policy and can initially push the optimal action backward. We study \emph{Deli…

• Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters

  Alexander Yukhimchuk, Mladen Kolar, Martin Tak\'a\v{c}, Sayantan Choudhury · 13 mai 2026

  Gradient clipping is a standard safeguard for training neural networks under noisy, heavy-tailed stochastic gradients; yet, most clipping rules treat all parameters as vectors and ignore the matrix structure of modern architectures. We show empirically that data outliers often amplify only a small n…

• Multi-Variable Conformal Prediction: Optimizing Prediction Sets without Data Splitting

  Laura L\"utzow, Simone Garatti, Marco C. Campi, Lars Lindemann, Matthias Althoff · 13 mai 2026

  Conformal prediction constructs prediction sets with finite-sample coverage guarantees, but its calibration stage is structurally constrained to a scalar score function and a single threshold variable - forcing shapes of prediction sets to be fixed before calibration, typically through data splittin…

• Taking the Road Less Scheduled with Adaptive Polyak Steps

  Dimitris Oikonomou, Matthew Buchholz, Yuen-Man Pun, Robert M. Gower, Nicolas Loizou · 13 mai 2026

  Schedule-Free SGD, proposed in [Defazio et al., 2024], achieves optimal convergence rates without requiring the training horizon in advance, by replacing learning rate schedules with a principled form of iterate averaging. However, the method still requires tuning a base learning rate whose optimal …

• BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization

  Hengrui Zhang, Boao Kong, Engao Zhang, Kun Yuan · 13 mai 2026

  Stochastic bilevel optimization (SBO) has become a standard framework for hyperparameter learning, data reweighting, representation learning, and data-mixture optimization in deep learning. Existing exact single-loop SBO methods and memory-efficient surrogate SBO methods either create severe memory …

• Error whitening: Why Gauss-Newton outperforms Newton

  Maricela Best McKay, Nathan P. Lawrence, Brian Wetton, R. Bhushan Gopaluni · 13 mai 2026

  The Gauss-Newton matrix is widely viewed as a positive semidefinite approximation of the Hessian, yet mounting empirical evidence shows that Gauss-Newton descent outperforms Newton's method. We adopt a function space perspective to analyze this phenomenon. We show that the generalized Gauss-Newton (…

• The tractability landscape of diffusion alignment: regularization, rewards, and computational primitives

  Ankur Moitra, Andrej Risteski, Dhruv Rohatgi · 13 mai 2026

  Inference-time reward alignment asks how to turn a pre-trained diffusion model with base law $p$ into a sampler that favors a reward $r$ while remaining close to $p$. Since there is no canonical distributional distance for this closeness constraint, different choices lead to different "reward-aligne…

• FastUMAP: Scalable Dimensionality Reduction via Bipartite Landmark Sampling

  Hongmin Li · 13 mai 2026

  Exploratory analysis of high-dimensional data rarely stops at a single embedding. In practice, analysts rerun dimensionality reduction after changing preprocessing, subsets, or hyperparameters, and standard nonlinear methods can quickly become the bottleneck. We introduce FastUMAP (Bipartite Manifol…

• LLM4Branch: Large Language Model for Discovering Efficient Branching Policies of Integer Programs

  Zhinan Hou, Xingchen Li, Yankai Zhang, Tianxun Li, Keyou You · 12 mai 2026

  Efficient branching policies are essential for accelerating Mixed Integer Linear Programming (MILP) solvers. Their design has long relied on hand-crafted heuristics, and now machine learning has emerged as a promising paradigm to automate this process. However, existing learning-based methods are of…

• AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

  Ziyun Liu, Fengmiao Bian, Jian-Feng Cai · 12 mai 2026

  Low-Rank Adaptation (LoRA) reparameterizes a weight update as a product of two low-rank factors, but the Jacobian $J_{G}$ of the generator mapping the factors to the weight matrix is rank-deficient, so the factor-space preconditioner $J_{G}^* {F}_t J_{G}$ induced by any ${W}$-space preconditioner ${…

• Scaling Limits of Long-Context Transformers

  Giuseppe Bruno, Shi Chen, Zhengjiang Lin, Yury Polyanskiy, Philippe Rigollet · 12 mai 2026

  We study the long-context limit of softmax self-attention with a fixed query and a random context of $n$ i.i.d. keys on the sphere, viewing the inverse temperature $\beta_n$ as the scaling parameter that decides whether attention degenerates into uniform averaging or collapses onto the single closes…

• cuRegOT: A GPU-Accelerated Solver for Entropic-Regularized Optimal Transport

  Yixuan Qiu · 12 mai 2026

  Optimal transport (OT) has emerged as a fundamental tool in modern machine learning, yet its computational cost remains a significant bottleneck for large-scale applications. While harnessing the massive parallelism of modern GPU hardware is critical for efficiency, the de facto standard Sinkhorn al…

• Feature Repulsion and Spectral Lock-in: An Empirical Study of Two-Layer Network Grokking

  Yongzhong Xu · 12 mai 2026

  Tian (2025) proves a repulsion theorem (Theorem 6) for the matrix $ B = (\widetilde{F}^\top \widetilde{F} + \eta I)^{-1} $ during the interactive feature-learning stage of grokking: similar features have negative off-diagonal entries $ B_{j\ell} $, producing an effective repulsive force that drives …

• Quantile Geometry Regularization for Distributional Reinforcement Learning

  Zhaofan Zhang, Minghao Yang, Rufeng Chen, Sihong Xie, Hui Xiong · 12 mai 2026

  Quantile-based distributional reinforcement learning methods learn return distributions through sampled quantile regression, but their bootstrapped target quantiles may induce distorted or degenerate distribution estimates. We propose Robust Quantile-based Implicit Quantile Networks (RQIQN), a light…

• Select-then-differentiate: Solving Bilevel Optimization with Manifold Lower-level Solution Sets

  Saeed Masiha, Zebang Shen, Negar Kiyavash, Niao He · 12 mai 2026

  We study optimistic bilevel optimization when the lower-level problem has a non-isolated manifold of minimizers. In this setting, the hyper-objective may be non-differentiable because the upper-level criterion must choose among multiple lower-level solutions. Under a local Polyak--{\L}ojasiewicz (P{…