Stochastic Gradient Optimization Techniques

Neueste Paper

• Contraction-Gauge Preconditioning for Quantized Matrix Multiplication

  Piyush Sao, Narasinga Miniskar, Pedro Valero-Lara, Keita Teranishi, Sudip Seal · 22. Juli 2026

  We study low-precision computation of C=AB with both factors quantized. We derive an exact finite-dimensional identity for the expected squared product error under independent, zero-mean entrywise errors with known variance fields; it holds exactly for non-overloading subtractive dither and for inde…

• Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization

  Anuragine S A, Prem Jagadeesan · 22. Juli 2026

  The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions abou…

• Federated Lightweight Fine-Tuning

  Radhakrishna Achanta, Will Reed · 22. Juli 2026

  Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor. We take a different lever. Mapping networks generate a network's weights from a small trainable lat…

• Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

  Nuemaan Malik · 22. Juli 2026

  Optimizer state is the largest single line item in the memory budget of mixture-of-experts (MoE) training: on a 6.78B-parameter MoE language model, AdamW keeps 50.6 GB of first and second moments to update 12.6 GB of bfloat16 weights. We study SkewAdam, an optimizer built on the observation that the…

• 1-Lipschitz Neural Networks on Hadamard Manifolds

  Davide Murari, Marta Ghirardelli, Ben Adcock, Elena Celledoni, Brynjulf Owren, Carola-Bibiane Sch\"onlieb · 22. Juli 2026

  Controlling the Lipschitz constant of a neural network is a standard way to promote robustness and stability. Most existing constraining strategies are designed for Euclidean spaces. In this work, we construct and analyze a class of 1-Lipschitz neural networks on Hadamard manifolds. Our layers are o…

• PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors

  Nicola Aladrah, Fabio Anselmi · 22. Juli 2026

  Overparameterized models often have continuous parameter symmetries, so different parameters define the same predictor. We show that PAC--Bayesian analysis should be performed on the quotient predictor space: pushing a prior and posterior to the quotient preserves the empirical and population Gibbs …

• Effects of width-dependent model hyperparameters and $\ell_2$-regularization on the loss landscape of two-layer ReLU networks

  Haruka Eshima, Makoto Yamada · 21. Juli 2026

  Understanding deep neural networks remains a central challenge in machine learning. In particular, the theoretical properties of even two-layer ReLU networks, especially in the presence of weight decay, remain poorly understood. To this end, we derive a sufficient condition on the hyperparameter set…

• Reducing Per-Sample Harm in Stochastic Optimization

  Apostolos Avranas · 21. Juli 2026

  Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments. While highly effective, aggregating across the batch and incorporating this history can produce parameter updates that increase the loss of individual samples. We…

• Scaling Limits of Constant-Stepsize SGD at Flat Minima

  Jingyi Zhang, Cheng Mao, Debankur Mukherjee · 21. Juli 2026

  For stochastic gradient descent (SGD) with a constant stepsize $\alpha$, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar $\sqrt{\alpha}$ scaling and a Gaussia…

• After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation

  Kwan Soo Shin, In Seok Kang, Munho Lee · 21. Juli 2026

  Expert domains are trees; the Euclidean transformer is not, diluting parent-child structure exponentially at depth. The hyperbolic turn left one question unasked: not how much of a network to curve, but where curvature may touch the gradient. Placement is a law, not a knob: the same geometry on a tr…

• PoLoRA: A Preconditioned Orthogonalized LoRA Optimizer

  Nikhil Ghosh, Tetiana Parshakova, Robert M. Gower · 21. Juli 2026

  Low-rank adaptation (LoRA) makes finetuning large language models cheaper by adding to each weight matrix a trainable low-rank update parameterized as the product of two matrices. These matrices are usually trained with Adam, which treats them as a single flat vector of parameters and ignores both t…

• Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets

  Javier Maass, L\'ena\"ic Chizat · 21. Juli 2026

  Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leav…

• OR Else: A Differentiable Trust Region for Policy Optimization

  Chinmay Rane, Kanishka Tyagi, Michael Manry · 21. Juli 2026

  PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative. We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model po…

• Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness

  R\'ois\'in Luo, James McDermott, Colm O'Riordan · 21. Juli 2026

  Lipschitz continuity is a fundamental property of neural networks that characterizes their sensitivity to input perturbations. It plays a pivotal role in deep learning, governing \textbf{robustness}, \textbf{generalization} and \textbf{optimization dynamics}. Despite its importance, research on Lips…

• De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction

  Peng Zhao · 21. Juli 2026

  Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues. We study clean Gaussian random designs in which the aggregate covariance tail creates a nearly sca…

• First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

  Irina Piontkovskaia, Sergey Nikolenko · 21. Juli 2026

  Task arithmetic, sequential fine-tuning, activation steering, and first-order random search all operate through relatively small perturbations around an already trained checkpoint, and they rely on different local approximations: individual perturbations should be first-order predictable, task updat…

• Optimizing the Preconditioner: A Black-box Online-to-Nonconvex Conversion with Static Regret Minimization Oracles

  Haichen Hu, David Simchi-Levi · 21. Juli 2026

  We study whether stochastic nonconvex optimization can be reduced to ordinary static regret minimization in online convex optimization in a black-box manner. For smooth nonconvex objectives, our reduction maintains a predictable gradient tracker, while a black-box online learner selects a preconditi…

• Regularity-Aware Stochastic MGDA with Adaptive Conflict-Avoidant Update Direction Control

  Chentong Huang, Lisha Chen · 20. Juli 2026

  Multi-objective learning (MOL) aims to optimize multiple objectives simultaneously. The multi-gradient descent algorithm (MGDA) is a workhorse that iteratively updates along a common descent or conflict-avoidant (CA) direction across objectives. In stochastic settings, however, the vanilla stochasti…

• Prediction-Only Distillation in Linear and Logistic Regression

  Hien Dang, Pratik Patil, Alessandro Rinaldo · 20. Juli 2026

  Self-distillation (SD) is typically studied when the student is retrained on the teacher's original training inputs. In many practical deployments, however, the labeled training data are no longer available, and one has access only to the trained predictor and fresh unlabeled covariates. We study SD…

• What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity

  Kumar Kshitij Patel, Rustem Islamov, Sebastian U Stich, Aurelien Lucchi, Eduard Gorbunov, Lingxiao Wang · 17. Juli 2026

  Local SGD, also known as Federated Averaging, is a widely used distributed optimization algorithm. Although Local SGD often outperforms alternatives such as Mini-batch SGD in practice, theory still only partially explains when and why local updates help under realistic data heterogeneity. Recent wor…

• Interleaved Noise Injection Improves Clean, Corrupted, and OOD Performance

  Matt L. Wiemann, Peter Melchior, Andrew K. Saydjari · 17. Juli 2026

  Noise injection is a well-known technique in stochastic optimization. We report its surprising effectiveness with an interleaved (on-off-on-off...) rather than the usual monotonic decay schedule. We present a theoretical analysis of noise injection, which confirms that corruption by impulse noise ap…

• New universal operator approximation theorem for encoder-decoder architectures

  Janek G\"odeke, Pascal Fernsel · 16. Juli 2026

  Motivated by the rapidly growing field of mathematics for operator approximation with neural networks, we present a novel universal operator approximation theorem for broad classes of encoder-decoder architectures and a wide range of input and output spaces. In this study, we focus on the approximat…

• Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

  Katie Everett · 16. Juli 2026

  We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across …

• ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level

  Chethan Reddy G. P · 16. Juli 2026

  We introduce ExTernD (Expanded-rank Ternary Decomposition), a post-training factorization of each LLM weight matrix $A \in \mathbb{R}^{m \times n}$ into $A \approx B \mathrm{diag}(D) C$ with ternary factors $B \in \{-1,0,+1\}^{m \times k}$, $C \in \{-1,0,+1\}^{k \times n}$ and a real scale vector $D…

• DeepLoop: Depth Scaling for Looped Transformers

  Shuzhen Li, Yifan Zhang, Jiacheng Guo, Quanquan Gu, Mengdi Wang · 16. Juli 2026

  Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies it…