Stochastic Gradient Optimization Techniques

Últimos artículos

• Sequential Group Composition: A Window into the Mechanics of Deep Learning

  Giovanni Luca Marchetti, Daniel Kunin, Adele Myers, Francisco Acosta, Nina Miolane · 4 de febrero de 2026

  How do neural networks trained over sequences acquire the ability to perform structured operations, such as arithmetic, geometric, and algorithmic computation? To gain insight into this question, we introduce the sequential group composition task. In this task, networks receive a sequence of element…

• When pre-training hurts LoRA fine-tuning: a dynamical analysis via single-index models

  Gibbs Nwemadji, Bruno Loureiro, Jean Barbier · 4 de febrero de 2026

  Pre-training on a source task is usually expected to facilitate fine-tuning on similar downstream problems. In this work, we mathematically show that this naive intuition is not always true: excessive pre-training can computationally slow down fine-tuning optimization. We study this phenomenon for l…

• Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent

  Hiroki Naganuma, Shagun Gupta, Youssef Briki, Ioannis Mitliagkas, Irina Rish, Parameswaran Raman, Hao-Jun Michael Shi · 4 de febrero de 2026

  To maximize hardware utilization, modern machine learning systems typically employ large constant or manually tuned batch size schedules, relying on heuristics that are brittle and costly to tune. Existing adaptive strategies based on gradient noise scale (GNS) offer a principled alternative. Howeve…

• Soft-Radial Projection for Constrained End-to-End Learning

  Philipp J. Schneider, Daniel Kuhn · 4 de febrero de 2026

  Integrating hard constraints into deep learning is essential for safety-critical systems. Yet existing constructive layers that project predictions onto constraint boundaries face a fundamental bottleneck: gradient saturation. By collapsing exterior points onto lower-dimensional surfaces, standard o…

• Admissibility of Stein Shrinkage for Batch Normalization in the Presence of Adversarial Attacks

  Sofia Ivolgina, P. Thomas Fletcher, Baba C. Vemuri · 4 de febrero de 2026

  Batch normalization (BN) is a ubiquitous operation in deep neural networks, primarily used to improve stability and regularization during training. BN centers and scales feature maps using sample means and variances, which are naturally suited for Stein's shrinkage estimation. Applying such shrinkag…

• Controlled disagreement improves generalization in decentralized training

  Zesen Wang, Mikael Johansson · 4 de febrero de 2026

  Decentralized training is often regarded as inferior to centralized training because the consensus errors between workers are thought to undermine convergence and generalization, even with homogeneous data distributions. This work challenges this view by introducing decentralized SGD with Adaptive C…

• A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimization

  Xiyuan Wei, Linli Zhou, Bokun Wang, Chih-Jen Lin, Tianbao Yang · 4 de febrero de 2026

  This paper studies optimization for a family of problems termed $\textbf{compositional entropic risk minimization}$, in which each data's loss is formulated as a Log-Expectation-Exponential (Log-E-Exp) function. The Log-E-Exp formulation serves as an abstraction of the Log-Sum-Exponential (LogSumExp…

• Why Some Models Resist Unlearning: A Linear Stability Perspective

  Wei-Kai Chang, Rajiv Khanna · 4 de febrero de 2026

  Machine unlearning, the ability to erase the effect of specific training samples without retraining from scratch, is critical for privacy, regulation, and efficiency. However, most progress in unlearning has been empirical, with little theoretical understanding of when and why unlearning works. We t…

• FlashSinkhorn: IO-Aware Entropic Optimal Transport

  Felix X. -F. Ye, Xingjie Li, An Yu, Ming-Ching Chang, Linsong Chu, Davis Wertheimer · 4 de febrero de 2026

  Entropic optimal transport (EOT) via Sinkhorn iterations is widely used in modern machine learning, yet GPU solvers remain inefficient at scale. Tensorized implementations suffer quadratic HBM traffic from dense $n\times m$ interactions, while existing online backends avoid storing dense matrices bu…

• NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learning

  Zhen Fang, Miao Yang, Zehang Lin, Zheng Lin, Zihan Fang, Zongyuan Zhang, Tianyang Duan, Dong Huang, Shunzhi Zhu · 4 de febrero de 2026

  The expanding scale of neural networks poses a major challenge for distributed machine learning, particularly under limited communication resources. While split learning (SL) alleviates client computational burden by distributing model layers between clients and server, it incurs substantial communi…

• Sparse Training of Neural Networks based on Multilevel Mirror Descent

  Yannick Lunk, Sebastian J. Scott, Leon Bungert · 4 de febrero de 2026

  We introduce a dynamic sparse training algorithm based on linearized Bregman iterations / mirror descent that exploits the naturally incurred sparsity by alternating between periods of static and dynamic sparsity pattern updates. The key idea is to combine sparsity-inducing Bregman iterations with a…

• Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Training

  Jie Hao, Xiaochuan Gong, Jie Xu, Zhengdao Wang, Mingrui Liu · 4 de febrero de 2026

  Geometry-aware optimization algorithms, such as Muon, have achieved remarkable success in training deep neural networks (DNNs). These methods leverage the underlying geometry of DNNs by selecting appropriate norms for different layers and updating parameters via norm-constrained linear minimization …

• Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More

  Jinwoo Lim, Suhyun Kim, Soo-Mook Moon · 4 de febrero de 2026

  One of the chronic problems of deep-learning models is shortcut learning. In a case where the majority of training data are dominated by a certain feature, neural networks prefer to learn such a feature even if the feature is not generalizable outside the training set. Based on the framework of Neur…

• 3D-Learning: Diffusion-Augmented Distributionally Robust Decision-Focused Learning

  Jiaqi Wen, Lei Fan, Jianyi Yang · 4 de febrero de 2026

  Predict-then-Optimize (PTO) pipelines are widely employed in computing and networked systems, where Machine Learning (ML) models are used to predict critical contextual information for downstream decision-making tasks such as cloud LLM serving, data center demand response, and edge workload scheduli…

• Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression

  Ali Abbasi, Chayne Thrash, Haoran Qin, Shansita Sharma, Sepehr Seifi, Soheil Kolouri · 4 de febrero de 2026

  Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storage and can speed up inference via low-rank factors, yet performance depends on how rank is allocated under a global compr…

• Depth, Not Data: An Analysis of Hessian Spectral Bifurcation

  Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Yaoqing Yang · 3 de febrero de 2026

  The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones,…

• Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization

  Taesun Yeom, Taehyeok Ha, Jaeho Lee · 3 de febrero de 2026

  Feature learning strength (FLS), i.e., the inverse of the effective output scaling of a model, plays a critical role in shaping the optimization dynamics of neural nets. While its impact has been extensively studied under the asymptotic regimes -- both in training time and FLS -- existing theory off…

• Exact Instance Compression for Convex Empirical Risk Minimization via Color Refinement

  Bryan Zhu, Ziang Chen · 3 de febrero de 2026

  Empirical risk minimization (ERM) can be computationally expensive, with standard solvers scaling poorly even in the convex setting. We propose a novel lossless compression framework for convex ERM based on color refinement, extending prior work from linear programs and convex quadratic programs to …

• High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation

  Yusuf Kalyoncuoglu, Ratmir Miftachov · 3 de febrero de 2026

  State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random pr…

• SimpleGPT: Improving GPT via A Simple Normalization Strategy

  Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao · 3 de febrero de 2026

  In this work, we revisit Transformer optimization through the lens of second-order geometry and establish a direct connection between architectural design, activation scale, the Hessian matrix, and the maximum tolerable learning rate. We introduce a simple normalization strategy, termed SimpleNorm, …

• Hessian Spectral Analysis at Foundation Model Scale

  Diego Granziol, Khurshid Juarev · 3 de febrero de 2026

  Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful spectral analysis of the true Hessian is tractable at frontier scale. Using shard-local finite-difference Hessian vecto…

• Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching

  Tianhao Miao, Zhongyuan Bao, Lejun Zhang · 3 de febrero de 2026

  Training efficiency in large-scale models is typically assessed through memory consumption, training time, and model performance. Current methods often exhibit trade-offs among these metrics, as optimizing one generally degrades at least one of the others. Addressing this trade-off remains a central…

• Stein-Rule Shrinkage for Stochastic Gradient Estimation in High Dimensions

  M. Arashi, M. Amintoosi · 3 de febrero de 2026

  Stochastic gradient methods are central to large-scale learning, yet their analysis typically treats mini-batch gradients as unbiased estimators of the population gradient. In high-dimensional settings, however, classical results from statistical decision theory show that unbiased estimators are gen…

• Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability

  Eric Regis, Sinho Chewi · 3 de febrero de 2026

  How can we understand gradient-based training over non-convex landscapes? The edge of stability phenomenon, introduced in Cohen et al. (2021), indicates that the answer is not so simple: namely, gradient descent (GD) with large step sizes often diverges away from the gradient flow. In this regime, t…

• OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases

  Zixiao Wang, Yifei Shen, Huishuai Zhang · 3 de febrero de 2026

  Many optimizers can be interpreted as steepest-descent methods under norm-induced geometries, and thus inherit corresponding implicit biases. We introduce \nameA{} (\fullname{}), which combines spectral control from orthogonalized update directions with $\ell_\infty$-style coordinate control from si…