Stochastic Gradient Optimization Techniques

Neueste Paper

• Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective

  Joel Wendin, Claudio Altafini · 14. November 2025

  The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size goes to 0) of deep neural networks missing the activation funct…

• Global Convergence of Four-Layer Matrix Factorization under Random Initialization

  Minrui Luo, Weihang Xu, Xiang Gao, Maryam Fazel, Simon Shaolei Du · 14. November 2025

  Gradient descent dynamics on the deep matrix factorization problem is extensively studied as a simplified theoretical model for deep neural networks. Although the convergence theory for two-layer matrix factorization is well-established, no global convergence guarantee for general deep matrix factor…

• On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks

  Arthur Castello Branco de Oliveira, Dhruv Jatkar, Eduardo Sontag · 14. November 2025

  This paper investigates how the compositional structure of neural networks shapes their optimization landscape and training dynamics. We analyze the gradient flow associated with overparameterized optimization problems, which can be interpreted as training a neural network with linear activations. R…

• Global Solutions to Non-Convex Functional Constrained Problems with Hidden Convexity

  Ilyas Fatkhullin, Niao He, Guanghui Lan, Florian Wolf · 14. November 2025

  Constrained non-convex optimization is fundamentally challenging, as global solutions are generally intractable and constraint qualifications may not hold. However, in many applications, including safe policy optimization in control and reinforcement learning, such problems possess hidden convexity,…

• FedPM: Federated Learning Using Second-order Optimization with Preconditioned Mixing of Local Parameters

  Hiro Ishii, Kenta Niwa, Hiroshi Sawada, Akinori Fujino, Noboru Harada, Rio Yokota · 13. November 2025

  We propose Federated Preconditioned Mixing (FedPM), a novel Federated Learning (FL) method that leverages second-order optimization. Prior methods--such as LocalNewton, LTDA, and FedSophia--have incorporated second-order optimization in FL by performing iterative local updates on clients and applyin…

• Factorization-in-Loop: Proximal Fill-in Minimization for Sparse Matrix Reordering

  Ziwei Li, Shuzi Niu, Tao Yuan, Huiyuan Li, Wenjia Wu · 13. November 2025

  Fill-ins are new nonzero elements in the summation of the upper and lower triangular factors generated during LU factorization. For large sparse matrices, they will increase the memory usage and computational time, and be reduced through proper row or column arrangement, namely matrix reordering. Fi…

• Scalable Mixed-Integer Optimization with Neural Constraints via Dual Decomposition

  Shuli Zeng, Sijia Zhang, Feng Wu, Shaojie Tang, Xiang-Yang Li · 13. November 2025

  Embedding deep neural networks (NNs) into mixed-integer programs (MIPs) is attractive for decision making with learned constraints, yet state-of-the-art monolithic linearisations blow up in size and quickly become intractable. In this paper, we introduce a novel dual-decomposition framework that rel…

• Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning

  Hyunseok Seung, Jaewoo Lee, Hyunsuk Ko · 12. November 2025

  We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach ad…

• Zeroth-Order Optimization Finds Flat Minima

  Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil, Sewoong Oh, Michael Muehlebach, Niao He · 12. November 2025

  Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning, and language model fine-tuning. Existing optimization theory focuses on convergence to an arbitrary stationary point, …

• Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $\lambda$-Effectiveness

  Halyun Jeong, Palle E. T. Jorgensen, Hyun-Kyoung Kwon, Myung-Sin Song · 12. November 2025

  We present a variety of projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance. We study the role of the relaxation parameter in generalized Kaczmarz algorithms and establish a priori regret bounds with explicit $\lambda$-depend…

• Multi-Objective Bilevel Learning

  Zhiyao Zhang, Zhuqing Liu, Xin Zhang, Wen-Yen Chen, Jiyan Yang, Jia Liu · 12. November 2025

  As machine learning (ML) applications grow increasingly complex in recent years, modern ML frameworks often need to address multiple potentially conflicting objectives with coupled decision variables across different layers. This creates a compelling need for multi-objective bilevel learning (MOBL).…

• Pruning at Initialization -- A Sketching Perspective

  Noga Bar, Raja Giryes · 12. November 2025

  The lottery ticket hypothesis (LTH) has increased attention to pruning neural networks at initialization. We study this problem in the linear setting. We show that finding a sparse mask at initialization is equivalent to the sketching problem introduced for efficient matrix multiplication. This give…

• Test-time Diverse Reasoning by Riemannian Activation Steering

  Ly Tran Ho Khanh, Dongxuan Zhu, Man-Chung Yue, Viet Anh Nguyen · 12. November 2025

  Best-of-$N$ reasoning improves the accuracy of language models in solving complex tasks by sampling multiple candidate solutions and then selecting the best one based on some criteria. A critical bottleneck for this strategy is the output diversity limit, which occurs when the model generates simila…

• Schedulers for Schedule-free: Theoretically inspired hyperparameters

  Yuen-Man Pun, Matthew Buchholz, Robert M. Gower · 12. November 2025

  The recently proposed schedule-free method has been shown to achieve strong performance when hyperparameter tuning is limited. The current theory for schedule-free only supports a constant learning rate, where-as the implementation used in practice uses a warm-up schedule. We show how to extend the …

• Error Estimate and Convergence Analysis for Data Valuation

  Zhangyong Liang, Huanhuan Gao, Ji Zhang · 11. November 2025

  Data valuation quantifies data importance, but existing methods cannot ensure validity in a single training process. The neural dynamic data valuation (NDDV) method [3] addresses this limitation. Based on NDDV, we are the first to explore error estimation and convergence analysis in data valuation. …

• Optimistic Online-to-Batch Conversions for Accelerated Convergence and Universality

  Yu-Hu Yan, Peng Zhao, Zhi-Hua Zhou · 11. November 2025

  In this work, we study offline convex optimization with smooth objectives, where the classical Nesterov's Accelerated Gradient (NAG) method achieves the optimal accelerated convergence. Extensive research has aimed to understand NAG from various perspectives, and a recent line of work approaches thi…

• Understanding the role of depth in the neural tangent kernel for overparameterized neural networks

  William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi · 11. November 2025

  Overparameterized fully-connected neural networks have been shown to behave like kernel models when trained with gradient descent, under mild conditions on the width, the learning rate, and the parameter initialization. In the limit of infinitely large widths and small learning rate, the kernel that…

• Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

  Steffen Dereich, Thang Do, Arnulf Jentzen, Philippe von Wurstemberger · 11. November 2025

  Beside the standard stochastic gradient descent (SGD) method, the Adam optimizer due to Kingma & Ba (2014) is currently probably the best-known optimization method for the training of deep neural networks in artificial intelligence (AI) systems. Despite the popularity and the success of Adam it rema…

• Bilevel Learning via Inexact Stochastic Gradient Descent

  Mohammad Sadegh Salehi, Subhadip Mukherjee, Lindon Roberts, Matthias J. Ehrhardt · 11. November 2025

  Bilevel optimization is a central tool in machine learning for high-dimensional hyperparameter tuning. Its applications are vast; for instance, in imaging it can be used for learning data-adaptive regularizers and optimizing forward operators in variational regularization. These problems are large i…

• On the Convergence of Continual Federated Learning Using Incrementally Aggregated Gradients

  Satish Kumar Keshri, Nazreen Shah, Ranjitha Prasad · 11. November 2025

  The holy grail of machine learning is to enable Continual Federated Learning (CFL) to enhance the efficiency, privacy, and scalability of AI systems while learning from streaming data. The primary challenge of a CFL system is to overcome global catastrophic forgetting, wherein the accuracy of the gl…

• PyLO: Towards Accessible Learned Optimizers in PyTorch

  Paul Janson, Benjamin Therien, Quentin Anthony, Xiaolong Huang, Abhinav Moudgil, Eugene Belilovsky · 11. November 2025

  Learned optimizers have been an active research topic over the past decade, with increasing progress toward practical, general-purpose optimizers that can serve as drop-in replacements for widely used methods like Adam. However, recent advances -- such as VeLO, which was meta-trained for 4000 TPU-mo…

• Minimum Width of Deep Narrow Networks for Universal Approximation

  Xiao-Song Yang, Qi Zhou, Xuan Zhou · 11. November 2025

  Determining the minimum width of fully connected neural networks has become a fundamental problem in recent theoretical studies of deep neural networks. In this paper, we study the lower bounds and upper bounds of the minimum width required for fully connected neural networks in order to have univer…

• ANO : Faster is Better in Noisy Landscape

  Adrien Kegreisz · 11. November 2025

  Stochastic optimizers are central to deep learning, yet widely used methods such as Adam and Adan can degrade in non-stationary or noisy environments, partly due to their reliance on momentum-based magnitude estimates. We introduce Ano, a novel optimizer that decouples direction and magnitude: momen…

• Local properties of neural networks through the lens of layer-wise Hessians

  Maxim Bolshim (ITMO University, Saint Petersburg, Russia), Alexander Kugaevskikh (ITMO University, Saint Petersburg, Russia) · 11. November 2025

  We introduce a methodology for analyzing neural networks through the lens of layer-wise Hessian matrices. The local Hessian of each functional block (layer) is defined as the matrix of second derivatives of a scalar function with respect to the parameters of that layer. This concept provides a forma…

• Revisiting Stochastic Approximation and Stochastic Gradient Descent

  Rajeeva Laxman Karandikar, Bhamidi Visweswara Rao, Mathukumalli Vidyasagar · 11. November 2025

  In this paper, we introduce a new approach to proving the convergence of the Stochastic Approximation (SA) and the Stochastic Gradient Descent (SGD) algorithms. The new approach is based on a concept called GSLLN (Generalized Strong Law of Large Numbers), which extends the traditional SLLN. Using th…