Muon and positron interactions and applications

Derniers papiers

• MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training

  Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Yupeng Su, Liyan Tan, Zheng Zhang · 15 mai 2026

  Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition is that polar iterations, by flattening the singular spectrum to all ones, should also eliminate column- and row-wise no…

• Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

  Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, Trung Le · 14 mai 2026

  Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and…

• MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization

  Yupeng Su, Ruijie Zhang, Ziyue Liu, Yequan Zhao, Zheng Zhang · 13 mai 2026

  The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization. However, Muon's optimizer state is more sensitive to quantization errors: because the orthogonalization discards the m…

• Muon is Not That Special: Random or Inverted Spectra Work Just as Well

  Zakhar Shumaylov, Natha\"el Da Costa, Peter Zaika, B\'alint Mucs\'anyi, Alex Massucco, Yoav Gelberg, Carola-Bibiane Sch\"onlieb, Yarin Gal, Philipp Hennig · 13 mai 2026

  The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonst…

• Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization

  Ruotong Sun, Ermin Wei · 8 mai 2026

  Optimizers that exploit the matrix structure of gradients are central to modern LLM pre-training, with two distinct frontiers: explicit Kronecker-factored preconditioning -- most recently KL-Shampoo, which estimates the preconditioner via KL divergence minimization -- and orthogonalization of the gr…

• On the Convergence Analysis of Muon

  Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, Jiawei Zhang · 15 avril 2026

  The majority of parameters in neural networks are naturally represented as matrices. However, most commonly used optimizers treat these matrix parameters as flattened vectors during optimization, potentially overlooking their inherent structural properties. Recently, an optimizer called Muon has bee…

• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning

  Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Yequan Zhao, Yupeng Su, Zi Yang, Zheng Zhang · 14 avril 2026

  Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, its practical efficiency is limited by the need for multiple Newton--Schulz (NS) iterations per optimizat…

• The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm

  Noah Amsel, David Persson, Christopher Musco, Robert M. Gower · 8 avril 2026

  Computing the polar decomposition and the related matrix sign function has been a well-studied problem in numerical analysis for decades. Recently, it has emerged as an important subroutine within the Muon optimizer for training deep neural networks. However, the requirements of this application dif…

• Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Training

  Abdelrahman Abouzeid (Georgia Institute of Technology) · 3 avril 2026

  In LLM training, normalization layers and optimizers are typically treated as independent design choices. In a 3x2 factorial at 1B parameters and 1000 training steps, we show this assumption can fail: Dynamic Erf (Derf; Chen & Liu, 2025) suffers a large negative interaction with Muon (Jordan, 2024),…

• The Newton-Muon Optimizer

  Zhehang Du, Weijie Su · 3 avril 2026

  The Muon optimizer has received considerable attention for its strong performance in training large language models, yet the design principle behind its matrix-gradient orthogonalization remains largely elusive. In this paper, we introduce a surrogate model that not only sheds new light on the desig…

• MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration

  Da Chang, Qiankun Shi, Lvgang Zhang, Yu Li, Ruijie Zhang, Yao Lu, Yongxiang Liu, Ganzhao Yuan · 31 mars 2026

  Orthogonalized-update optimizers such as Muon improve training of matrix-valued parameters, but existing extensions mostly act either after orthogonalization by rescaling updates or before it with heavier whitening-based preconditioners. We introduce {\method}, a lightweight family of pre-orthogonal…

• HTMuon: Improving Muon via Heavy-Tailed Spectral Correction

  Tianyu Pang, Yujie Fang, Zihang Liu, Shenyang Deng, Lei Hsiung, Shuhua Yu, Yaoqing Yang · 12 mars 2026

  Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. Motivated by the H…

• To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters

  Sara Dragutinovi\'c, Rajesh Ranganath · 3 mars 2026

  For a long period of time, Adam has served as the ubiquitous default choice for training deep neural networks. Recently, many new optimizers have been introduced, out of which Muon has perhaps gained the highest popularity due to its superior training speed. While many papers set out to validate the…

• Muon+: Towards Better Muon via One Additional Normalization Step

  Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Zheng Zhang · 26 février 2026

  The Muon optimizer has demonstrated promising performance in pre-training large language models through gradient (or momentum) orthogonalization. In this work, we propose a simple yet effective enhancement to Muon, namely Muon+, which introduces an additional normalization step after orthogonalizati…

• Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum

  Minxin Zhang, Yuxuan Liu, Hayden Scheaffer · 20 février 2026

  Efficient stochastic optimization typically integrates an update direction that performs well in the deterministic regime with a mechanism adapting to stochastic perturbations. While Adam uses adaptive moment estimates to promote stability, Muon utilizes the weight layers' matrix structure via ortho…

• Effective Quantization of Muon Optimizer States

  Aman Gupta, Rafael Celente, Abhishek Shivanna, D. T. Braithwaite, Gregory Dexter, Shao Tang, Hiroto Udagawa, Daniel Silva, Rohan Ramanath, S. Sathiya Keerthi · 17 février 2026

  The Muon optimizer, based on matrix orthogonalization, has recently shown faster convergence and better computational efficiency over AdamW in LLM pre-training. However, the memory overhead of maintaining high-precision optimizer states remains a challenge for large-scale deployment. In this paper, …

• Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy

  Zhendong Huang, Hengjie Cao, Fang Dong, Ruijun Huang, Mengyi Chen, Yifeng Yang, Xin Zhang, Anrui Chen, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Fan Yang, Tun Lu, Li Shang · 13 février 2026

  Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific information resides in a long tail. We show that this spike tail separation persists throughout training, with the spik…

• Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization

  Shuntaro Nagashima, Hideaki Iiduka · 28 janvier 2026

  The Muon optimizer has recently attracted attention due to its orthogonalized first-order updates, and a deeper theoretical understanding of its convergence behavior is essential for guiding practical applications; however, existing convergence guarantees are either coarse or obtained under restrict…

• Convergence of Muon with Newton-Schulz

  Gyu Yeol Kim, Min-hwan Oh · 28 janvier 2026

  We analyze Muon as originally proposed and used in practice -- using the momentum orthogonalization with a few Newton-Schulz steps. The prior theoretical results replace this key step in Muon with an exact SVD-based polar factor. We prove that Muon with Newton-Schulz converges to a stationary point …

• Preconditioning Benefits of Spectral Orthogonalization in Muon

  Jianhao Ma, Yu Huang, Yuejie Chi, Yuxin Chen · 21 janvier 2026

  The Muon optimizer, a matrix-structured algorithm that leverages spectral orthogonalization of gradients, is a milestone in the pretraining of large language models. However, the underlying mechanisms of Muon -- particularly the role of gradient orthogonalization -- remain poorly understood, with ve…

• Controlled LLM Training on Spectral Sphere

  Tian Xie, Haoming Luo, Haoyu Tang, Yiwen Hu, Jason Klein Liu, Qingnan Ren, Yang Wang, Wayne Xin Zhao, Rui Yan, Bing Su, Chong Luo, Baining Guo · 14 janvier 2026

  Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization ($\boldsymbol{\mu}$P) provides a theoretical safeguard for width-invariant $\Theta(1)$ activation control, whereas emerging optimizers like Muon are only ``half-a…

• Towards a Principled Muon under $\mu\mathsf{P}$: Ensuring Spectral Conditions throughout Training

  John Zhao · 6 janvier 2026

  The $\mu$-parameterization ($\mu$P) provides a principled foundation for large language model (LLM) training by prescribing width-independent learning dynamics, which in turn enables predictable scaling behavior and robust hyperparameter transfer across model sizes. A central requirement of $\mu$P i…

• Dion2: A Simple Method to Shrink Matrix in Muon

  Kwangjun Ahn, Noah Amsel, John Langford · 22 décembre 2025

  The Muon optimizer enjoys strong empirical performance and theoretical grounding. However, the super-linear cost of its orthonormalization step introduces increasing overhead with scale. To alleviate this cost, several works have attempted to reduce the size of the matrix entering the orthonormaliza…

• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data

  Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, Christos Thrampoulidis · 15 décembre 2025

  The growing adoption of spectrum-aware matrix-valued optimizers such as Muon and Shampoo in deep learning motivates a systematic study of their generalization properties and, in particular, when they might outperform competitive algorithms. We approach this question by introducing appropriate simpli…

• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data

  Bhavya Vasudeva, Puneesh Deora, Yize Zhao, Vatsal Sharan, Christos Thrampoulidis · 2 décembre 2025

  The growing adoption of spectrum-aware matrix-valued optimizers such as Muon and Shampoo in deep learning motivates a systematic study of their generalization properties and, in particular, when they might outperform competitive algorithms. We approach this question by introducing appropriate simpli…