Power Transformer Diagnostics and Insulation

Neueste Paper

• Predictive Geometry of Hidden Trajectories in Transformers

  Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova, Radu State · 1. Oktober 2026

  Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state through the fixed downstream computation. We formalize this constraint by studying layerwise loss-to-go functions: the terminal loss obtained by continui…

• VC Dimension and Expressivity of Real-Valued Transformers

  Gavin Dooley, Andy Yang, Yijia Jessica Zhu, David Chiang, Peter Cholak, Anand Pillay · 29. September 2026

  Whereas previous results on abilities and limitations of transformers have restricted the definition of transformers in various ways, here we study softmax-attention, multi-layer transformers operating on real values, with very few additional assumptions. Applying results from real geometry, we obta…

• Spectral Tail Interventions in Decoder-Only Language Models: Reasoning-Sensitive Weight Structure from Controlled Surgery

  Ibne Farabi Shihab, Sanjida Akhter, Md Najmus Swaqeeb, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Anuj Sharma · 23. September 2026

  Weight-space structure often correlates with language-model behavior, but correlation alone does not establish computational involvement. We study concentrated upper spectral tails in decoder-only transformers through controlled interventions. At a fixed relative offset, we derive a finite-width con…

• Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

  Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara, Suranadi De Silva · 23. September 2026

  Structured pruning of attention heads provides a hardware-friendly way to compress Transformer language models. However, existing methods for measuring head-level importance require calibration data, gradient computation, or Hessian estimation. These requirements add extra overhead and make the meth…

• Certified Mechanistic Interpretability: Lifting Single-Input Findings to Bounded Neighbourhoods

  Zhen Zhang, Yanliang Huang, Peng Xie, Wenyuan Wu, Amr Alanwar · 23. September 2026

  Mechanistic interpretability reverse-engineers transformer circuits one input at a time, leaving observed mechanisms without guarantees over bounded input neighbourhoods. We address this gap with a framework based on constrained polynomial-zonotope (CPZ) propagation that lifts mechanistic-interpreta…

• Transformer Heads Looking for Order

  Jasper van Doornmalen, Alexander Kozachinskiy, Corinna Mathwieser, Tomasz Steifer, Felipe Urrutia, Jos\'{e} Verschae, Przemys{\l}aw Andrzej Wa{\l}\c{e}ga · 23. September 2026

  In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.…

• Complex-valued Phase-Coherent Transformers

  Leona Hioki · 22. September 2026

  Complex-valued Transformers have inherited softmax attention over the raw complex inner product. Outside natively complex domains this standard form stays near chance, and no complex attention had been shown to correct it. We show that the match must be a scaled cosine score: L2-normalise queries an…

• Transformer fault diagnosis using an efficient simulation-driven variational quantum classifier with domain-aware feature encoding

  Huy Hoang Le, Ba Tu Phung, Dai Huynh, Kim-Anh Nguyen · 18. September 2026

  Early transformer fault diagnosis is challenged by nonlinear dissolved-gas interactions, overlapping fault signatures, and limited labeled data, while practical deployment further requires reliable performance under realistic computational constraints. This paper presents a simulation-driven modelin…

• What Does Layer-Importance Reveal About Transformers and State-Space Models?

  Istabrak Abbes, Nizar Islah, Irina Rish, Sarath Chandar · 16. September 2026

  Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical knowledge built for transformers transfers to SSMs. We address this through the lens of layer importance which underpins compression, selective fine-tuni…

• Training Trajectories Determine Circuit Removability in Annealable Soft-Prior Transformers

  Zonglin Yang, Ziming Zhao, Wei Tang, Xunyu Jiang, Yihong Liu, Tailin Chen, Zifu Yu, Jiayu Liu · 10. September 2026

  Soft positional priors can help small Transformers learn retrieval circuits, but it is unclear whether the resulting circuits remain functional once the prior is removed. We test this with an annealable soft-prior Transformer whose attention biases can be learned, faded, or zeroed during training an…

• Through the Looking Glass: Directly Reading and Writing Transformers

  Mark Oskin · 10. September 2026

  How many of a transformer's components decide a token? Counted by the absolute value of each unit's and channel's contribution to the logit, one prediction rests on thousands to hundreds of thousands of them. But contributions are signed, and across eighteen models the mass pushing away from the pre…

• One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Context

  Skanda Athreya, Yutong Wang · 2. September 2026

  We extend recent work establishing an equivalence between one-layer transformers and nearest-neighbor classifiers in the binary setting to the multiclass case. By leveraging the simplex encoding, we show that one-layer transformers with an argmax classification head behave identically to a one-neare…

• Multi-Head Self Attention is a Parameter Identification Mechanism

  W. Ross Morrow · 2. September 2026

  We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to the total number of parameters scales like the reciprocal of the number of heads ($1/2 \to 1/(2H)$), meaning models with more heads are structurally m…

• Measuring Optimal Transport in Transformer Depth

  Alexandre Quemy · 2. September 2026

  A transformer carries each token's state from layer to layer, and the whole vocabulary carried together forms a cloud that moves with depth. We ask whether a trained network moves this cloud the way optimal transport would: at the cheapest cost, and along the map that pairs each token with its optim…

• The Communication Map of a Transformer

  Richard Zhe Wang · 25. August 2026

  The components of a transformer communicate by writing to and reading from a shared residual stream, and mechanistic interpretability has mapped these connections by hand, one circuit at a time. We present the communication map, which charts every potential communication channel in a language model …

• Optimized Fuzzy Logic Approach with the IEEE Key Gas Method for Diagnosing Power Transformer Faults Using Dissolved Gas Analysis

  Kim-Anh Nguyen, Huy Hoang Le, Ba Tu Phung · 20. August 2026

  Reliable transformer fault diagnosis is essential for maintaining power system stability. The IEEE Key Gas Method (KGM), a widely utilized approach in Dissolved Gas Analysis (DGA), exhibits limitations in addressing ambiguous data and ensuring high diagnostic accuracy. This study presents An enhance…

• Virtual Temperature Sensors in Power Transformers Using Neural Ordinary Differential Equations

  Berk Hadzhamolla, Alexander Johannes Stasik, Signe Riemer-S{\o}rensen · 14. August 2026

  Accurate modeling and forecasting of power transformer thermal behavior are critical for reliability, asset lifetime, and optimized power system operation. Numerical approaches such as finite element methods (FEM) and computational fluid dynamics (CFD) offer high fidelity but are computationally exp…

• Transformer Circuits Can Realize Clustering Algorithms

  Kenneth L. Clarkson, Lior Horesh, Takuya Ito, Charlotte Park, Parikshit Ram · 11. August 2026

  Although transformers are most commonly optimized as statistical sequence models, it is unclear to what extent they can implement and learn exact algorithmic computations. Here, we specify a transformer implementation from first principles that executes a fundamental and widely used method for $k$-m…

• Stability of Transformers under Layer Normalization

  Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Krishna Kumar, Markos A. Katsoulakis · 10. August 2026

  Despite their widespread use, training deep Transformers can be unstable. Layer normalization, a standard component, improves training stability, but its placement has often been ad-hoc. In this paper, we conduct a principled study on the forward (hidden states) and backward (gradient) stability of …

• The Ignition Index: Measuring Global Workspace Dynamics in Language Models

  Saman Rahbar · 7. August 2026

  We introduce the Ignition Index (I), a validated scalar metric that operationalizes Global Workspace Theory's (GWT) all-or-none ignition prediction in transformer language models. The metric fits a four-parameter sigmoid to per-layer linear probe accuracy as a function of input signal strength, extr…

• One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse

  Shuxiao Xie, Shuyang Xie, Yuan Cao, Dezhi Ran, Wei Yang, Tao Xie · 4. August 2026

  A bfloat16 transformer can train normally for many steps and then collapse abruptly. Distinct low-precision errors can trigger the same failure, leaving unclear whether each source needs its own repair or one shared route can be blocked. We isolate a reproduced GPT-2-class collapse to the streaming-…

• An adaptive multi-fuzzy logic model for diagnosing transformer faults using dynamic weight optimization

  Kim-Anh Nguyen, Huy Hoang Le, Ba Tu Phung · 28. Juli 2026

  Dissolved gas analysis (DGA) is crucial for diagnosing early power transformer failures. Traditional DGA interpretation methods like Duval Triangle, IEC ratio, Roger ratio, Doernenburg ratio and Key Gas are inconsistent and vary in accuracy, especially for multiple fault conditions. We propose an Ad…

• LiFT: Local Search via Linear Programming for Overfitting-Controlled Transformers

  Abhishek Shukla, Anikeit Khanna, Ankur Sinha, Faiz Hamid · 16. Juni 2026

  This paper proposes a Linear Programming (LP)-based local search framework for fine-tuning pretrained transformer models with explicit control against overfitting. The approach formulates transformer fine-tuning as a bilevel optimization-based regularization problem, in which model parameters and re…

• A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions

  Tiexin Ding · 20. Mai 2026

  We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit …

• Supernodes and Halos: Loss-Critical Hubs in LLM Feed-Forward Layers

  Audrey Cherilyn, Houman Safaai · 28. April 2026

  We study the organization of channel-level importance in transformer feed-forward networks (FFNs). Using a Fisher-style loss proxy (LP) based on activation-gradient second moments, we show that loss sensitivity is concentrated in a small set of channels within each layer. In Llama-3.1-8B, the top 1%…