Sensor Technology and Measurement Systems

Neueste Paper

• What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

  Jesse Ponnock · 20. August 2026

  Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift, as recent work has shown. What the copy delivers has never been measured directly in the attention structu…

• On the Benefits of Weight Normalization for Overparameterized Matrix Sensing

  Yudong Wei, Liang Zhang, Bingcong Li, Niao He · 16. Juni 2026

  While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian op…

• Limitations of Normalization in Attention Mechanism

  Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova, Radu State · 8. Juni 2026

  This paper investigates the limitations of the normalization in attention mechanisms. We begin with a theoretical framework that enables the identification of the model's selective ability and the geometric separation involved in token selection. Our analysis includes explicit bounds on distances an…

• Measuring multi-calibration

  Ido Guy, Daniel Haimovich, Fridolin Linder, Nastaran Okati, Lorenzo Perini, Niek Tax, Mark Tygert · 17. April 2026

  A suitable scalar metric can help measure multi-calibration, defined as follows. When the expected values of observed responses are equal to corresponding predicted probabilities, the probabilistic predictions are known as "perfectly calibrated." When the predicted probabilities are perfectly calibr…

• Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning

  Magauiya Zhussip, Dmitriy Shopkhoev, Ammar Ali, Stamatios Lefkimmiatis · 23. Februar 2026

  Large language models have revolutionized AI applications, yet their high computational and memory demands hinder their widespread deployment. Existing compression techniques focus on intra-block optimizations (e.g., low-rank approximation or attention pruning), while the repetitive layered structur…

• Can Distillation Mitigate Backdoor Attacks in Pre-trained Encoders?

  TIngxu Han, Wei Song, Weisong Sun, Ziqi Ding, Yebo Feng, Chunrong Fang, Jun Li, Hanwei Qian, Zhenyu Chen, Yang Liu · 2. Februar 2026

  Self-Supervised Learning (SSL) has become a prominent paradigm for pre-training encoders to learning general-purpose representations from unlabeled data and releasing them on third-party platforms for broad downstream deep learning tasks. However, SSL is vulnerable to backdoor attacks, where an adve…

• Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Encoder-Only and Decoder-Only Transformers

  Marko Karbevski, Antonij Mijoski · 2. Februar 2026

  We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that Query weights are redundant and can be replaced with the identity matrix, reducing attention parameters by $25\%$. This also s…

• Shared DIFF Transformer

  Yueyang Cang, Yuhang Liu, Xiaoteng Zhang, Li Shi, Wenge Que · 17. Dezember 2025

  DIFF Transformer improves attention allocation by enhancing focus on relevant context while suppressing noise. It introduces a differential attention mechanism that calculates the difference between two independently generated attention distributions, effectively reducing noise and promoting sparse …

• Iwin Transformer: Hierarchical Vision Transformer using Interleaved Windows

  Simin Huo, Ning Li · 9. Dezember 2025

  We introduce Iwin Transformer, a novel position-embedding-free hierarchical vision transformer, which can be fine-tuned directly from low to high resolution, through the collaboration of innovative interleaved window attention and depthwise separable convolution. This approach uses attention to conn…

• A Control Perspective on Training PINNs

  Matthieu Barreau, Haoming Shen · 8. Dezember 2025

  We investigate the training of Physics-Informed Neural Networks (PINNs) from a control-theoretic perspective. Using gradient descent with resampling, we interpret the training dynamics as asymptotically equivalent to a stochastic control-affine system, where sampling effects act as process disturban…

• IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning

  Xiandong Zou, Pan Zhou · 30. Oktober 2025

  Normalization is fundamental to deep learning, but existing approaches such as BatchNorm, LayerNorm, and RMSNorm are variance-centric by enforcing zero mean and unit variance, stabilizing training without controlling how representations capture task-relevant information. We propose IB-Inspired Norma…