Adversarial Robustness in Machine Learning

Neueste Paper

• PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses

  Chenlong Yin, Runpeng Geng, Yanting Wang, Jinyuan Jia · 16. März 2026

  Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents. Although many defenses have been proposed, their robustness against adaptive attacks remains insufficiently evaluated, potentially creating a false sense of security. In this work, we propos…

• SpectralGuard: Detecting Memory Collapse Attacks in State Space Models

  Davi Bonetto · 16. März 2026

  State Space Models (SSMs) such as Mamba achieve linear-time sequence processing through input-dependent recurrence, but this mechanism introduces a critical safety vulnerability. We show that the spectral radius rho(A-bar) of the discretized transition operator governs effective memory horizon: when…

• Disentangling Recall and Reasoning in Transformer Models through Layer-wise Attention and Activation Analysis

  Harshwardhan Fartale, Ashish Kattamuri, Rahul Raja, Arpita Vats, Ishita Prasad, Akshata Kishore Moharir · 16. März 2026

  Transformer-based language models excel at both recall (retrieving memorized facts) and reasoning (performing multi-step inference), but whether these abilities rely on distinct internal mechanisms remains unclear. Distinguishing recall from reasoning is crucial for predicting model generalization, …

• Altered Thoughts, Altered Actions: Probing Chain-of-Thought Vulnerabilities in VLA Robotic Manipulation

  Tuan Duong Trinh, Naveed Akhtar, Basim Azam · 16. März 2026

  Recent Vision-Language-Action (VLA) models increasingly adopt chain-of-thought (CoT) reasoning, generating a natural-language plan before decoding motor commands. This internal text channel between the reasoning module and the action decoder has received no adversarial scrutiny. We ask: which proper…

• Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems

  Yihe Fan, Wenqi Zhang, Xudong Pan, Min Yang · 16. März 2026

  As foundation models grow increasingly more intelligent, reliable and trustworthy safety evaluation becomes more indispensable than ever. However, an important question arises: Whether and how an advanced AI system would perceive the situation of being evaluated, and lead to the broken integrity of …

• Purify Once, Edit Freely: Breaking Image Protections under Model Mismatch

  Qichen Zhao, Shengfang Zhai, Xinjian Bai, Qingni Shen, Qiqi Lin, Yansong Gao, Zhonghai Wu · 16. März 2026

  Diffusion models enable high-fidelity image editing but can also be misused for unauthorized style imitation and harmful content generation. To mitigate these risks, proactive image protection methods embed small, often imperceptible adversarial perturbations into images before sharing to disrupt do…

• STRAP-ViT: Segregated Tokens with Randomized -- Transformations for Defense against Adversarial Patches in ViTs

  Nandish Chattopadhyay, Anadi Goyal, Chandan Karfa, Anupam Chattopadhyay · 16. März 2026

  Adversarial patches are physically realizable localized noise, which are able to hijack Vision Transformers (ViT) self-attention, pulling focus toward a small, high-contrast region and corrupting the class token to force confident misclassifications. In this paper, we claim that the tokens which cor…

• Learnability and Privacy Vulnerability are Entangled in a Few Critical Weights

  Xingli Fang, Jung-Eun Kim · 16. März 2026

  Prior approaches for membership privacy preservation usually update or retrain all weights in neural networks, which is costly and can lead to unnecessary utility loss or even more serious misalignment in predictions between training data and non-training data. In this work, we observed three insigh…

• Cascade: Composing Software-Hardware Attack Gadgets for Adversarial Threat Amplification in Compound AI Systems

  Sarbartha Banerjee, Prateek Sahu, Anjo Vahldiek-Oberwagner, Jose Sanchez Vicarte, Mohit Tiwari · 13. März 2026

  Rapid progress in generative AI has given rise to Compound AI systems - pipelines comprised of multiple large language models (LLM), software tools and database systems. Compound AI systems are constructed on a layered traditional software stack running on a distributed hardware infrastructure. Many…

• The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey

  Juhee Kim, Xiaoyuan Liu, Zhun Wang, Shi Qiu, Bo Li, Wenbo Guo, Dawn Song · 13. März 2026

  AI agents that combine large language models with non-AI system components are rapidly emerging in real-world applications, offering unprecedented automation and flexibility. However, this unprecedented flexibility introduces complex security challenges fundamentally different from those in traditio…

• Delayed Backdoor Attacks: Exploring the Temporal Dimension as a New Attack Surface in Pre-Trained Models

  Zikang Ding, Haomiao Yang, Meng Hao, Wenbo Jiang, Kunlan Xiang, Runmeng Du, Yijing Liu, Ruichen Zhang, Dusit Niyato · 13. März 2026

  Backdoor attacks against pre-trained models (PTMs) have traditionally operated under an ``immediacy assumption,'' where malicious behavior manifests instantly upon trigger occurrence. This work revisits and challenges this paradigm by introducing \textit{\textbf{Delayed Backdoor Attacks (DBA)}}, a n…

• Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats

  Xinhao Deng, Yixiang Zhang, Jiaqing Wu, Jiaqi Bai, Sibo Yi, Zhuoheng Zou, Yue Xiao, Rennai Qiu, Jianan Ma, Jialuo Chen, Xiaohu Du, Xiaofang Yang, Shiwen Cui, Changhua Meng, Weiqiang Wang, Jiaxing Song, Ke Xu, Qi Li · 13. März 2026

  Autonomous Large Language Model (LLM) agents, exemplified by OpenClaw, demonstrate remarkable capabilities in executing complex, long-horizon tasks. However, their tightly coupled instant-messaging interaction paradigm and high-privilege execution capabilities substantially expand the system attack …

• The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning

  Raj Sanjay Shah, Jing Huang, Keerthiram Murugesan, Nathalie Baracaldo, Diyi Yang · 13. März 2026

  Unlearning in Large Language Models (LLMs) aims to enhance safety, mitigate biases, and comply with legal mandates, such as the right to be forgotten. However, existing unlearning methods are brittle: minor query modifications, such as multi-hop reasoning and entity aliasing, can recover supposedly …

• Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generation

  Kaichao Jiang, He Wang, Xiaoshuai Hao, Xiulong Yang, Ajian Liu, Qi Chu, Yunfeng Diao, Richang Hong · 13. März 2026

  Joint Energy-based Models (JEMs) are well known for their ability to unify classification and generation within a single framework. Despite their promising generative and discriminative performance, their robustness remains far inferior to adversarial training (AT), which, conversely, achieves stron…

• Deep regression learning from dependent observations with minimum error entropy principle

  William Kengne, Modou Wade · 13. März 2026

  This paper considers nonparametric regression from strongly mixing observations. The proposed approach is based on deep neural networks with minimum error entropy (MEE) principle. We study two estimators: the non-penalized deep neural network (NPDNN) and the sparse-penalized deep neural network (SPD…

• OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure

  Chuancheng Shi, Wenhua Wu, Fei Shen, Xiaogang Zhu, Kun Hu, Zhiyong Wang · 13. März 2026

  Text-to-image (T2I) models face significant safety risks from adversarial induction, yet current concept erasure methods often cause collateral damage to benign attributes when suppressing selected neurons entirely. This occurs because sensitive and benign semantics exhibit non-orthogonal superposit…

• Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models

  Xiangwen Wang, Ananth Balashankar, Varun Chandrasekaran · 13. März 2026

  Large language models remain vulnerable to jailbreak attacks, yet we still lack a systematic understanding of how jailbreak success scales with attacker effort across methods, model families, and harm types. We initiate a scaling-law framework for jailbreaks by treating each attack as a compute-boun…

• The Mirror Design Pattern: Strict Data Geometry over Model Scale for Prompt Injection Detection

  J Alex Corll · 13. März 2026

  Prompt injection defenses are often framed as semantic understanding problems and delegated to increasingly large neural detectors. For the first screening layer, however, the requirements are different: the detector runs on every request and therefore must be fast, deterministic, non-promptable, an…

• Taming the Adversary: Stable Minimax Deep Deterministic Policy Gradient via Fractional Objectives

  Taeho Lee, Donghwan Lee · 13. März 2026

  Reinforcement learning (RL) has achieved remarkable success in a wide range of control and decision-making tasks. However, RL agents often exhibit unstable or degraded performance when deployed in environments subject to unexpected external disturbances and model uncertainties. Consequently, ensurin…

• Hiding in Plain Sight: A Steganographic Approach to Stealthy LLM Jailbreaks

  Jianing Geng, Biao Yi, Zekun Fei, Ruiqi He, Lihai Nie, Tong Li, Zheli Liu · 13. März 2026

  Jailbreak attacks pose a serious threat to Large Language Models (LLMs) by bypassing their safety mechanisms. A truly advanced jailbreak is defined not only by its effectiveness but, more critically, by its stealthiness. However, existing methods face a fundamental trade-off between semantic stealth…

• Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment

  Zhiyu Xue, Zimo Qi, Guangliang Liu, Bocheng Chen, Ramtin Pedarsani · 13. März 2026

  Safety alignment aims to ensure that large language models (LLMs) refuse harmful requests by post-training on harmful queries paired with refusal answers. Although safety alignment is widely adopted in industry, the overrefusal problem where aligned LLMs also reject benign queries after safety align…

• RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents

  Yonas Atinafu, Robin Cohen · 13. März 2026

  LLM agents increasingly perform end-to-end ML engineering tasks where success is judged by a single scalar test metric. This creates a structural vulnerability: an agent can increase the reported score by compromising the evaluation pipeline rather than improving the model. We introduce RewardHackin…

• A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis

  Bo Hu, Jose C Principe · 13. März 2026

  Statistical dependence measures like mutual information is ideal for analyzing autoencoders, but it can be ill-posed for deterministic, static, noise-free networks. We adopt the variational (Gaussian) formulation that makes dependence among inputs, latents, and reconstructions measurable, and we pro…

• Incremental Neural Network Verification via Learned Conflicts

  Raya Elsaleh, Liam Davis, Haoze Wu, Guy Katz · 13. März 2026

  Neural network verification is often used as a core component within larger analysis procedures, which generate sequences of closely related verification queries over the same network. In existing neural network verifiers, each query is typically solved independently, and information learned during …

• Adaptive Activation Cancellation for Hallucination Mitigation in Large Language Models

  Eric Yocam, Varghese Vaidyan, Gurcan Comert, Paris Kalathas, Yong Wang, Judith L. Mwakalonge · 12. März 2026

  Large Language Models frequently generate fluent but factually incorrect text. We propose Adaptive Activation Cancellation (AAC), a real-time inference-time framework that treats hallucination-associated neural activations as structured interference within the transformer residual stream, drawing an…