Adversarial Robustness in Machine Learning

Neueste Paper

• A Hazard-Informed Data Pipeline for Robotics Physical Safety

  Alexei Odinokov, Rostislav Yavorskiy · 9. März 2026

  This report presents a structured Robotics Physical Safety Framework based on explicit asset declaration, systematic vulnerability enumeration, and hazard-driven synthetic data generation. The approach bridges classical risk engineering with modern machine learning pipelines, enabling safety envelop…

• Traversal-as-Policy: Log-Distilled Gated Behavior Trees as Externalized, Verifiable Policies for Safe, Robust, and Efficient Agents

  Peiran Li, Jiashuo Sun, Fangzhou Lin, Shuo Xing, Tianfu Fu, Suofei Feng, Chaoqun Ni, Zhengzhong Tu · 9. März 2026

  Autonomous LLM agents fail because long-horizon policy remains implicit in model weights and transcripts, while safety is retrofitted post hoc. We propose Traversal-as-Policy: distill sandboxed OpenHands execution logs into a single executable Gated Behavior Tree (GBT) and treat tree traversal -- ra…

• BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviation

  Feiran Li, Qianqian Xu, Shilong Bao, Zhiyong Yang, Xilin Zhao, Xiaochun Cao, Qingming Huang · 9. März 2026

  This paper investigates the challenging task of detecting backdoored text-to-image models under black-box settings and introduces a novel detection framework BlackMirror. Existing approaches typically rely on analyzing image-level similarity, under the assumption that backdoor-triggered generations …

• Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts

  Kartik Sharma, Yiqiao Jin, Vineeth Rakesh, Yingtong Dou, Menghai Pan, Mahashweta Das, Srijan Kumar · 9. März 2026

  As large language models (LLMs) are deployed in safety-critical settings, it is essential to ensure that their responses comply with safety standards. Prior research has revealed that LLMs often fail to grasp the notion of safe behaviors, resulting in either unjustified refusals to harmless prompts …

• Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Models

  Jinman Wu, Yi Xie, Shen Lin, Shiqian Zhao, Xiaofeng Chen · 9. März 2026

  Safety alignment is often conceptualized as a monolithic process wherein harmfulness detection automatically triggers refusal. However, the persistence of jailbreak attacks suggests a fundamental mechanistic decoupling. We propose the \textbf{\underline{D}}isentangled \textbf{\underline{S}}afety \te…

• When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models

  Qitong Wang, Haoran Dai, Haotian Zhang, Christopher Rasmussen, Binghui Wang · 9. März 2026

  While diffusion models have revolutionized visual content generation, their rapid adoption has underscored the critical need to investigate vulnerabilities, e.g., to backdoor attacks. In multimodal diffusion models, it is natural to expect that attacking multiple modalities simultaneously (e.g., tex…

• Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

  Jinman Wu, Yi Xie, Shiqian Zhao, Xiaofeng Chen · 9. März 2026

  Currently, open-sourced large language models (OSLLMs) have demonstrated remarkable generative performance. However, as their structure and weights are made public, they are exposed to jailbreak attacks even after alignment. Existing attacks operate primarily at shallow levels, such as the prompt or…

• Proof-of-Guardrail in AI Agents and What (Not) to Trust from It

  Xisen Jin, Michael Duan, Qin Lin, Aaron Chan, Zhenglun Chen, Junyi Du, Xiang Ren · 9. März 2026

  As AI agents become widely deployed as online services, users often rely on an agent developer's claim about how safety is enforced, which introduces a threat where safety measures are falsely advertised. To address the threat, we propose proof-of-guardrail, a system that enables developers to provi…

• Identifying Adversary Characteristics from an Observed Attack

  Soyon Choi, Scott Alfeld, Meiyi Ma · 9. März 2026

  When used in automated decision-making systems, machine learning (ML) models are vulnerable to data-manipulation attacks. Some defense mechanisms (e.g., adversarial regularization) directly affect the ML models while others (e.g., anomaly detection) act within the broader system. In this paper we co…

• Osmosis Distillation: Model Hijacking with the Fewest Samples

  Yuchen Shi, Huajie Chen, Heng Xu, Zhiquan Liu, Jialiang Shen, Chi Liu, Shuai Zhou, Tianqing Zhu, Wanlei Zhou · 6. März 2026

  Transfer learning is devised to leverage knowledge from pre-trained models to solve new tasks with limited data and computational resources. Meanwhile, dataset distillation has emerged to synthesize a compact dataset that preserves critical information from the original large dataset. Therefore, a c…

• Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation

  Benjamin Feuer, Lucas Rosenblatt, Oussama Elachqar · 6. März 2026

  As AI models progress beyond simple chatbots into more complex workflows, we draw ever closer to the event horizon beyond which AI systems will be utilized in autonomous, self-maintaining feedback loops. Any autonomous AI system will depend on automated, verifiable rewards and feedback; in settings …

• Knowledge Divergence and the Value of Debate for Scalable Oversight

  Robin Young · 6. März 2026

  AI safety via debate and reinforcement learning from AI feedback (RLAIF) are both proposed methods for scalable oversight of advanced AI systems, yet no formal framework relates them or characterizes when debate offers an advantage. We analyze this by parameterizing debate's value through the geomet…

• SalamahBench: Toward Standardized Safety Evaluation for Arabic Language Models

  Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh, Ihsen Alouani, Mohammed E. Fouda · 6. März 2026

  Safety alignment in Language Models (LMs) is fundamental for trustworthy AI. However, while different stakeholders are trying to leverage Arabic Language Models (ALMs), systematic safety evaluation of ALMs remains largely underexplored, limiting their mainstream uptake. Existing safety benchmarks an…

• BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

  Yuan Li, Bo Wang, Yufei Gao, Yuqian Yao, Xinyuan Wang, Zhangyue Yin, Xipeng Qiu · 6. März 2026

  Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of l…

• Parameter Stress Analysis in Reinforcement Learning: Applying Synaptic Filtering to Policy Networks

  Zain ul Abdeen, Ming Jin · 6. März 2026

  This paper explores reinforcement learning (RL) policy robustness by systematically analyzing network parameters under internal and external stresses. \textcolor{black}{We apply synaptic filtering methods using high-pass, low-pass, and pulse-wave filters from} \citep{pravin2024fragility}, as an inte…

• Why Is RLHF Alignment Shallow? A Gradient Analysis

  Robin Young · 6. März 2026

  Why is safety alignment in LLMs shallow? We prove that gradient-based alignment inherently concentrates on positions where harm is decided and vanishes beyond. Using a martingale decomposition of sequence-level harm, we derive an exact characterization of alignment gradients. The gradient at positio…

• Good-Enough LLM Obfuscation (GELO)

  Anatoly Belikov, Ilya Fedotov · 6. März 2026

  Large Language Models (LLMs) are increasingly served on shared accelerators where an adversary with read access to device memory can observe KV caches and hidden states, threatening prompt privacy for open-source models. Cryptographic protections such as MPC and FHE offer strong guarantees but remai…

• Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions

  Neha Nagaraja, Lan Zhang, Zhilong Wang, Bo Zhang, Pawan Patil · 5. März 2026

  Multimodal Large Language Models (MLLMs) integrate vision and text to power applications, but this integration introduces new vulnerabilities. We study Image-based Prompt Injection (IPI), a black-box attack in which adversarial instructions are embedded into natural images to override model behavior…

• Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information

  Yifan Zhu, Yibo Miao, Yinpeng Dong, Xiao-Shan Gao · 5. März 2026

  The volume of freely scraped data on the Internet has driven the tremendous success of deep learning. Along with this comes the growing concern about data privacy and security. Numerous methods for generating unlearnable examples have been proposed to prevent data from being illicitly learned by una…

• Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinement

  Yuxiao Lu, Lin Xu, Yang Sun, Wenjun Li, Jie Shi · 5. März 2026

  Large language models (LLMs) aligned for safety often suffer from over-refusal, the tendency to reject seemingly toxic or benign prompts by misclassifying them as toxic. This behavior undermines models' helpfulness and restricts usability in sensitive or nuanced contexts. While prior work has propos…

• Sleeper Cell: Injecting Latent Malice Temporal Backdoors into Tool-Using LLMs

  Bhanu Pallakonda, Mikkel Hindsbo, Sina Ehsani, Prag Mishra · 5. März 2026

  The proliferation of open-weight Large Language Models (LLMs) has democratized agentic AI, yet fine-tuned weights are frequently shared and adopted with limited scrutiny beyond leaderboard performance. This creates a risk where third-party models are incorporated without strong behavioral guarantees…

• In-Context Environments Induce Evaluation-Awareness in Language Models

  Maheep Chaudhary · 5. März 2026

  Humans often become more self-aware under threat, yet can lose self-awareness when absorbed in a task; we hypothesize that language models exhibit environment-dependent \textit{evaluation awareness}. This raises concerns that models could strategically underperform, or \textit{sandbag}, to avoid tri…

• Structure-Aware Distributed Backdoor Attacks in Federated Learning

  Wang Jian, Shen Hong, Ke Wei, Liu Xue Hua · 5. März 2026

  While federated learning protects data privacy, it also makes the model update process vulnerable to long-term stealthy perturbations. Existing studies on backdoor attacks in federated learning mainly focus on trigger design or poisoning strategies, typically assuming that identical perturbations be…

• From Misclassifications to Outliers: Joint Reliability Assessment in Classification

  Yang Li, Youyang Sha, Yinzhi Wang, Timothy Hospedales, Xi Shen, Shell Xu Hu, Xuanlong Yu · 5. März 2026

  Building reliable classifiers is a fundamental challenge for deploying machine learning in real-world applications. A reliable system should not only detect out-of-distribution (OOD) inputs but also anticipate in-distribution (ID) errors by assigning low confidence to potentially misclassified sampl…

• Dynamic Adversarial Reinforcement Learning for Robust Multimodal Large Language Models

  Yicheng Bao, Xuhong Wang, Qiaosheng Zhang, Chaochao Lu, Xia Hu, Xin Tan · 5. März 2026

  Despite their impressive capabilities, Multimodal Large Language Models (MLLMs) exhibit perceptual fragility when confronted with visually complex scenes. This weakness stems from a reliance on finite training datasets, which are prohibitively expensive to scale and impose a ceiling on model robustn…