Adversarial Robustness in Machine Learning

Neueste Paper

• TRAM: Test-Time Risk Adaptation with Mixture of Agents

  Mohamad Fares El Hajj Chehade, Amrit Singh Bedi, Amy Zhang, Hao Zhu · 21. Mai 2026

  Deployed reinforcement learning agents often face safety requirements that are specified only after training, such as new hazard maps, revised risk thresholds, or behavioral alignment constraints. We study zero-update deployment-time adaptation, where a fixed library of risk-neutral source policies …

• Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor

  Yang Zhao, Jiahao Lu, Bin Huang, Guhua Zhang, Jie Zhou · 21. Mai 2026

  Narang et al. (2021) evaluated 40+ Transformer modifications at T5-base scale and concluded that most did not transfer. Five years later, the typical working regime has moved to 1-3B parameters, downstream evaluation has replaced pretraining perplexity, and a substantially different catalogue of mod…

• Mechanistic Interpretability for Learning Assurance of a Vision-Based Landing System

  Romeo Valentin, Olivia Beyer Bruvik, Marc R. Schlichting, Mykel J. Kochenderfer · 21. Mai 2026

  EASA's learning-assurance guidance requires data-driven aviation systems to build and monitor their own situation representation, yet for neural networks the technical means to provide such evidence remain an open problem. We address this gap for a vision-based aircraft landing system: we propose th…

• Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving

  Yang Wu, Qiang Meng, Zhaojiang Liu, Youquan Liu, Jian Yang, Jin Xie · 21. Mai 2026

  Current end-to-end autonomous driving models are fundamentally constrained by the behavioral cloning ceiling of imitation learning. While reinforcement learning offers a path to smarter autonomy, it demands two missing pieces of infrastructure: (1) a cognitive foundation that understands traffic sem…

• \ECUAS{n}: A family of metrics for principled evaluation of uncertainty-augmented systems

  Lautaro Estienne, Erik Ernst, Mat\'ias Vera, Pablo Piantanida, Luciana Ferrer · 21. Mai 2026

  In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems -- i.e., systems that output both predic…

• A Mechanistic Study of Tabular Foundation Models

  Marin Bilo\v{s}, James T. Wilson, Anderson Schneider, Yuriy Nevmyvaka · 21. Mai 2026

  Tabular foundation models with different architectures converge in accuracy across a range of classification and regression tasks. This raises questions a leaderboard cannot answer: (i) whether the models execute the same in-context algorithm, (ii) where row, column, and class-permutation invariance…

• Backchaining Loss of Control Mitigations from Mission-Specific Benchmarks in National Security

  Matteo Pistillo, Samantha Faraone, Joshua Herman · 21. Mai 2026

  Affordances and permissions are promising and timely safety levers for mitigating Loss of Control (LoC) threats in high-stakes deployment contexts, such as national security. Deployers in defense and intelligence could rely on several approaches to identify which affordances and permissions should b…

• Online Conformal Prediction with Corrupted Feedback

  Bowen Wang, Matteo Zecchin, Osvaldo Simeone · 21. Mai 2026

  Modern artificial intelligence systems require calibrated uncertainty estimates that remain reliable in sequential and non-stationary environments. Online conformal prediction (OCP) addresses this challenge through adaptively updated prediction sets that provide deterministic long-run miscoverage gu…

• PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment

  Richa Verma, Bavish Kulur, Sanjay Chawla, Balaraman Ravindran · 21. Mai 2026

  We address the problem of making a pre-trained reinforcement learning (RL) policy safety-aware by incorporating cost constraints without retraining it from scratch. While costs could be numerically encoded, we assume a more general setting is when costs are provided as preferences. Given a reward-op…

• SDM: A Powerful Tool for Evaluating Model Robustness

  Xinlei Liu, Tao Hu, Jichao Xie, Peng Yi, Hailong Ma, Baolin Li · 21. Mai 2026

  Gradient-based attacks are important methods for evaluating model robustness. However, since the proposal of APGD, it has been difficult for such methods to achieve significant breakthroughs. To achieve such an effect, we first analyze the issue of "high-loss non-adversarial examples" that degrades …

• Adversarial Robustness in One-Stage Learning-to-Defer

  Yannis Montreuil, Letian Yu, Axel Carlier, Lai Xing Ng, Wei Tsang Ooi · 21. Mai 2026

  Learning-to-Defer (L2D) enables hybrid decision-making by routing inputs either to a predictor or to external experts. While promising, L2D is highly vulnerable to adversarial perturbations, which can not only flip predictions but also manipulate deferral decisions. Prior robustness analyses focus s…

• LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models

  Fanfei Li, Thomas Klein, Wieland Brendel, Robert Geirhos, Roland S. Zimmermann · 21. Mai 2026

  Out-of-distribution (OOD) robustness is a desired property of computer vision models. Improving model robustness requires high-quality signals from robustness benchmarks to quantify progress. While various benchmark datasets such as ImageNet-C were proposed in the ImageNet era, most ImageNet-C corru…

• REFLECTOR: Internalizing Step-wise Reflection against Indirect Jailbreak

  Jiachen Ma, Jiawen Zhang, Xiangtian Li, Bo Zou, Chaochao Lu, Chao Yang · 21. Mai 2026

  While Large Language Models (LLMs) demonstrate remarkable capabilities, they remain susceptible to sophisticated, multi-step jailbreak attacks that circumvent conventional surface-level safety alignment by exploiting the internal generation process. To address these vulnerabilities, we propose Refle…

• A New Framework to Analyse the Distributional Robustness of Deep Neural Networks

  Divij Khaitan, Subhashis Banerjee · 21. Mai 2026

  Deep neural networks have achieved impressive performance on a variety of tasks, but their brittleness to distributional shifts remains a significant barrier to real-world deployment. In this paper, we propose a framework to analyse and quantify the distributional robustness of neural networks by st…

• When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity

  Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu · 21. Mai 2026

  Agent Skills, structured packages of procedural knowledge loaded into an LLM agent at inference time, are widely reported to improve task pass rates by an average of 16.2~percentage points across diverse domains. Yet the same benchmarks show wide variance, with 16 of 84 tasks suffering negative delt…

• Verifiable Provenance and Watermarking for Generative AI: An Evidentiary Framework for International Operational Law and Domestic Courts

  Gustav Olaf Yunus Laitinen-Fredriksson Lundstr\"om-Imanov, Nurana Abdullayeva · 21. Mai 2026

  Generative artificial intelligence now synthesizes photorealistic imagery, audio, and video at a cost that defeats traditional forensic intuition. The legal consequences span three regimes studied so far in isolation: international operational law, domestic procedure, and product regulation. This ar…

• Geometry-Lite: Interpretable Safety Probing via Layer-Wise Margin Geometry

  Woo Seob Sim, Yu Rang Park · 21. Mai 2026

  Prompt-level safety probes for large language models use hidden-state representations to separate safe from unsafe prompts, but strong average detection performance does not explain the geometry of this separation. In particular, it remains unclear how safety evidence is formed across layers, which …

• Quadratic Characterizations for Reachability Analysis of Neural Networks

  Elias Khalife, Mazen Farhood, Pierre-Loic Garoche · 21. Mai 2026

  Quadratic constraints (QCs) are widely used to characterize nonlinearities and uncertainties, but generic analytical characterizations can be conservative on bounded domains. This paper develops a framework for constructing verified quadratic characterizations of scalar relations in the two-dimensio…

• Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale

  Amit Roth, Ankur Samanta, Matan Halevy, Yoav Levine, Yonathan Efroni · 21. Mai 2026

  Aligning autonomous agents with human intent remains a central challenge in modern AI. A key manifestation of this challenge is reward hacking, whereby agents appear successful under the evaluation signal while violating the intended objective. Reward hacking has been observed across a wide range of…

• The Evaluation Game: Beyond Static LLM Benchmarking

  Paul Wang, Jade Garcia-Bourr\'ee, Anne-Marie Kermarrec, Vincent Corruble · 20. Mai 2026

  As jailbreaks, adversarially crafted inputs that bypass safety constraints, continue to be discovered in Large Language Models, practitioners increasingly rely on fine-tuning as a defensive strategy. Yet the theoretical foundations underlying this robustness fine-tuning remain underexplored. We intr…

• When Skills Don't Help: A Negative Result on Procedural Knowledge for Tool-Grounded Agents in Offensive Cybersecurity

  Samuel Jacob Chacko, James Hugglestone, Chashi Mahiul Islam, Xiuwen Liu · 20. Mai 2026

  Agent Skills, structured packages of procedural knowledge loaded into an LLM agent at inference time, are widely reported to improve task pass rates by an average of 16.2~percentage points across diverse domains. Yet the same benchmarks show wide variance, with 16 of 84 tasks suffering negative delt…

• Jailbreaking on Text-to-Video Models via Scene Splitting Strategy

  Wonjun Lee, Haon Park, Doehyeon Lee, Bumsub Ham, Suhyun Kim · 20. Mai 2026

  Along with the rapid advancement of numerous Text-to-Video (T2V) models, growing concerns have emerged regarding their safety risks. While recent studies have explored vulnerabilities in models like LLMs, VLMs, and Text-to-Image (T2I) models through jailbreak attacks, T2V models remain largely unexp…

• Targeted Downstream-Agnostic Attack

  Zhuxin Lei, Ziyuan Yang, Yi Zhang · 20. Mai 2026

  Recently, pre-trained encoders have gained widespread use due to their strong capability in representation extraction. However, they are vulnerable to downstream-agnostic attacks (DAAs). Existing DAA methods operate under a permissive threat model, where an attack is successful if the generated down…

• ESLD (External Surrogate Latent Defense): A Latent-Space Architecture for Faster, Stronger Prompt-Injection Defense

  Yash Narendra · 20. Mai 2026

  Modern AI assistants are agentic. To answer a single user request, the underlying language model pulls in information from many sources, such as web searches, retrieved documents, tool outputs, and user follow-ups, and reasons over them across several steps. Any of these inputs can carry malicious c…

• The Capability Paradox: How Smarter Auditors Make Multi-Agent Systems Less Secure

  Qiqi Liu, Thorsten Holz, Shilin Ye, Runhan Song · 20. Mai 2026

  Multi-agent systems extend large language models (LLMs) by decomposing tasks among specialized agents, but their distributed decision process creates new attack surfaces. We identify semantic hijacking, an attack in which harmful requests are concealed within domain-specific narratives and propagate…