Adversarial Robustness in Machine Learning

Neueste Paper

• Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoning

  Chenyu Zhang, Lanjun Wang, Yiwen Ma, Wenhui Li, An-An Liu · 24. November 2025

  Text-to-Image(T2I) models typically deploy safety filters to prevent the generation of sensitive images. Unfortunately, recent jailbreaking attack methods manually design instructions for the LLM to generate adversarial prompts, which effectively bypass safety filters while producing sensitive image…

• AutoBackdoor: Automating Backdoor Attacks via LLM Agents

  Yige Li, Zhe Li, Wei Zhao, Nay Myat Min, Hanxun Huang, Xingjun Ma, Jun Sun · 24. November 2025

  Backdoor attacks pose a serious threat to the secure deployment of large language models (LLMs), enabling adversaries to implant hidden behaviors triggered by specific inputs. However, existing methods often rely on manually crafted triggers and static data pipelines, which are rigid, labor-intensiv…

• Detecting and Steering LLMs' Empathy in Action

  Juan P. Cadile · 24. November 2025

  We investigate empathy-in-action -- the willingness to sacrifice task efficiency to address human needs -- as a linear direction in LLM activation space. Using contrastive prompts grounded in the Empathy-in-Action (EIA) benchmark, we test detection and steering across Phi-3-mini-4k (3.8B), Qwen2.5-7…

• Non-Parametric Probabilistic Robustness: A Conservative Metric with Optimized Perturbation Distributions

  Zheng Wang, Yi Zhang, Siddartha Khastgir, Carsten Maple, Xingyu Zhao · 24. November 2025

  Deep learning (DL) models, despite their remarkable success, remain vulnerable to small input perturbations that can cause erroneous outputs, motivating the recent proposal of probabilistic robustness (PR) as a complementary alternative to adversarial robustness (AR). However, existing PR formulatio…

• Monte Carlo Expected Threat (MOCET) Scoring

  Joseph Kim, Saahith Potluri · 24. November 2025

  Evaluating and measuring AI Safety Level (ASL) threats are crucial for guiding stakeholders to implement safeguards that keep risks within acceptable limits. ASL-3+ models present a unique risk in their ability to uplift novice non-state actors, especially in the realm of biosecurity. Existing evalu…

• SHIELD: Secure Hypernetworks for Incremental Expansion Learning Defense

  Patryk Krukowski, {\L}ukasz Gorczyca, Piotr Helm, Kamil Ksi\k{a}\.zek, Przemys{\l}aw Spurek · 24. November 2025

  Continual learning under adversarial conditions remains an open problem, as existing methods often compromise either robustness, scalability, or both. We propose a novel framework that integrates Interval Bound Propagation (IBP) with a hypernetwork-based architecture to enable certifiably robust con…

• A Closer Look at Adversarial Suffix Learning for Jailbreaking LLMs: Augmented Adversarial Trigger Learning

  Zhe Wang, Yanjun Qi · 21. November 2025

  Gradient optimization-based adversarial attack methods automate the learning of adversarial triggers to generate jailbreak prompts or leak system prompts. In this work, we take a closer look at the optimization objective of adversarial trigger learning and propose ATLA: Adversarial Trigger Learning …

• Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models

  Shuang Liang, Zhihao Xu, Jialing Tao, Hui Xue, Xiting Wang · 21. November 2025

  Despite extensive alignment efforts, Large Vision-Language Models (LVLMs) remain vulnerable to jailbreak attacks, posing serious safety risks. To address this, existing detection methods either learn attack-specific parameters, which hinders generalization to unseen attacks, or rely on heuristically…

• Decoding Deception: Understanding Automatic Speech Recognition Vulnerabilities in Evasion and Poisoning Attacks

  Aravindhan G, Yuvaraj Govindarajulu, Parin Shah · 21. November 2025

  Recent studies have demonstrated the vulnerability of Automatic Speech Recognition systems to adversarial examples, which can deceive these systems into misinterpreting input speech commands. While previous research has primarily focused on white-box attacks with constrained optimizations, and trans…

• Do Not Merge My Model! Safeguarding Open-Source LLMs Against Unauthorized Model Merging

  Qinfeng Li, Miao Pan, Jintao Chen, Fu Teng, Zhiqiang Shen, Ge Su, Hao Peng, Xuhong Zhang · 21. November 2025

  Model merging has emerged as an efficient technique for expanding large language models (LLMs) by integrating specialized expert models. However, it also introduces a new threat: model merging stealing, where free-riders exploit models through unauthorized model merging. Unfortunately, existing defe…

• Identifying the Supply Chain of AI for Trustworthiness and Risk Management in Critical Applications

  Raymond K. Sheh, Karen Geappen · 21. November 2025

  Risks associated with the use of AI, ranging from algorithmic bias to model hallucinations, have received much attention and extensive research across the AI community, from researchers to end-users. However, a gap exists in the systematic assessment of supply chain risks associated with the complex…

• Physically Realistic Sequence-Level Adversarial Clothing for Robust Human-Detection Evasion

  Dingkun Zhou, Patrick P. K. Chan, Hengxu Wu, Shikang Zheng, Ruiqi Huang, Yuanjie Zhao · 21. November 2025

  Deep neural networks used for human detection are highly vulnerable to adversarial manipulation, creating safety and privacy risks in real surveillance environments. Wearable attacks offer a realistic threat model, yet existing approaches usually optimize textures frame by frame and therefore fail t…

• Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models

  Piercosma Bisconti, Matteo Prandi, Federico Pierucci, Francesco Giarrusso, Marcantonio Bracale, Marcello Galisai, Vincenzo Suriani, Olga Sorokoletova, Federico Sartore, Daniele Nardi · 21. November 2025

  We present evidence that adversarial poetry functions as a universal single-turn jailbreak technique for Large Language Models (LLMs). Across 25 frontier proprietary and open-weight models, curated poetic prompts yielded high attack-success rates (ASR), with some providers exceeding 90%. Mapping pro…

• SafeRBench: A Comprehensive Benchmark for Safety Assessment in Large Reasoning Models

  Xin Gao, Shaohan Yu, Zerui Chen, Yueming Lyu, Weichen Yu, Guanghao Li, Jiyao Liu, Jianxiong Gao, Jian Liang, Ziwei Liu, Chenyang Si · 21. November 2025

  Large Reasoning Models (LRMs) improve answer quality through explicit chain-of-thought, yet this very capability introduces new safety risks: harmful content can be subtly injected, surface gradually, or be justified by misleading rationales within the reasoning trace. Existing safety evaluations, h…

• "To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios

  Zhen Sun, Zongmin Zhang, Deqi Liang, Han Sun, Yule Liu, Yun Shen, Xiangshan Gao, Yilong Yang, Shuai Liu, Yutao Yue, Xinlei He · 21. November 2025

  As LLMs become more common, non-expert users can pose risks, prompting extensive research into jailbreak attacks. However, most existing black-box jailbreak attacks rely on hand-crafted heuristics or narrow search spaces, which limit scalability. Compared with prior attacks, we propose Game-Theory A…

• Large Language Model-Based Reward Design for Deep Reinforcement Learning-Driven Autonomous Cyber Defense

  Sayak Mukherjee, Samrat Chatterjee, Emilie Purvine, Ted Fujimoto, Tegan Emerson · 21. November 2025

  Designing rewards for autonomous cyber attack and defense learning agents in a complex, dynamic environment is a challenging task for subject matter experts. We propose a large language model (LLM)-based reward design approach to generate autonomous cyber defense policies in a deep reinforcement lea…

• Dataset Distillation for Pre-Trained Self-Supervised Vision Models

  George Cazenavette, Antonio Torralba, Vincent Sitzmann · 21. November 2025

  The task of dataset distillation aims to find a small set of synthetic images such that training a model on them reproduces the performance of the same model trained on a much larger dataset of real samples. Existing distillation methods focus on synthesizing datasets that enable training randomly i…

• SeSE: A Structural Information-Guided Uncertainty Quantification Framework for Hallucination Detection in LLMs

  Xingtao Zhao, Hao Peng, Dingli Su, Xianghua Zeng, Chunyang Liu, Jinzhi Liao, Philip S. Yu · 21. November 2025

  Reliable uncertainty quantification (UQ) is essential for deploying large language models (LLMs) in safety-critical scenarios, as it enables them to abstain from responding when uncertain, thereby avoiding hallucinating falsehoods. However, state-of-the-art UQ methods primarily rely on semantic prob…

• Sparse-PGD: A Unified Framework for Sparse Adversarial Perturbations Generation

  Xuyang Zhong, Chen Liu · 21. November 2025

  This work studies sparse adversarial perturbations, including both unstructured and structured ones. We propose a framework based on a white-box PGD-like attack method named Sparse-PGD to effectively and efficiently generate such perturbations. Furthermore, we combine Sparse-PGD with a black-box att…

• TopoReformer: Mitigating Adversarial Attacks Using Topological Purification in OCR Models

  Bhagyesh Kumar, A S Aravinthakashan, Akshat Satyanarayan, Ishaan Gakhar, Ujjwal Verma · 21. November 2025

  Adversarially perturbed images of text can cause sophisticated OCR systems to produce misleading or incorrect transcriptions from seemingly invisible changes to humans. Some of these perturbations even survive physical capture, posing security risks to high-stakes applications such as document proce…

• Securing AI Agents Against Prompt Injection Attacks

  Badrinath Ramakrishnan, Akshaya Balaji · 21. November 2025

  Retrieval-augmented generation (RAG) systems have become widely used for enhancing large language model capabilities, but they introduce significant security vulnerabilities through prompt injection attacks. We present a comprehensive benchmark for evaluating prompt injection risks in RAG-enabled AI…

• Detecting Sleeper Agents in Large Language Models via Semantic Drift Analysis

  Shahin Zanbaghi, Ryan Rostampour, Farhan Abid, Salim Al Jarmakani · 21. November 2025

  Large Language Models (LLMs) can be backdoored to exhibit malicious behavior under specific deployment conditions while appearing safe during training a phenomenon known as "sleeper agents." Recent work by Hubinger et al. demonstrated that these backdoors persist through safety training, yet no prac…

• Revisiting Model Inversion Evaluation: From Misleading Standards to Reliable Privacy Assessment

  Sy-Tuyen Ho, Koh Jun Hao, Ngoc-Bao Nguyen, Alexander Binder, Ngai-Man Cheung · 21. November 2025

  Model Inversion (MI) attacks aim to reconstruct information from private training data by exploiting access to machine learning models T. To evaluate such attacks, the standard evaluation framework relies on an evaluation model E, trained under the same task design as T. This framework has become th…

• LightDefense: A Lightweight Uncertainty-Driven Defense against Jailbreaks via Shifted Token Distribution

  Zhuoran Yang, Yanyong Zhang · 21. November 2025

  Large Language Models (LLMs) face threats from jailbreak prompts. Existing methods for defending against jailbreak attacks are primarily based on auxiliary models. These strategies, however, often require extensive data collection or training. We propose LightDefense, a lightweight defense mechanism…

• Active Measurement: Efficient Estimation at Scale

  Max Hamilton, Jinlin Lai, Wenlong Zhao, Subhransu Maji, Daniel Sheldon · 21. November 2025

  AI has the potential to transform scientific discovery by analyzing vast datasets with little human effort. However, current workflows often do not provide the accuracy or statistical guarantees that are needed. We introduce active measurement, a human-in-the-loop AI framework for scientific measure…