Adversarial Robustness in Machine Learning

Últimos artículos

• MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content

  Ruoqi Guo, Yi Liu, Gelei Deng, Yiheng Xiong, Yuekang Li, Ying Zhang, Leo Yu Zhang, Lida Zhao, Ji Jie, Yuxiao Lu · 28 de mayo de 2026

  Mobile graphical user interface (GUI) agents driven by vision-language models (VLMs) perceive the screen as rendered pixels and choose actions from what they see, so they cannot reliably separate trusted interface elements from user-generated content. We present MIRAGE (Mobile Injection of Realistic…

• Mitigating Adaptive Attacks against Reasoning Models with Activation Consistency Training

  Avidan Shah, Jannik Brinkmann, Rico Angell · 28 de mayo de 2026

  As LLMs gain stronger reasoning capabilities, their extended chain-of-thought introduces new degrees of complexity for defending against adversarial jailbreaks and prompt injection. We study consistency training, a family of fine-tuning objectives that enforce identical behavior on clean prompts and…

• Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deployment

  Gengyue Han, Yiheng Feng · 28 de mayo de 2026

  Due to limited resources and public safety concerns, deep reinforcement learning (RL) agents for many cyber-physical systems (e.g., autonomous vehicles) are first trained in simulators. However, when deployed in real world environments, they often suffer from performance degradation or safety violat…

• SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection

  Shuhao Chen, Weisen Jiang, Yeqi Gong, Shengda Luo, Chengxiang Zhuo, Zang Li, James T. Kwok, Yu Zhang · 28 de mayo de 2026

  Fine-tuning large language models often undermines their safety alignment, a problem further amplified by harmful fine-tuning attacks in which adversarial data removes safeguards and induces unsafe behaviors. We propose SPARD, a defense framework that integrates Safety-Projected Alternating optimiza…

• Blind PRNG Hijacking: An Undetectable Integrity-Preserving Attack Against LLM Watermarking

  Ziyang You, Huilong He, Xiaoke Yang, Xuxing Lu · 28 de mayo de 2026

  Cryptographic watermarking is a leading defense for attributing text generated by large language models (LLMs). Existing schemes, including KGW, Unigram, and DipMark, derive their security guarantees from the assumption that the underlying pseudo-random number generator (PRNG) is trustworthy. This w…

• Density-aware Sample-specific Attack

  Qiyuan Wang, Yao Li, Raymond K. W. Wong · 28 de mayo de 2026

  Despite recent progress in backdoor attacks, existing methods remain susceptible to post-training defenses that erase the backdoor through fine-tuning or pruning. We revisit the core objectives of backdoor attacks and derive principled criteria characterizing optimal sample-specific trigger construc…

• Cross-Entropy Games and Frost Training

  Arthur Renard, Franck Gabriel, Valentin Hartmann, Cl\'ement Hongler · 28 de mayo de 2026

  We present Frost Training, a method for improving Monte Carlo-based policy optimization for a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The key idea is to exploit the gradient of the reward function in embedding space. This signal is used in the Greedy Coordinate Gradient (GCG…

• Hallucination Behavior in Multimodal LLMs Across Agricultural Image Interpretation and Generation Tasks

  Partho Ghose, Al Bashir, Prem Raj, Azlan Zahid · 28 de mayo de 2026

  Large Language Models (LLMs) are being rapidly adopted in agricultural imaging applications, ranging from crop interpretation to synthetic field image generation. However, these models frequently exhibit hallucinations outputs that appear confident yet deviate from biological or environmental realit…

• Structure-Guided Visual Perturbation Neutralization for LVLMs

  Yuanhe Zhang, Xueting Wang, YanBin Ren, Haoran Gao, Xinhan Zheng, Zhenhong Zhou, Fanyu Meng, Li Sun, Sen Su · 28 de mayo de 2026

  Image inputs enable Large Vision Language Models (LVLMs) to perceive fine-grained visual information, but also introduce a pixel-level attack surface through which adversarial perturbations can elicit unsafe model behaviors. However, most existing defenses are designed for traditional computer visio…

• Disentangling Adversarial Prompts: A Semantic-Graph Defense for Robust LLM Security

  Xiang Fang, Wanlong Fang · 28 de mayo de 2026

  Large Language Models (LLMs) are increasingly vulnerable to adversarial prompts that exploit semantic ambiguities to bypass safety mechanisms, resulting in harmful or inappropriate outputs. Such attacks, including jailbreaking and prompt injection, pose significant risks to the integrity and availab…

• Models That Know How Evaluations Are Designed Score Safer

  Katharina Deckenbach, Haritz Puerto, Jonas Geiping, Sahar Abdelnabi · 28 de mayo de 2026

  The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings. Prior work has identified test-time contextual cues, such as hypothetical scenarios, as a source of verbalized evaluation awareness and subsequent behavioral shift. In this paper,…

• Operational AI Deployment Assurance: Governance-State Orchestration Under Threshold-Sensitive Deployment Conditions -- A Governance Framework for High-Stakes AI Systems

  Khalid Adnan Alsayed · 28 de mayo de 2026

  AI governance frameworks increasingly emphasize fairness, transparency, accountability, and lifecycle risk management in high-stakes domains. However, many current approaches remain observational, relying on static metric reporting, post-hoc auditing, and monitoring dashboards without directly gover…

• CRaFT: Circuit-Guided Refusal Feature Selection via Cross-Layer Transcoders

  Su-Hyeon Kim, Hyundong Jin, Yejin Lee, Yo-Sub Han · 28 de mayo de 2026

  While modern LLMs are aligned to refuse harmful requests, it is essential to understand the underlying mechanistic basis of this refusal behavior for model safety analysis. For example, steering-based jailbreak attacks exploit this by identifying and manipulating sparse, neuron-like refusal features…

• Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions

  Yuming Xu, Mingtao Zhang, Zhuohan Ge, Haoyang Li, Nicole Hu, Yongqi Zhang, Zhiyuan Wen, Jason Chen Zhang, Qing Li, Lei Chen · 28 de mayo de 2026

  Retrieval-augmented generation (RAG) extends large language models (LLMs) with external knowledge, but this access path also introduces security risks that existing work often conflates with inherent LLM flaws. We frame secure RAG as securing external knowledge access and organize the literature wit…

• Rethinking Calibration for Early-Exit Neural Networks

  Piotr Kubaty, Filip Szatkowski, Grzegorz Choczy\'nski, Eric Nalisnick, Bartosz W\'ojcik · 28 de mayo de 2026

  Early-exit neural networks (EENNs) accelerate inference by allowing intermediate classifiers to stop computation once predictions are confident enough. Most methods rely on confidence thresholds for exiting, and consequently, improving classifier calibration is widely assumed to improve performance.…

• When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models

  Dasol Choi, Alex Kwon · 28 de mayo de 2026

  Safety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often adhere to rigid rules even when a situational update flips which action is safe. We term this failure brittle safety. To diagnose it, we introduce context-flip evaluation, testing 12 models acr…

• Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents

  Yongxiang Li, Moxin Li, Zhixin Ma, Fengbin Zhu, Dongrui Liu, Wenjie Wang, Fuli Feng · 28 de mayo de 2026

  Large Language Model (LLM) agents remain vulnerable to safety threats from the external environment, where attackers inject adversarial content into external observations such as tool-returned data, webpages, or MCP context, causing harmful agentic behaviors such as unsafe actions or incorrect outpu…

• Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations

  Matteo Gioele Collu, Riccardo Conte, Alberto Giaretta, Denis Kleyko, Mauro Conti, Matteo Zavatteri, Roberto Confalonieri · 28 de mayo de 2026

  In this paper, we investigate whether refusal behavior can be predicted from LLM intermediate activations before decoding using linear probes trained on residual stream activations at each transformer block. We find that refusal is linearly decodable well before the final layer, indicating that safe…

• Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference

  Alan Ferrari · 28 de mayo de 2026

  Standard transformer architectures apply a single attention mechanism uniformly across all tokens and sequence positions, irrespective of local context or computational budget. We propose Meta-Attention, a framework that dynamically routes each token to the most appropriate attention strategy -- ful…

• Test-Time Collective Action: Proxy-Based Perturbations for Correcting Algorithmic Harms

  Meghana Bhange, Ulrich A\"ivodji, Elliot Creager · 28 de mayo de 2026

  When machine learning systems under-perform for particular subgroups, affected users typically have no way to correct these disparities without relying on platform-level fixes. Existing approaches to algorithmic fairness rely on provider-centric approaches to correct these failures, leaving users wi…

• TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling

  Jiaqian Li, Yanshu Li, Boxuan Zhang, Ruixiang Tang, Kuan-Hao Huang · 28 de mayo de 2026

  LLM agents increasingly operate through multi-turn tool use and environment interaction, where safety risks often emerge from intermediate steps long before they surface in the final outcome. Reactive auditing is therefore insufficient: post-hoc diagnosis frequently misses the chance to flag risks w…

• Entropy Distribution as a Fingerprint for Hallucinations in Generative Models

  Mattia J. Villani, Pranav Deshpande, Akshay Seshadri, Romina Yalovetzky, Niraj Kumar · 28 de mayo de 2026

  Large Language Models (LLMs) often generate factually incorrect outputs, commonly termed hallucinations, that undermine trust and limit deployment in high-stakes settings. Existing hallucination detection methods typically require multiple forward passes, or access to model internals. In this work, …

• SNARE: Adaptive Scenario Synthesis for Eliciting Overeager Behavior in Coding Agents

  Yubin Qu, Yi Liu, Gelei Deng, Yanjun Zhang, Yuekang Li, Ying Zhang, Leo Yu Zhang · 28 de mayo de 2026

  A coding agent executes a benign task as a sequence of shell, file, and network actions, any of which can quietly exceed the authorized scope while the task still completes. We call this overeager behavior: the prompt is not adversarial and the run succeeds, yet an out-of-scope step can leak credent…

• MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks

  Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-Wei Chang, Daniel Kang, Heng Ji · 28 de mayo de 2026

  Retrieval-augmented generation (RAG) has become a common practice in multimodal large language models (MLLM) to enhance factual grounding and reduce hallucination. Yet, its reliance on retrieval exposes MLLMs to knowledge poisoning attacks, in which adversaries deliberately inject malicious multimod…

• Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection

  Vijeta Deshpande, Tootiya Giyahchi, Veena Padmanabhan, Leman Akoglu, Anna Rumshisky · 28 de mayo de 2026

  Safety detection models require examples of HHH (Helpful, Harmless, Honest)-violating outputs for robust generalization, however such examples are scarce. Activation Steering (AS) has emerged as a data-efficient method for generating target-concept-aligned responses. We investigate whether AS can ge…