Adversarial Robustness in Machine Learning

Últimos artículos

• Attention Is Where You Attack

  Aviral Srivastava, Sourav Panda · 4 de mayo de 2026

  Safety-aligned large language models rely on RLHF and instruction tuning to refuse harmful requests, yet the internal mechanisms implementing safety behavior remain poorly understood. We introduce the Attention Redistribution Attack (ARA), a white-box adversarial attack that identifies safety-critic…

• Possibilistic Predictive Uncertainty for Deep Learning

  Yao Ni, Jeremie Houssineau, Yew Soon Ong, Piotr Koniusz · 4 de mayo de 2026

  Deep neural networks achieve impressive results across diverse applications, yet their overconfidence on unseen inputs necessitates reliable epistemic uncertainty modelling. Existing methods for uncertainty modelling face a fundamental dilemma: Bayesian approaches provide principled estimates but re…

• Language models recognize dropout and Gaussian noise applied to their activations

  Damiano Fornasiere, Mirko Bronzi, Spencer Kitts, Alessandro Palmas, Yoshua Bengio, Oliver Richardson · 4 de mayo de 2026

  We provide evidence that language models can detect, localize and, to a certain degree, verbalize the difference between perturbations applied to their activations. More precisely, we either (a) mask activations, simulating dropout, or (b) add Gaussian noise to them, at a target sentence. We then as…

• Skills as Verifiable Artifacts: A Trust Schema and a Biconditional Correctness Criterion for Human-in-the-Loop Agent Runtimes

  Alfredo Metere · 4 de mayo de 2026

  Agent skills -- structured packages of instructions, scripts, and references that augment a large language model (LLM) without modifying the model itself -- have moved from convenience to first-class deployment artifact. The runtime that loads them inherits the same problem package managers and oper…

• SHIFT: Robust Double Machine Learning for Average Dose-Response Functions under Heavy-Tailed Contamination

  Eichi Uehara · 4 de mayo de 2026

  Double-machine-learning pipelines for the Average Dose-Response Function rely on kernel-weighted local-linear smoothers, which inherit unbounded functional influence: a single outlier within a kernel window biases the curve across the entire window. We introduce SHIFT (Self-calibrated Heavy-tail Inl…

• Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Models

  Shubham Kumar, Narendra Ahuja · 4 de mayo de 2026

  Safety trained large language models (LLMs) can often be induced to answer harmful requests through jailbreak prompts. Because we lack a robust understanding of why LLMs are susceptible to jailbreaks, future frontier models operating more autonomously in higher-stakes settings may similarly be vulne…

• ARMOR 2025: A Military-Aligned Benchmark for Evaluating Large Language Model Safety Beyond Civilian Contexts

  Sydney Johns, Heng Jin, Chaoyu Zhang, Y. Thomas Hou, Wenjing Lou · 4 de mayo de 2026

  Large language models (LLMs) are now being explored for defense applications that require reliable and legally compliant decision support. They also hold significant potential to enhance decision making, coordination, and operational efficiency in military contexts. These uses demand evaluation meth…

• Jailbreaking Vision-Language Models Through the Visual Modality

  Aharon Azulay, Jan Dubi\'nski, Zhuoyun Li, Atharv Mittal, Yossi Gandelsman · 4 de mayo de 2026

  The visual modality of vision-language models (VLMs) is an underexplored attack surface for bypassing safety alignment. We introduce four jailbreak attacks exploiting the vision component: (1) encoding harmful instructions as visual symbol sequences with a decoding legend, (2) replacing harmful obje…

• Sentra-Guard: A Real-Time Multilingual Defense Against Adversarial LLM Prompts

  Md. Mehedi Hasan, Sk Tanzir Mehedi, Ziaur Rahman, Rafid Mostafiz, Md. Abir Hossain · 4 de mayo de 2026

  This paper presents a real-time modular defense system named Sentra-Guard. The system detects and mitigates jailbreak and prompt injection attacks targeting large language models (LLMs). The framework uses a hybrid architecture with FAISS-indexed SBERT embedding representations that capture the sema…

• Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance

  Minchan Kwon, Sunghyun Baek, Minseo Kim, Jaemyung Yu, Dongyoon Han, Junmo Kim · 4 de mayo de 2026

  Large Language Model (LLM) Red-Teaming, which proactively identifies vulnerabilities of LLMs, is an essential process for ensuring safety. Finding effective and diverse attacks in red-teaming is important, but achieving both is challenging. Generative Flow Networks (GFNs) that perform distribution m…

• Jailbroken Frontier Models Retain Their Capabilities

  Daniel Zhu, Zihan Wang, Jenny Bao, Jerry Wei · 4 de mayo de 2026

  As language model safeguards become more robust, attackers are pushed toward developing increasingly complex jailbreaks. Prior work has found that this complexity imposes a "jailbreak tax" that degrades the target model's task performance. We show that this tax scales inversely with model capability…

• Autonomous Systems Dependability in the era of AI: Design Challenges in Safety, Security, Reliability and Certification

  Behnaz Ranjbar, Kirankumar Raveendiran, Sudeep Pasricha, Samarjit Chakraborty, Cecilia Carbonelli, Akash Kumar · 4 de mayo de 2026

  The design of embedded safety-critical systems such as those used in next-generation automotive and autonomous platforms, is increasingly challenged by escalating system complexity, hardware-software heterogeneity, and the integration of intelligent, data-driven components. Ensuring dependability in…

• Why Self-Supervised Encoders Want to Be Normal

  Yuval Domb · 1 de mayo de 2026

  We develop a geometric and information-theoretic framework for encoder-decoder learning built on the Information Bottleneck (IB) principle. Recasting IB as a rate-distortion problem with Kullback-Leibler (KL) divergence as distortion, we show that the optimal representation at any distortion level i…

• Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection

  Prashant Kulkarni · 1 de mayo de 2026

  Multi-turn prompt injection follows a known attack path -- trust-building, pivoting, escalation but text-level defenses miss covert attacks where individual turns appear benign. We show this attack path leaves an activation-level signature in the model's residual stream: each phase shift moves the a…

• Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations

  Mingqian Zheng, Malia Morgan, Liwei Jiang, Carolyn Rose, Maarten Sap · 1 de mayo de 2026

  Current LLM safety alignment techniques improve model robustness against adversarial attacks, but overlook whether and how LLMs can recover helpfulness when benign users clarify their intent. We introduce CarryOnBench, the first interactive benchmark that measures whether LLMs can revise their inter…

• Security Attack and Defense Strategies for Autonomous Agent Frameworks: A Layered Review with OpenClaw as a Case Study

  Luyao Xu, Xiang Chen · 1 de mayo de 2026

  Autonomous agent frameworks built upon large language models (LLMs) are evolving into complex, tool-integrated, and continuously operating systems, introducing security risks beyond traditional prompt-level vulnerabilities. As this paradigm is still at an early stage of development, a timely and sys…

• Policy-Grounded Safety Evaluation of 20 Large Language Models

  Juan Manuel Contreras · 1 de mayo de 2026

  As large language models (LLMs) become increasingly integrated into real-world applications, scalable and rigorous safety evaluation is essential. This paper introduces Aymara AI, a programmatic platform for generating and administering customized, policy-grounded safety evaluations. Aymara AI trans…

• Understanding Adversarial Transferability in Vision-Language Models for Autonomous Driving: A Cross-Architecture Analysis

  David Fernandez, Pedram MohajerAnsari, Amir Salarpour, Mert D. Pese · 1 de mayo de 2026

  Vision-language models (VLMs) are increasingly used in autonomous driving because they combine visual perception with language-based reasoning, supporting more interpretable decision-making, yet their robustness to physical adversarial attacks, especially whether such attacks transfer across differe…

• From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation

  Guang Yang, Xing Hu, Xiang Chen, Xin Xi · 1 de mayo de 2026

  Multimodal large language models (MLLMs) are increasingly used to translate visual artifacts into code, from UI mockups into HTML to scientific plots into Python scripts. A circuit diagram can be viewed as a visual domain-specific language for hardware: it encodes timing, topology, and bit level sem…

• Focus Session: Autonomous Systems Dependability in the era of AI: Design Challenges in Safety, Security, Reliability and Certification

  Behnaz Ranjbar, Kirankumar Raveendiran, Sudeep Pasricha, Samarjit Chakraborty, Cecilia Carbonelli, Akash Kumar · 1 de mayo de 2026

  The design of embedded safety-critical systems such as those used in next-generation automotive and autonomous platforms, is increasingly challenged by escalating system complexity, hardware-software heterogeneity, and the integration of intelligent, data-driven components. Ensuring dependability in…

• CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs

  Elham Nasarian, Abhilash Neog, Kwok-Leung Tsui, Niyousha HosseiniChimeh · 1 de mayo de 2026

  Integrating large language models (LLMs) into patient-facing healthcare systems offers significant potential to improve access to medical information. However, ensuring clinical safety and factual reliability remains a critical challenge. In practice, AI-generated responses may be conditionally corr…

• Optimized Deferral for Imbalanced Settings

  Corinna Cortes, Anqi Mao, Mehryar Mohri, Yutao Zhong · 1 de mayo de 2026

  Learning algorithms can be significantly improved by routing complex or uncertain inputs to specialized experts, balancing accuracy with computational cost. This approach, known as learning to defer, is essential in domains like natural language generation, medical diagnosis, and computer vision, wh…

• From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systems

  Neha Nagaraja, Hayretdin Bahsi, Carlo R. da Cunha · 1 de mayo de 2026

  As large language models are integrated into autonomous robotic systems for task planning and control, compromised inputs or unsafe model outputs can propagate through the planning pipeline to physical-world consequences. Although prior work has studied robotic cybersecurity, adversarial perception …

• What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design

  Ivan Bercovich · 1 de mayo de 2026

  Terminal-agent benchmarks have become a primary signal for measuring the coding and system-administration capabilities of large language models. As the market for evaluation environments grows, so does the pressure to ship tasks quickly, often without thorough adversarial review of the verification …

• GAVEL: Towards Rule-Based Safety Through Activation Monitoring

  Shir Rozenfeld, Rahul Pankajakshan, Itay Zloczower, Eyal Lenga, Gilad Gressel, Yisroel Mirsky · 1 de mayo de 2026

  Large language models (LLMs) are increasingly paired with activation-based monitoring to detect and prevent harmful behaviors that may not be apparent at the surface-text level. However, existing activation safety approaches, trained on broad misuse datasets, struggle with poor precision, limited fl…