Adversarial Robustness in Machine Learning

Últimos artículos

• Conformalized Percentile Interval: Finite Sample Validity and Improved Conditional Performance

  Ran Zou, Wanrong Zhu, Bin Nan · 6 de mayo de 2026

  Conformal prediction provides distribution-free predictive intervals with finite-sample marginal coverage. However, achieving conditional validity and interval efficiency (in terms of short interval length) remains challenging, particularly in complex settings with heteroskedasticity, skewed respons…

• GEM-FI: Gated Evidential Mixtures with Fisher Modulation

  Marco Mustafa Mohammed, Fatemeh Daneshfar, Pietro Li\`o · 6 de mayo de 2026

  Evidential Deep Learning (EDL) enables single-pass uncertainty estimation by predicting Dirichlet evidence, but it can remain overconfident and poorly calibrated, and it often fails to represent multi-modal epistemic uncertainty. We introduce Gated Evidential Mixtures (GEM), a family of models that …

• Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use

  Kunvar Thaman · 6 de mayo de 2026

  Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic short…

• Tracing the Dynamics of Refusal: Exploiting Latent Refusal Trajectories for Robust Jailbreak Detection

  Xulin Hu, Che Wang, Wei Yang Bryan Lim, Jianbo Gao, Zhong Chen · 6 de mayo de 2026

  Representation Engineering typically relies on static refusal vectors derived from terminal representations. We move beyond this paradigm, demonstrating that refusal is a dynamic and sparse process rather than a localized outcome. Using Causal Tracing, we uncover the Refusal Trajectory-a persistent …

• Adversarial Flow Matching for Imperceptible Attacks on End-to-End Autonomous Driving

  Xinyu Zeng, Xiangkun He, Lei Tao, Chen Lv, Hong Cheng · 6 de mayo de 2026

  Autonomous driving (AD) is evolving towards end-to-end (E2E) frameworks through two primary paradigms: monolithic models exemplified by Vision-Language-Action (VLA), and specialized modular architectures. Despite their divergent designs, both paradigms increasingly rely on Transformer backbones for …

• Self-Correction as Feedback Control: Error Dynamics, Stability Thresholds, and Prompt Interventions in LLMs

  Aofan Liu, Jingxiang Meng · 6 de mayo de 2026

  Iterative self-correction is increasingly deployed in agentic LLM systems, yet whether repeated refinement improves or degrades performance remains inconsistent across models. We recast self-correction as a closed-loop feedback-control problem in which the same model is both controller and plant, an…

• Improving Model Safety by Targeted Error Correction

  Abolfazl Mohammadi-Seif, Ricardo Baeza-Yates · 6 de mayo de 2026

  The widespread adoption of machine learning in critical applications demands techniques to mitigate high-consequence errors. Our method utilizes a dual-classifier GBDT pipeline to distinguish routine human-like errors from high-risk non-human misclassifications. Evaluated across three domains, anima…

• RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs

  Zhiyuan Xu, Joseph Gardiner, Sana Belguith, Lichao Wu · 6 de mayo de 2026

  Safety alignment is critical for the responsible deployment of large language models (LLMs). As Mixture-of-Experts (MoE) architectures are increasingly adopted to scale model capacity, understanding their safety robustness becomes essential. Existing adversarial attacks, however, have notable limita…

• Self-Mined Hardness for Safety Fine-Tuning

  Prakhar Gupta, Garv Shah, Donghua Zhang · 6 de mayo de 2026

  Safety fine-tuning of language models typically requires a curated adversarial dataset. We take a different approach: score each candidate prompt's difficulty by how often the target model's own rollouts are judged harmful, then fine-tune on the hardest prompts paired with the model's own non-jailbr…

• EvoJail: Evolutionary Diverse Jailbreak Prompt Generation for Large Language Models

  Rui Tang, Kaiyu Xu, Pengsen Cheng, Hao Ren, Haizhou Wang, Shuyu Jiang · 6 de mayo de 2026

  As LLMs continue to shape real-world applications, automated jailbreak generation becomes essential to reveal safety weaknesses and guide model improvement. Existing automatic jailbreak generation methods have not yet fully considered two important aspects: adaptability to evolving safety-finetuned …

• Exposing LLM Safety Gaps Through Mathematical Encoding:New Attacks and Systematic Analysis

  Haoyu Zhang, Mohammad Zandsalimy, Shanu Sushmita · 6 de mayo de 2026

  Large language models (LLMs) employ safety mechanisms to prevent harmful outputs, yet these defenses primarily rely on semantic pattern matching. We show that encoding harmful prompts as coherent mathematical problems -- using formalisms such as set theory, formal logic, and quantum mechanics -- byp…

• MEMSAD: Gradient-Coupled Anomaly Detection for Memory Poisoning in Retrieval-Augmented Agents

  Ishrith Gowda (University of California, Berkeley) · 6 de mayo de 2026

  Persistent external memory enables LLM agents to maintain context across sessions, yet its security properties remain formally uncharacterized. We formalize memory poisoning attacks on retrieval-augmented agents as a Stackelberg game with a unified evaluation framework spanning three attack classes …

• Set-Based Training of Neural Barrier Certificates for Safety Verification of Dynamical Systems

  Miriam Kranzlm\"uller, Lukas Koller, Tobias Ladner, Matthias Althoff · 6 de mayo de 2026

  Barrier certificates are scalar functions over the state space of dynamical systems that separate all unsafe states from all reachable states. The existence of a barrier certificate formally verifies the safety of the dynamical system. Recent approaches synthesize barrier certificates by iteratively…

• TsallisPGD: Adaptive Gradient Weighting for Adversarial Attacks on Semantic Segmentation

  Alexander Matyasko, Xin Lou, Indriyati Atmosukarto, Wei Zhang · 6 de mayo de 2026

  Attacking semantic segmentation models is significantly harder than image classification models because an attacker must flip thousands of pixel predictions simultaneously. Standard pixel-wise cross-entropy (CE) is ill-suited to this setting: it tends to overemphasize already-misclassified pixels, w…

• Two Calls, Two Moments, and the Vote-Accuracy Curve of Repeated LLM Inference

  Yi Liu · 6 de mayo de 2026

  Repeated sampling is a standard way to spend test-time compute, but its benefit is controlled by the latent distribution of correctness across examples, not by one-call accuracy alone. We study the binary correctness layer of repeated LLM inference under conditional-i.i.d. calls. One labeled call id…

• A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents

  Sheldon Yu, Yingcheng Sun, Hanqing Guo, Julian McAuley, Qianqian Tong · 6 de mayo de 2026

  Large Language Model (LLM)-powered agents demonstrate strong capabilities in autonomous task execution, tool use, and multi-step reasoning. However, their increasing autonomy also introduces a new attack surface: adversarial interactions can manipulate agent behavior through direct prompt injection,…

• Repurposing and Evaluating the (In)Feasibility of Dataset Poisoning enabled Watermarking for Contrastive Learning

  Zhiyang Dai, Yansong Gao, Boyu Kuang, Haodong Li, Qi Chang, Gaurav Varshney, Derek Abbott, Anmin Fu · 6 de mayo de 2026

  Contrastive learning (CL) reduces annotation cost via auto-derived supervisory signals. Since large-scale in-house CL datasets are infeasible, reliance on third-party or internet data is common. Recent studies show CL models are vulnerable to data-poisoning backdoor attacks, but their generalization…

• When Safety Geometry Collapses: Fine-Tuning Vulnerabilities in Agentic Guard Models

  Ismail Hossain, Sai Puppala, Jannatul Ferdaus, Md Jahangir Alam, Yoonpyo Lee, Syed Bahauddin Alam, Sajedul Talukder · 6 de mayo de 2026

  A guard model fine-tuned on entirely benign data can lose all safety alignment -- not through adversarial manipulation, but through standard domain specialization. We demonstrate this failure across three purpose-built safety classifiers -- LlamaGuard, WildGuard, and Granite Guardian -- deployed as …

• Multilingual Safety Alignment via Self-Distillation

  Ruiyang Qin, Qingzhuo Wang, Dongrui Liu, Qiang Li, Zhihua Wei, Wen Shen · 6 de mayo de 2026

  Large language models (LLMs) exhibit severe multilingual safety misalignment: they possess strong safeguards in high-resource languages but remain highly vulnerable to jailbreak attacks in low-resource languages. Current safety alignment methods generally rely on high-quality response data for each …

• Gated Subspace Inference for Transformer Acceleration

  Stephen J. Thomas · 6 de mayo de 2026

  A method is presented for accelerating inference in transformer language models by exploiting the low effective rank of the token activation manifold at each layer. The method decomposes each activation vector into a subspace component and a residual, computes the linear-layer output on the subspace…

• Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness

  Tung-Ling Li, Hongliang Liu · 5 de mayo de 2026

  RLHF-style alignment trains language models to refuse unsafe requests, but how much operational margin does this refusal rest on? We introduce the refusal-affirmation logit gap: the difference between the top refusal-token logit and the top affirmative-token logit at the first decoding step. This si…

• Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Data

  Ruiqi Xue, Lei Yuan, Kainuo Cheng, Jing-Wen Yang, Yang Yu · 5 de mayo de 2026

  Learning constraint-satisfying policies from offline data without risky online interaction is crucial for safety-critical decision making. Conventional methods typically learn cost value functions from abundant unsafe samples to define safety boundaries and penalize violations. However, in high-stak…

• Efficient Preference Poisoning Attack on Offline RLHF

  Chenye Yang, Weiyu Xu, Lifeng Lai · 5 de mayo de 2026

  Offline Reinforcement Learning from Human Feedback (RLHF) pipelines such as Direct Preference Optimization (DPO) train on a pre-collected preference dataset, which makes them vulnerable to preference poisoning attack. We study label flip attacks against log-linear DPO. We first illustrate that flipp…

• From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity

  Yee Zhing Liew, Andrew Huey Ping Tan, Anwar P. P Abdul Majeed · 5 de mayo de 2026

  Traditional hallucination detection fails on "Stubborn Hallucinations" -- errors where LLMs are confidently wrong. We propose a geometric solution: Embedding-Perturbed Gradient Sensitivity (EPGS). We hypothesize that while robust facts reside in flat minima, stubborn hallucinations sit in sharp mini…

• GEODE: Angle-Adaptive OOD Detection with Universal Scorer Compatibility

  Bruno Abrahao · 5 de mayo de 2026

  Outlier Exposure (OE) is among the strongest training-based OOD detectors on standard benchmarks but exhibits scorer-dependent tradeoffs (e.g., strong on MSP, weak on KNN) and requires curated auxiliary data. We show why OE works: its features sit at the same geometric locus as real near-OOD data, w…