Adversarial Robustness in Machine Learning

Neueste Paper

• Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization

  Alexander Long, Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Yan Zuo, Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi, Sameera Ramasinghe · 25. Mai 2026

  We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model. In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to an…

• The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems

  Tanzim Ahad, Ismail Hossain, Md Jahangir Alam, Sai Puppala, Syed Bahauddin Alam, Sajedul Talukder · 25. Mai 2026

  Multi-agent AI pipelines typically assume that agent misconduct originates from model misalignment. We identify a structural failure in this assumption, the \emph{Misattribution Gap}, where memory-layer attacks produce behaviors indistinguishable from model failure, causing defenders to apply the wr…

• ArcMark: Distortion-Free Multi-Byte LLM Watermark via Optimal Transport

  Atefeh Gilani, Sajani Vithana, Carol Xuan Long, Oliver Kosut, Lalitha Sankar, Flavio P. Calmon · 25. Mai 2026

  Watermarking is an important tool for promoting the responsible use of large language models (LLMs). Existing watermarks insert a signal into generated tokens that either flags LLM-generated text (zero-bit watermarking) or encodes more complex messages (multi-bit watermarking). Though a number of re…

• WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents

  Zhixiang Guo, Siyuan Liang, Shi Fu, Cheng Guo, Andras Balogh, Mark Jelasity, Dacheng Tao · 25. Mai 2026

  Despite the growing use of world models as decision-making agents, their adversarial robustness remains underexplored due to the lack of dedicated automated evaluation methods. A key obstacle is that attack evaluation must be both accurate and efficient: weak manually tuned attacks can overestimate …

• Smoothed Elicitation Complexity for Approximate $\Gamma$-calibration of Discrete Classification Tasks

  Jessica Finocchiaro, Victor Ganson, Drona Khurana · 25. Mai 2026

  One prominent method of evaluating machine learning model trustworthiness is the notion of calibration. In the binary outcome setting, a probabilistic predictor is calibrated if outcomes are realized according to a model's distributional prediction, conditioned on this prediction. Straightforward ex…

• NeuroNL2LTL: A Neurosymbolic Framework for Natural Language Translation of Linear Temporal Logic

  Paapa Kwesi Quansah, Ernest Bonnah · 25. Mai 2026

  Effectively translating between natural language (NL) and formal logics like Linear Temporal Logic (LTL) requires expertise that limits formal verification's reach in safety-critical development. Template-based approaches sacrifice expressiveness for reliability; neural methods achieve fluency but p…

• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models

  Samar Fares, Klea Ziu, Toluwani Aremu, Nikita Durasov, Martin Tak\'a\v{c}, Pascal Fua, Karthik Nandakumar, Ivan Laptev · 25. Mai 2026

  Vision-Language Models (VLMs) are increasingly susceptible to sophisticated adversarial attacks, including adaptive strategies specifically designed to bypass existing defenses. To address this vulnerability, we propose MirrorCheck, a robust and model-agnostic detection framework that operates effec…

• Deja Vu in Plots: Leveraging Cross-Session Evidence with Retrieval-Augmented LLMs for Live Streaming Risk Assessment

  Yiran Qiao, Xiang Ao, Jing Chen, Yang Liu, Qiwei Zhong, Qing He · 25. Mai 2026

  The rise of live streaming has transformed online interaction, enabling massive real-time engagement but also exposing platforms to complex risks such as scams and coordinated malicious behaviors. Detecting these risks is challenging because harmful actions often accumulate gradually and recur acros…

• Boundary-targeted Membership Inference Attacks on Safety Classifiers

  Anthony Hughes, Alexander Goldberg, Prince Jha, Adam Perer, Nikolaos Aletras, Niloofar Mireshghallah · 25. Mai 2026

  Safety classifiers are essential safeguards within generative AI systems, filtering harmful content or identifying at-risk users when interacting with large language models. Despite their necessity, these models are trained on sensitive datasets including discussions of self-harm and mental health, …

• GradingAttack: Exposing Security Vulnerabilities in LLM Based Educational Grading Agents

  Xueyi Li, Zhuoneng Zhou, Zitao Liu, Yongdong Wu · 25. Mai 2026

  Large language models (LLMs) are increasingly deployed as educational agents for automatic short answer grading (ASAG) in real-world educational environments, significantly boosting assessment efficiency and scalability. However, when these grading agents operate ``in the wild'', their vulnerability…

• MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

  Joss Armstrong · 25. Mai 2026

  Foundation model agents increasingly operate in multi-agent deployments where a coordinator must decide which agent's response to trust. The standard approach weights agents by their self-reported confidence, but recent evidence shows that foundation model confidence is systematically mis-calibrated…

• CHASD: Language Increment-Calibrated Contrastive Decoding against Hallucination in LVLMs

  Xiaoyi Huang, Kejia Zhang, Zhiming Luo · 25. Mai 2026

  Large Vision-Language Models have shown strong multimodal reasoning capabilities, yet they remain susceptible to object hallucinations when language priors dominate insufficient or misaligned visual evidence. Training-free contrastive decoding methods mitigate this issue by comparing predictions fro…

• Low-Cost Hard-Label Adversarial Attack with Theoretical Foundations

  Jun Liu, Leo Yu Zhang, Fengpeng Li, Isao Echizen, Jiantao Zhou · 25. Mai 2026

  Hard-label black-box attacks, relying solely on top-1 predictions, represent one of the most challenging yet practically threat models. Despite recent progress, existing approaches face two key limitations: (1) they overlook the critical role of initialization, focusing primarily on optimization str…

• Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization

  Artur Back de Luca, Kimon Fountoulakis · 25. Mai 2026

  As state-of-the-art neural networks are deployed on reasoning and algorithmic tasks, exactness guarantees become increasingly important. However, high average-case accuracy can still mask inconsistent behaviors. This motivates exact certification, which asks for the smallest set of labeled examples …

• PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs

  Luze Sun, Anshuman Suri, Harsh Chaudhari, Cristina Nita-Rotaru, Alina Oprea · 25. Mai 2026

  When practitioners fine-tune LLMs on unvetted datasets, an adversary can exploit the data supply chain through task-level poisoning: inserting a small number of crafted instruction-response pairs that cause the model to embed attacker-specified entities, such as a country, in outputs for a targeted …

• Dirichlet-Based Monte Carlo Dropout for Uncertainty Estimation in Neural Networks

  Rouaa Hoblos (FEMTO-ST), Noura Dridi (FEMTO-ST), Noureddine Zerhouni (FEMTO-ST), Zeina Al Masry (FEMTO-ST) · 25. Mai 2026

  Traditional neural networks provide deterministic predictions without inherent uncertainty estimates. While Bayesian Neural Networks (BNNs) offer a principled approach to uncertainty quantification, their computational complexity limits scalability. Monte Carlo (MC) Dropout, initially introduced as …

• Dithering Defense: Adversarial Robustness of Vision Foundation Models via Multi-Level Floyd-Steinberg Dithering

  Yury Belousov, Brian Pulfer, Vitaliy Kinakh, Slava Voloshynovskiy · 25. Mai 2026

  Vision foundation models are widely used as frozen backbones across many downstream tasks, making them a single point of failure under adversarial attack. We study multi-level Floyd-Steinberg error-diffusion dithering as a lightweight, model-agnostic input transformation that disrupts adversarial pe…

• Lipschitz Optimization for Formal Verification of Homographies

  Jean-Guillaume Durand, Panagiotis Kouvaros, Maxime Gariel, Alessio Lomuscio · 25. Mai 2026

  The adoption of vision neural networks in regulated industries requires formal robustness guarantees, especially in safety-critical domains such as healthcare, autonomous vehicles, and aerospace. However, current approaches are confined to incomplete statistical verification or robustness to $\ell_p…

• Test-Time Training Undermines Safety Guardrails

  Simone Antonelli, Sadegh Akhondzadeh, Aleksandar Bojchevski · 25. Mai 2026

  Test-Time Training (TTT) is an emerging paradigm that enables models to adapt their parameters during inference, improving performance on tasks such as few-shot learning, retrieval-augmented generation, and complex reasoning. However, this dynamic adaptation introduces new vulnerabilities that adver…

• Sample-wise Targeted Adversarial Attacks on Test-time Adaptation

  Phuc Duc Nguyen, Quang Duc Nguyen · 25. Mai 2026

  Test-time adaptation (TTA) effectively counters distribution shifts but exposes models to adversarial manipulation via the unlabeled test stream. Existing class-wise targeted attacks remain impractical for stealthy exploitation in this setting: since TTA operates on batches, forcing a subset of samp…

• Through the Stealth Lens: Attention-Aware Defenses Against Poisoning in RAG

  Sarthak Choudhary, Nils Palumbo, Ashish Hooda, Krishnamurthy Dj Dvijotham, Somesh Jha · 25. Mai 2026

  Retrieval-augmented generation (RAG) systems are vulnerable to attacks that inject poisoned passages into the retrieved context, even at low corruption rates. We show that existing attacks are not designed to be stealthy, allowing reliable detection and mitigation. We formalize a distinguishability-…

• AI Security Research Should Better Incentivize Defense Research

  Youqian Zhang · 25. Mai 2026

  This work examines an imbalance in artificial intelligence (AI) security research: the field tends to produce more work on attacking AI systems than on defending them. Drawing on related academic papers, we find biased attack-to-defense ratios across subfields, including federated learning, speech r…

• Enhancing Deep Neural Network Reliability with Refinement and Calibration

  Ramya Hebbalaguppe, Ajay Shastry, Soumya Suvra Ghosal, Chetan Arora · 25. Mai 2026

  Although deep neural networks (DNNs) achieve high predictive accuracy, their confidence estimates are often unreliable, potentially compromising user trust in their decisions. This has motivated research on calibrated models, where calibration measures how well a model's predicted confidence aligns …

• SafeHarbor: Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

  Zhe Liu, Zonghao Ying, Wenxin Zhang, Quanchen Zou, Deyue Zhang, Dongdong Yang, Xiangzheng Zhang, Hao Peng · 25. Mai 2026

  Recent advances in foundation models have transformed LLMs from passive conversational systems into autonomous agents capable of reasoning and tool execution. While these capabilities unlock substantial practical value, they also introduce new security risks, as adversaries can manipulate agents int…

• $ECUAS_n$: A family of metrics for principled evaluation of uncertainty-augmented systems

  Lautaro Estienne, Erik Ernst, Mat\'ias Vera, Pablo Piantanida, Luciana Ferrer · 22. Mai 2026

  In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems -- i.e., systems that output both predic…