Adversarial Robustness in Machine Learning

Neueste Paper

• Security in LLM-as-a-Judge: A Comprehensive SoK

  Aiman Almasoud, Antony Anju, Marco Arazzi, Mert Cihangiroglu, Vignesh Kumar Kembu, Serena Nicolazzo, Antonino Nocera, Vinod P., Saraga Sakthidharan · 1. April 2026

  LLM-as-a-Judge (LaaJ) is a novel paradigm in which powerful language models are used to assess the quality, safety, or correctness of generated outputs. While this paradigm has significantly improved the scalability and efficiency of evaluation processes, it also introduces novel security risks and …

• Hallucination-aware intermediate representation edit in large vision-language models

  Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang, Yanning Zhang · 1. April 2026

  Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retrain…

• Trojan-Speak: Bypassing Constitutional Classifiers with No Jailbreak Tax via Adversarial Finetuning

  Bilgehan Sel, Xuanli He, Alwin Peng, Ming Jin, Jerry Wei · 1. April 2026

  Fine-tuning APIs offered by major AI providers create new attack surfaces where adversaries can bypass safety measures through targeted fine-tuning. We introduce Trojan-Speak, an adversarial fine-tuning method that bypasses Anthropic's Constitutional Classifiers. Our approach uses curriculum learnin…

• BenchScope: How Many Independent Signals Does Your Benchmark Provide?

  Tommy Sha, Stella Zhao · 1. April 2026

  AI evaluation suites often report many scores without checking whether those scores carry independent information. We introduce Effective Dimensionality (ED), the participation ratio of a centered benchmark-score spectrum, as a fast, population-conditional upper-bound diagnostic of measurement bread…

• GUARD-SLM: Token Activation-Based Defense Against Jailbreak Attacks for Small Language Models

  Md Jueal Mia, Joaquin Molto, Yanzhao Wu, M. Hadi Amini · 1. April 2026

  Small Language Models (SLMs) are emerging as efficient and economically viable alternatives to Large Language Models (LLMs), offering competitive performance with significantly lower computational costs and latency. These advantages make SLMs suitable for resource-constrained and efficient deploymen…

• SNEAKDOOR: Stealthy Backdoor Attacks against Distribution Matching-based Dataset Condensation

  He Yang, Dongyi Lv, Song Ma, Wei Xi, Jizhong Zhao · 1. April 2026

  Dataset condensation aims to synthesize compact yet informative datasets that retain the training efficacy of full-scale data, offering substantial gains in efficiency. Recent studies reveal that the condensation process can be vulnerable to backdoor attacks, where malicious triggers are injected in…

• AGFT: Alignment-Guided Fine-Tuning for Zero-Shot Adversarial Robustness of Vision-Language Models

  Yubo Cui, Xianchao Guan, Zijun Xiong, Zheng Zhang · 1. April 2026

  Pre-trained vision-language models (VLMs) exhibit strong zero-shot generalization but remain vulnerable to adversarial perturbations. Existing classification-guided adversarial fine-tuning methods often disrupt pre-trained cross-modal alignment, weakening visual-textual correspondence and degrading …

• Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks

  Chong Xiang, Drew Zagieboylo, Shaona Ghosh, Sanjay Kariyappa, Kai Greshake, Hanshen Xiao, Chaowei Xiao, G. Edward Suh · 1. April 2026

  AI agents, predominantly powered by large language models (LLMs), are vulnerable to indirect prompt injection, in which malicious instructions embedded in untrusted data can trigger dangerous agent actions. This position paper discusses our vision for system-level defenses against indirect prompt in…

• Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

  Yicheng Cai, Mitchell John DeStefano, Guodong Dong, Pulkit Handa, Peng Liu, Tejas Singhal, Peiyu Tseng, Winston Jen White · 1. April 2026

  As Large Language Models (LLMs) and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to ac…

• Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Design Implications for Reward-Hacking Mitigation

  Nathan Heath · 1. April 2026

  Myopic Optimization with Non-myopic Approval (MONA) mitigates multi-step reward hacking by restricting the agent's planning horizon while supplying far-sighted approval as a training signal~\cite{farquhar2025mona}. The original paper identifies a critical open question: how the method of constructin…

• Adversarial Prompt Injection Attack on Multimodal Large Language Models

  Meiwen Ding, Song Xia, Chenqi Kong, Xudong Jiang · 1. April 2026

  Although multimodal large language models (MLLMs) are increasingly deployed in real-world applications, their instruction-following behavior leaves them vulnerable to prompt injection attacks. Existing prompt injection methods predominantly rely on textual prompts or perceptible visual prompts that …

• Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs

  Seine A. Shintani · 31. März 2026

  Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generaliz…

• AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback

  Oliver D\"urr · 31. März 2026

  We present AutoStan, a framework in which a command-line interface (CLI) coding agent autonomously builds and iteratively improves Bayesian models written in Stan. The agent operates in a loop, writing a Stan model file, executing MCMC sampling, then deciding whether to keep or revert each change ba…

• Lipschitz verification of neural networks through training

  Simon Kuang, Yuezhu Xu, S. Sivaranjani, Xinfan Lin · 31. März 2026

  The global Lipschitz constant of a neural network governs both adversarial robustness and generalization. Conventional approaches to ``certified training" typically follow a train-then-verify paradigm: they train a network and then attempt to bound its Lipschitz constant. Because the efficient `…

• Detection of Adversarial Attacks in Robotic Perception

  Ziad Sharawy, Mohammad Nakshbandiand, Sorin Mihai Grigorescu · 31. März 2026

  Deep Neural Networks (DNNs) achieve strong performance in semantic segmentation for robotic perception but remain vulnerable to adversarial attacks, threatening safety-critical applications. While robustness has been studied for image classification, semantic segmentation in robotic contexts require…

• InkDrop: Invisible Backdoor Attacks Against Dataset Condensation

  He Yang, Dongyi Lv, Song Ma, Wei Xi, Zhi Wang, Hanlin Gu, Yajie Wang · 31. März 2026

  Dataset Condensation (DC) is a data-efficient learning paradigm that synthesizes small yet informative datasets, enabling models to match the performance of full-data training. However, recent work exposes a critical vulnerability of DC to backdoor attacks, where malicious patterns (\textit{e.g.}, t…

• Energy Score-Guided Neural Gaussian Mixture Model for Predictive Uncertainty Quantification

  Yang Yang, Chunlin Ji, Haoyang Li, Ke Deng · 31. März 2026

  Quantifying predictive uncertainty is essential for real world machine learning applications, especially in scenarios requiring reliable and interpretable predictions. Many common parametric approaches rely on neural networks to estimate distribution parameters by optimizing the negative log likelih…

• Next-Token Prediction and Regret Minimization

  Mehryar Mohri, Clayton Sanford, Jon Schneider, Kiran Vodrahalli, Yifan Wu · 31. März 2026

  We consider the question of how to employ next-token prediction algorithms in adversarial online decision-making environments. Specifically, if we train a next-token prediction model on a distribution $\mathcal{D}$ over sequences of opponent actions, when is it the case that the induced online decis…

• Squish and Release: Exposing Hidden Hallucinations by Making Them Surface as Safety Signals

  Nathaniel Oh, Paul Attie · 31. März 2026

  Language models detect false premises when asked directly but absorb them under conversational pressure, producing authoritative professional output built on errors they already identified. This failure - order-gap hallucination - is invisible to output inspection because the error migrates into the…

• Information-Theoretic Limits of Safety Verification for Self-Improving Systems

  Arsenios Scrivens · 31. März 2026

  Can a safety gate permit unbounded beneficial self-modification while maintaining bounded cumulative risk? We formalize this question through dual conditions -- requiring sum delta_n < infinity (bounded risk) and sum TPR_n = infinity (unbounded utility) -- and establish a theory of their (in)compati…

• Robust Smart Contract Vulnerability Detection via Contrastive Learning-Enhanced Granular-ball Training

  Zeli Wang, Qingxuan Yang, Shuyin Xia, Yueming Wu, Bo Liu, Longlong Lin · 31. März 2026

  Deep neural networks (DNNs) have emerged as a prominent approach for detecting smart contract vulnerabilities, driven by the growing contract datasets and advanced deep learning techniques. However, DNNs typically require large-scale labeled datasets to model the relationships between contract featu…

• Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey

  Bhavuk Jain, Sercan \"O. Ar{\i}k, Hardeo K. Thakur · 31. März 2026

  Multimodal large language models (MLLMs) integrate information from multiple modalities such as text, images, audio, and video, enabling complex capabilities such as visual question answering and audio translation. While powerful, this increased expressiveness introduces new and amplified vulnerabil…

• A Survey of Zero-Knowledge Proof Based Verifiable Machine Learning

  Zhizhi Peng, Chonghe Zhao, Taotao Wang, Guofu Liao, Zibin Lin, Yifeng Liu, Bin Cao, Long Shi, Qing Yang, Shengli Zhang · 31. März 2026

  Machine learning is increasingly deployed through outsourced and cloud-based pipelines, which improve accessibility but also raise concerns about computational integrity, data privacy, and model confidentiality. Zero-knowledge proofs (ZKPs) provide a compelling foundation for verifiable machine lear…

• FL-PBM: Pre-Training Backdoor Mitigation for Federated Learning

  Osama Wehbi, Sarhad Arisdakessian, Omar Abdel Wahab, Azzam Mourad, Hadi Otrok, Jamal Bentahar · 31. März 2026

  Backdoor attacks pose a significant threat to the integrity and reliability of Artificial Intelligence (AI) models, enabling adversaries to manipulate model behavior by injecting poisoned data with hidden triggers. These attacks can lead to severe consequences, especially in critical applications su…

• Evasion Adversarial Attacks Remain Impractical Against ML-based Network Intrusion Detection Systems, Especially Dynamic Ones

  Mohamed elShehaby, Ashraf Matrawy · 31. März 2026

  Machine Learning (ML) has become pervasive, and its deployment in Network Intrusion Detection Systems (NIDS) is inevitable due to its automated nature and high accuracy compared to traditional models in processing and classifying large volumes of data. However, ML has been found to have several flaw…