Adversarial Robustness in Machine Learning

Últimos artículos

• STRIATUM-CTF: A Protocol-Driven Agentic Framework for General-Purpose CTF Solving

  James Hugglestone, Samuel Jacob Chacko, Dawson Stoller, Ryan Schmidt, Xiuwen Liu · 25 de marzo de 2026

  Large Language Models (LLMs) have demonstrated potential in code generation, yet they struggle with the multi-step, stateful reasoning required for offensive cybersecurity operations. Existing research often relies on static benchmarks that fail to capture the dynamic nature of real-world vulnerabil…

• Not All Tokens Are Created Equal: Query-Efficient Jailbreak Fuzzing for LLMs

  Wenyu Chen, Xiangtao Meng, Chuanchao Zang, Li Wang, Xinyu Gao, Jianing Wang, Peng Zhan, Zheng Li, Shanqing Guo · 25 de marzo de 2026

  Large Language Models(LLMs) are widely deployed, yet are vulnerable to jailbreak prompts that elicit policy-violating outputs. Although prior studies have uncovered these risks, they typically treat all tokens as equally important during prompt mutation, overlooking the varying contributions of indi…

• Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

  Janaka Chathuranga Brahmanage, Akshat Kumar · 25 de marzo de 2026

  Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, th…

• T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search

  Hyomin Lee, Sangwoo Park, Yumin Choi, Sohyun An, Seanie Lee, Sung Ju Hwang · 25 de marzo de 2026

  While prior red-teaming efforts have focused on eliciting harmful text outputs from large language models (LLMs), such approaches fail to capture agent-specific vulnerabilities that emerge through multi-step tool execution, particularly in rapidly growing ecosystems such as the Model Context Protoco…

• Robust Safety Monitoring of Language Models via Activation Watermarking

  Toluwani Aremu, Daniil Ognev, Samuele Poppi, Nils Lukas · 25 de marzo de 2026

  Large language models (LLMs) can be misused to reveal sensitive information, such as weapon-making instructions or writing malware. LLM providers rely on $\emph{monitoring}$ to detect and flag unsafe behavior during inference. An open security challenge is $\emph{adaptive}$ adversaries who craft att…

• Adversarial Vulnerabilities in Neural Operator Digital Twins: Gradient-Free Attacks on Nuclear Thermal-Hydraulic Surrogates

  Samrendra Roy, Kazuma Kobayashi, Souvik Chakraborty, Rizwan-uddin, Syed Bahauddin Alam · 25 de marzo de 2026

  Operator learning models are rapidly emerging as the predictive core of digital twins for nuclear and energy systems, promising real-time field reconstruction from sparse sensor measurements. Yet their robustness to adversarial perturbations remains uncharacterized, a critical gap for deployment in …

• Improving Safety Alignment via Balanced Direct Preference Optimization

  Shiji Zhao, Mengyang Wang, Shukun Xiong, Fangzhou Chen, Qihui Zhu, Shouwei Ruan, Yisong Xiao, Ranjie Duan, Xun Chen, XingXing Wei · 25 de marzo de 2026

  With the rapid development and widespread application of Large Language Models (LLMs), their potential safety risks have attracted widespread attention. Reinforcement Learning from Human Feedback (RLHF) has been adopted to enhance the safety performance of LLMs. As a simple and effective alternative…

• Chain-of-Authorization: Internalizing Authorization into Large Language Models via Reasoning Trajectories

  Yang Li, Yule Liu, Xinlei He, Youjian Zhao, Qi Li, Ke Xu · 25 de marzo de 2026

  Large Language Models (LLMs) have become core cognitive components in modern artificial intelligence (AI) systems, combining internal knowledge with external context to perform complex tasks. However, LLMs typically treat all accessible data indiscriminately, lacking inherent awareness of knowledge …

• Sample Transform Cost-Based Training-Free Hallucination Detector for Large Language Models

  Zeyang Ding, Xinglin Hu, Jicong Fan · 25 de marzo de 2026

  Hallucinations in large language models (LLMs) remain a central obstacle to trustworthy deployment, motivating detectors that are accurate, lightweight, and broadly applicable. Since an LLM with a prompt defines a conditional distribution, we argue that the complexity of the distribution is an indic…

• Beyond the Mean: Distribution-Aware Loss Functions for Bimodal Regression

  Abolfazl Mohammadi-Seif, Carlos Soares, Rita P. Ribeiro, Ricardo Baeza-Yates · 25 de marzo de 2026

  Despite the strong predictive performance achieved by machine learning models across many application domains, assessing their trustworthiness through reliable estimates of predictive confidence remains a critical challenge. This issue arises in scenarios where the likelihood of error inferred from …

• TreeTeaming: Autonomous Red-Teaming of Vision-Language Models via Hierarchical Strategy Exploration

  Chunxiao Li, Lijun Li, Jing Shao · 25 de marzo de 2026

  The rapid advancement of Vision-Language Models (VLMs) has brought their safety vulnerabilities into sharp focus. However, existing red teaming methods are fundamentally constrained by an inherent linear exploration paradigm, confining them to optimizing within a predefined strategy set and preventi…

• TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space

  Gianmarco De Vita, Nargiz Humbatova, Paolo Tonella · 25 de marzo de 2026

  Testing Deep Learning (DL)-based systems is an open challenge. Although it is relatively easy to find inputs that cause a DL model to misbehave, the grouping of inputs by features that make the DL model under test fail is largely unexplored. Existing approaches for DL testing introduce perturbations…

• SafeSeek: Universal Attribution of Safety Circuits in Language Models

  Miao Yu, Siyuan Fu, Moayad Aloqaily, Zhenhong Zhou, Safa Otoum, Xing fan, Kun Wang, Yufei Guo, Qingsong Wen · 25 de marzo de 2026

  Mechanistic interpretability reveals that safety-critical behaviors (e.g., alignment, jailbreak, backdoor) in Large Language Models (LLMs) are grounded in specialized functional components. However, existing safety attribution methods struggle with generalization and reliability due to their relianc…

• Concept-based explanations of Segmentation and Detection models in Natural Disaster Management

  Samar Heydari, Jawher Said, Galip \"Umit Yolcu, Evgenii Kortukov, Elena Golimblevskaia, Evgenios Vlachos, Vasileios Mygdalis, Ioannis Pitas, Sebastian Lapuschkin, Leila Arras · 25 de marzo de 2026

  Deep learning models for flood and wildfire segmentation and object detection enable precise, real-time disaster localization when deployed on embedded drone platforms. However, in natural disaster management, the lack of transparency in their decision-making process hinders human trust required for…

• Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks

  Mat\'ias Pizarro, Raghavan Narasimhan, Asja Fischer · 25 de marzo de 2026

  With the increasing deployment of automated and agentic systems, ensuring the adversarial robustness of automatic speech recognition (ASR) models has become critical. We observe that changing the precision of an ASR model during inference reduces the likelihood of adversarial attacks succeeding. We …

• RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models

  Jiale Ding, Xiang Zheng, Yutao Wu, Cong Wang, Wei-Bin Lee, Ling Pan, Xingjun Ma, Yu-Gang Jiang · 25 de marzo de 2026

  As large language models (LLMs) are increasingly deployed as black-box components in real-world applications, red teaming has become essential for identifying potential risks. It tests LLMs with adversarial prompts to uncover vulnerabilities and improve safety alignment. Ideally, effective red teami…

• Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions

  Adri\'an Detavernier, Jasper De Bock · 25 de marzo de 2026

  We consider two approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We explain the conceptual differences between the two approaches, compare both approaches on a number of benchmark datasets and …

• CSTS: A Canonical Security Telemetry Substrate for AI-Native Cyber Detection

  Abdul Rahman · 25 de marzo de 2026

  AI-driven cybersecurity systems often fail under cross-environment deployment due to fragmented, event-centric telemetry representations. We introduce the Canonical Security Telemetry Substrate (CSTS), an entity-relational abstraction that enforces identity persistence, typed relationships, and temp…

• Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scores

  Zvi N. Badash, Yonatan Belinkov, Moti Freiman · 25 de marzo de 2026

  Large language models (LLMs) are often confidently wrong, making reliable uncertainty estimation (UE) essential. Output-based heuristics are cheap but brittle, while probing internal representations is effective yet high-dimensional and hard to transfer. We propose a compact, per-instance UE metho…

• Metaphor-based Jailbreak Attacks on Text-to-Image Models

  Chenyu Zhang, Lanjun Wang, Yiwen Ma, Wenhui Li, Yi Tu, An-An Liu · 25 de marzo de 2026

  Text-to-image (T2I) models commonly incorporate defense mechanisms to prevent the generation of sensitive images. Unfortunately, recent jailbreak attacks have shown that adversarial prompts can effectively bypass these mechanisms and induce T2I models to produce sensitive content, revealing critical…

• OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation

  Aarush Aggarwal, Akshat Tomar, Amritanshu Tiwari, Sargam Goyal · 24 de marzo de 2026

  Robust semantic segmentation is crucial for safe autonomous driving, yet deployed models remain vulnerable to black-box adversarial attacks when target weights are unknown. Most existing approaches either craft image-wide perturbations or optimize patches for a single architecture, which limits thei…

• CAMA: Exploring Collusive Adversarial Attacks in c-MARL

  Men Niu, Xinxin Fan, Quanliang Jing, Shaoye Luo, Yunfeng Lu · 24 de marzo de 2026

  Cooperative multi-agent reinforcement learning (c-MARL) has been widely deployed in real-world applications, such as social robots, embodied intelligence, UAV swarms, etc. Nevertheless, many adversarial attacks still exist to threaten various c-MARL systems. At present, the studies mainly focus on s…

• Before the Tool Call: Deterministic Pre-Action Authorization for Autonomous AI Agents

  Uchi Uchibeke · 24 de marzo de 2026

  AI agents today have passwords but no permission slips. They execute tool calls (fund transfers, database queries, shell commands, sub-agent delegation) with no standard mechanism to enforce authorization before the action executes. Current safety architectures rely on model alignment (probabilistic…

• Profit is the Red Team: Stress-Testing Agents in Strategic Economic Interactions

  Shouqiao Wang, Marcello Politi, Samuele Marro, Davide Crapis · 24 de marzo de 2026

  As agentic systems move into real-world deployments, their decisions increasingly depend on external inputs such as retrieved content, tool outputs, and information provided by other actors. When these inputs can be strategically shaped by adversaries, the relevant security risk extends beyond a fix…

• WARBENCH: A Comprehensive Benchmark for Evaluating LLMs in Military Decision-Making

  Zongjie Li, Chaozheng Wang, Yuchong Xie, Pingchuan Ma, Shuai Wang · 24 de marzo de 2026

  Large Language Models are increasingly being considered for deployment in safety-critical military applications. However, current benchmarks suffer from structural blindspots that systematically overestimate model capabilities in real-world tactical scenarios. Existing frameworks typically ignore st…