Adversarial Robustness in Machine Learning

Últimos artículos

• On the use of LLMs to generate a dataset of Neural Networks

  Nadia Daoudi, Jordi Cabot · 5 de febrero de 2026

  Neural networks are increasingly used to support decision-making. To verify their reliability and adaptability, researchers and practitioners have proposed a variety of tools and methods for tasks such as NN code verification, refactoring, and migration. These tools play a crucial role in guaranteei…

• GeoIB: Geometry-Aware Information Bottleneck via Statistical-Manifold Compression

  Weiqi Wang, Zhiyi Tian, Chenhan Zhang, Shui Yu · 5 de febrero de 2026

  Information Bottleneck (IB) is widely used, but in deep learning, it is usually implemented through tractable surrogates, such as variational bounds or neural mutual information (MI) estimators, rather than directly controlling the MI I(X;Z) itself. The looseness and estimator-dependent bias can mak…

• RASA: Routing-Aware Safety Alignment for Mixture-of-Experts Models

  Jiacheng Liang, Yuhui Wang, Tanqiu Jiang, Ting Wang · 5 de febrero de 2026

  Mixture-of-Experts (MoE) language models introduce unique challenges for safety alignment due to their sparse routing mechanisms, which can enable degenerate optimization behaviors under standard full-parameter fine-tuning. In our preliminary experiments, we observe that naively applying full-parame…

• Generative Neural Operators through Diffusion Last Layer

  Sungwon Park, Anthony Zhou, Hongjoong Kim, Amir Barati Farimani · 5 de febrero de 2026

  Neural operators have emerged as a powerful paradigm for learning discretization-invariant function-to-function mappings in scientific computing. However, many practical systems are inherently stochastic, making principled uncertainty quantification essential for reliable deployment. To address this…

• Anytime-Valid Conformal Risk Control

  Bror Hultberg, Dave Zachariah, Ant\^onio H. Ribeiro · 5 de febrero de 2026

  Prediction sets provide a means of quantifying the uncertainty in predictive tasks. Using held out calibration data, conformal prediction and risk control can produce prediction sets that exhibit statistically valid error control in a computationally efficient manner. However, in the standard formul…

• Cascading Robustness Verification: Toward Efficient Model-Agnostic Certification

  Mohammadreza Maleki, Rushendra Sidibomma, Arman Adibi, Reza Samavi · 5 de febrero de 2026

  Certifying neural network robustness against adversarial examples is challenging, as formal guarantees often require solving non-convex problems. Hence, incomplete verifiers are widely used because they scale efficiently and substantially reduce the cost of robustness verification compared to comple…

• From Data to Behavior: Predicting Unintended Model Behaviors Before Training

  Mengru Wang, Zhenqian Xu, Junfeng Fang, Yunzhi Yao, Shumin Deng, Huajun Chen, Ningyu Zhang · 5 de febrero de 2026

  Large Language Models (LLMs) can acquire unintended biases from seemingly benign training data even without explicit cues or malicious content. Existing methods struggle to detect such risks before fine-tuning, making post hoc evaluation costly and inefficient. To address this challenge, we introduc…

• How Few-shot Demonstrations Affect Prompt-based Defenses Against LLM Jailbreak Attacks

  Yanshu Wang, Shuaishuai Yang, Jingjing He, Tong Yang · 5 de febrero de 2026

  Large Language Models (LLMs) face increasing threats from jailbreak attacks that bypass safety alignment. While prompt-based defenses such as Role-Oriented Prompts (RoP) and Task-Oriented Prompts (ToP) have shown effectiveness, the role of few-shot demonstrations in these defense strategies remains …

• Comparative Insights on Adversarial Machine Learning from Industry and Academia: A User-Study Approach

  Vishruti Kakkad (Carnegie Mellon University), Paul Chung (University of California, San Diego), Hanan Hibshi (Carnegie Mellon University, King Abdulaziz University), Maverick Woo (Carnegie Mellon University) · 5 de febrero de 2026

  An exponential growth of Machine Learning and its Generative AI applications brings with it significant security challenges, often referred to as Adversarial Machine Learning (AML). In this paper, we conducted two comprehensive studies to explore the perspectives of industry professionals and studen…

• Representation Geometry as a Diagnostic for Out-of-Distribution Robustness

  Ali Zia, Farid Hazratian · 5 de febrero de 2026

  Robust generalization under distribution shift remains difficult to monitor and optimize in the absence of target-domain labels, as models with similar in-distribution accuracy can exhibit markedly different out-of-distribution (OOD) performance. While prior work has focused on training-time regular…

• ZKBoost: Zero-Knowledge Verifiable Training for XGBoost

  Nikolas Melissaris, Jiayi Xu, Antigoni Polychroniadou, Akira Takahashi, Chenkai Weng · 5 de febrero de 2026

  Gradient boosted decision trees, particularly XGBoost, are among the most effective methods for tabular data. As deployment in sensitive settings increases, cryptographic guarantees of model integrity become essential. We present ZKBoost, the first zero-knowledge proof of training (zkPoT) protocol f…

• The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment

  Zhexin Zhang, Yida Lu, Junfeng Fang, Junxiao Yang, Shiyao Cui, Hao Zhou, Fandong Meng, Jie Zhou, Hongning Wang, Minlie Huang, Tat-Seng Chua · 5 de febrero de 2026

  Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcem…

• A principled framework for uncertainty decomposition in TabPFN

  Sandra Fortini, Kenyon Ng, Sonia Petrone, Judith Rousseau, Susan Wei · 5 de febrero de 2026

  TabPFN is a transformer that achieves state-of-the-art performance on supervised tabular tasks by amortizing Bayesian prediction into a single forward pass. However, there is currently no method for uncertainty decomposition in TabPFN. Because it behaves, in an idealised limit, as a Bayesian in-cont…

• Inference-Time Backdoors via Hidden Instructions in LLM Chat Templates

  Ariel Fogel, Omer Hofman, Eilon Cohen, Roman Vainshtein · 5 de febrero de 2026

  Open-weight language models are increasingly used in production settings, raising new security challenges. One prominent threat in this context is backdoor attacks, in which adversaries embed hidden behaviors in language models that activate under specific conditions. Previous work has assumed that …

• Active Asymmetric Multi-Agent Multimodal Learning under Uncertainty

  Rui Liu, Pratap Tokekar, Ming Lin · 5 de febrero de 2026

  Multi-agent systems are increasingly equipped with heterogeneous multimodal sensors, enabling richer perception but introducing modality-specific and agent-dependent uncertainty. Existing multi-agent collaboration frameworks typically reason at the agent level, assume homogeneous sensing, and handle…

• Generative Adversarial Evasion and Out-of-Distribution Detection for UAV Cyber-Attacks

  Deepak Kumar Panda, Weisi Guo · 5 de febrero de 2026

  The growing integration of UAVs into civilian airspace underscores the need for resilient and intelligent intrusion detection systems (IDS), as traditional anomaly detection methods often fail to identify novel threats. A common approach treats unfamiliar attacks as out-of-distribution (OOD) samples…

• RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoning

  Zeming Wei, Qiaosheng Zhang, Xia Hu, Xingcheng Xu · 5 de febrero de 2026

  Large Reasoning Models (LRMs) have achieved tremendous success with their chain-of-thought (CoT) reasoning, yet also face safety issues similar to those of basic language models. In particular, while algorithms are designed to guide them to deliberately refuse harmful prompts with safe reasoning, th…

• When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs

  Hiskias Dingeto, Taeyoun Kwon, Dasol Choi, Bodam Kim, DongGeon Lee, Haon Park, JaeHoon Lee, Jongho Shin · 5 de febrero de 2026

  As large language models (LLMs) become increasingly integrated into daily life, audio has emerged as a key interface for human-AI interaction. However, this convenience also introduces new vulnerabilities, making audio a potential attack surface for adversaries. Our research introduces WhisperInject…

• Trust The Typical

  Debargha Ganguly, Sreehari Sankar, Biyao Zhang, Vikash Singh, Kanan Gupta, Harshini Kavuru, Alan Luo, Weicong Chen, Warren Morningstar, Raghu Machiraju, Vipin Chaudhary · 5 de febrero de 2026

  Current approaches to LLM safety fundamentally rely on a brittle cat-and-mouse game of identifying and blocking known threats via guardrails. We argue for a fresh approach: robust safety comes not from enumerating what is harmful, but from deeply understanding what is safe. We introduce Trust The Ty…

• Alignment Drift in Multimodal LLMs: A Two-Phase, Longitudinal Evaluation of Harm Across Eight Model Releases

  Casey Ford, Madison Van Doren, Emily Dix · 5 de febrero de 2026

  Multimodal large language models (MLLMs) are increasingly deployed in real-world systems, yet their safety under adversarial prompting remains underexplored. We present a two-phase evaluation of MLLM harmlessness using a fixed benchmark of 726 adversarial prompts authored by 26 professional red team…

• Beyond Rewards in Reinforcement Learning for Cyber Defence

  Elizabeth Bates, Chris Hicks, Vasilios Mavroudis · 5 de febrero de 2026

  Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and ince…

• Quantifying Risks in Multi-turn Conversation with Large Language Models

  Chengxiao Wang, Isha Chaudhary, Qian Hu, Weitong Ruan, Rahul Gupta, Gagandeep Singh · 5 de febrero de 2026

  Large Language Models (LLMs) can produce catastrophic responses in conversational settings that pose serious risks to public safety and security.Existing evaluations often fail to fully reveal these vulnerabilities because they rely on fixed attack prompt sequences, lack statistical guarantees, and …

• MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems

  Jonathan N\"other, Adish Singla, Goran Radanovic · 5 de febrero de 2026

  LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. …

• Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention

  Rakshith Vasudev, Melisa Russak, Dan Bikel, Waseem Alshikh · 4 de febrero de 2026

  Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentag…

• Relaxed Triangle Inequality for Kullback-Leibler Divergence Between Multivariate Gaussian Distributions

  Shiji Xiao, Yufeng Zhang, Chubo Liu, Yan Ding, Keqin Li, Kenli Li · 4 de febrero de 2026

  The Kullback-Leibler (KL) divergence is not a proper distance metric and does not satisfy the triangle inequality, posing theoretical challenges in certain practical applications. Existing work has demonstrated that KL divergence between multivariate Gaussian distributions follows a relaxed triangle…