Adversarial Robustness in Machine Learning

Últimos artículos

• Certified Learning under Distribution Shift: Sound Verification and Identifiable Structure

  Chandrasekhar Gokavarapu (Mathematics, Government College), Sudhakar Gadde (Mathematics, Government College), Y. Rajasekhar (Mathematics, Government College), S. R. Bhargava (Mathematics, Government College) · 23 de febrero de 2026

  Proposition. Let $f$ be a predictor trained on a distribution $P$ and evaluated on a shifted distribution $Q$. Under verifiable regularity and complexity constraints, the excess risk under shift admits an explicit upper bound determined by a computable shift metric and model parameters. We develop a…

• On the Adversarial Robustness of Discrete Image Tokenizers

  Rishika Bhagwatkar, Irina Rish, Nicolas Flammarion, Francesco Croce · 23 de febrero de 2026

  Discrete image tokenizers encode visual inputs as sequences of tokens from a finite vocabulary and are gaining popularity in multimodal systems, including encoder-only, encoder-decoder, and decoder-only models. However, unlike CLIP encoders, their vulnerability to adversarial attacks has not been ex…

• Can LLM Safety Be Ensured by Constraining Parameter Regions?

  Zongmin Li, Jian Su, Farah Benamara, Aixin Sun · 23 de febrero de 2026

  Large language models (LLMs) are often assumed to contain ``safety regions'' -- parameter subsets whose modification directly influences safety behaviors. We conduct a systematic evaluation of four safety region identification methods spanning different parameter granularities, from individual weigh…

• Uncertainty Estimation by Flexible Evidential Deep Learning

  Taeseong Yoon, Heeyoung Kim · 23 de febrero de 2026

  Uncertainty quantification (UQ) is crucial for deploying machine learning models in high-stakes applications, where overconfident predictions can lead to serious consequences. An effective UQ method must balance computational efficiency with the ability to generalize across diverse scenarios. Eviden…

• Trojans in Artificial Intelligence (TrojAI) Final Report

  Kristopher W. Reese, Taylor Kulp-McDowall, Michael Majurski, Tim Blattner, Derek Juba, Peter Bajcsy, Antonio Cardone, Philippe Dessauw, Alden Dima, Anthony J. Kearsley, Melinda Kleczynski, Joel Vasanth, Walid Keyrouz, Chace Ashcraft, Neil Fendley, Ted Staley, Trevor Stout, Josh Carney, Greg Canal, Will Redman, Aurora Schmidt, Cameron Hickert, William Paul, Jared Markowitz, Nathan Drenkow, David Shriver, Marissa Connor, Keltin Grimes, Marco Christiani, Hayden Moore, Jordan Widjaja, Kasimir Gabert, Uma Balakrishnan, Satyanadh Gundimada, John Jacobellis, Sandya Lakkur, Vitus Leung, Jon Roose, Casey Battaglino, Farinaz Koushanfar, Greg Fields, Xihe Gu, Yaman Jandali, Xinqiao Zhang, Akash Vartak, Tim Oates, Ben Erichson, Michael Mahoney, Rauf Izmailov, Xiangyu Zhang, Guangyu Shen, Siyuan Cheng, Shiqing Ma, XiaoFeng Wang, Haixu Tang, Di Tang, Xiaoyi Chen, Zihao Wang, Rui Zhu, Susmit Jha, Xiao Lin, Manoj Acharya, Wenchao Li, Chao Chen · 23 de febrero de 2026

  The Intelligence Advanced Research Projects Activity (IARPA) launched the TrojAI program to confront an emerging vulnerability in modern artificial intelligence: the threat of AI Trojans. These AI trojans are malicious, hidden backdoors intentionally embedded within an AI model that can cause a syst…

• Provable Adversarial Robustness in In-Context Learning

  Di Zhang · 23 de febrero de 2026

  Large language models adapt to new tasks through in-context learning (ICL) without parameter updates. Current theoretical explanations for this capability assume test tasks are drawn from a distribution similar to that seen during pretraining. This assumption overlooks adversarial distribution shift…

• A Geometric Probe of the Accuracy-Robustness Trade-off: Sharp Boundaries in Symmetry-Breaking Dimensional Expansion

  Yu Bai, Zhe Wang, Jiarui Zhang, Dong-Xiao Zhang, Yinjun Gao, Jun-Jie Zhang · 23 de febrero de 2026

  The trade-off between clean accuracy and adversarial robustness is a pervasive phenomenon in deep learning, yet its geometric origin remains elusive. In this work, we utilize Symmetry-Breaking Dimensional Expansion (SBDE) as a controlled probe to investigate the mechanism underlying this trade-off. …

• PRISM-FCP: Byzantine-Resilient Federated Conformal Prediction via Partial Sharing

  Ehsan Lari, Reza Arablouei, Stefan Werner · 23 de febrero de 2026

  We propose PRISM-FCP (Partial shaRing and robust calIbration with Statistical Margins for Federated Conformal Prediction), a Byzantine-resilient federated conformal prediction framework that utilizes partial model sharing to improve robustness against Byzantine attacks during both model training and…

• FENCE: A Financial and Multimodal Jailbreak Detection Dataset

  Mirae Kim, Seonghun Jeong, Youngjun Kwak · 23 de febrero de 2026

  Jailbreaking poses a significant risk to the deployment of Large Language Models (LLMs) and Vision Language Models (VLMs). VLMs are particularly vulnerable because they process both text and images, creating broader attack surfaces. However, available resources for jailbreak detection are scarce, pa…

• Asking Forever: Universal Activations Behind Turn Amplification in Conversational LLMs

  Zachary Coalson, Bo Fang, Sanghyun Hong · 23 de febrero de 2026

  Multi-turn interaction length is a dominant factor in the operational costs of conversational LLMs. In this work, we present a new failure mode in conversational LLMs: turn amplification, in which a model consistently prolongs multi-turn interactions without completing the underlying task. We show t…

• NeST: Neuron Selective Tuning for LLM Safety

  Sasha Behrouzi, Lichao Wu, Mohamadreza Rostami, Ahmad-Reza Sadeghi · 20 de febrero de 2026

  Safety alignment is essential for the responsible deployment of large language models (LLMs). Yet, existing approaches often rely on heavyweight fine-tuning that is costly to update, audit, and maintain across model families. Full fine-tuning incurs substantial computational and storage overhead, wh…

• SoK: DARPA's AI Cyber Challenge (AIxCC): Competition Design, Architectures, and Lessons Learned

  Cen Zhang, Younggi Park, Fabian Fleischer, Yu-Fu Fu, Jiho Kim, Dongkwan Kim, Youngjoon Kim, Qingxiao Xu, Andrew Chin, Ze Sheng, Hanqing Zhao, Brian J. Lee, Joshua Wang, Michael Pelican, David J. Musliner, Jeff Huang, Jon Silliman, Mikel Mcdaniel, Jefferson Casavant, Isaac Goldthwaite, Nicholas Vidovich, Matthew Lehman, Taesoo Kim · 20 de febrero de 2026

  DARPA's AI Cyber Challenge (AIxCC, 2023--2025) is the largest competition to date for building fully autonomous cyber reasoning systems (CRSs) that leverage recent advances in AI -- particularly large language models (LLMs) -- to discover and remediate vulnerabilities in real-world open-source softw…

• Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis

  Scott Thornton · 20 de febrero de 2026

  AI-assisted code review is widely used to detect vulnerabilities before production release. Prior work shows that adversarial prompt manipulation can degrade large language model (LLM) performance in code generation. We test whether similar comment-based manipulation misleads LLMs during vulnerabili…

• Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding

  Rahul Thomas, Teo Kitanovski, Micah Goldblum, Arka Pal · 20 de febrero de 2026

  Multi-path speculative decoding accelerates lossless sampling from a target model by using a cheaper draft model to generate a draft tree of tokens, and then applies a verification algorithm that accepts a subset of these. While prior work has proposed various verification algorithms for i.i.d rollo…

• Cert-SSBD: Certified Backdoor Defense with Sample-Specific Smoothing Noises

  Ting Qiao, Yingjia Wang, Xing Liu, Sixing Wu, Jianbin Li, Yiming Li · 20 de febrero de 2026

  Deep neural networks (DNNs) are vulnerable to backdoor attacks, where an attacker manipulates a small portion of the training data to implant hidden backdoors into the model. The compromised model behaves normally on clean samples but misclassifies backdoored samples into the attacker-specified targ…

• Fail-Closed Alignment for Large Language Models

  Zachary Coalson, Beth Sohler, Aiden Gabriel, Sanghyun Hong · 20 de febrero de 2026

  We identify a structural weakness in current large language model (LLM) alignment: modern refusal mechanisms are fail-open. While existing approaches encode refusal behaviors across multiple latent features, suppressing a single dominant feature$-$via prompt-based jailbreaks$-$can cause alignment to…

• Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning

  Jyotin Goel, Souvik Maji, Pratik Mazumder · 20 de febrero de 2026

  Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates. Existing defenses often offer limited protection or force a trade-off between safety and utility. We introduce a training…

• IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

  Priyaranjan Pattnayak, Sanchari Chowdhuri · 20 de febrero de 2026

  Safety alignment of large language models (LLMs) is mostly evaluated in English and contract-bound, leaving multilingual vulnerabilities understudied. We introduce \textbf{Indic Jailbreak Robustness (IJR)}, a judge-free benchmark for adversarial safety across 12 Indic and South Asian languages (2.1 …

• LLM4Cov: Execution-Aware Agentic Learning for High-coverage Testbench Generation

  Hejia Zhang, Zhongming Yu, Chia-Tung Ho, Haoxing Ren, Brucek Khailany, Jishen Zhao · 20 de febrero de 2026

  Execution-aware LLM agents offer a promising paradigm for learning from tool feedback, but such feedback is often expensive and slow to obtain, making online reinforcement learning (RL) impractical. High-coverage hardware verification exemplifies this challenge due to its reliance on industrial simu…

• Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioning

  Vishal Srivastava · 20 de febrero de 2026

  Black-box safety evaluation of AI systems assumes model behavior on test distributions reliably predicts deployment performance. We formalize and challenge this assumption through latent context-conditioned policies -- models whose outputs depend on unobserved internal variables that are rare under …

• Automating Agent Hijacking via Structural Template Injection

  Xinhao Deng, Jiaqing Wu, Miao Chen, Yue Xiao, Ke Xu, Qi Li · 20 de febrero de 2026

  Agent hijacking, highlighted by OWASP as a critical threat to the Large Language Model (LLM) ecosystem, enables adversaries to manipulate execution by injecting malicious instructions into retrieved content. Most existing attacks rely on manually crafted, semantics-driven prompt manipulation, which …

• Discovering Universal Activation Directions for PII Leakage in Language Models

  Leo Marchyok, Zachary Coalson, Sungho Keum, Sooel Son, Sanghyun Hong · 20 de febrero de 2026

  Modern language models exhibit rich internal structure, yet little is known about how privacy-sensitive behaviors, such as personally identifiable information (PII) leakage, are represented and modulated within their hidden states. We present UniLeak, a mechanistic-interpretability framework that id…

• Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study

  Dylan Bouchard, Mohit Singh Chauhan, Viren Bajaj, David Skarbrevik · 20 de febrero de 2026

  Uncertainty quantification has emerged as an effective approach to closed-book hallucination detection for LLMs, but existing methods are largely designed for short-form outputs and do not generalize well to long-form generation. We introduce a taxonomy for fine-grained uncertainty quantification in…

• Jolt Atlas: Verifiable Inference via Lookup Arguments in Zero Knowledge

  Wyatt Benno, Alberto Centelles, Antoine Douchet, Khalil Gibran · 20 de febrero de 2026

  We present Jolt Atlas, a zero-knowledge machine learning (zkML) framework that extends the Jolt proving system to model inference. Unlike zkVMs (zero-knowledge virtual machines), which emulate CPU instruction execution, Jolt Atlas adapts Jolt's lookup-centric approach and applies it directly to ONNX…

• Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

  Xiaohan Zhao, Zhaoyi Li, Yaxin Luo, Jiacheng Cui, Zhiqiang Shen · 20 de febrero de 2026

  Black-box adversarial attacks on Large Vision-Language Models (LVLMs) are challenging due to missing gradients and complex multimodal boundaries. While prior state-of-the-art transfer-based approaches like M-Attack perform well using local crop-level matching between source and target images, we fin…