Natural Language Processing Techniques

Neueste Paper

• Can I Have Your Order? Monte-Carlo Tree Search for Slot Filling Ordering in Diffusion Language Models

  Joshua Ong Jun Leang, Yu Zhao, Mihaela C\u{a}t\u{a}lina Stoian, Wenda Li, Shay B. Cohen, Eleonora Giunchiglia · 16. Februar 2026

  While plan-and-infill decoding in Masked Diffusion Models (MDMs) shows promise for mathematical and code reasoning, performance remains highly sensitive to slot infilling order, often yielding substantial output variance. We introduce McDiffuSE, a framework that formulates slot selection as decision…

• Soft Contamination Means Benchmarks Test Shallow Generalization

  Ari Spiesberger, Juan J. Vazquez, Nicky Pochinkov, Tom\'a\v{s} Gaven\v{c}iak, Peli Grietzer, Gavin Leech, Nandi Schoots · 16. Februar 2026

  If LLM training data is polluted with benchmark test data, then benchmark performance gives biased estimates of out-of-distribution (OOD) generalization. Typical decontamination filters use n-gram matching which fail to detect semantic duplicates: sentences with equivalent (or near-equivalent) conte…

• Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Models

  Hanwool Lee, Dasol Choi, Sooyong Kim, Ilgyun Jeong, Sangwon Baek, Guijin Son, Inseon Hwang, Naeun Lee, Seunghyeok Hong · 16. Februar 2026

  Recent advancements in Korean large language models (LLMs) have driven numerous benchmarks and evaluation methods, yet inconsistent protocols cause up to 10 p.p performance gaps across institutions. Overcoming these reproducibility gaps does not mean enforcing a one-size-fits-all evaluation. Rather,…

• Measure-to-measure interpolation using Transformers

  Borjan Geshkovski, Philippe Rigollet, Dom\`enec Ruiz-Balet · 16. Februar 2026

  Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-point maps, a Transformer acts as a measure-to-measure map implemented as specific interacting particle system on the uni…

• TyphoonMLA: A Mixed Naive-Absorb MLA Kernel For Shared Prefix

  Ahmet Caner Y\"uz\"ug\"uler, Ahmet \c{C}elik, Jiawei Zhuang, Lukas Cavigelli · 13. Februar 2026

  Multi-Head Latent Attention (MLA) is a recent attention mechanism adopted in state-of-the-art LLMs such as DeepSeek-v3 and Kimi K2. Thanks to its novel formulation, MLA allows two functionally equivalent but computationally distinct kernel implementations: naive and absorb. While the naive kernels (…

• Olmix: A Framework for Data Mixing Throughout LM Development

  Mayee F. Chen, Tyler Murray, David Heineman, Matt Jordan, Hannaneh Hajishirzi, Christopher R\'e, Luca Soldaini, Kyle Lo · 13. Februar 2026

  Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, they fall short when applied during real-world LM development. We present Olmix, a framework that addresses two such challe…

• Improving HPC Code Generation Capability of LLMs via Online Reinforcement Learning with Real-Machine Benchmark Rewards

  Ryo Mikasa, Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri · 13. Februar 2026

  Large language models (LLMs) have demonstrated strong code generation capabilities, yet the runtime performance of generated code is not guaranteed, and there have been few attempts to train LLMs using runtime performance as a reward in the HPC domain. We propose an online reinforcement learning app…

• The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models

  Aradhya Dixit, Shreem Dixit · 13. Februar 2026

  Pretrained multilingual language models are often assumed to be script-agnostic, yet their tokenizers can impose systematic costs on certain writing systems. We quantify this script tax by comparing two orthographic variants with identical linguistic content. Across mBERT and XLM-R, the higher-fragm…

• Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding

  Lin Long, Changdae Oh, Seongheon Park, Sharon Li · 12. Februar 2026

  Large vision-language models (LVLMs) achieve strong performance on multimodal tasks, yet they often default to their language prior (LP) -- memorized textual patterns from pre-training while under-utilizing visual evidence. Prior analyses of LP mostly rely on input-output probing, which fails to rev…

• SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

  Masataka Yoneda, Yusuke Matsushita, Go Kamoda, Kohei Suenaga, Takuya Akiba, Masaki Waga, Sho Yokoi · 12. Februar 2026

  We present an ultra-fast and flexible search algorithm that enables search over trillion-scale natural language corpora in under 0.3 seconds while handling semantic variations (substitution, insertion, and deletion). Our approach employs string matching based on suffix arrays that scales well with c…

• Context-level Language Modeling by Learning Predictive Context Embeddings

  Beiya Dai, Yuliang Liu, Daozheng Xue, Yunchong Song, Qipeng Guo, Kai Chen, Xinbing Wang, Bowen Zhou, Zhouhan Lin · 12. Februar 2026

  We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by p…

• Distribution-Aligned Decoding for Efficient LLM Task Adaptation

  Senkang Hu, Xudong Han, Jinqi Jiang, Yihang Tao, Zihan Fang, Yong Dai, Sam Tak Wu Kwong, Yuguang Fang · 11. Februar 2026

  Adapting billion-parameter language models to a downstream task is still costly, even with parameter-efficient fine-tuning (PEFT). We re-cast task adaptation as output-distribution alignment: the objective is to steer the output distribution toward the task distribution directly during decoding rath…

• Noisy-Pair Robust Representation Alignment for Positive-Unlabeled Learning

  Hengwei Zhao, Zhengzhong Tu, Zhuo Zheng, Wei Wang, Junjue Wang, Rusty Feagin, Wenzhe Jiao · 11. Februar 2026

  Positive-Unlabeled (PU) learning aims to train a binary classifier (positive vs. negative) where only limited positive data and abundant unlabeled data are available. While widely applicable, state-of-the-art PU learning methods substantially underperform their supervised counterparts on complex dat…

• Generalizing Scaling Laws for Dense and Sparse Large Language Models

  Md Arafat Hossain, Xingfu Wu, Valerie Taylor, Ali Jannesari · 11. Februar 2026

  Despite recent advancements of large language models (LLMs), optimally predicting the model size for LLM pretraining or allocating optimal resources still remains a challenge. Several efforts have addressed the challenge by proposing different empirical scaling laws, but almost all of them are archi…

• Aligning Tree-Search Policies with Fixed Token Budgets in Test-Time Scaling of LLMs

  Sora Miyamoto, Daisuke Oba, Naoaki Okazaki · 11. Februar 2026

  Tree-search decoding is an effective form of test-time scaling for large language models (LLMs), but real-world deployment imposes a fixed per-query token budget that varies across settings. Existing tree-search policies are largely budget-agnostic, treating the budget as a termination condition, wh…

• Maastricht University at AMIYA: Adapting LLMs for Dialectal Arabic using Fine-tuning and MBR Decoding

  Abdulhai Alali, Abderrahmane Issam · 11. Februar 2026

  Large Language Models (LLMs) are becoming increasingly multilingual, supporting hundreds of languages, especially high resource ones. Unfortunately, Dialect variations are still underrepresented due to limited data and linguistic variation. In this work, we adapt a pre-trained LLM to improve dialect…

• UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

  Jonathan von Rad, Yong Cao, Andreas Geiger · 11. Februar 2026

  Model compression is increasingly essential for deploying large language models (LLMs), yet existing evaluations are limited in method coverage and focus primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and kno…

• Toward Efficient Exploration by Large Language Model Agents

  Dilip Arumugam, Thomas L. Griffiths · 10. Februar 2026

  A burgeoning area within reinforcement learning (RL) is the design of sequential decision-making agents centered around large language models (LLMs). While autonomous decision-making agents powered by modern LLMs could facilitate numerous real-world applications, such successes demand agents that ar…

• Efficient Attention Mechanisms for Large Language Models: A Survey

  Yutao Sun, Zhenyu Li, Yike Zhang, Tengyu Pan, Bowen Dong, Yuyi Guo, Jianyong Wang · 10. Februar 2026

  Transformer-based architectures have become the prevailing backbone of large language models. However, the quadratic time and memory complexity of self-attention remains a fundamental obstacle to efficient long-context modeling. To address this limitation, recent research has introduced two principa…

• Copy-Paste to Mitigate Large Language Model Hallucinations

  Yongchao Long, Xian Wu, Yingying Zhang, Xianbin Wen, Yuxi Zhou, Shenda Hong · 10. Februar 2026

  While Retrieval-Augmented Generation (RAG) enables large language models (LLMs) to generate contextually grounded responses, contextual faithfulness remains challenging as LLMs may not consistently trust provided context, leading to hallucinations that undermine reliability. We observe an inverse co…

• Towards Open-Ended Discovery for Low-Resource NLP

  Bonaventure F. P. Dossou, Henri A\"idasso · 10. Februar 2026

  Natural Language Processing (NLP) for low-resource languages remains fundamentally constrained by the lack of textual corpora, standardized orthographies, and scalable annotation pipelines. While recent advances in large language models have improved cross-lingual transfer, they remain inaccessible …

• GPTOpt: Teaching LLMs to do Interpretable Black-Box Optimization

  Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konakovi\'c Lukovi\'c · 10. Februar 2026

  Global optimization of expensive, derivative-free black-box functions demands extreme sample efficiency and decision interpretability. While Large Language Models (LLMs) have shown broad capabilities, even state-of-the-art models remain limited in solving continuous black-box optimization tasks and …

• On Generation in Metric Spaces

  Jiaxun Li, Vinod Raman, Ambuj Tewari · 10. Februar 2026

  We study generation in separable metric instance spaces. We extend the language generation framework from Kleinberg and Mullainathan [2024] beyond countable domains by defining novelty through metric separation and allowing asymmetric novelty parameters for the adversary and the generator. We introd…

• d2: Improved Techniques for Training Reasoning Diffusion Language Models

  Guanghan Wang, Gilad Turok, Yair Schiff, Marianne Arriola, Volodymyr Kuleshov · 10. Februar 2026

  While diffusion language models (DLMs) have achieved competitive performance in text generation, improving their reasoning ability with reinforcement learning remains an active research area. Here, we introduce d2, a reasoning framework tailored for masked DLMs. Central to our framework is a new pol…

• Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction

  Jang-Hyun Kim, Dongyoon Han, Sangdoo Yun · 10. Februar 2026

  Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for froz…