Natural Language Processing Techniques

Derniers papiers

• Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models

  Etrit Haxholli, Yeti Z. Gurbuz, Ogul Can, Eli Waxman · 9 février 2026

  While continuous diffusion models excel in modeling continuous distributions, their application to categorical data has been less effective. Recent work has shown that ratio-matching through score-entropy within a continuous-time discrete Markov chain (CTMC) framework serves as a competitive alterna…

• Stopping Computation for Converged Tokens in Masked Diffusion-LM Decoding

  Daisuke Oba, Danushka Bollegala, Masahiro Kaneko, Naoaki Okazaki · 9 février 2026

  Masked Diffusion Language Models generate sequences via iterative sampling that progressively unmasks tokens. However, they still recompute the attention and feed-forward blocks for every token position at every step -- even when many unmasked tokens are essentially fixed, resulting in substantial w…

• MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew

  Andy Rosenbaum, Assaf Siani, Ilan Kernerman · 9 février 2026

  We release MTQE.en-he: to our knowledge, the first publicly available English-Hebrew benchmark for Machine Translation Quality Estimation. MTQE.en-he contains 959 English segments from WMT24++, each paired with a machine translation into Hebrew, and Direct Assessment scores of the translation qualit…

• Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinement

  Yu-Che Tsai, Kuan-Yu Chen, Yuan-Chi Li, Yuan-Hao Chen, Ching-Yu Tsai, Shou-De Lin · 9 février 2026

  Existing large language model (LLM)-based embeddings typically adopt an encoder-only paradigm, treating LLMs as static feature extractors and overlooking their core generative strengths. We introduce GIRCSE (Generative Iterative Refinement for Contrastive Sentence Embeddings), a novel framework that…

• Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay

  Duygu Altinok · 9 février 2026

  Tokenization is a pivotal design choice for neural language modeling in morphologically rich languages (MRLs) such as Turkish, where productive agglutination challenges both vocabulary efficiency and morphological fidelity. Prior studies have explored tokenizer families and vocabulary sizes but typi…

• Deep networks learn to parse uniform-depth context-free languages from local statistics

  Jack T. Parley, Francesco Cagnetta, Matthieu Wyart · 9 février 2026

  Understanding how the structure of language can be learned from sentences alone is a central question in both cognitive science and machine learning. Studies of the internal representations of Large Language Models (LLMs) support their ability to parse text when predicting the next word, while repre…

• code_transformed: The Influence of Large Language Models on Code

  Yuliang Xu, Siming Huang, Mingmeng Geng, Yao Wan, Xuanhua Shi, Dongping Chen · 9 février 2026

  Coding remains one of the most fundamental modes of interaction between humans and machines. With the rapid advancement of Large Language Models (LLMs), code generation capabilities have begun to significantly reshape programming practices. This development prompts a central question: Have LLMs tran…

• Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Models

  Michael Browder, Kevin Duh, J. David Harris, Vince Lyzinski, Paul McNamee, Youngser Park, Carey E. Priebe, Peter Viechnicki · 6 février 2026

  Scarcity of labeled training data remains the long pole in the tent for building performant language technology and generative AI models. Transformer models -- particularly LLMs -- are increasingly being used to mitigate the data scarcity problem via synthetic data generation. However, because the m…

• Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers

  Wooseok Seo, Seungju Han, Jaehun Jung, Benjamin Newman, Seungwon Lim, Seungbeen Lee, Ximing Lu, Yejin Choi, Youngjae Yu · 6 février 2026

  Fact verification is essential for ensuring the reliability of LLM applications. In this study, we evaluate 12 pre-trained LLMs and one specialized fact-verifier, including frontier LLMs and open-weight reasoning LLMs, using a collection of examples from 14 fact-checking benchmarks. We share three f…

• Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory

  Haozhen Zhang, Haodong Yue, Tao Feng, Quanyu Long, Jianzhu Bao, Bowen Jin, Weizhi Zhang, Xiao Li, Jiaxuan You, Chengwei Qin, Wenya Wang · 6 février 2026

  Memory is increasingly central to Large Language Model (LLM) agents operating beyond a single context window, yet most existing systems rely on offline, query-agnostic memory construction that can be inefficient and may discard query-critical information. Although runtime memory utilization is a nat…

• EuroLLM-22B: Technical Report

  Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef, Jo\~ao Alves, Pedro Henrique Martins, Patrick Fernandes, Jos\'e Pombal, Nuno M. Guerreiro, Ricardo Rei, Nicolas Boizard, Amin Farajian, Mateusz Klimaszewski, Jos\'e G. C. de Souza, Barry Haddow, Fran\c{c}ois Yvon, Pierre Colombo, Alexandra Birch, Andr\'e F. T. Martins · 6 février 2026

  This report presents EuroLLM-22B, a large language model trained from scratch to support the needs of European citizens by covering all 24 official European Union languages and 11 additional languages. EuroLLM addresses the issue of European languages being underrepresented and underserved in existi…

• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models

  Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, Shaohui Lin · 6 février 2026

  DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to acti…

• Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping

  Xinzhe Zheng, Zhen-Qun Yang, Zishan Liu, Haoran Xie, S. Joe Qin, Arlene Chen, Fangzhen Lin · 6 février 2026

  Large Language Models (LLMs) deliver strong performance but are difficult to deploy under tight memory and compute constraints. Low-bit post-training quantization (PTQ) is a promising direction; however, it typically relies on calibration data, auxiliary transformations, and GPU tools. To address th…

• Proxy Compression for Language Modeling

  Lin Zheng, Xinyu Li, Qian Liu, Xiachong Feng, Lingpeng Kong · 5 février 2026

  Modern language models are trained almost exclusively on token sequences produced by a fixed tokenizer, an external lossless compressor often over UTF-8 byte sequences, thereby coupling the model to that compressor. This work introduces proxy compression, an alternative training scheme that preserve…

• Rethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMs

  Letian Cheng, Junyan Wang, Yan Gao, Elliott Wen, Ting Dang, Hong Jia · 5 février 2026

  Perplexity is a widely adopted metric for assessing the predictive quality of large language models (LLMs) and often serves as a reference metric for downstream evaluations. However, recent evidence shows that perplexity can be unreliable, especially when irrelevant long inputs are used, raising con…

• Improving Detection of Watermarked Language Models

  Dara Bahri, John Wieting · 5 février 2026

  Watermarking has recently emerged as an effective strategy for detecting the generations of large language models (LLMs). The strength of a watermark typically depends strongly on the entropy afforded by the language model and the set of input prompts. However, entropy can be quite limited in practi…

• Adaptive Test-Time Compute Allocation via Learned Heuristics over Categorical Structure

  Shuhui Qu · 5 février 2026

  Test-time computation has become a primary driver of progress in large language model (LLM) reasoning, but it is increasingly bottlenecked by expensive verification. In many reasoning systems, a large fraction of verifier calls are spent on redundant or unpromising intermediate hypotheses. We study …

• From Lemmas to Dependencies: What Signals Drive Light Verbs Classification?

  Sercan Karaka\c{s}, Yusuf \c{S}im\c{s}ek · 5 février 2026

  Light verb constructions (LVCs) are a challenging class of verbal multiword expressions, especially in Turkish, where rich morphology and productive complex predicates create minimal contrasts between idiomatic predicate meanings and literal verb--argument uses. This paper asks what signals drive LV…

• No One-Size-Fits-All: Building Systems For Translation to Bashkir, Kazakh, Kyrgyz, Tatar and Chuvash Using Synthetic And Original Data

  Dmitry Karpov · 5 février 2026

  We explore machine translation for five Turkic language pairs: Russian-Bashkir, Russian-Kazakh, Russian-Kyrgyz, English-Tatar, English-Chuvash. Fine-tuning nllb-200-distilled-600M with LoRA on synthetic data achieved chrF++ 49.71 for Kazakh and 46.94 for Bashkir. Prompting DeepSeek-V3.2 with retriev…

• Multiple Choice Learning of Low-Rank Adapters for Language Modeling

  Victor Letzelter, Hugo Malard, Mathieu Fontaine, Ga\"el Richard, Slim Essid, Andrei Bursuc, Patrick P\'erez · 5 février 2026

  We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple ``futures'' may be…

• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

  Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Hanhui Li, Yiwei Wang, Xiaodan Liang, Jing Tang · 4 février 2026

  Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models, yet its full potential is hindered by two under-explored dimensions: Depth-the hardest problem a model can sample; Breadth-the number of instances co…

• BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

  Jie Hao, Rui Yu, Wei Zhang, Huixia Wang, Jie Xu, Mingrui Liu · 4 février 2026

  Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks. However, existing approaches often require leveraging external pretrained models, making it difficult to disentangle the effects of data selectio…

• AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation

  Sara Papi, Marco Turchi, Matteo Negri · 4 février 2026

  Attention is the core mechanism of today's most used architectures for natural language processing and has been analyzed from many perspectives, including its effectiveness for machine translation-related tasks. Among these studies, attention resulted to be a useful source of information to get insi…

• Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

  Jenny Y. Huang, Yunyi Shen, Dennis Wei, Tamara Broderick · 4 février 2026

  We propose a method for evaluating the robustness of widely used LLM ranking systems -- variants of a Bradley--Terry model -- to dropping a worst-case very small fraction of preference data. Our approach is computationally fast and easy to adopt. When we apply our method to matchups from popular LLM…

• Reusing Overtrained Language Models Saturates Scaling

  Seng Pei Liew, Takuya Kato · 4 février 2026

  Reusing pretrained base models for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch. However, the effectiveness remains unclear, especially when applied to overtrained base models. In this work, we empirica…