Natural Language Processing Techniques

Derniers papiers

• ShIOEnv: A Command Evaluation Environment for Grammar-Constrained Synthesis and Execution Behavior Modeling

  Jarrod Ragsdale, Rajendra Boppana · 6 mars 2026

  Modeling of command-line interface (CLI) interaction has enabled flexible, execution-free output presentation. However, current approaches struggle to model inputs with complex compositions and inputs whose execution behavior depends on system characteristics. This is due to a lack of shell input-ou…

• SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models

  Thinh Pham, Nguyen Nguyen, Pratibha Zunjare, Weiyuan Chen, Yu-Min Tseng, Tu Vu · 6 mars 2026

  We introduce SealQA, a new challenge benchmark for evaluating SEarch-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors: (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and reasoning c…

• SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference

  Luchang Li, Dongfang Li, Bozhao Gong, Yu Zhang · 6 mars 2026

  Prefill-Decode (P/D) disaggregation has emerged as a widely adopted optimization strategy for Large Language Model (LLM) inference. However, there currently exists no well-established methodology for determining the optimal number of P/D hardware resources, subject to constraints on total throughput…

• Best-of-$\infty$ -- Asymptotic Performance of Test-Time LLM Ensembling

  Junpei Komiyama, Daisuke Oba, Masafumi Oyamada · 5 mars 2026

  We study best-of-$N$ for large language models (LLMs) where the selection is based on majority voting. In particular, we analyze the limit $N \to \infty$, which we denote as \boinflower. While this approach achieves impressive performance in the limit, it requires an infinite test-time budget. To ad…

• Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque

  Lukas Arana, Julen Etxaniz, Ander Salaberria, Gorka Azkune · 5 mars 2026

  Current Multimodal Large Language Models exhibit very strong performance for several demanding tasks. While commercial MLLMs deliver acceptable performance in low-resource languages, comparable results remain unattained within the open science community. In this paper, we aim to develop a strong MLL…

• ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

  Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard, Besnik Fetahu, Nasser Zalmout, Xian Li · 5 mars 2026

  Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce \textbf{ByteFlow…

• Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

  Shiza Fatimah, Aniket Sen, Sophia Falk, Florian Mai, Lucie Flek, Nicholas Kluge Corr\^ea · 5 mars 2026

  The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address t…

• Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model

  Jakub Prejzner · 5 mars 2026

  We present Bielik-Q2-Sharp, the first systematic academic evaluation of extreme 2-bit quantization applied to a Polish large language model. Using Bielik-11B-v2.3-Instruct (11B parameters, Mistral architecture) as our base model, we compare six state-of-the-art post-training quantization methods -- …

• Parallel Test-Time Scaling with Multi-Sequence Verifiers

  Yegon Kim, Seungyoo Lee, Chaeyun Jang, Hyungi Lee, Juho Lee · 5 mars 2026

  Parallel test-time scaling, which generates multiple candidate solutions for a single problem, is a powerful technique for improving large language model performance. However, it is hindered by two key bottlenecks: accurately selecting the correct solution from the candidate pool, and the high infer…

• LMUnit: Fine-grained Evaluation with Natural Language Unit Tests

  Jon Saad-Falcon, Rajan Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, Shikib Mehri · 5 mars 2026

  As language models become integral to critical workflows, assessing their behavior remains a fundamental challenge -- human evaluation is costly and noisy, while automated metrics provide only coarse, difficult-to-interpret signals. We introduce natural language unit tests, a paradigm that decompose…

• GPUTOK: GPU Accelerated Byte Level BPE Tokenization

  Venu Gopal Kadamba, Kanishkha Jaisankar · 4 mars 2026

  As large language models move toward million-token context windows, CPU tokenizers become a major slowdown because they process text one step at a time while powerful GPUs sit unused. We built a GPU-based byte-level BPE tokenizer that follows GPT-2's merge rules. It includes a basic BlockBPE-style k…

• Topic-Based Watermarks for Large Language Models

  Alexander Nemecek, Yuzhou Jiang, Erman Ayday · 4 mars 2026

  The indistinguishability of large language model (LLM) output from human-authored content poses significant challenges, raising concerns about potential misuse of AI-generated text and its influence on future model training. Watermarking algorithms offer a viable solution by embedding detectable sig…

• No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models

  Omer Sela · 4 mars 2026

  CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contami…

• ScaleDoc: Scaling LLM-based Predicates over Large Document Collections

  Hengrui Zhang, Yulong Hui, Yihao Liu, Huanchen Zhang · 4 mars 2026

  Predicates are foundational components in data analysis systems. However, modern workloads increasingly involve unstructured documents, which demands semantic understanding, beyond traditional value-based predicates. Given enormous documents and ad-hoc queries, while Large Language Models (LLMs) dem…

• CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

  Jiace Zhu, Wentao Chen, Qi Fan, Zhixing Ren, Junying Wu, Xing Zhe Chai, Chotiwit Rungrueangwutthinon, Yehan Ma, An Zou · 4 mars 2026

  Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-speci…

• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward

  Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, Enhong Chen · 4 mars 2026

  Enhancing the multimodal reasoning capabilities of Multimodal Large Language Models (MLLMs) is a challenging task that has attracted increasing attention in the community. Recently, several studies have applied Reinforcement Learning with Verifiable Rewards (RLVR) to the multimodal domain in order t…

• Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding

  Shijing Hu, Jingyang Li, Zhihui Lu, Pan Zhou · 3 mars 2026

  Speculative decoding accelerates large language model (LLM) inference by letting a lightweight draft model propose multiple tokens that the target model verifies in parallel. Yet existing training objectives optimize only a single greedy draft path, while decoding follows a tree policy that re-ranks…

• EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

  Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fi\v{s}el, Tanel Alum\"ae, Eleri Aedmaa, Krister Kruusmaa, Kairit Sirts · 3 mars 2026

  Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We study whether continued pretraining (CPT) can substantially improve Estonian capabilities in a pretrained multilingual LLM while preserving its English and genera…

• AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size

  Guanxi Lu, Hao Mark Chen, Yuto Karashima, Zhican Wang, Daichi Fujiki, Hongxiang Fan · 3 mars 2026

  Diffusion-based large language models (dLLMs) are gaining attention for their inherent capacity for parallel decoding, offering a compelling alternative to autoregressive LLMs. Among various decoding strategies, block-wise semi-autoregressive (semi-AR) approaches are widely adopted due to their supp…

• Language steering in latent space to mitigate unintended code-switching

  Andrey Goncharov, Nikolai Kondusov, Alexey Zaytsev · 3 mars 2026

  Multilingual Large Language Models (LLMs) often exhibit unintended code-switching, reducing reliability in downstream tasks. We propose latent-space language steering, a lightweight inference-time method that identifies language directions via PCA on parallel translations and steers token embeddings…

• LiTS: A Modular Framework for LLM Tree Search

  Xinzhe Li, Yaguang Tao · 3 mars 2026

  LiTS is a modular Python framework for LLM reasoning via tree search. It decomposes tree search into three reusable components (Policy, Transition, and RewardModel) that plug into algorithms like MCTS and BFS. A decorator-based registry enables domain experts to extend to new domains by registering …

• Distillation of Large Language Models via Concrete Score Matching

  Yeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na, Il-Chul Moon · 3 mars 2026

  Large language models (LLMs) deliver remarkable performance but are costly to deploy, motivating knowledge distillation (KD) for efficient inference. Existing KD objectives typically match student and teacher probabilities via softmax, which blurs valuable logit information. While direct logit disti…

• Train Once, Answer All: Many Pretraining Experiments for the Cost of One

  Sebastian Bordt, Martin Pawelczyk · 3 mars 2026

  Recent work has demonstrated that controlled pretraining experiments are a powerful tool for studying the relationship between training data and large language model (LLM) behavior. However, the computational cost of pretraining presents a significant constraint. To overcome this constraint, we prop…

• Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models

  Junhong Lin, Xinyue Zeng, Jie Zhu, Song Wang, Julian Shun, Jun Wu, Dawei Zhou · 3 mars 2026

  Large Language Models (LLMs) have achieved remarkable success in complex reasoning tasks, but their inference remains computationally inefficient. We observe a common failure mode in many prevalent LLMs, overthinking, where models generate verbose and tangential reasoning traces even for simple quer…

• Cognitive models can reveal interpretable value trade-offs in language models

  Sonia K. Murthy, Rosie Zhao, Jennifer Hu, Sham Kakade, Markus Wulfmeier, Peng Qian, Tomer Ullman · 3 mars 2026

  Value trade-offs are an integral part of human decision-making and language use, however, current tools for interpreting such dynamic and multi-faceted notions of values in language models are limited. In cognitive science, so-called "cognitive models" provide formal accounts of such trade-offs in h…