Natural Language Processing Techniques

Latest papers

• How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective

  Shimao Zhang, Zhejian Lai, Xiang Liu, Shuaijie She, Xiao Liu, Yeyun Gong, Shujian Huang, Jiajun Chen · 3 April 2026

  Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some research on language-specific neurons provides a new perspective to analyze …

• How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective

  Shimao Zhang, Zhejian Lai, Xiang Liu, Shuaijie She, Xiao Liu, Yeyun Gong, Shujian Huang, Jiajun Chen · 2 April 2026

  Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some research on language-specific neurons provides a new perspective to analyze …

• Adaptive Parallel Monte Carlo Tree Search for Efficient Test-time Compute Scaling

  Hongbeen Kim, Juhyun Lee, Sanghyeon Lee, Kwanghoon Choi, Jaehyuk Huh · 2 April 2026

  Monte Carlo Tree Search (MCTS) is an effective test-time compute scaling (TTCS) method for improving the reasoning performance of large language models, but its highly variable execution time leads to severe long-tail latency in practice. Existing optimizations such as positive early exit, reduce la…

• Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

  Thomas F Burns, Letitia Parcalabescu, Stephan W\"aldchen, Michael Barlow, Gregor Ziegltrum, Volker Stampa, Bastian Harren, Bj\"orn Deiseroth · 1 April 2026

  Scaling data quantity is essential for large language models (LLMs), yet recent findings show that data quality can significantly boost performance and training efficiency. We introduce a German-language dataset curation pipeline that combines heuristic and model-based filtering techniques with synt…

• M-MiniGPT4: Multilingual VLLM Alignment via Translated Data

  Seung Hun Han, Youssef Mohamed, Mohamed Elhoseiny · 1 April 2026

  This paper presents a Multilingual Vision Large Language Model, named M-MiniGPT4. Our model exhibits strong vision-language understanding (VLU) capabilities across 11 languages. We utilize a mixture of native multilingual and translated data to push the multilingual VLU performance of the MiniGPT4 a…

• OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

  Haiyue Song, Masao Utiyama · 1 April 2026

  Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. In this work, we pr…

• Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks

  Chunyang Jiang, Yonggang Zhang, Yiyang Cai, Chi-Min Chan, Yulong Liu, Mingming Chen, Wei Xue, Yike Guo · 1 April 2026

  The rising cost of acquiring supervised data has driven significant interest in self-improvement for large language models (LLMs). Straightforward unsupervised signals like majority voting have proven effective in generating pseudo-labels for verifiable tasks, while their applicability to unverifiab…

• Understand and Accelerate Memory Processing Pipeline for Disaggregated LLM Inference

  Zifan He, Rui Ma, Yizhou Sun, Jason Cong · 1 April 2026

  Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified i…

• Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles

  Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Puyu Zeng, Yuxuan Wang, Biqing Qi, Dongrui Liu, Linfeng Zhang · 1 April 2026

  Diffusion-based language models (dLLMs) have emerged as a promising alternative to traditional autoregressive LLMs by enabling parallel token generation and significantly reducing inference latency. However, existing sampling strategies for dLLMs, such as confidence-based or semi-autoregressive deco…

• ShishuLM : Achieving Optimal and Efficient Parameterization with Low Attention Transformer Models

  Shivanshu Kumar, Gopalakrishnan Srinivasan · 1 April 2026

  While the transformer architecture has achieved state-of-the-art performance on natural language processing tasks, these models impose substantial memory and computational overhead. Recent research has identified significant architectural redundancies within these models, particularly in the attenti…

• ATLAS-RTC: Closing the Loop on LLM Agent Output with Token-Level Runtime Control

  Christopher Cruz · 31 March 2026

  We present ATLAS-RTC, a runtime control system for autoregressive language models that enforces structured output during decoding. ATLAS-RTC monitors generation at each step, detects drift from output contracts using lightweight signals, and applies targeted interventions such as biasing, masking, a…

• Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints

  Jelena Markovic-Voronov, Kayhan Behdin, Yuanda Xu, Zhengze Zhou, Zhipeng Wang, Rahul Mazumder · 31 March 2026

  We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, res…

• Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generation

  Seamus Somerstep, Vinod Raman, Unique Subedi, Yuekai Sun · 31 March 2026

  Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as supervised fine-tuning, involves training a new next token predictor on good generations. The second method, Best-of-N, tr…

• Multi-Sample Prompting and Actor-Critic Prompt Optimization for Diverse Synthetic Data Generation

  Abdelkarim El-Hajjami, Camille Salinesi · 31 March 2026

  High-quality labeled datasets are fundamental for training and evaluating machine learning models, yet domains such as healthcare and Requirements Engineering (RE) face persistent barriers due to data scarcity, privacy constraints, or proprietary restrictions. While Large Language Models (LLMs) offe…

• DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

  Hao Liang, Zhengyang Zhao, Meiyi Qiang, Mingrui Chen, Lu Ma, Rongyi Yu, Hengyi Feng, Shixuan Sun, Zimo Meng, Xiaochen Ma, Xuanlin Yang, Qifeng Cai, Ruichuan An, Bohan Zeng, Zhen Hao Wong, Chengyu Shen, Runming He, Zhaoyang Han, Yaowei Zheng, Fangcheng Fu, Conghui He, Bin Cui, Zhiyu Li, Weinan E, Wentao Zhang · 30 March 2026

  Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimiza…

• ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

  In\^es Vieira, In\^es Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Gl\'oria-Silva, David Semedo, Jo\~ao Magalh\~aes · 30 March 2026

  As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). T…

• AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese

  Afonso Simpl\'icio, Gon\c{c}alo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, In\^es Calvo, In\^es Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Gl\'oria-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, Andr\'e Martins, Jo\~ao Magalh\~aes · 30 March 2026

  Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant's linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that p…

• Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages

  Danlu Chen, Ka Sing He, Jiahe Tian, Chenghao Xiao, Zhaofeng Wu, Taylor Berg-Kirkpatrick, Freda Shi · 27 March 2026

  The landscape of extremely low-resource machine translation (MT) is characterized by perplexing variability in reported performance, often making results across different language pairs difficult to contextualize. For researchers focused on specific language groups -- such as ancient languages -- it…

• Estimating near-verbatim extraction risk in language models with decoding-constrained beam search

  A. Feder Cooper, Mark A. Lemley, Christopher De Sa, Lea Duesterwald, Allison Casasola, Jamie Hayes, Katherine Lee, Daniel E. Ho, Percy Liang · 27 March 2026

  Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -- computing the probability of generating a target suffix given a prefix under a decoding scheme -- addresses this, but …

• Multi-LLM Query Optimization

  Arlen Dean, Zijin Zhang, Stefanus Jasin, Yuqing Liu · 27 March 2026

  Deploying multiple large language models (LLMs) in parallel to classify an unknown ground-truth label is a common practice, yet the problem of optimally allocating queries across heterogeneous models remains poorly understood. In this paper, we formulate a robust, offline query-planning problem that…

• Demystifying When Pruning Works via Representation Hierarchies

  Shwai He, Guoheng Sun, Haichao Zhang, Yun Fu, Ang Li · 27 March 2026

  Network pruning, which removes less important parameters or architectures, is often expected to improve efficiency while preserving performance. However, this expectation does not consistently hold across language tasks: pruned models can perform well on non-generative tasks but frequently fail in g…

• LogitScope: A Framework for Analyzing LLM Uncertainty Through Information Metrics

  Farhan Ahmed, Yuya Jeremy Ong, Chad DeLuca · 27 March 2026

  Understanding and quantifying uncertainty in large language model (LLM) outputs is critical for reliable deployment. However, traditional evaluation approaches provide limited insight into model confidence at individual token positions during generation. To address this issue, we introduce LogitScop…

• GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs

  Selim An, Il hong Suh, Yeseong Kim · 27 March 2026

  Quantization techniques such as BitsAndBytes, AWQ, and GPTQ are widely used as a standard method in deploying large language models but often degrades accuracy when using low-bit representations, e.g., 4 bits. Low-rank correction methods (e.g., LQER, QERA, ASER) has been proposed to mitigate this is…

• Toward domain-specific machine translation and quality estimation systems

  Javad Pourmostafa Roshan Sharami · 27 March 2026

  Machine Translation (MT) and Quality Estimation (QE) perform well in general domains but degrade under domain mismatch. This dissertation studies how to adapt MT and QE systems to specialized domains through a set of data-focused contributions. Chapter 2 presents a similarity-based data selection me…

• The Limits of Inference Scaling Through Resampling

  Benedikt Stroebl, Sayash Kapoor, Arvind Narayanan · 27 March 2026

  Recent research has generated hope that inference scaling, such as resampling solutions until they pass verifiers like unit tests, could allow weaker models to match stronger ones. Beyond inference, this approach also enables training reasoning models, where data is curated using rejection sampling …