Natural Language Processing Techniques

Latest papers

• IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

  Ali Abbasi, Chayne Thrash, Haoran Qin, Hamed Pirsiavash, Soheil Kolouri · 18 May 2026

  Large language models deliver strong performance across language and reasoning tasks, but their storage and compute costs remain major barriers to deployment in resource-constrained and latency-sensitive settings. SVD-based post-training compression offers a hardware-agnostic way to reduce model siz…

• Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective

  Ernesto Garcia-Estrada, Carlos Escolano, Jos\'e A. R. Fonallosa · 18 May 2026

  Production machine translation relies overwhelmingly on encoder-decoder Seq2Seq models, yet reinforcement learning approaches to MT fine-tuning have largely targeted decoder-only LLMs at $\geq$7B parameters, with limited systematic study of encoder-decoder architectures. We apply Group Relative Poli…

• CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs

  Kamil Guttmann, Zofia Fra\'s, Artur Nowakowski, Krzysztof Jassem · 18 May 2026

  Current state-of-the-art Quality Estimation (QE) in machine translation relies on massive, proprietary LLMs, raising data privacy concerns. We demonstrate that smaller, open-source LLMs (<30B parameters) are a viable, cost-effective and privacy-preserving alternative. Using a single-pass prompting s…

• AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs

  Haizhong Zheng, Yizhuo Di, Jiahui Wang, Shuowei Jin, Xueshen Liu, Yongji Wu, Z. Morley Mao, Ion Stoica, Jiawei Zhao, Beidi Chen · 18 May 2026

  Reinforcement learning (RL) is increasingly used to improve the reasoning, coding, and tool-use capabilities of large language models, but agentic RL remains prohibitively expensive. Scaling RL to agentic LLMs requires supporting complex workloads, including multi-policy collaborative training, whil…

• Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Models

  Sasha Cui, Zhongren Chen · 18 May 2026

  Language models (LMs) are typically post-trained for desired capabilities and behaviors via weight-based or prompt-based steering, but the former is time-consuming and expensive, and the latter is not precisely controllable and often requires manual trial-and-error. While activation steering (AS) pr…

• From Flat Language Labels to Typological Priors: Structured Language Conditioning for Multilingual Speech-to-Speech Translation

  Yu Pan, Yang Hou, Xiongfei Wu, Liang Zhang, Yves Le Traon, Lei Ma, Jianjun Zhao · 18 May 2026

  Compositional speech-to-speech translation (S2ST) systems built upon speech large language models (SpeechLLMs) have recently shown promising performance. However, existing S2ST systems often either neglect source-language information or encode it through a language-as-label paradigm, representing ea…

• paper.json: A Coordination Convention for LLM-Agent-Actionable Papers

  Arquimedes Canedo · 18 May 2026

  LLM agents routinely serve as first (and sometimes only) readers of academic papers, skimming for sub-claims, extracting reproducibility steps, and generalizing scope. Standard prose papers produce recurring failures in this role: sub-claims that cannot be cited at sub-paper granularity, scope overe…

• GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

  Fanxu Meng · 18 May 2026

  Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly. Its trained weights, however, expose only one decoding path - an absorbed MQA form - which ties efficient inference to H1…

• SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization

  Jamison Meindl, Yunsheng Tian, Tony Cui, Veronika Thost, Zhang-Wei Hong, Jie Chen, Wojciech Matusik, Mina Konakovi\'c Lukovi\'c · 18 May 2026

  Optimizing an experimental system can be extremely challenging when each experiment is expensive, time-consuming, or difficult to perform. Existing optimizers for expensive black-box problems, such as Bayesian optimization, are typically limited to numerical or categorical observations. They do not …

• RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

  Yufeng Du, Phillip Harris, Minyang Tian, Eliu A Huerta, Srikanth Ronanki, Subendhu Rongali, Aram Galstyan, Hao Peng · 18 May 2026

  We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We prove that as context length increases, RoPE-based attention…

• Merging Methods for Multilingual Knowledge Editing for Large Language Models: An Empirical Odyssey

  Kunil Lee, Ki-Young Shin, Jong-Hyeok Lee, Young-Joo Suh · 15 May 2026

  Multilingual knowledge editing (MKE) remains challenging because language-specific edits interfere with one another, even when locate-then-edit methods work well in monolingual settings. This paper focuses on three issues: the effectiveness of vector merging methods for MKE, the extent to which Task…

• Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuning

  Siyuan Liu (IIIS, Tsinghua University), Tinghong Chen (College of AI, Tsinghua University,Shanghai Qi Zhi Institute), Xinghan Li (IIIS, Tsinghua University), Yifei Wang (Amazon AGI SF Lab), Jingzhao Zhang (IIIS, Tsinghua University,Shanghai Qi Zhi Institute) · 14 May 2026

  Data selection during supervised fine-tuning (SFT) can critically change the behavior of large language models (LLMs). Although existing work has studied the effect of selecting data based on heuristics such as perplexity, difficulty, or length, the reported findings are often inconsistent or contex…

• LIFT: Last-Mile Fine-Tuning for Table Explicitation

  Divij Khaitan, Ashish Tiwari · 14 May 2026

  We propose last-mile fine-tuning, or Lift, a pipeline in which a pre-trained large language model extracts an initial table from unstructured clipboard text, and a fine-tuned small language model (1B-24B parameters SLM) repairs errors in the extracted table. On a benchmark of 2,596 tables from three…

• WARDEN: Endangered Indigenous Language Transcription and Translation with 6 Hours of Training Data

  Ziheng Zhang, Yunzhong Hou, Naijing Liu, Liang Zheng · 14 May 2026

  This paper introduces WARDEN, an early language model system capable of transcribing and translating Wardaman, an endangered Australian indigenous language into English. The significant challenge we face is the lack of large-scale training data: in fact, we only have 6 hours of annotated audio. Ther…

• Sampling from Flow Language Models via Marginal-Conditioned Bridges

  Iskander Azangulov, Leo Zhang · 14 May 2026

  Flow Language Models (FLMs) are a recently introduced class of language models which adapt continuous flow matching for one-hot encoded token sequences. Their denoisers have a special structure absent from generic continuous diffusion models: each block of the denoising mean is a posterior marginal …

• DocAtlas: Multilingual Document Understanding Across 80+ Languages

  Ahmed Heakl, Youssef Mohamed, Abdullah Sohail, Rania Elbadry, Ahmed Nassar, Peter W. J. Staar, Fahad Shahbaz Khan, Imran Razzak, Salman Khan · 14 May 2026

  Multilingual document understanding remains limited for low-resource languages due to scarce training data and model-based annotation pipelines that perpetuate existing biases. We introduce DocAtlas, a framework that constructs high-fidelity OCR datasets and benchmarks covering 82 languages and 9 ev…

• Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition

  Yuu Jinnai · 14 May 2026

  Recent work has shown that sample-based Minimum Bayes Risk (MBR) decoding outperforms beam search in text-to-text generation tasks, such as machine translation, text summarization, and image captioning. On the other hand, beam search is the current practice for speech-to-text tasks such as automatic…

• Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models

  Omer Luxembourg, Haim Permuter, Eliya Nachmani · 14 May 2026

  Masked diffusion language models (MDLMs) promise fast, non-autoregressive text generation, yet existing samplers, which pick tokens to unmask based on model confidence, ignore interactions when unmasking multiple positions in parallel and effectively reduce to slow, autoregressive behavior. We propo…

• Pretraining large language models with MXFP4 on Native FP4 Hardware

  Musa Cim, Poovaiah Palangappa, Miro Hodak, Ravi Dwivedula, Meena Arunachalam, Mahmut Taylan Kandemir · 14 May 2026

  Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable? We address this question through a controlled study of MXFP4 quantization in transformer training, progressively enabling FP4 across forward propagation (…

• Scaling Laws for Mixture Pretraining Under Data Constraints

  Anastasiia Sedova, Skyler Seto, Natalie Schluter, Pierre Ablin · 14 May 2026

  As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundam…

• MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs

  Yan Sun, Qixin Zhang, Zhiyuan Yu, Xikun Zhang, Li Shen, Dacheng Tao · 14 May 2026

  The rapid scaling of large language models~(LLMs) has made inference efficiency a primary bottleneck in the practical deployment. To address this, semi-structured sparsity offers a promising solution by strategically retaining $N$ elements out of every $M$ weights, thereby enabling hardware-friendly…

• Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

  Ruan Visser, Trienko Grobler, Marcel Dunaiski · 14 May 2026

  Subword regularization methods such as BPE dropout are typically applied only during fine-tuning, while pretraining is usually done with deterministic tokenization. This creates a potential segmentation mismatch between pretraining and fine-tuning. We investigate whether applying BPE dropout during …

• Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

  Paul Jeha, Anastasiia Sedova, Louis B\'ethune, Skyler Seto, Jes Frellsen, Pierre Ablin, Natalie Schluter · 14 May 2026

  For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resourc…

• ChatSR: Multimodal Large Language Models for Scientific Formula Discovery

  Yanjie Li, Lina Yu, Weijun Li, Min Wu, Liping Zhang, Jingyi Liu, Yusong Deng, Mingzhu Wan, Xin Ning · 14 May 2026

  Current multimodal large language models (MLLMs) are mainly focused on the understanding and processing of perceptual modalities such as images and videos, while their capability for scientific data understanding remains insufficient. To this end, we propose ChatSR, a novel multimodal large language…

• Do Activation Verbalization Methods Convey Privileged Information?

  Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, Byron C. Wallace · 14 May 2026

  Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such activation verbalization approaches actually pr…