Natural Language Processing Techniques

Neueste Paper

• Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps

  Gabriel Chua, Leanne Tan, Ziyu Ge, Roy Ka-Wei Lee · 3. Februar 2026

  Large language models (LLMs) often fail to maintain safety in low-resource language varieties, such as code-mixed vernaculars and regional dialects. We introduce RabakBench, a multilingual safety benchmark and scalable pipeline localized to Singapore's unique linguistic landscape, covering Singlish,…

• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence

  Qizhen Zhang, Ankush Garg, Jakob Foerster, Niladri Chatterji, Kshitiz Malik, Mike Lewis · 3. Februar 2026

  Large-scale pretraining datasets drive the success of large language models (LLMs). However, these web-scale corpora inevitably contain large amounts of noisy data due to unregulated web content or randomness inherent in data. Although LLM pretrainers often speculate that such noise contributes to i…

• On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating

  Huy Nguyen, Thong T. Doan, Quang Pham, Nghi D. Q. Bui, Nhat Ho, Alessandro Rinaldo · 3. Februar 2026

  Mixture of experts (MoE) methods are a key component in most large language model architectures, including the recent series of DeepSeek models. Compared to other MoE implementations, DeepSeekMoE stands out because of two unique features: the deployment of a shared expert strategy and of the normali…

• How does a Multilingual LM Handle Multiple Languages?

  Santhosh Kakarla, Gautama Shastry Bulusu Venkata, Aishwarya Gaddam, Maheedhar Sai Omtri Mohan · 3. Februar 2026

  Multilingual language models have significantly advanced due to rapid progress in natural language processing. Models like BLOOM 1.7B, trained on diverse multilingual datasets, aim to bridge linguistic gaps. However, their effectiveness in capturing linguistic knowledge, particularly for low-resourc…

• Entropy-Lens: Uncovering Decision Strategies in LLMs

  Riccardo Ali, Francesco Caso, Christopher Irwin, Pietro Li\`o · 3. Februar 2026

  In large language models (LLMs), each block operates on the residual stream to map input token sequences to output token distributions. However, most of the interpretability literature focuses on internal latent representations, leaving token-space dynamics underexplored. The high dimensionality and…

• Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference

  Nikhil Gopal, Kostis Kaffes · 3. Februar 2026

  Large Language Model (LLM) inference is increasingly constrained by GPU memory capacity rather than compute throughput, driven by growing model sizes and the linear growth of the key-value (KV) cache during autoregressive decoding. Existing approaches mitigate memory pressure by offloading model sta…

• Simple Policy Gradients for Reasoning with Diffusion Language Models

  Anthony Zhan · 3. Februar 2026

  Diffusion large language models (dLLMs), which offer a promising alternative to traditional autoregressive LLMs, have recently shown strong results in pretraining. However, due to their lack of tractable sequence-level likelihoods, they have yet to benefit from modern LLM post-training techniques su…

• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction

  Xingshan Zeng, Weiwen Liu, Lingzhi Wang, Liangyou Li, Fei Mi, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu · 3. Februar 2026

  Agentic task-solving with Large Language Models (LLMs) requires multi-turn, multi-step interactions, often involving complex function calls and dynamic user-agent exchanges. Existing simulation-based data generation methods for such scenarios rely heavily on costly autoregressive interactions betwee…

• Do Large Language Models (Really) Need Statistical Foundations?

  Weijie Su · 3. Februar 2026

  Large language models (LLMs) represent a new paradigm for processing unstructured data, with applications across an unprecedented range of domains. In this paper, we address, through two arguments, whether the development and application of LLMs would genuinely benefit from foundational contribution…

• On the Spectral Flattening of Quantized Embeddings

  Junlin Huang, Wenyi Fang, Zhenheng Tang, Yuxin Wang, Xueze Kang, Yang Zheng, Bo Li, Xiaowen Chu · 3. Februar 2026

  Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic heavy-tailed spectral nature of linguistic data. By formalizing the connection between Zipfian statistics and random ma…

• Pretraining Scaling Laws for Generative Evaluations of Language Models

  Rylan Schaeffer, Noam Levi, Brando Miranda, Sanmi Koyejo · 3. Februar 2026

  Neural scaling laws have driven the field's ever-expanding exponential growth in parameters, data and compute. While scaling behaviors for pretraining losses and discriminative benchmarks are well established, generative benchmarks such as mathematical problem-solving or software engineering remain …

• SVIP: Towards Verifiable Inference of Open-source Large Language Models

  Yifan Sun, Yuhang Li, Yue Zhang, Yuchen Jin, Huan Zhang · 3. Februar 2026

  The ever-increasing size of open-source Large Language Models (LLMs) renders local deployment impractical for individual users. Decentralized computing has emerged as a cost-effective solution, allowing individuals and small companies to perform LLM inference for users using surplus computational po…

• p-less Sampling: A Robust Hyperparameter-Free Approach for LLM Decoding

  Runyan Tan, Shuang Wu, Phillip Howard · 3. Februar 2026

  Obtaining high-quality outputs from Large Language Models (LLMs) often depends upon the choice of a sampling-based decoding strategy to probabilistically choose the next token at each generation step. While a variety of such sampling methods have been proposed, their performance can be sensitive to …

• P-EAGLE: Parallel-Drafting EAGLE with Scalable Training

  Mude Hui, Xin Huang, Jaime Campos Salas, Yue Sun, Nathan Pemberton, Xiang Song, Ashish Khetan, George Karypis · 3. Februar 2026

  Reasoning LLMs produce longer outputs, requiring speculative decoding drafters trained on extended sequences. Parallel drafting - predicting multiple tokens per forward pass - offers latency benefits over sequential generation, but training complexity scales quadratically with the product of sequenc…

• From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference Optimization

  Chaoqun Cui, Shijing Wang, Liangbin Huang, Qingqing Gu, Zhaolong Huang, Xiao Zeng, Wenji Mao · 3. Februar 2026

  The rapid development of Large Language Models (LLMs) has significantly enhanced the general capabilities of machine translation. However, as application scenarios become more complex, the limitations of LLMs in vertical domain translations are gradually becoming apparent. In this study, we focus on…

• VoxServe: Streaming-Centric Serving System for Speech Language Models

  Keisuke Kamahori, Wei-Tzu Lee, Atindra Jha, Rohan Kadekodi, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci · 3. Februar 2026

  Deploying modern Speech Language Models (SpeechLMs) in streaming settings requires systems that provide low latency, high throughput, and strong guarantees of streamability. Existing systems fall short of supporting diverse models flexibly and efficiently. We present VoxServe, a unified serving syst…

• ECCO: Evidence-Driven Causal Reasoning for Compiler Optimization

  Haolin Pan, Lianghong Huang, Jinyuan Dong, Mingjie Xing, Yanjun Wu · 3. Februar 2026

  Compiler auto-tuning faces a dichotomy between traditional black-box search methods, which lack semantic guidance, and recent Large Language Model (LLM) approaches, which often suffer from superficial pattern matching and causal opacity. In this paper, we introduce ECCO, a framework that bridges int…

• Trust by Design: Skill Profiles for Transparent, Cost-Aware LLM Routing

  Mika Okamoto, Ansel Kaplan Erol, Glenn Matlin · 3. Februar 2026

  How should Large Language Model (LLM) practitioners select the right model for a task without wasting money? We introduce BELLA (Budget-Efficient LLM Selection via Automated skill-profiling), a framework that recommends optimal LLM selection for tasks through interpretable skill-based model selectio…

• Code-Mixed Phonetic Perturbations for Red-Teaming LLMs

  Darpan Aswal, Siddharth D Jaiswal · 3. Februar 2026

  Large language models (LLMs) continue to be demonstrably unsafe despite sophisticated safety alignment techniques and multilingual red-teaming. However, recent red-teaming work has focused on incremental gains in attack success over identifying underlying architectural vulnerabilities in models. In …

• LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference

  Guangyuan Ma, Yongliang Ma, Xuanrui Gou, Zhenpeng Su, Ming Zhou, Songlin Hu · 2. Februar 2026

  Large Language Models (LLMs)-based text retrieval retrieves documents relevant to search queries based on vector similarities. Documents are pre-encoded offline, while queries arrive in real-time, necessitating an efficient online query encoder. Although LLMs significantly enhance retrieval capabili…

• ElectriQ: A Benchmark for Assessing the Response Capability of Large Language Models in Power Marketing

  Jinzhi Wang, Qingke Peng, Haozhou Li, Zeyuan Zeng, Jiangbo Zhang, Kaixuan Yang, Ningyong Wu, Qinfeng Song, Ruimeng Li, Biyi Zhou · 2. Februar 2026

  As power systems decarbonise and digitalise, high penetrations of distributed energy resources and flexible tariffs make electric power marketing (EPM) a key interface between regulation, system operation and sustainable-energy deployment. Many utilities still rely on human agents and rule- or inten…

• Think Less, Label Better: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunications

  Chenhua Shi, Gregor Macdonald, Bhavika Jalli, Wanlu Lei, John Zou, Mridul Jain, Joji Philip · 2. Februar 2026

  The success of large language models (LLMs) depends heavily on large-scale, high-quality instruction-following and reinforcement datasets. However, generating such data through human annotation is prohibitively time-consuming particularly for domain-specific tasks like telecom network troubleshootin…

• Clipping-Free Policy Optimization for Large Language Models

  \"Omer Veysel \c{C}a\u{g}atan, Bar{\i}\c{s} Akg\"un, G\"ozde G\"ul \c{S}ahin, Xuandong Zhao · 2. Februar 2026

  Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (…

• Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization

  Kanishk Awadhiya · 2. Februar 2026

  Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation. However, we identify a significant limitation we term ''Spectral Rigidity'': standard RoPE utilizes a fixed geometric decay ($\t…

• MetaLint: Generalizable Idiomatic Code Quality Analysis through Instruction-Following and Easy-to-Hard Generalization

  Atharva Naik, Lawanya Baghel, Dhakshin Govindarajan, Darsh Agrawal, Daniel Fried, Carolyn Rose · 2. Februar 2026

  Large Language Models excel at code generation but struggle with code quality analysis, where best practices evolve and cannot be fully captured by static training data. We introduce MetaLint, a training framework that treats code quality analysis as detecting best practice violations from high-leve…