Natural Language Processing Techniques

Latest papers

• Integrating Symbolic Natural Language Understanding and Language Models for Word Sense Disambiguation

  Kexin Zhao, Ken Forbus · 21 November 2025

  Word sense disambiguation is a fundamental challenge in natural language understanding. Current methods are primarily aimed at coarse-grained representations (e.g. WordNet synsets or FrameNet frames) and require hand-annotated training data to construct. This makes it difficult to automatically disa…

• Constraint-Guided Prediction Refinement via Deterministic Diffusion Trajectories

  Pantelis Dogoulis, Fabien Bernier, F\'elix Fourreau, Karim Tit, Maxime Cordy · 21 November 2025

  Many real-world machine learning tasks require outputs that satisfy hard constraints, such as physical conservation laws, structured dependencies in graphs, or column-level relationships in tabular data. Existing approaches rely either on domain-specific architectures and losses or on strong assumpt…

• Anatomy of an Idiom: Tracing Non-Compositionality in Language Models

  Andrew Gomes · 21 November 2025

  We investigate the processing of idiomatic expressions in transformer-based language models using a novel set of techniques for circuit discovery and analysis. First discovering circuits via a modified path patching algorithm, we find that idiom processing exhibits distinct computational patterns. W…

• CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners

  Yunzhi Yao, Jizhan Fang, Jia-Chen Gu, Ningyu Zhang, Shumin Deng, Huajun Chen, Nanyun Peng · 21 November 2025

  Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can update isolated facts, they often fail to generalize these updates to multi-hop reasoning tasks that rely on the modified knowledge. Through an analysis…

• Eliciting Reasoning in Language Models with Cognitive Tools

  Brown Ebouky, Andrea Bartezzaghi, Mattia Rigotti · 21 November 2025

  The recent advent of reasoning models like OpenAI's o1 was met with excited speculation by the AI community about the mechanisms underlying these capabilities in closed models, followed by a rush of replication efforts, particularly from the open source community. These speculations were largely set…

• PRIMUS: Pretraining IMU Encoders with Multimodal Self-Supervision

  Arnav M. Das, Chi Ian Tang, Fahim Kawsar, Mohammad Malekzadeh · 19 November 2025

  Sensing human motions through Inertial Measurement Units (IMUs) embedded in personal devices has enabled significant applications in health and wellness. Labeled IMU data is scarce, however, unlabeled or weakly labeled IMU data can be used to model human motions. For video or text modalities, the "p…

• IPAD: Inverse Prompt for AI Detection - A Robust and Interpretable LLM-Generated Text Detector

  Zheng Chen, Yushi Feng, Jisheng Dang, Yue Deng, Changyang He, Hongxi Pu, Haoxuan Li, Bo Li · 19 November 2025

  Large Language Models (LLMs) have attained human-level fluency in text generation, which complicates the distinguishing between human-written and LLM-generated texts. This increases the risk of misuse and highlights the need for reliable detectors. Yet, existing detectors exhibit poor robustness on …

• AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

  Mohammad Zbib, Hasan Abed Al Kader Hammoud, Sina Mukalled, Nadine Rizk, Fatima Karnib, Issam Lakkis, Ammar Mohanna, Bernard Ghanem · 19 November 2025

  We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 expert-designed multiple choice question…

• PIP: Perturbation-based Iterative Pruning for Large Language Models

  Yi Cao, Wei-Jie Xu, Yucheng Shen, Weijie Shi, Chi-Min Chan, Jianfeng Qu, Jiajie Xu · 18 November 2025

  The rapid increase in the parameter counts of Large Language Models (LLMs), which often reach into the billions or even trillions, presents significant challenges for their practical deployment, particularly in resource-constrained environments. To address this issue, we propose PIP (Perturbation-ba…

• What You See Is Not Always What You Get: Evaluating GPT's Comprehension of Source Code

  Jiawen Wen, Bangshuo Zhu, Huaming Chen · 18 November 2025

  Recent studies have demonstrated outstanding capabilities of large language models (LLMs) in software engineering tasks, including code generation and comprehension. While LLMs have shown significant potential in assisting with coding, LLMs are vulnerable to adversarial attacks. In this paper, we in…

• Fine-Grained Uncertainty Decomposition in Large Language Models: A Spectral Approach

  Nassim Walha, Sebastian G. Gruber, Thomas Decker, Yinchong Yang, Alireza Javanmardi, Eyke H\"ullermeier, Florian Buettner · 18 November 2025

  As Large Language Models (LLMs) are increasingly integrated in diverse applications, obtaining reliable measures of their predictive uncertainty has become critically important. A precise distinction between aleatoric uncertainty, arising from inherent ambiguities within input data, and epistemic un…

• Larger Datasets Can Be Repeated More: A Theoretical Analysis of Multi-Epoch Scaling in Linear Regression

  Tingkai Yan, Haodong Wen, Binghui Li, Kairong Luo, Wenguang Chen, Kaifeng Lyu · 18 November 2025

  While data scaling laws of large language models (LLMs) have been widely examined in the one-pass regime with massive corpora, their form under limited data and repeated epochs remains largely unexplored. This paper presents a theoretical analysis of how a common workaround, training for multiple ep…

• On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning

  Simon Kurz, Jian-Jia Chen, Lucie Flek, Zhixue Zhao · 18 November 2025

  Recent advances in large language model (LLM) pruning have shown state-of-the-art (SotA) compression results in post-training and retraining-free settings while maintaining high predictive performance. However, previous research mainly considered calibrating based on English text, despite the multil…

• On the Fundamental Limits of LLMs at Scale

  Muhammad Ahmed Mohsin, Muhammad Umer, Ahsan Bilal, Zeeshan Memon, Muhammad Ibtsaam Qadir, Sagnik Bhattacharya, Hassan Rizwan, Abhiram R. Gorle, Maahe Zehra Kazmi, Ayesha Mohsin, Muhammad Usman Rafique, Zihao He, Pulkit Mehta, Muhammad Ali Jamshed, John M. Cioffi · 18 November 2025

  Large Language Models (LLMs) have benefited enormously from scaling, yet these gains are bounded by five fundamental limitations: (1) hallucination, (2) context compression, (3) reasoning degradation, (4) retrieval fragility, and (5) multimodal misalignment. While existing surveys describe these phe…

• ICL-Router: In-Context Learned Model Representations for LLM Routing

  Chenxu Wang, Hao Li, Yiqun Zhang, Linyao Chen, Jianhao Chen, Ping Jian, Peng Ye, Qiaosheng Zhang, Shuyue Hu · 17 November 2025

  Large language models (LLMs) often exhibit complementary strengths. Model routing harnesses these strengths by dynamically directing each query to the most suitable model, given a candidate model pool. However, routing performance relies on accurate model representations, and adding new models typic…

• EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training

  Qingao Yi, Jiaang Duan, Hanwen Hu, Qin Hua, Haiyan Zhao, Shiyou Qian, Dingyu Yang, Jian Cao, Jinghua Tang, Yinghao Yu, Chenzhi Liao, Kangjin Wang, Liping Zhang · 14 November 2025

  Training large language models (LLMs) poses significant challenges regarding computational resources and memory capacity. Although distributed training techniques help mitigate these issues, they still suffer from considerable communication overhead. Existing approaches primarily rely on static grad…

• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

  Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas Shakkottai · 14 November 2025

  Inference-time alignment enables large language models (LLMs) to generate outputs aligned with end-user preferences without further training. Recent post-training methods achieve this by using small guidance models to modify token generation during inference. These methods typically optimize a rewar…

• NSL-MT: Linguistically Informed Negative Samples for Efficient Machine Translation in Low-Resource Languages

  Mamadou K. Keita, Christopher Homan, Huy Le · 13 November 2025

  We introduce Negative Space Learning MT (NSL-MT), a training method that teaches models what not to generate by encoding linguistic constraints as severity-weighted penalties in the loss function. NSL-MT increases limited parallel data with synthetically generated violations of target language gramm…

• SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning

  Huanxuan Liao, Yixing Xu, Shizhu He, Guanchen Li, Xuanwu Yin, Dong Li, Emad Barsoum, Jun Zhao, Kang Liu · 13 November 2025

  Long-context inference in large language models (LLMs) is increasingly constrained by the KV cache bottleneck: memory usage grows linearly with sequence length, while attention computation scales quadratically. Existing approaches address this issue by compressing the KV cache along the temporal axi…

• LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls

  Kangning Zhang, Wenxiang Jiao, Kounianhua Du, Yuan Lu, Weiwen Liu, Weinan Zhang, Lei Zhang, Yong Yu · 13 November 2025

  Augmenting Large Language Models (LLMs) with external tools enables them to execute complex, multi-step tasks. However, tool learning is hampered by the static synthetic data pipelines where data generation and model training are executed as two separate, non-interactive processes. This approach fai…

• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity

  Susav Shrestha, Brad Settlemyer, Nikoli Dryden, Narasimha Reddy · 13 November 2025

  Accelerating large language model (LLM) inference is critical for real-world deployments requiring high throughput and low latency. Contextual sparsity, where each token dynamically activates only a small subset of the model parameters, shows promise but does not scale to large batch sizes due to un…

• Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning

  Yurun Yuan, Fan Chen, Zeyu Jia, Alexander Rakhlin, Tengyang Xie · 13 November 2025

  Policy-based methods currently dominate reinforcement learning (RL) pipelines for large language model (LLM) reasoning, leaving value-based approaches largely unexplored. We revisit the classical paradigm of Bellman Residual Minimization and introduce Trajectory Bellman Residual Minimization (TBRM),…

• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning

  Wenlong Deng, Yi Ren, Yushu Li, Boying Gong, Danica J. Sutherland, Xiaoxiao Li, Christos Thrampoulidis · 13 November 2025

  Reinforcement learning with verifiable rewards has significantly advanced the reasoning capabilities of large language models, yet how to explicitly steer training toward exploration or exploitation remains an open problem. We introduce Token Hidden Reward (THR), a token-level metric that quantifies…

• Training and Evaluating Language Models with Template-based Data Generation

  Yifan Zhang · 13 November 2025

  The rapid advancement of large language models (LLMs) such as GPT-3, PaLM, and Llama has significantly transformed natural language processing, showcasing remarkable capabilities in understanding and generating language. However, a fundamental bottleneck persists: these models often struggle with ta…

• MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages

  Chenxi Whitehouse, Sebastian Ruder, Tony Lin, Oksana Kurylo, Haruka Takagi, Janice Lam, Nicol\`o Busetto, Denise Diaz, Francisco Guzm\'an · 12 November 2025

  Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a datas…