Natural Language Processing Techniques

Latest papers

• Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position

  Zhixin Xie, Xurui Song, Jun Luo · 27 November 2025

  Diffusion Large Language Models (dLLMs) have recently emerged as a competitive non-autoregressive paradigm due to their unique training and inference approach. However, there is currently a lack of safety study on this novel architecture. In this paper, we present the first analysis of dLLMs' safety…

• RosettaSpeech: Zero-Shot Speech-to-Speech Translation from Monolingual Data

  Zhisheng Zheng, Xiaohang Sun, Tuan Dinh, Abhishek Yanamandra, Abhinav Jain, Zhu Liu, Sunil Hadap, Vimal Bhat, Manoj Aggarwal, Gerard Medioni, David Harwath · 27 November 2025

  The scarcity of parallel speech corpora critically hampers speech-to-speech translation (S2ST), often forcing reliance on complex, multi-stage pipelines. This paper introduces RosettaSpeech, a novel and simplified framework for zero-shot S2ST that is trained on monolingual speech-text data augmented…

• Alignment of large language models with constrained learning

  Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, Alejandro Ribeiro · 27 November 2025

  We study the problem of computing an optimal large language model (LLM) policy for the constrained alignment problem, where the goal is to maximize a primary reward objective while satisfying constraints on secondary utilities. Despite the popularity of Lagrangian-based LLM policy search in constrai…

• Special-Character Adversarial Attacks on Open-Source Language Model

  Ephraiem Sarabamoun · 27 November 2025

  Large language models (LLMs) have achieved remarkable performance across diverse natural language processing tasks, yet their vulnerability to character-level adversarial manipulations presents significant security challenges for real-world deployments. This paper presents a study of different speci…

• Uncovering Implicit Bias in Large Language Models with Concept Learning Dataset

  Leroy Z. Wang · 27 November 2025

  We introduce a dataset of concept learning tasks that helps uncover implicit biases in large language models. Using in-context concept learning experiments, we found that language models may have a bias toward upward monotonicity in quantifiers; such bias is less apparent when the model is tested by…

• BoundingDocs: a Unified Dataset for Document Question Answering with Spatial Annotations

  Simone Giovannini, Fabio Coppini, Andrea Gemelli, Simone Marinai · 27 November 2025

  We present a unified dataset for document Question-Answering (QA), which is obtained combining several public datasets related to Document AI and visually rich document understanding (VRDU). Our main contribution is twofold: on the one hand we reformulate existing Document AI tasks, such as Informat…

• Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining

  Mikey Shechter, Yair Carmon · 27 November 2025

  We introduce Filter Like You Test (FLYT), an algorithm for curating large-scale vision-language datasets that learns the usefulness of each data point as a pretraining example. FLYT trains a scoring model that learns to weigh each example's features using gradient signals from downstream tasks train…

• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning

  Jingchu Gai, Guanning Zeng, Huaqing Zhang, Aditi Raghunathan · 26 November 2025

  It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to \textit{diversity collapse}, where outputs lack variety. Prior work has proposed a range of heuristics to counteract this effect, but these methods are ad hoc: they frequently trade off corre…

• "When Data is Scarce, Prompt Smarter"... Approaches to Grammatical Error Correction in Low-Resource Settings

  Somsubhra De, Harsh Kumar, Arun Prakash A · 26 November 2025

  Grammatical error correction (GEC) is an important task in Natural Language Processing that aims to automatically detect and correct grammatical mistakes in text. While recent advances in transformer-based models and large annotated datasets have greatly improved GEC performance for high-resource la…

• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

  Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, Yi Wu · 26 November 2025

  Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are sy…

• Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unification

  Xin Wang, Yuwei Zhou, Bin Huang, Hong Chen, Wenwu Zhu · 26 November 2025

  Multi-modal generative AI (Artificial Intelligence) has attracted increasing attention from both academia and industry. Particularly, two dominant families of techniques have emerged: i) Multi-modal large language models (LLMs) demonstrate impressive ability for multi-modal understanding; and ii) Di…

• Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation

  Hao Wang, Yuanfeng Song, Xiaoming Yin, Xing Chen · 25 November 2025

  Text-to-SQL datasets are essential for training and evaluating text-to-SQL models, but existing datasets often suffer from limited coverage and fail to capture the diversity of real-world applications. To address this, we propose a novel taxonomy for text-to-SQL classification based on dimensions in…

• Equivalence of Context and Parameter Updates in Modern Transformer Blocks

  Adrian Goldwaser, Michael Munn, Javier Gonzalvo, Benoit Dherin · 25 November 2025

  Recent research has established that the impact of context in a vanilla transformer can be represented implicitly by forming a token-dependent, rank-1 patch to its MLP weights. This work extends that foundational theory to the diverse architectures of modern Large Language Models. We first demonstra…

• How does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective

  Shimao Zhang, Zhejian Lai, Xiang Liu, Shuaijie She, Xiao Liu, Yeyun Gong, Shujian Huang, Jiajun Chen · 25 November 2025

  Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some research on language-specific neurons provides a new perspective to analyze …

• From Projection to Prediction: Beyond Logits for Scalable Language Models

  Jianbing Dong, Jianbin Chang · 25 November 2025

  Training Large Language Models (LLMs) typically involves a two-stage pipeline at the output layer: hidden states are projected into vocabulary logits via a linear transformation (lm_head), followed by cross-entropy loss computation against target tokens. While conceptually simple, this design incurs…

• Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25

  Meng Lu, Catherine Chen, Carsten Eickhoff · 25 November 2025

  Mechanistic interpretation has greatly contributed to a more detailed understanding of generative language models, enabling significant progress in identifying structures that implement key behaviors through interactions between internal components. In contrast, interpretability in information retri…

• ChineseErrorCorrector3-4B: State-of-the-Art Chinese Spelling and Grammar Corrector

  Wei Tian, YuhaoZhou · 25 November 2025

  This paper introduces ChineseErrorCorrector3-4B, a unified model for Chinese spelling and grammatical error correction based on Qwen3-4B. The model demonstrates outstanding performance in general text correction tasks and achieves state-of-the-art results in both spelling correction (CSC) and gramma…

• Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets

  Gowtham, Sai Rupesh, Sanjay Kumar, Saravanan, Venkata Chaithanya · 25 November 2025

  High-quality training data is fundamental to large language model (LLM) performance, yet existing preprocessing pipelines often struggle to effectively remove noise and unstructured content from web-scale corpora. This paper presents Blu-WERP, a novel data preprocessing pipeline designed to optimize…

• SmolKalam: Ensemble Quality-Filtered Translation at Scale for High Quality Arabic Post-Training Data

  Sultan Alrashed, Chadi Helwe, Francesco Orabona · 25 November 2025

  Although the community has tackled the acquisition of high-quality Arabic pretraining data, we still lack large-scale, multi-turn Arabic datasets that include reasoning and tool calling. Naive translation can work at the pretraining scale, but post-training demands much higher quality, which require…

• OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph

  Michael J. Bommarito II · 25 November 2025

  We present OpenGloss, a synthetic encyclopedic dictionary and semantic knowledge graph for English that integrates lexicographic definitions, encyclopedic context, etymological histories, and semantic relationships in a unified resource. OpenGloss contains 537K senses across 150K lexemes, on par wit…

• Skeletons Matter: Dynamic Data Augmentation for Text-to-Query

  Yuchen Ji, Bo Xu, Jie Shi, Jiaqing Liang, Deqing Yang, Yu Mao, Hai Chen, Yanghua Xiao · 25 November 2025

  The task of translating natural language questions into query languages has long been a central focus in semantic parsing. Recent advancements in Large Language Models (LLMs) have significantly accelerated progress in this field. However, existing studies typically focus on a single query language, …

• How Well Do LLMs Understand Tunisian Arabic?

  Mohamed Mahdi · 24 November 2025

  Large Language Models (LLMs) are the engines driving today's AI agents. The better these models understand human languages, the more natural and user-friendly the interaction with AI becomes, from everyday devices like computers and smartwatches to any tool that can act intelligently. Yet, the abili…

• Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge

  Maximilian Abstreiter, Sasu Tarkoma, Roberto Morabito · 24 November 2025

  The rapid rise of Language Models (LMs) has expanded the capabilities of natural language processing, powering applications from text generation to complex decision-making. While state-of-the-art LMs often boast hundreds of billions of parameters and are primarily deployed in data centers, recent tr…

• Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning

  Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Ding Wang, Botian Shi · 24 November 2025

  Multimodal reasoning in Large Language Models (LLMs) struggles with incomplete knowledge and hallucination artifacts, challenges that textual Knowledge Graphs (KGs) only partially mitigate due to their modality isolation. While Multimodal Knowledge Graphs (MMKGs) promise enhanced cross-modal underst…

• Towards Formal Verification of LLM-Generated Code from Natural Language Prompts

  Aaron Councilman, David Jiahao Fu, Aryan Gupta, Chengxiao Wang, David Grove, Yu-Xiong Wang, Vikram Adve · 24 November 2025

  In the past few years LLMs have emerged as a tool that can aid programmers by taking natural language descriptions and generating code based on it. However, the reliability of LLM code generation and current validation techniques for it are far from strong enough to be used for mission-critical or s…