Natural Language Processing Techniques

Neueste Paper

• Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries

  Divyat Mahajan, Sachin Goyal, Badr Youbi Idrissi, Mohammad Pezeshki, Ioannis Mitliagkas, David Lopez-Paz, Kartik Ahuja · 26. März 2026

  Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by pre…

• Phrase-Instance Alignment for Generalized Referring Segmentation

  E-Ro Nguyen, Hieu Le, Dimitris Samaras, Michael S. Ryoo · 26. März 2026

  Generalized Referring expressions can describe one object, several related objects, or none at all. Existing generalized referring segmentation (GRES) models treat all cases alike, predicting a single binary mask and ignoring how linguistic phrases correspond to distinct visual instances. To this en…

• MDKeyChunker: Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAG

  Bhavik Mangla · 26. März 2026

  RAG pipelines typically rely on fixed-size chunking, which ignores document structure, fragments semantic units across boundaries, and requires multiple LLM calls per chunk for metadata extraction. We present MDKeyChunker, a three-stage pipeline for Markdown documents that (1) performs structure-awa…

• You only need 4 extra tokens: Synergistic Test-time Adaptation for LLMs

  Yijie Xu, Huizai Yao, Zhiyu Guo, Pengteng Li, Aiwei Liu, Xuming Hu, Weiyu Guo, Hui Xiong · 26. März 2026

  Large language models (LLMs) are increasingly deployed in specialized domains such as finance, medicine, and agriculture, where they face significant distribution shifts from their training data. Domain-specific fine-tuning can mitigate this challenge but relies on high-quality labeled data that is …

• Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation

  Reza Habibi, Darian Lee, Magy Seif El-Nasr · 26. März 2026

  Accuracy-based evaluation cannot reliably distinguish genuine generalization from shortcuts like memorization, leakage, or brittle heuristics, especially in small-data regimes. In this position paper, we argue for mechanism-aware evaluation that combines task-relevant symbolic rules with mechanistic…

• LatentQA: Teaching LLMs to Decode Activations Into Natural Language

  Alexander Pan, Lijie Chen, Jacob Steinhardt · 25. März 2026

  Top-down transparency typically analyzes language model activations using probes with scalar or single-token outputs, limiting the range of behaviors that can be captured. To alleviate this issue, we develop a more expressive probe that can directly output natural language, performing LatentQA: the …

• Happiness is Sharing a Vocabulary: A Study of Transliteration Methods

  Haeji Jung, Jinju Kim, Kyungjin Kim, Youjeong Roh, David R. Mortensen · 25. März 2026

  Transliteration has emerged as a promising means to bridge the gap between various languages in multilingual NLP, showing promising results especially for languages using non-Latin scripts. We investigate the degree to which shared script, overlapping token vocabularies, and shared phonology contrib…

• BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactions

  Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, Reynold Cheng · 25. März 2026

  Large language models (LLMs) have demonstrated remarkable performance on single-turn text-to-SQL tasks, but real-world database applications predominantly require multi-turn interactions to handle ambiguous queries, execution errors, and evolving user requirements. Existing multi-turn benchmarks fal…

• Toward a Theory of Hierarchical Memory for Language Agents

  Yashar Talebirad, Ali Parsaee, Csongor Y. Szepesvari, Amirhossein Nadiri, Osmar Zaiane · 24. März 2026

  Many recent long-context and agentic systems address context-length limitations by adding hierarchical memory: they extract atomic units from raw data, build multi-level representatives by grouping and compression, and traverse this structure to retrieve content under a token budget. Despite recurri…

• MKA: Memory-Keyed Attention for Efficient Long-Context Reasoning

  Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu · 24. März 2026

  As long-context language modeling becomes increasingly important, the cost of maintaining and attending to large Key/Value (KV) caches grows rapidly, becoming a major bottleneck in both training and inference. While prior works such as Multi-Query Attention (MQA) and Multi-Latent Attention (MLA) red…

• RoboMorph: Evolving Robot Morphology using Large Language Models

  Kevin Qiu, W{\l}adys{\l}aw Pa{\l}ucki, Krzysztof Ciebiera, Pawe{\l} Fija{\l}kowski, Marek Cygan, {\L}ukasz Kuci\'nski · 24. März 2026

  We introduce RoboMorph, an automated approach for generating and optimizing modular robot designs using large language models (LLMs) and evolutionary algorithms. Each robot design is represented by a structured grammar, and we use LLMs to efficiently explore this design space. Traditionally, such ex…

• Task-Specific Efficiency Analysis: When Small Language Models Outperform Large Language Models

  Jinghan Cao, Yu Ma, Xinjin Li, Qingyang Ren, Xiangyun Chen · 24. März 2026

  Large Language Models achieve remarkable performance but incur substantial computational costs unsuitable for resource-constrained deployments. This paper presents the first comprehensive task-specific efficiency analysis comparing 16 language models across five diverse NLP tasks. We introduce the P…

• Epistemic Observability in Language Models

  Tony Mason · 24. März 2026

  We find that models report highest confidence precisely when they are fabricating. Across four model families (OLMo-3, Llama-3.1, Qwen3, Mistral), self-reported confidence inversely correlates with accuracy, with AUC ranging from 0.28 to 0.36 where 0.5 is random guessing. We prove, under explicit …

• A Training-free Method for LLM Text Attribution

  Tara Radvand, Mojtaba Abdolmaleki, Mohamed Mostagir, Ambuj Tewari · 24. März 2026

  Verifying the provenance of content is crucial to the functioning of many organizations, e.g., educational institutions, social media platforms, and firms. This problem is becoming increasingly challenging as text generated by Large Language Models (LLMs) becomes almost indistinguishable from human-…

• Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala

  Minuri Rajapakse, Ruvan Weerasinghe · 24. März 2026

  The performance of Language Models (LMs) on low-resource, morphologically rich languages like Sinhala remains largely unexplored, particularly regarding script variation in digital communication. Sinhala exhibits script duality, with Unicode used in formal contexts and Romanized text dominating soci…

• Efficient Fine-Tuning Methods for Portuguese Question Answering: A Comparative Study of PEFT on BERTimbau and Exploratory Evaluation of Generative LLMs

  Mariela M. Nina, Caio Veloso Costa, Lilian Berton, Didier A. Vega-Oliveros · 24. März 2026

  Although large language models have transformed natural language processing, their computational costs create accessibility barriers for low-resource languages such as Brazilian Portuguese. This work presents a systematic evaluation of Parameter-Efficient Fine-Tuning (PEFT) and quantization techniqu…

• Enhancing Document-Level Machine Translation via Filtered Synthetic Corpora and Two-Stage LLM Adaptation

  Ireh Kim, Tesia Sker, Chanwoo Kim · 24. März 2026

  In Machine Translation, Large Language Models (LLMs) have generally underperformed compared to conventional encoder-decoder systems and thus see limited adoption. However, LLMs excel at modeling contextual information, making them a natural fit for document-level translation tasks where coherence ac…

• LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs

  Zifan He, Shengyu Ye, Rui Ma, Yang Wang, Jason Cong · 24. März 2026

  The rapid development of large language models (LLM) has greatly enhanced everyday applications. While many FPGA-based accelerators, with flexibility for fine-grained data control, exhibit superior speed and energy efficiency compared to GPUs, recent GPU-specific optimizations have diminished this a…

• Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF

  K. M. Jubair Sami, Dipto Sumit, Ariyan Hossain, Farig Sadeque · 24. März 2026

  Large language models (LLMs) frequently exhibit performance biases against regional dialects of low-resource languages. However, frameworks to quantify these disparities remain scarce. We propose a two-phase framework to evaluate dialectal bias in LLM question-answering across nine Bengali dialects.…

• On the Failure of Topic-Matched Contrast Baselines in Multi-Directional Refusal Abliteration

  Valentin Petrov · 24. März 2026

  Inasmuch as the removal of refusal behavior from instruction-tuned language models by directional abliteration requires the extraction of refusal-mediating directions from the residual stream activation space, and inasmuch as the construction of the contrast baseline against which harmful prompt act…

• Learning to Interpret Weight Differences in Language Models

  Avichal Goel, Yoon Kim, Nir Shavit, Tony T. Wang · 24. März 2026

  Finetuning (pretrained) language models is a standard approach for updating their internal parametric knowledge and specializing them to new tasks and domains. However, the corresponding model weight changes ("weight diffs") are not generally interpretable. While inspecting the finetuning dataset ca…

• Breaking the Silence: A Dataset and Benchmark for Bangla Text-to-Gloss Translation

  Sharif Mohammad Abdullah, Abhijit Paul, Shubhashis Roy Dipta, Zarif Masud, Shebuti Rayana, Ahmedul Kabir · 24. März 2026

  Gloss is a written approximation that bridges Sign Language (SL) and its corresponding spoken language. Despite a deaf and hard-of-hearing population of at least 3 million in Bangladesh, Bangla Sign Language (BdSL) remains largely understudied, with no prior work on Bangla text-to-gloss translation …

• Significance-Gain Pair Encoding for LLMs: A Statistical Alternative to Frequency-Based Subword Merging

  Azam Nouri · 23. März 2026

  Subword tokenization is a key design choice for modern language models, including large language models (LLMs), with byte- and character-level BPE serving as a widely used baseline. Standard BPE selects merges by raw pair frequency, which favors compression but can conflate true adjacency cohesion w…

• The $\mathbf{Y}$-Combinator for LLMs: Solving Long-Context Rot with $\lambda$-Calculus

  Amartya Roy, Rasul Tutunov, Xiaotong Ji, Matthieu Zimmer, Haitham Bou-Ammar · 23. März 2026

  LLMs are increasingly used as general-purpose reasoners, but long inputs remain bottlenecked by a fixed context window. Recursive Language Models (RLMs) address this by externalising the prompt and recursively solving subproblems. Yet existing RLMs depend on an open-ended read-eval-print loop (REPL)…

• LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages

  Godwin Abuh Faruna · 23. März 2026

  Safety alignment in large language models relies predominantly on English-language training data. When harmful intent is expressed in low-resource languages, refusal mechanisms that hold in English frequently fail to activate. We introduce LSR (Linguistic Safety Robustness), the first systematic ben…