Natural Language Processing Techniques

Neueste Paper

• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation

  Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu · 15. Dezember 2025

  In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in \textit{Multi-Linguality}, \textit{Multi-Functionality}, and \textit{Multi-Granularity}. It provides a uniform support for the semantic retrieval of more than 100 working languages. I…

• Large Continual Instruction Assistant

  Jingyang Qiao, Zhizhong Zhang, Xin Tan, Yanyun Qu, Shouhong Ding, Yuan Xie · 15. Dezember 2025

  Continual Instruction Tuning (CIT) is adopted to continually instruct Large Models to follow human intent data by data. It is observed that existing gradient update would heavily destroy the performance on previous datasets during CIT process. Instead, Exponential Moving Average (EMA), owns the abil…

• SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning

  Huanyu Liu, Ge Li, Jia Li, Hao Zhu, Kechi Zhang, Yihong Dong · 15. Dezember 2025

  How to design reinforcement learning (RL) tasks that effectively unleash the reasoning capability of large language models (LLMs) remains an open question. Existing RL tasks (e.g., math, programming, and constructing reasoning tasks) suffer from three key limitations: (1) Scalability. They rely heav…

• Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs

  Mikhail Menschikov, Alexander Kharitonov, Maiia Kotyga, Vadim Porvatov, Anna Zhukovskaya, David Kagramanyan, Egor Shvetsov, Evgeny Burnaev · 15. Dezember 2025

  Large Language Models (LLMs) exhibit position bias systematically underweighting information based on its location in the context but how this bias varies across languages and models remains unclear. We conduct a multilingual study across five typologically diverse languages (English, Russian, Germa…

• MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Data

  Christopher Driggers-Ellis, Detravious Brinkley, Ray Chen, Aashish Dhawan, Daisy Zhe Wang, Christan Grant · 15. Dezember 2025

  Multi30k is frequently cited in the multimodal machine translation (MMT) literature, offering parallel text data for training and fine-tuning deep learning models. However, it is limited to four languages: Czech, English, French, and German. This restriction has led many researchers to focus their i…

• Grammar-Aligned Decoding

  Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, Loris D'Antoni · 15. Dezember 2025

  Large Language Models (LLMs) struggle with reliably generating highly structured outputs, such as program code, mathematical formulas, or well-formed markup. Constrained decoding approaches mitigate this problem by greedily restricting what tokens an LLM can output at each step to guarantee that the…

• Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence

  Sophia Hager, David Mueller, Kevin Duh, Nicholas Andrews · 15. Dezember 2025

  As large language models (LLMs) are increasingly used for factual question-answering, it becomes more important for LLMs to have the capability to communicate the likelihood that their answer is correct. For these verbalized expressions of uncertainty to be meaningful, they should reflect the error …

• Same model, better performance: the impact of shuffling on DNA Language Models benchmarking

  Davide Greco, Konrad Rawlik · 12. Dezember 2025

  Large Language Models are increasingly popular in genomics due to their potential to decode complex biological sequences. Hence, researchers require a standardized benchmark to evaluate DNA Language Models (DNA LMs) capabilities. However, evaluating DNA LMs is a complex task that intersects genomic'…

• Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning

  Tianle Zhang, Wanlong Fang, Jonathan Woo, Paridhi Latawa, Deepak A. Subramanian, Alvin Chan · 12. Dezember 2025

  The remarkable performance of Large Language Models (LLMs) can be enhanced with test-time computation, which relies on external tools and even other deep learning models. However, existing approaches for integrating non-text modality representations into LLMs typically require additional costly supe…

• PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Code

  Itay Dreyfuss, Antonio Abu Nassar, Samuel Ackerman, Axel Ben David, Rami Katan, Orna Raz, Marcel Zalmanovici · 12. Dezember 2025

  Large Language Model (LLM)-based code assistants have emerged as a powerful application of generative AI, demonstrating impressive capabilities in code generation and comprehension. A key requirement for these systems is their ability to accurately follow user instructions. We present Precise Automa…

• Local LLM Ensembles for Zero-shot Portuguese Named Entity Recognition

  Jo\~ao Lucas Luz Lima Sarcinelli, Diego Furtado Silva · 12. Dezember 2025

  Large Language Models (LLMs) excel in many Natural Language Processing (NLP) tasks through in-context learning but often under-perform in Named Entity Recognition (NER), especially for lower-resource languages like Portuguese. While open-weight LLMs enable local deployment, no single model dominates…

• Token Sample Complexity of Attention

  L\'ea Bohbot, Cyril Letrouit, Gabriel Peyr\'e, Fran\c{c}ois-Xavier Vialard · 12. Dezember 2025

  As context windows in large language models continue to expand, it is essential to characterize how attention behaves at extreme sequence lengths. We introduce token-sample complexity: the rate at which attention computed on $n$ tokens converges to its infinite-token limit. We estimate finite-$n$ co…

• Provably Learning from Modern Language Models via Low Logit Rank

  Noah Golowich, Allen Liu, Abhishek Shetty · 11. Dezember 2025

  While modern language models and their inner workings are incredibly complex, recent work (Golowich, Liu & Shetty; 2025) has proposed a simple and potentially tractable abstraction for them through the observation that empirically, these language models all seem to have approximately low logit rank.…

• Attention Sinks in Diffusion Language Models

  Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michielon, Fabrizio Silvestri, Simone Scardapane, Alessio Devoto · 11. Dezember 2025

  Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs). DLMs employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. Although their efficiency…

• Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search

  Ekaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii, Roman Vashurin, Artem Shelmanov, Preslav Nakov, Mrinmaya Sachan, Maxim Panov · 11. Dezember 2025

  Consistency-based methods have emerged as an effective approach to uncertainty quantification (UQ) in large language models. These methods typically rely on several generations obtained via multinomial sampling, measuring their agreement level. However, in short-form QA, multinomial sampling is pron…

• LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL

  Dzmitry Pihulski, Karol Charchut, Viktoria Novogrodskaia, Jan Koco\'n · 10. Dezember 2025

  Converting natural language questions into SQL queries enables non-expert users to interact with relational databases and has long been a central task for natural language interfaces to data. While the WikiSQL dataset played a key role in early text-to-SQL research, its usage has declined due to str…

• Do Natural Language Descriptions of Model Activations Convey Privileged Information?

  Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers, Naomi Saphra, Byron C. Wallace · 10. Dezember 2025

  Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illuminate how the target model represents and operates on inputs. But do such activation verbalization approaches actually pr…

• MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

  Yufei Gao, Jiaying Fei, Nuo Chen, Ruirui Chen, Guohang Yan, Yunshi Lan, Botian Shi · 10. Dezember 2025

  Multimodal Large Language Models (MLLMs) have shown remarkable performance in high-resource languages. However, their effectiveness diminishes significantly in the contexts of low-resource languages. Current multilingual enhancement methods are often limited to text modality or rely solely on machin…

• Short-Context Dominance: How Much Local Context Natural Language Actually Needs?

  Vala Vakilian, Zimeng Wang, Ankit Singh Rawat, Christos Thrampoulidis · 10. Dezember 2025

  We investigate the short-context dominance hypothesis: that for most sequences, a small local prefix suffices to predict their next tokens. Using large language models as statistical oracles, we measure the minimum context length (MCL) needed to reproduce accurate full-context predictions across dat…

• Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages

  David Samuel, Lilja {\O}vrelid, Erik Velldal, Andrey Kutuzov · 10. Dezember 2025

  We propose a post-training method for lower-resource languages that preserves fluency of language models even when aligned by disfluent reward models. Preference-optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languag…

• Optimizing LLMs Using Quantization for Mobile Execution

  Agatsya Yadav, Renta Chintala Bhargavi · 9. Dezember 2025

  Large Language Models (LLMs) offer powerful capabilities, but their significant size and computational requirements hinder deployment on resource-constrained mobile devices. This paper investigates Post-Training Quantization (PTQ) for compressing LLMs for mobile execution. We apply 4-bit PTQ using t…

• KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models

  Sourjya Roy, Shrihari Sridharan, Surya Selvam, Anand Raghunathan · 9. Dezember 2025

  As Large Language Models (LLMs) scale in size and context length, the memory requirements of the key value (KV) cache have emerged as a major bottleneck during autoregressive decoding. The KV cache grows with sequence length and embedding dimension, often exceeding the memory footprint of the model …

• Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation

  Boxuan Lyu, Haiyue Song, Hidetaka Kamigaito, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Kotaro Funakoshi, Manabu Okumura · 9. Dezember 2025

  Error Span Detection (ESD) is a subtask of automatic machine translation evaluation that localizes error spans in translations and labels their severity. State-of-the-art generative ESD methods typically decode using Maximum a Posteriori (MAP), assuming that model-estimated probabilities are perfect…

• Mastering AI: Big Data, Deep Learning, and the Evolution of Large Language Models -- AutoML from Basics to State-of-the-Art Techniques

  Pohsun Feng, Ziqian Bi, Yizhu Wen, Benji Peng, Junyu Liu, Caitlyn Heqi Yin, Tianyang Wang, Keyu Chen, Sen Zhang, Ming Li, Jiawei Xu, Ming Liu, Xuanhe Pan, Jinlang Wang, Xinyuan Song, Qian Niu · 9. Dezember 2025

  A comprehensive guide to Automated Machine Learning (AutoML) is presented, covering fundamental principles, practical implementations, and future trends. The paper is structured to assist both beginners and experienced practitioners, with detailed discussions on popular AutoML tools such as TPOT, Au…

• A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning

  Ruiyi Wang, Prithviraj Ammanabrolu · 9. Dezember 2025

  We study what actually works and what doesn't for training large language models as agents via multi-turn reinforcement learning. Despite rapid progress, existing frameworks and definitions are fragmented, and there is no systematic formulation or analysis of which design choices matter across tasks…