Natural Language Processing Techniques

Derniers papiers

• Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM

  Zehao Fan, Garrett Gagnon, Zhenyu Liu, Liu Liu · 13 mai 2026

  Transformer-based models are the foundation of modern machine learning, but their execution, particularly during autoregressive decoding in large language models (LLMs), places significant pressure on memory systems due to frequent memory accesses and growing key-value (KV) caches. This creates a bo…

• Modality-Inconsistent Continual Learning of Multimodal Large Language Models

  Weiguo Pian, Shijian Deng, Shentong Mo, Mingrui Liu, Yunhui Guo, Yapeng Tian · 13 mai 2026

  In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with inconsistent modalities (image, audio, or video) and varying task types (captioning or question-answering). Unlike exis…

• CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration

  Yuning Han, Yangchenchen Jin, Dylan Zhao, Jingwei Sun · 13 mai 2026

  Auto-regressive decoding in Large Language Models (LLMs) is inherently memory-bound: every generation step requires loading the model weights and intermediate results from memory (e.g., High-Bandwidth Memory (HBM) for GPU servers), making throughput bottlenecked by memory bandwidth rather than compu…

• TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

  Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa Doan, Trung Le · 13 mai 2026

  Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bra…

• Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

  Yan Gao, Yazheng Yang, Zhibin Lan, Yidong Chen, Min Zhang, Daimeng Wei, Derek F. Wong, Jinsong Su · 13 mai 2026

  Code-switching (CS) speech translation (ST) aims to translate speech that alternates between multiple languages into a target language text, posing significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies mainly rely on the models themselves to …

• It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

  Yong-eun Cho · 13 mai 2026

  This paper experimentally analyzes how the level of harness engineering affects the operational performance of small language models (SLMs, 2-3B parameters). Three harness conditions - model-only (raw prompt), minimal-shell (wrapper tags), and a 4-stage pipeline (plan->execute->verify->recover) - ar…

• Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobs

  Eitan Wagner, Omri Abend · 13 mai 2026

  Language modeling has shifted in recent years from a distribution over strings to prediction models with textual inputs and outputs for general-purpose tasks. This position paper highlights the often overlooked implications of this shift for the use of large language models (LLMs) as probability est…

• Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies

  Jingze Song, Zihao Chen, Wenqing Chen, Zibin Zheng · 12 mai 2026

  Data selection is a key component of efficient instruction tuning for large language models, as recent work has shown that data quality often matters more than data quantity. Accordingly, prior studies have introduced various multi-dimensional heuristics to evaluate and filter instruction data. Howe…

• On Distinguishing Capability Elicitation from Capability Creation in Post-Training: A Free-Energy Perspective

  Yuhao Li, Shengchao Liu · 12 mai 2026

  Debates about large language model post-training often treat supervised fine-tuning (SFT) as imitation and reinforcement learning (RL) as discovery. But this distinction is too coarse. What matters is whether a training procedure increases the probability of behaviors the pretrained model could alre…

• Budget-Efficient Automatic Algorithm Design via Code Graph

  Maxime Bouscary, Manxi Wu, Saurabh Amin · 12 mai 2026

  Large language models (LLMs) have emerged as powerful tools for automatic algorithm design (AAD). However, existing pipelines remain inefficient. They operate at the granularity of full algorithms, redundantly rewriting recurring substructures and discarding low-fitness candidates that may contain v…

• RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache

  Junkai Zhang, Hang Guo, Luca Benini, Yawei Li · 12 mai 2026

  Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-Value (KV) cache size grows linearly with sequence length and needs to be re-read from off-chip high-bandwidth memory (H…

• From Traditional Taggers to LLMs: A Comparative Study of POS Tagging for Medieval Romance Languages

  Matthias Sch\"offel, Esteban Garces Arias · 12 mai 2026

  Part-of-speech (POS) tagging for Medieval Romance languages remains challenging due to orthographic variation, morphological complexity, and limited annotated resources. This paper presents a systematic empirical evaluation of large language models (LLMs) for POS tagging across three medieval variet…

• CauSim: Scaling Causal Reasoning with Increasingly Complex Causal Simulators

  Nicol\'as Astorga, Anita Kriz, Mihaela van der Schaar · 12 mai 2026

  Despite surpassing human performance across mathematics, coding, and other knowledge-intensive tasks, large language models (LLMs) continue to struggle with causal reasoning. A core obstacle is the target data itself: causal systems are complex and often expressed in non-executable forms, while grou…

• HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing

  Yuan Fang, Yi Xie, Xuming Ran · 12 mai 2026

  Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment, yet retraining is costly prohibitive, motivating model editing in lifelong settings that updates targeted behavior without harming the rest of the model. One line of work installs new …

• Future Validity is the Missing Statistic: From Impossibility to $\Phi$-Estimation for Grammar-Faithful Speculative Decoding

  Wenhua Nie, Zijie Meng, Kun Zou, Zheng Lin, Ziwei Li, Haoran Zheng, Jyh-Shing Roger Jang, Hao Zhang · 11 mai 2026

  Grammar-constrained generation is often combined with local vocabulary masking and speculative decoding, but the resulting sampling law is not the grammar-conditional distribution users usually intend. We show that any speculative decoder with local mask access, Leviathan rejection, and rollback sou…

• Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation

  Amin Karimi Monsefi, Dominic Culver, Nikhil Bhendawade, Manuel R. Ciosici, Yizhe Zhang, Irina Belousova · 11 mai 2026

  Discrete flow matching generates text by iteratively transforming noise tokens into coherent language, but may require hundreds of forward passes. Distillation uses the multi-step trajectory to train a student to reproduce the process in a few steps. When the student underperforms, the usual explana…

• CA-SQL: Complexity-Aware Inference Time Reasoning for Text-to-SQL via Exploration and Compute Budget Allocation

  James Petullo, Nianwen Xue · 11 mai 2026

  While recent advancements in inference-time learning have improved LLM reasoning on Text-to-SQL tasks, current solutions still struggle to perform well on the most challenging tasks in the Bird-Bench (BIRD) benchmark. This is due to inadequate solution space exploration, which is necessary to uncove…

• Conformal Agent Error Attribution

  Naihe Feng, Yi Sui, Shiyi Hou, Ga Wu, Jesse C. Cresswell · 11 mai 2026

  When multi-agent systems (MAS) fail, identifying where the decisive error occurred is the first step for automated recovery to an earlier state. Error attribution remains a fundamental challenge due to the long interaction traces that large language model-based MAS generate. This paper presents a fr…

• When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification

  Wenjie Guan, Jelena Bradic · 11 mai 2026

  Template tasks have emerged as a clean testbed for asking whether transformers reason with abstract symbols rather than concrete token names. We study the fixed-label classification version of this problem, where train and test examples share latent templates but may use disjoint vocabularies. Unlik…

• FastOmniTMAE: Parallel Clause Learning for Scalable and Hardware-Efficient Tsetlin Embeddings

  Ahmed K. Kadhim, Lei Jiao, Rishad Shafik, Ole-Christoffer Granmo, Mayur Kishor Shende · 11 mai 2026

  Embedding models in natural language processing (NLP) increasingly rely on deep architectures such as BERT, while simpler models such as Word2Vec provide efficient representations but limited interpretability. The Tsetlin Machine (TM) offers an alternative logic-based learning paradigm. Omni TM Auto…

• How Do Language Models Compose Functions?

  Apoorv Khandelwal, Ellie Pavlick · 11 mai 2026

  While large language models (LLMs) appear to be increasingly capable of solving compositional tasks, it is an open question whether they do so using compositional mechanisms. In this work, we investigate how feedforward LLMs solve two-hop factual recall tasks, which can be expressed compositionally …

• MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference

  Ruijie Zhou, Fanxu Meng, Yufei Xu, Tongxuan Liu, Guangming Lu, Muhan Zhang, Wenjie Pei · 11 mai 2026

  DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix token and selects the most relevant ones for the main attention. To remain expressive, the indexer uses many query heads (fo…

• TIDE: Every Layer Knows the Token Beneath the Context

  Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho · 8 mai 2026

  We revisit a universally accepted but under-examined design choice in every modern LLM: a token index is looked up once at the input embedding layer and then permanently discarded. This single-injection assumption induces two structural failures: (i) the Rare Token Problem, where a Zipf-type distrib…

• Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search

  Ekaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii, Roman Vashurin, Artem Shelmanov, Preslav Nakov, Mrinmaya Sachan, Maxim Panov · 8 mai 2026

  Consistency-based methods have emerged as an effective approach to uncertainty quantification (UQ) in large language models. These methods typically rely on several generations obtained via multinomial sampling, measuring their agreement level. However, in short-form QA, multinomial sampling is pron…

• Teaching Metric Distance to Discrete Autoregressive Language Models

  Jiwan Chung, Saejin Kim, Yongrae Jo, Jaewoo Park, Dongjun Min, Youngjae Yu · 8 mai 2026

  Large language models (LLMs) operate as autoregressive predictors over discrete token vocabularies, a formulation that has enabled their adaptation far beyond natural language to vision, robotics, and multimodal reasoning. However, training against one-hot targets disregards metric relationships bet…