Natural Language Processing Techniques

Últimos artículos

• Heterogeneous Parallelism for Multimodal Large Language Model Training

  Yashaswi Karnati, Kamran Jafari, Akash Mehra, Li Ding, Pranav Prashant Thombre, Ali Roshan Ghias, Shifang Xu, Parth Mannan, Yu Yao, Hao Wu, Eric Harper, Ashwath Aithal, Nima Tajbakhsh · 28 de mayo de 2026

  Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context windows grow, and encoder LLM scales diverge, a single LLM-centric TP/CP/PP/DP/EP layout increasingly limits throughput. This coupling forces encoders to …

• Learning the Error Patterns of Language Models

  Jinwoo Kim, Taylor Berg-KirkPatrick, Loris D'Antoni · 28 de mayo de 2026

  When generating outputs for domains with specific validity constraints (e.g., a program should compile), LLMs often fail in a small number of focused ways: for example, by using Python function names when generating TypeScript. We observe that these error patterns can be represented using a small nu…

• Cultural Fidelity in English-to-Hindi Translation: A Preservation-Fluency Frontier for Gender Recoverability

  Samyak Savi, Chavi Gupta, Shreyas Gantayet, Tanay Sodha, Dhruv Kumar · 28 de mayo de 2026

  Generative translation systems are cultural technologies because they decide how socially meaningful cues are rendered within culturally specific grammatical systems. We study one concrete notion of successful cultural translation: when an English source explicitly encodes gender, an English-to-Hind…

• Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning

  Mingze Wu, Abhinav Anand, Shweta Verma, Mira Mezini · 28 de mayo de 2026

  Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models. However, online RL for code generation involves LLM inference and verification of the generated output, which can take considerable time and resources. In this paper, we e…

• Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning

  Minkyu Kim, Vincent-Daniel Yun, Youngrae Kim, Suin Cho, Woosang Lim, Sunwoo Lee · 28 de mayo de 2026

  Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work typically treats layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search algorithms to identify removable layers. In this…

• Sense Representations Are Inducible Interfaces

  Jan Christian Blaise Cruz, Alham Fikri Aji · 28 de mayo de 2026

  Sense representations (explicit, per-token meaning decompositions) are useful for disambiguation, steering, and cross-lingual alignment, but existing approaches require models to be pretrained with sense structure baked in. We introduce ACROS, which induces an explicit sense pathway into a frozen pr…

• AGORA: Adapter-Grounded Observation-Action Retention for Inference-Free Prompt Compression in LLM Agents

  Haoran Zhang, Zhaohua Sun · 27 de mayo de 2026

  The token-level extractive compressors widely used for general LM context are structurally inappropriate for LLM agents: across 17 (env, backbone, method) cells spanning two independent token-level method families, every cell collapses to mean reward <= 0.05 despite 1.3-13.3x realized compression. W…

• CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

  Mike Zhang, Ali Basirat, Desmond Elliott · 27 de mayo de 2026

  Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource …

• ParsVoice: A Large-Scale Multi-Speaker Persian Speech Corpus for Text-to-Speech Synthesis

  Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili, Azadeh Shakery · 27 de mayo de 2026

  Persian remains substantially underrepresented in open speech-text resources, limiting progress in multi-speaker text-to-speech (TTS), speech-language modelling, and low-resource speech processing. We introduce ParsVoice, the largest publicly available Persian speech-text corpus tailored for trainin…

• CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Credit

  Kangyu Wang, Zhiyun Jiang, Haibo Feng, Weijia Zhao, Lin Liu, Jianguo Li, Zhenzhong Lan, Weiyao Lin · 27 de mayo de 2026

  Diffusion large language models (dLLMs) generate text through iterative denoising. In commonly adopted parallel decoding schemes, each step confirms only high-confidence positions while remasking the others. By analyzing dLLM denoising traces, we uncover a key inefficiency: models often predict the …

• GEM: Geometric Entropy Mixing for Optimal LLM Data Curation

  Yue Min, Ziyun Qiao, Ruining Chen, Yujun Li · 27 de mayo de 2026

  LLM pre-training efficacy increasingly depends on data composition rather than sheer volume. Yet, optimal mixing is hindered by categorization flaws: human taxonomies suffer from ontological misalignment, and Euclidean clustering fails to address embedding anisotropy. We introduce GEM (Geometric Ent…

• VIDA: A dataset for Visually Dependent Ambiguity in Multimodal Machine Translation

  Jingheng Pan, Xintong Wang, Longyue Wang, Liang Ding, Weihua Luo, Chris Biemann · 27 de mayo de 2026

  Ambiguity resolution is a key challenge in multimodal machine translation (MMT), where models must genuinely leverage visual input to map an ambiguous expression to its intended meaning. Although prior work has proposed disambiguation-oriented benchmarks probing the role of vision, we observe that e…

• Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning

  Ye Mo, Kai Ye, Xianwei Mao, Zirui Shao, Gang Huang, Bo Zhang, Hangdi Xing, Kehan Chen, Huan Zhou, Zixu Yan, Jiajun Bu, Sheng Zhou · 27 de mayo de 2026

  Document understanding aims to perform question answering and information extraction over document images, where the visual content is highly information-dense and most queries rely on only a few relevant layout regions. However, existing methods either adopt a one-pass strategy that implicitly assu…

• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior

  Zeyi Huang, Xuehai He, LiLiang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen · 27 de mayo de 2026

  We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this source state is already computed during ordinary decoding, LRT adds a…

• HiSpec: Hierarchical Speculative Decoding for LLMs

  Avinash Kumar, Sujay Sanghavi, Poulami Das · 27 de mayo de 2026

  Speculative decoding accelerates LLM inference by using a smaller draft model to speculate tokens that a larger target model verifies. Verification is often the bottleneck (e.g. verification is $4\times$ slower than token generation when a 3B model speculates for a 70B target model), but most prior …

• BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translation

  Param Thakkar, Anushka Yadav, Michael Tiemann, Abhi Mehta, Akshita Bhasin, Shrinivas Khedkar · 27 de mayo de 2026

  We present BhashaSetu, a linguistically enriched English--Marathi parallel dataset addressing persistent data limitations in low-resource neural machine translation (NMT). Marathi, spoken by over 95 million people, remains underrepresented in high-quality parallel corpora across diverse domains. Our…

• Tool-Schema Compression Enables Agentic RAG Under Constrained Context Budgets

  Furkan Sakizli · 27 de mayo de 2026

  Agentic RAG systems that equip language models with dozens to hundreds of tool definitions face a critical resource conflict: tool schemas consume the same context window needed for retrieval-augmented generation. We present the first systematic study of this tool-context trade-off, evaluating 14 mo…

• Large Language Model Selection with Limited Annotations

  Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch, Torsten Hoefler, Nezihe Merve G\"urel · 26 de mayo de 2026

  Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims…

• MinerU-Popo: Universal Post-Processing Model for Structured Document Parsing

  Bangrui Xu, Ziyang Miao, Xuanhe Zhou, Yiming Lin, Zirui Tang, Xiaomeng Zhao, Fan Wu, Cheng Tan, Fan Wu, Bin Wang, Conghui He · 26 de mayo de 2026

  VLM-based OCR models have become the de facto choice for document parsing, as they can accurately extract page-level elements (e.g., paragraphs within individual pages) together with their bounding boxes and textual content. However, downstream applications such as RAG require coherent document-leve…

• Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

  Bo Li, Tianyu Dong, Shaolin Zhu, Deyi Xiong · 26 de mayo de 2026

  Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT), even with limited bilingual supervision. However, fine-tuning LLMs with parallel corpora presents major challenges, namely parameter interference. To address these issues, we propose Mix-MoE, a mixed Mixt…

• InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

  Yanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Fei Wu, Hongxia Yang · 26 de mayo de 2026

  Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on model fusion focus primarily on supervised fine-tuning (SFT), leaving preference alignment (PA) --a critical phase for e…

• How Much Thinking is Enough? Quantifying and Understanding Redundancy in LLM Reasoning

  Zhiyuan Zhai, Xinkai You, Wenjing Yan, Xin Wang · 26 de mayo de 2026

  Reasoning-capable large language models solve hard problems by emitting long chains of thought, paying heavily in latency, GPU time, and energy. Casual inspection of their traces reveals extensive reformulation, verification, and circular self-reflection, yet how much of this deliberation is actuall…

• EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture

  Bowen Duan, Cong Guo, Chiyue Wei, Haoxuan Shan, Yuzhe Fu, Xinhua Chen, Yifan Xu, Ziyue Zhang, Changchun Zhou, Hai Li, Yiran Chen · 26 de mayo de 2026

  Large Language Models (LLMs) have achieved impressive performance across diverse domains but remain inefficient during the autoregressive decoding phase. Unlike the prefill stage, which employs compute-bound GEMM operations, decoding executes a sequence of small GEMV-like computations that are memor…

• How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws

  Zhitao Zhu, Xili Wang, Shizhe Wu, Jiawei Fu, Xiaoqing Liu · 26 de mayo de 2026

  High-quality data is scarce in large language model (LLM) training, yet how to schedule its use jointly with training dynamics lacks theoretical guidance. We extend functional scaling laws by incorporating a data-quality dimension, and solve the joint data-quality and batch-size scheduling problem i…

• Length Generalization with Log-Depth Recurrent Units

  Charles Pert, Dalal Alrajeh, Alessandra Russo · 26 de mayo de 2026

  Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are constrained by fixed computational depth. Regular languages provide a frequently used testbed for evaluating length generalization, as label predic…