Natural Language Processing Techniques

Últimos artículos

• Perturbation is All You Need for Extrapolating Language Models

  Zetai Cen, Jin Zhu, Xinwei Shen, Chengchun Shi · 7 de mayo de 2026

  We introduce a simple yet powerful framework for training large language models. In contrast to the standard autoregressive next-token prediction based on an exact prefix, we propose a perturbation-based procedure that first transforms the prefix into a semantic neighbor and then conditions on this …

• Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks

  Yaobo Zhang · 7 de mayo de 2026

  Relative positional encodings determine which functions of query-key lag can enter the primitive attention logit. RoPE supplies a rotary phase, while ALiBi supplies an additive distance bias. Motivated by group-theoretic views of linear translation-invariant positional encodings, we study a non-semi…

• UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding

  Yepeng Weng, Qiao Hu, Takehisa Yairi · 7 de mayo de 2026

  Speculative decoding accelerates Large Language Models via draft-then-verify, where verification can be framed as an Optimal Transport (OT) problem. Existing approaches typically handle multi-draft and multi-step aspects in isolation, applying either flat OT to single-step drafts or per-token reject…

• Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varieties

  Jinju Kim, Haeji Jung, Youjeong Roh, Jong Hwan Ko, David R. Mortensen · 7 de mayo de 2026

  Low-resource language varieties used by specific groups remain neglected in the development of Multilingual Language Models. A great deal of cross-lingual research focuses on inter-lingual language transfer which strives to align allied varieties and minimize differences between them. However, for l…

• Parallel Prefix Verification for Speculative Generation

  Yuncheng Yao, Yuxuan Xia, Shengjie Wang, Danyang Zhuo · 7 de mayo de 2026

  We introduce PARSE (PArallel pRefix Speculative Engine), a speculative generation framework that accelerates large language model (LLM) inference by parallelizing prefix verification on a semantic level. Existing speculative decoding methods are fundamentally limited by token-level equivalence: the …

• Emergent Hierarchical Structure in Large Language Models: An Information-Theoretic Framework for Multi-Scale Representation

  Yukin Zhang, Qi Dong, Kemu Xu · 7 de mayo de 2026

  Why do language models from different architecture families respond so differently to the same perturbation? We argue that the answer is not scale, but \emph{how architecture shapes information compression}. Analyzing eight Transformer models (7B--70B parameters) from the Llama and Qwen families, we…

• Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

  Rohan Surana, Gagan Mundada, Xunyi Jiang, Chuhan Wang, Zhenwei Tang, Difan Jiao, Zihan Huang, Yuxin Xiong, Junda Wu, Sheldon Yu, Xintong Li, Raghav Jain, Nikki Kuang, Sizhe Zhou, Bowen Jin, Zhendong Chu, Tong Yu, Ryan Rossi, Kuan-Hao Huang, Jingbo Shang, Jiawei Han, Julian McAuley · 6 de mayo de 2026

  Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajectory sampled from a prompt to termination, including intermediate reasoning steps and optional tool or environment inter…

• A Multimodal Dataset for Visually Grounded Ambiguity in Machine Translation

  Jingheng Pan, Xintong Wang, Longyue Wang, Liang Ding, Weihua Luo, Chris Biemann · 6 de mayo de 2026

  Ambiguity resolution is a key challenge in multimodal machine translation (MMT), where models must genuinely leverage visual input to map an ambiguous expression to its intended meaning. Although prior work has proposed disambiguation-oriented benchmarks that provide supportive evidence for the role…

• Generative Interfaces for Language Models

  Jiaqi Chen, Yanzhe Zhang, Yutong Zhang, Yijia Shao, Diyi Yang · 6 de mayo de 2026

  Large language models (LLMs) are increasingly seen as assistants, copilots, and consultants, capable of supporting a wide range of tasks through natural conversation. However, most systems remain constrained by a linear request-response format that often makes interactions inefficient in multi-turn,…

• GLEAN: Active Generalized Category Discovery with Diverse LLM Feedback

  Henry Peng Zou, Siffi Singh, Yi Nian, Jianfeng He, Jason Cai, Saab Mansour, Hang Su · 6 de mayo de 2026

  Generalized Category Discovery (GCD) is a practical and challenging open-world task that aims to recognize both known and novel categories in unlabeled data using limited labeled data from known categories. Due to the lack of supervision, previous GCD methods face significant challenges, such as dif…

• Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training

  Luong N. Nguyen · 6 de mayo de 2026

  How reliably can a small language model estimate its own correctness? The answer determines whether local-to-cloud routing-escalating queries a cheap local model cannot handle-can work without supervised training data. As inference costs dominate large language model (LLM) deployment budgets, routin…

• Attention Sinks in Massively Multilingual Neural Machine Translation:Discovery, Analysis, and Mitigation

  Hillary Mutisya, John Mugane · 5 de mayo de 2026

  Cross-attention patterns in neural machine translation (NMT) are widely used to study how multilingual models align linguistic structure. We report a systematic artifact in cross-attention analysis of NLLB-200 (600M): non-content tokens - primarily end-of-sequence tokens, language tags, and punctuat…

• Submodular Benchmark Selection

  Alexander Smola · 5 de mayo de 2026

  Evaluating large language models across many benchmarks is expensive, yet many benchmarks are highly correlated. We formalize the selection of a small, informative subset as submodular maximization under a multivariate Gaussian model. Entropy (log-determinant covariance) and mutual information betwe…

• Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages

  Kevin Guan, Happy Buzaaba, Christiane Fellbaum · 5 de mayo de 2026

  Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-resource settings remains poorly understood. We evaluate four parsers -- the Biaffine LSTM, Stack-Pointer Network, AfroXLMR-large, and RemBERT -- ac…

• P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats

  Yuzong Chen, Chao Fang, Xilai Dai, Yuheng Wu, Thierry Tambe, Marian Verhelst, Mohamed S. Abdelfattah · 5 de mayo de 2026

  The substantial memory bandwidth and computational demands of large language models (LLMs) present critical challenges for efficient inference. To tackle this, the literature has explored heterogeneous systems that combine neural processing units (NPUs) with DRAM-based processing-in-memory (PIM) for…

• Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference

  Egor Sevriugov, Nikita Dragunov, Anton Razzhigaev, Andrey Kuznetsov, Ivan Oseledets · 5 de mayo de 2026

  Non-autoregressive (NAR) language models offer notable efficiency in text generation by circumventing the sequential bottleneck of autoregressive decoding. However, accurately modeling dependencies in discrete sequences remains challenging in this paradigm. In this work, we advance the field of NAR …

• Borrowed Geometry: Computational Reuse of Frozen Text-Pretrained Transformer Weights Across Modalities

  Abay Bektursun · 4 de mayo de 2026

  Frozen Gemma 4 31B weights pretrained exclusively on text tokens, unmodified, transfer across modality boundaries through a thin trainable interface. (1) OGBench scene-play-singletask-task1-v0: $+4.33$pt over published GCIQL at $n=3$ with std 0.74 -- a published-SOTA win on a robotic manipulation ta…

• NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

  Enzo S. N. Silva, Pablo B. Costa, Raphael C. Vlasman, Rosimeire P. Costa, Henrique L. P. Silva, Lucas F. A. O. Pellicer, Guilherme Rinaldo, Renato A. Almeida, Darian S. R. Rabbani, Cinthya O. Oestreich, Vinicius F. Carid\'a · 4 de mayo de 2026

  High-quality corpora are essential for advancing Natural Language Processing (NLP) in Portuguese. Building on previous encoder-only models such as BERTimbau and Albertina PT-BR, we introduce NorBERTo, a modern encoder based on the ModernBERT architecture, featuring long-context support and efficient…

• Bias in Large Language Models: Origin, Evaluation, and Mitigation

  Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, Shuo Shuo Liu · 4 de mayo de 2026

  Large Language Models (LLMs) have revolutionized natural language processing, but their susceptibility to biases poses significant challenges. This comprehensive review examines the landscape of bias in LLMs, from its origins to current mitigation strategies. We categorize biases as intrinsic and ex…

• Representation in large language models

  Cameron Yetman · 4 de mayo de 2026

  The extraordinary success of recent Large Language Models (LLMs) on a diverse array of tasks has led to an explosion of scientific and philosophical theorizing aimed at explaining how they do what they do. Unfortunately, disagreement over fundamental theoretical issues has led to stalemate, with ent…

• Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for LLMs

  Dylan Bouchard · 4 de mayo de 2026

  Bias and fairness risks in Large Language Models (LLMs) vary substantially across deployment contexts, yet existing approaches lack systematic guidance for selecting appropriate evaluation metrics. We present a decision framework that maps LLM use cases, characterized by a model and population of pr…

• Rethinking LLM Ensembling from the Perspective of Mixture Models

  Jiale Fu, Yuchu Jiang, Peijun Wu, Chonghan Liu, Joey Tianyi Zhou, Xu Yang · 4 de mayo de 2026

  Model ensembling is a well-established technique for improving the performance of machine learning models. Conventionally, this involves averaging the output distributions of multiple models and selecting the most probable label. This idea has been naturally extended to large language models (LLMs),…

• Lost in State Space: Probing Frozen Mamba Representations

  Bhagyashree Wagh, Akash Singh · 4 de mayo de 2026

  Mamba's recurrent state h_t is, by construction, a compressed summary of every token seen so far. This raises a tempting hypothesis: if we extract token-level outputs y_t at fixed patch boundaries, we obtain semantic sentence summaries for free, with no pooling head, no fine-tuning, and no [CLS] tok…

• Reliable Answers for Recurring Questions: Boosting Text-to-SQL Accuracy with Template Constrained Decoding

  Smit Jivani, Sarvam Maheshwari, Sunita Sarawagi · 1 de mayo de 2026

  Large language models (LLMs) have revolutionized Text-to-SQL generation, allowing users to query structured data using natural language with growing ease. Yet, real-world deployment remains challenging, especially in complex or unseen schemas, due to inconsistent accuracy and the risk of generating …

• Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

  Ansar Aynetdinov, Patrick Haller, Alan Akbik · 1 de mayo de 2026

  Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. However, for high-resource non-English languages like German, French, or Japanese, aggressive filtering creates a strategic dilemma: should practitioners priorit…