Natural Language Processing Techniques

Últimos artículos

• Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold Steering

  Craig Atkinson · 23 de febrero de 2026

  Quantized language models face a fundamental dilemma: low sampling temperatures yield repetitive, mode-collapsed outputs, while high temperatures (T > 2.0) cause trajectory divergence and semantic incoherence. We present HELIX, a geometric framework that decouples output entropy from hallucination b…

• Sign-SGD via Parameter-Free Optimization

  Daniil Medyakov, Sergey Stanko, Gleb Molodtsov, Philip Zmushko, Grigoriy Evseev, Egor Petrov, Aleksandr Beznosikov · 23 de febrero de 2026

  Large language models have achieved major advances across domains, yet training them remains extremely resource-intensive. We revisit Sign-SGD, which serves both as a memory-efficient optimizer for single-node training and as a gradient compression mechanism for distributed learning. This paper addr…

• Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning

  Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma · 23 de febrero de 2026

  Parameter-efficient fine-tuning methods such as LoRA enable practical adaptation of large language models but provide no principled uncertainty estimates, leading to poorly calibrated predictions and unreliable behavior under domain shift. We introduce Stiefel-Bayes Adapters (SBA), a Bayesian PEFT f…

• AstroMLab 4: Benchmark-Topping Performance in Astronomy Q&A with a 70B-Parameter Domain-Specialized Reasoning Model

  Tijmen de Haan, Yuan-Sen Ting, Tirthankar Ghosal, Tuan Dung Nguyen, Alberto Accomazzi, Emily Herron, Vanessa Lama, Rui Pan, Azton Wells, Nesar Ramachandra · 23 de febrero de 2026

  General-purpose large language models (LLMs), despite their broad capabilities, often struggle with specialized domain knowledge. This gap hinders their deployment as reliable research agents in demanding fields such as astronomy. Building on our prior work with AstroSage-Llama-3.1-8B, this study in…

• Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning

  Julian Minder, Cl\'ement Dumas, Caden Juang, Bilal Chugtai, Neel Nanda · 23 de febrero de 2026

  Model diffing is the study of how fine-tuning changes a model's representations and internal algorithms. Many behaviors of interest are introduced during fine-tuning, and model diffing offers a promising lens to interpret such behaviors. Crosscoders are a recent model diffing method that learns a sh…

• Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers

  Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, Haitham Bou-Ammar · 23 de febrero de 2026

  Decoding sits between a language model and everything we do with it, yet it is still treated as a heuristic knob-tuning exercise. We argue decoding should be understood as a principled optimisation layer: at each token, we solve a regularised problem over the probability simplex that trades off mode…

• PonderLM: Pretraining Language Models to Ponder in Continuous Space

  Boyi Zeng, Shixiang Song, Siyuan Huang, Yixuan Wang, He Li, Ziwei He, Xinbing Wang, Zhiyu Li, Zhouhan Lin · 23 de febrero de 2026

  Humans ponder before articulating complex sentence elements, enabling deeper cognitive processing through focused effort. In this work, we introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step. During pondering, instead…

• ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs

  Xinlin Li, Timothy Chou, Josh Fromm, Zichang Liu, Yunjie Pan, Christina Fragouli · 23 de febrero de 2026

  Post-training weight quantization is crucial for reducing the memory and inference cost of large language models (LLMs), yet pushing the average precision below 4 bits remains challenging due to highly non-uniform weight sensitivity and the lack of principled precision allocation. Existing solutions…

• Enhancing Multilingual LLM Pretraining with Model-Based Data Selection

  Bettina Messmer, Vinko Sabol\v{c}ec, Martin Jaggi · 20 de febrero de 2026

  Dataset curation has become a basis for strong large language model (LLM) performance. While various rule-based filtering heuristics exist for English and multilingual datasets, model-based filtering techniques have primarily focused on English. To address the disparity stemming from limited researc…

• State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?

  Taja Kuzman Punger\v{s}ek, Peter Rupnik, Ivan Porupski, Vuk Dini\'c, Nikola Ljube\v{s}i\'c · 20 de febrero de 2026

  Until recently, fine-tuned BERT-like models provided state-of-the-art performance on text classification tasks. With the rise of instruction-tuned decoder-only models, commonly known as large language models (LLMs), the field has increasingly moved toward zero-shot and few-shot prompting. However, t…

• Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization

  Sumedh Rasal · 20 de febrero de 2026

  We introduce Predictive Batch Scheduling (PBS), a novel training optimization technique that accelerates language model convergence by dynamically prioritizing high-loss samples during batch construction. Unlike curriculum learning approaches that require predefined difficulty metrics or hard exampl…

• Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

  Evgeniia Tokarchuk, Maya K. Nachesa, Sergey Troshin, Vlad Niculae · 20 de febrero de 2026

  Modern neural translation models based on the Transformer architecture are known for their high performance, particularly when trained on high-resource datasets. A standard next-token prediction training strategy, while widely adopted in practice, may lead to overlooked artifacts such as representat…

• Fast KV Compaction via Attention Matching

  Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim · 19 de febrero de 2026

  Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performa…

• LMSeg: Unleashing the Power of Large-Scale Models for Open-Vocabulary Semantic Segmentation

  Huadong Tang, Youpeng Zhao, Yan Huang, Min Xu, Jun Wang, Qiang Wu · 19 de febrero de 2026

  It is widely agreed that open-vocabulary-based approaches outperform classical closed-set training solutions for recognizing unseen objects in images for semantic segmentation. Existing open-vocabulary approaches leverage vision-language models, such as CLIP, to align visual features with rich seman…

• LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioning

  Tiago Fernandes Tavares · 18 de febrero de 2026

  The discovery of deep, steerable taxonomies in large text corpora is currently restricted by a trade-off between the surface-level efficiency of topic models and the prohibitive, non-scalable assignment costs of LLM-integrated frameworks. We introduce \textbf{LogiPart}, a scalable, hypothesis-first …

• NPG-Muse: Scaling Long Chain-of-Thought Reasoning with NP-Hard Graph Problems

  Yuyao Wang, Bowen Liu, Jianheng Tang, Nuo Chen, Yuhan Li, Qifan Zhang, Chenyi Zi, Chen Zhang, Jia Li · 18 de febrero de 2026

  Reasoning Large Language Models (RLLMs) have recently achieved remarkable progress on complex reasoning tasks, largely enabled by their long chain-of-thought (Long CoT) capabilities. However, developing these Long CoT behaviors relies heavily on post-training with high-quality datasets, which are ty…

• Far Out: Evaluating Language Models on Slang in Australian and Indian English

  Deniz Kaya Dilsiz, Dipankar Srirag, Aditya Joshi · 18 de febrero de 2026

  Language models exhibit systematic performance gaps when processing text in non-standard language varieties, yet their ability to comprehend variety-specific slang remains underexplored for several languages. We present a comprehensive evaluation of slang awareness in Indian English (en-IN) and Aust…

• Differentiating Between Human-Written and AI-Generated Texts Using Automatically Extracted Linguistic Features

  Georgios P. Georgiou · 18 de febrero de 2026

  While extensive research has focused on ChatGPT in recent years, very few studies have systematically quantified and compared linguistic features between human-written and artificial intelligence (AI)-generated language. This exploratory study aims to investigate how various linguistic components ar…

• Intermittent Semi-Working Mask: A New Masking Paradigm for LLMs

  HaoYuan Hu, Mingcong Lu, Di Luo, XinYa Wu, Jiangcai Zhu, Taoye Yin, Zheng Li, Hao Wang, Shusheng Zhang, KeZun Zhang, KaiLai Shao, Chao Chen, Feng Wang · 18 de febrero de 2026

  Multi-turn dialogues and context-intensive tasks challenge Large Language Models (LLMs) to integrate long histories without sacrificing generation quality. Although prefix LLMs can better exploit historical context via bidirectional attention on prefix tokens, they are rarely used in practice becaus…

• wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models

  Xiaohang Tang, Rares Dolga, Sangwoong Yoon, Ilija Bogunovic · 17 de febrero de 2026

  Improving the reasoning capabilities of diffusion-based large language models (dLLMs) through reinforcement learning (RL) remains an open problem. The intractability of dLLMs likelihood function necessitates approximating the current, old, and reference policy likelihoods at each policy optimization…

• Evolution of Concepts in Language Model Pre-Training

  Xuyang Ge, Wentao Shu, Jiaxing Wu, Yunhua Zhou, Zhengfu He, Xipeng Qiu · 17 de febrero de 2026

  Language models obtain extensive capabilities through pre-training. However, the pre-training process remains a black box. In this work, we track linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders. We find that most featu…

• On Representation Redundancy in Large-Scale Instruction Tuning Data Selection

  Youwei Shu, Shaomian Zheng, Dingnan Jin, Wenjie Qu, Ziyao Guo, Qing Cui, Jun Zhou, Jiaheng Zhang · 17 de febrero de 2026

  Data quality is a crucial factor in large language models training. While prior work has shown that models trained on smaller, high-quality datasets can outperform those trained on much larger but noisy or low-quality corpora, systematic methods for industrial-scale data selection in instruction tun…

• LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

  S\"onke Tenckhoff, Mario Koddenbrock, Erik Rodner · 17 de febrero de 2026

  We present LLMStructBench, a novel benchmark for evaluating Large Language Models (LLMs) on extracting structured data and generating valid JavaScript Object Notation (JSON) outputs from natural-language text. Our open dataset comprises diverse, manually verified parsing scenarios of varying complex…

• Predicting the Order of Upcoming Tokens Improves Language Modeling

  Zayd M. K. Zuhri, Erland Hilman Fuadi, Alham Fikri Aji · 17 de febrero de 2026

  Multi-token prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We found MTP's exact future token prediction to be too difficult as an auxiliary…

• RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech

  Zhisheng Zheng, Xiaohang Sun, Tuan Dinh, Abhishek Yanamandra, Abhinav Jain, Zhu Liu, Sunil Hadap, Vimal Bhat, Manoj Aggarwal, Gerard Medioni, David Harwath · 17 de febrero de 2026

  End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by mac…