Natural Language Processing Techniques

Últimos artículos

• Fairness Testing of Large Language Models in Role-Playing

  Xinyue Li, Zhenpeng Chen, Jie M. Zhang, Ying Xiao, Tianlin Li, Weisong Sun, Yang Liu, Yiling Lou, Xuanzhe Liu · 23 de abril de 2026

  Large Language Models (LLMs) have become foundational in modern language-driven software applications, profoundly influencing daily life. A critical technique in leveraging their potential is role-playing, where LLMs simulate diverse roles to enhance their real-world utility. However, while research…

• WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling

  Jiacheng Li, Jianchao Tan, Zhidong Yang, Pingwei Sun, Feiye Huo, Jiayu Qin, Xiangyu Zhang, Maoxin He, Yerui Sun, Yuchen Xie, Guangming Tan, Weile Jia, Xunliang Cai, Tong Zhao · 23 de abril de 2026

  Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns d…

• Transparent Screening for LLM Inference and Training Impacts

  Arnault Pachot, Thierry Petit · 23 de abril de 2026

  This paper presents a transparent screening framework for estimating inference and training impacts of current large language models under limited observability. The framework converts natural-language application descriptions into bounded environmental estimates and supports a comparative online ob…

• LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillation

  Siqing Song, Chuang Wang, Yong Lang, Yi Yang, Xu-Yao Zhang · 22 de abril de 2026

  Deploying large language models (LLMs) in resource-constrained environments is hindered by heavy computational and memory requirements. We present LBLLM, a lightweight binarization framework that achieves effective W(1+1)A4 quantization through a novel three-stage quantization strategy. The framewor…

• GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

  Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang Li, Rui Mao, Jianbin Qin · 22 de abril de 2026

  Large language models (LLMs) are expensive to serve because model parameters, attention computation, and KV caches impose substantial memory and latency costs. We present GRASPrune, a structured pruning framework applied after pretraining that jointly prunes FFN channels and KV head groups under a s…

• RepIt: Steering Language Models with Concept-Specific Refusal Vectors

  Vincent Siu, Nathan W. Henry, Nicholas Crispino, Yang Liu, Dawn Song, Chenguang Wang · 22 de abril de 2026

  Current safety evaluations of language models rely on benchmark-based assessments that may miss localized vulnerabilities. We present RepIt, a simple and data-efficient framework for isolating concept-specific representations in LM activations. While existing steering methods already achieve high at…

• MRS: Multi-Resolution Skills for HRL Agents

  Shashank Sharma, Janina Hoffmann, Vinay Namboodiri · 22 de abril de 2026

  Hierarchical reinforcement learning (HRL) decomposes the policy into a manager and a worker, enabling long-horizon planning but introducing a performance gap on tasks requiring agility. We identify a root cause: in subgoal-based HRL, the manager's goal representation is typically learned without con…

• Bootstrapping Code Translation with Weighted Multilanguage Exploration

  Yuhan Wu, Huan Zhang, Wei Cheng, Chen Shen, Jingyue Yang, Wei Hu · 22 de abril de 2026

  Code translation across multiple programming languages is essential yet challenging due to two vital obstacles: scarcity of parallel data paired with executable test oracles, and optimization imbalance when handling diverse language pairs. We propose BootTrans, a bootstrapping method that resolves b…

• Data Mixing for Large Language Models Pretraining: A Survey and Outlook

  Zhuo Chen, Yuxuan Miao, Supryadi, Deyi Xiong · 21 de abril de 2026

  Large language models (LLMs) rely on pretraining on massive and heterogeneous corpora, where training data composition has a decisive impact on training efficiency and downstream generalization under realistic compute and data budget constraints. Unlike sample-level data selection, data mixing optim…

• BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes

  Baichuan Huang, Ananth Balashankar, Amir Aminifar · 21 de abril de 2026

  Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. However, the link between fine-tuning different bias terms (i.e., $\boldsymbol{b}_q$, $\boldsymbol{b…

• MetaLint: Easy-to-Hard Generalization for Code Linting

  Atharva Naik, Lawanya Baghel, Dhakshin Govindarajan, Darsh Agrawal, Yiqing Xie, Daniel Fried, Carolyn Rose · 21 de abril de 2026

  Large language models excel at code generation but struggle with code linting, particularly in generalizing to unseen or evolving best practices beyond those observed during training. We introduce MetaLint, a meta-learning framework that formulates code linting as an instruction-following task, wher…

• ONTO: A Token-Efficient Columnar Notation for LLM Input Optimization

  Harshavardhanan Deekeswar · 21 de abril de 2026

  Serialization formats designed for document interchange impose structural overhead that becomes prohibitive when large language models consume operational data at scale. A modest dataset of 1,000 IoT sensor readings serialized as JSON requires approximately 80,000 tokens - the majority spent on repe…

• Predicting LLM Compression Degradation from Spectral Statistics

  Mingxue (Mercy), Xu · 21 de abril de 2026

  Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can be prohibitively expensive. Can compression-induced degradation be predicted before committing to this compute? We syste…

• Latent Phase-Shift Rollback: Inference-Time Error Correction via Residual Stream Monitoring and KV-Cache Steering

  Manan Gupta, Dhruv Kumar · 21 de abril de 2026

  Large language models frequently commit unrecoverable reasoning errors mid-generation: once a wrong step is taken, subsequent tokens compound the mistake rather than correct it. We introduce $\textbf{Latent Phase-Shift Rollback}$ (LPSR): at each generation step, we monitor the residual stream at a c…

• Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

  Jelena Markovic-Voronov, Wenhui Zhu, Bo Long, Zhipeng Wang, Suyash Gupta, Kayhan Behdin, Bee-Chung Chen, Deepak Agarwal · 21 de abril de 2026

  We introduce a principled probabilistic framework for reward-guided decoding in large language models, addressing the limitations of standard decoding methods that optimize token-level likelihood rather than sequence-level quality. Our method defines a reward-augmented target distribution over compl…

• Preparation of Fractal-Inspired Computational Architectures for Automated Neural Design Exploration

  Yash Mittal, Dmitry Ignatov, Radu Timofte · 21 de abril de 2026

  It introduces FractalNet, a fractal-inspired computational architectures for advanced large language model analysis that mainly challenges model diversity on a large scale in an efficient manner. The new set-up involves a template-driven generator, runner, and evaluation framework that, through syst…

• A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions

  Zhiyin Yu, Yuchen Mou, Juncheng Yan, Junyu Luo, Chunchun Chen, Xing Wei, Yunhui Liu, Hongru Sun, Yuxing Zhang, Jun Xu, Yatao Bian, Ming Zhang, Wei Ye, Tieke He, Jie Yang, Guanjie Zheng, Zhonghai Wu, Bo Zhang, Lei Bai, Xiao Luo · 21 de abril de 2026

  Reinforcement learning (RL) has emerged as a powerful post-training paradigm for enhancing the reasoning capabilities of large language models (LLMs). However, reinforcement learning for LLMs faces substantial data scarcity challenges, including the limited availability of high-quality external supe…

• SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe

  Yuxin Xiao, Shujian Zhang, Wenxuan Zhou, Marzyeh Ghassemi, Sanqiang Zhao · 21 de abril de 2026

  To acquire instruction-following capabilities, large language models (LLMs) undergo instruction tuning, where they are trained on instruction-response pairs using next-token prediction (NTP). Efforts to improve instruction tuning often focus on higher-quality supervised fine-tuning (SFT) datasets, t…

• ProTrain: Efficient LLM Training via Memory-Aware Techniques

  Hanmei Yang, Jin Zhou, Yao Fu, Xiaoqun Wang, Ramine Roane, Hui Guan, Tongping Liu · 21 de abril de 2026

  Memory pressure has emerged as a dominant constraint in scaling the training of large language models (LLMs), particularly in resource-constrained environments. While modern frameworks incorporate various memory-saving techniques, they often expose low-level configuration knobs that require manual t…

• Faster LLM Inference via Sequential Monte Carlo

  Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang, Cameron Freer, Tim Vieira, Ryan Cotterell, Mohamed S. Abdelfattah · 20 de abril de 2026

  Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge…

• Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization

  Zhiyuan Zhai, Bingcong Li, Bingnan Xiao, Ming Li, Xin Wang · 17 de abril de 2026

  Test-time compute scaling, the practice of spending extra computation during inference via repeated sampling, search, or extended reasoning, has become a powerful lever for improving large language model performance. Yet deploying these techniques under finite inference budgets requires a decision t…

• Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms

  Zeguan Xiao, Yun Chen, Guanhua Chen, Ke Tang · 17 de abril de 2026

  Direct Alignment Algorithms (DAAs), such as Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO), have emerged as efficient alternatives to Reinforcement Learning from Human Feedback (RLHF) algorithms for aligning large language models (LLMs) with human preferences. Howeve…

• Generating Concept Lexicalizations via Dictionary-Based Cross-Lingual Sense Projection

  David Basil, Chirooth Girigowda, Bradley Hauer, Sahir Momin, Ning Shi, Grzegorz Kondrak · 17 de abril de 2026

  We study the task of automatically expanding WordNet-style lexical resources to new languages through sense generation. We generate senses by associating target-language lemmas with existing lexical concepts via semantic projection. Given a sense-tagged English corpus and its translation, our method…

• ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving

  Yuseon Choi, Jingu Lee, Jungjun Oh, Sunjoo Whang, Byeongcheol Kim, Minsung Kim, Hoi-Jun Yoo, Sangjin Kim · 17 de abril de 2026

  Mixture-of-Experts (MoE) models have become the dominant architecture for large-scale language models, yet on-premises serving remains fundamentally memory-bound as batching turns sparse per-token compute into dense memory activation. Memory-centric architectures (PIM, NMP) improve bandwidth but lea…

• In Context Learning and Reasoning for Symbolic Regression with Large Language Models

  Samiha Sharlin, Tyler R. Josephson · 17 de abril de 2026

  Large Language Models (LLMs) are transformer-based machine learning models that have shown remarkable performance in tasks for which they were not explicitly trained. Here, we explore the potential of LLMs to perform symbolic regression -- a machine-learning method for finding simple and accurate eq…