Natural Language Processing Techniques

Últimos artículos

• EuroSpeech: A Multilingual Speech Corpus

  Samuel Pfisterer, Florian Gr\"otschla, Luca A. Lanzend\"orfer, Florian Yan, Roger Wattenhofer · 28 de octubre de 2025

  Recent progress in speech processing has highlighted that high-quality performance across languages requires substantial training data for each individual language. While existing multilingual datasets cover many languages, they often contain insufficient data for most languages. Thus, trained model…

• Principled Multimodal Representation Learning

  Xiaohao Liu, Xiaobo Xia, See-Kiong Ng, Tat-Seng Chua · 28 de octubre de 2025

  Multimodal representation learning seeks to create a unified representation space by integrating diverse data modalities to improve multimodal understanding. Traditional methods often depend on pairwise contrastive learning, which relies on a predefined anchor modality, restricting alignment across …

• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation

  Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun · 28 de octubre de 2025

  Scaling language models unlocks impressive capabilities, but the accompanying computational and memory demands make both training and deployment expensive. Existing efficiency efforts typically target either parameter sharing or adaptive computation, leaving open the question of how to attain both s…

• AttentionPredictor: Temporal Patterns Matter for KV Cache Compression

  Qingyue Yang, Jie Wang, Xing Li, Zhihai Wang, Chen Chen, Lei Chen, Xianzhi Yu, Wulong Liu, Jianye Hao, Mingxuan Yuan, Bin Li · 28 de octubre de 2025

  With the development of large language models (LLMs), efficient inference through Key-Value (KV) cache compression has attracted considerable attention, especially for long-context generation. To compress the KV cache, recent methods identify critical KV tokens through static modeling of attention s…

• Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide

  Marton Szep, Daniel Rueckert, R\"udiger von Eisenhart-Rothe, Florian Hinterwimmer · 28 de octubre de 2025

  Fine-tuning large language models (LLMs) with limited data poses a practical challenge in low-resource languages, specialized domains, and constrained deployment settings. While pre-trained LLMs provide strong foundations, effective adaptation under data scarcity requires focused and efficient fine-…

• Tahakom LLM Guidelines and Recipes: From Pre-training Data to an Arabic LLM

  Areej AlOtaibi, Lina Alyahya, Raghad Alshabanah, Shahad Alfawzan, Shuruq Alarefei, Reem Alsabti, Nouf Alsubaie, Abdulaziz Alhuzaymi, Lujain Alkhelb, Majd Alsayari, Waad Alahmed, Omar Talabay, Jalal Alowibdi, Salem Alelyani, Adel Bibi · 28 de octubre de 2025

  Large Language Models (LLMs) have significantly advanced the field of natural language processing, enhancing capabilities in both language understanding and generation across diverse domains. However, developing LLMs for Arabic presents unique challenges. This paper explores these challenges by focu…

• HOPSE: Scalable Higher-Order Positional and Structural Encoder for Combinatorial Representations

  Martin Carrasco, Guillermo Bernardez, Marco Montagna, Nina Miolane, Lev Telyatnikov · 28 de octubre de 2025

  While Graph Neural Networks (GNNs) have proven highly effective at modeling relational data, pairwise connections cannot fully capture multi-way relationships naturally present in complex real-world systems. In response to this, Topological Deep Learning (TDL) leverages more general combinatorial re…

• SALSA: Single-pass Autoregressive LLM Structured Classification

  Ruslan Berdichevsky, Shai Nahum-Gefen, Elad Ben Zaken · 28 de octubre de 2025

  Despite their impressive generalization capabilities, instruction-tuned Large Language Models often underperform on text classification benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning, thereby avoidin…

• Frustratingly Easy Task-aware Pruning for Large Language Models

  Yuanhe Tian, Junjie Liu, Xican Yang, Haishan Ye, Yan Song · 28 de octubre de 2025

  Pruning provides a practical solution to reduce the resources required to run large language models (LLMs) to benefit from their effective capabilities as well as control their cost for training and inference. Research on LLM pruning often ranks the importance of LLM parameters using their magnitude…

• Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search

  Kayhan Behdin, Qingquan Song, Sriram Vasudevan, Jian Sheng, Xiaojing Ma, Z Zhou, Chuanrui Zhu, Guoyao Li, Chanh Nguyen, Sayan Ghosh, Hejian Sang, Ata Fatahi Baarzi, Sundara Raman Ramachandran, Xiaoqing Wang, Qing Lan, Vinay Y S, Qi Guo, Caleb Johnson, Zhipeng Wang, Fedor Borisyuk · 28 de octubre de 2025

  Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements. In this work, …

• ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality

  Shayne Longpre, Sneha Kudugunta, Niklas Muennighoff, I-Hung Hsu, Isaac Caswell, Alex Pentland, Sercan Arik, Chen-Yu Lee, Sayna Ebrahimi · 28 de octubre de 2025

  Scaling laws research has focused overwhelmingly on English -- yet the most prominent AI models explicitly serve billions of international users. In this work, we undertake the largest multilingual scaling laws study to date, totaling 774 multilingual training experiments, spanning 10M-8B model para…

• Can Language Models Compose Skills In-Context?

  Zidong Liu, Zhuoyan Xu, Zhenmei Shi, Yingyu Liang · 28 de octubre de 2025

  Composing basic skills from simple tasks to accomplish composite tasks is crucial for modern intelligent systems. We investigate the in-context composition ability of language models to perform composite tasks that combine basic skills demonstrated in in-context examples. This is more challenging th…

• Restoring Pruned Large Language Models via Lost Component Compensation

  Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Tianjiao Li, Jia Jim Deryl Chua, Lee Onn Mak, Gee Wah Ng, Kezhi Mao · 28 de octubre de 2025

  Pruning is a widely used technique to reduce the size and inference cost of large language models (LLMs), but it often causes performance degradation. To mitigate this, existing restoration methods typically employ parameter-efficient fine-tuning (PEFT), such as LoRA, to recover the pruned model's p…

• Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation

  Mir Tafseer Nayeem, Sawsan Alqahtani, Md Tahmid Rahman Laskar, Tasnim Mohiuddin, M Saiful Bari · 28 de octubre de 2025

  Tokenization is a crucial but under-evaluated step in large language models (LLMs). The standard metric, fertility (the average number of tokens per word), captures compression efficiency but obscures how vocabularies are allocated across languages and domains. We analyze six widely used tokenizers …

• Aligning LLMs for Multilingual Consistency in Enterprise Applications

  Amit Agarwal, Hansa Meghwani, Hitesh Laxmichand Patel, Tao Sheng, Sujith Ravi, Dan Roth · 28 de octubre de 2025

  Large language models (LLMs) remain unreliable for global enterprise applications due to substantial performance gaps between high-resource and mid/low-resource languages, driven by English-centric pretraining and internal reasoning biases. This inconsistency undermines customer experience and opera…

• WolBanking77: Wolof Banking Speech Intent Classification Dataset

  Abdou Karim Kandji, Fr\'ed\'eric Precioso, Cheikh Ba, Samba Ndiaye, Augustin Ndione · 28 de octubre de 2025

  Intent classification models have made a significant progress in recent years. However, previous studies primarily focus on high-resource language datasets, which results in a gap for low-resource languages and for regions with high rates of illiteracy, where languages are more spoken than read or w…

• Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training

  Minhak Song, Beomhan Baek, Kwangjun Ahn, Chulhee Yun · 28 de octubre de 2025

  As both model and dataset sizes continue to scale rapidly, conventional pretraining strategies with fixed compute budgets-such as cosine learning rate schedules-are increasingly inadequate for large-scale training. Recent alternatives, including warmup-stable-decay (WSD) schedules and weight averagi…

• Less is More: Local Intrinsic Dimensions of Contextual Language Models

  Benjamin Matthias Ruppik, Julius von Rohrscheidt, Carel van Niekerk, Michael Heck, Renato Vukovic, Shutong Feng, Hsien-chin Lin, Nurul Lubis, Bastian Rieck, Marcus Zibrowius, Milica Ga\v{s}i\'c · 28 de octubre de 2025

  Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. In this paper, we introduce a novel perspective based on the geo…

• Better Estimation of the Kullback--Leibler Divergence Between Language Models

  Afra Amini, Tim Vieira, Ryan Cotterell · 28 de octubre de 2025

  Estimating the Kullback--Leibler (KL) divergence between language models has many applications, e.g., reinforcement learning from human feedback (RLHF), interpretability, and knowledge distillation. However, computing the exact KL divergence between two arbitrary language models is intractable. Thus…

• Beyond QA Pairs: Assessing Parameter-Efficient Fine-Tuning for Fact Embedding in LLMs

  Shivam Ratnakar, Abhiroop Talasila, Raghav Chamadiya, Nikhil Agarwal, Vinayak K Doifode · 28 de octubre de 2025

  This paper presents an extensive examination of Parameter-Efficient Fine-Tuning (PEFT) for embedding domain specific facts into Large Language Models (LLMs), focusing on improving the fine-tuning process by categorizing question-answer (QA) pairs into Factual and Conceptual classes using a BERT-base…

• Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling

  Yihong Dong, Ge Li, Xue Jiang, Yongding Tao, Kechi Zhang, Hao Zhu, Huanyu Liu, Jiazheng Ding, Jia Li, Jinliang Deng, Hong Mei · 28 de octubre de 2025

  Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficienc…

• How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?

  Atsuki Yamaguchi, Aline Villavicencio, Nikolaos Aletras · 28 de octubre de 2025

  Large language models (LLMs) have shown remarkable capabilities in many languages beyond English. Yet, LLMs require more inference steps when generating non-English text due to their reliance on English-centric tokenizers and vocabulary, resulting in higher usage costs to non-English speakers. Vocab…

• Lost in Transmission: When and Why LLMs Fail to Reason Globally

  Tobias Schnabel, Kiran Tomlinson, Adith Swaminathan, Jennifer Neville · 28 de octubre de 2025

  Despite their many successes, transformer-based large language models (LLMs) continue to struggle with tasks that require complex reasoning over large parts of their input. We argue that these failures arise due to capacity limits on the accurate flow of information within LLMs. To formalize this is…

• Beyond Direct Generation: A Decomposed Approach to Well-Crafted Screenwriting with LLMs

  Hang Lei, Shengyi Zong, Zhaoyan Li, Ziren Zhou, Hao Liu · 28 de octubre de 2025

  The screenplay serves as the foundation for television production, defining narrative structure, character development, and dialogue. While Large Language Models (LLMs) show great potential in creative writing, direct end-to-end generation approaches often fail to produce well-crafted screenplays. W…

• Quality-Aware Translation Tagging in Multilingual RAG system

  Hoyeon Moon, Byeolhee Kim, Nikhil Verma · 28 de octubre de 2025

  Multilingual Retrieval-Augmented Generation (mRAG) often retrieves English documents and translates them into the query language for low-resource settings. However, poor translation quality degrades response generation performance. Existing approaches either assume sufficient translation quality or …