Natural Language Processing Techniques

Últimos artículos

• Explanations of Large Language Models Explain Language Representations in the Brain

  Maryam Rahimi, Mohammad Reza Daliri, Yadollah Yaghoobzadeh · 7 de agosto de 2026

  Large Language Model (LLM) representations are known to align with brain activity during language processing, but it remains unclear what drives this alignment. We test whether explainable AI (XAI) can help answer this: using attribution methods, we quantify the contribution of each input word to an…

• SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models

  Hoda Fakharzadehjahromy, Emil Wiman, Andreas Bueff, Hafsteinn Einarsson, Fredrik Heintz · 7 de agosto de 2026

  Preference optimisation has proven effective for improving large language models but typically relies on costly human preference annotations. Extending these methods to morphologically rich, low-resource languages remains challenging because such annotations are scarce. We present SAGA (Score-weight…

• Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers

  Haris Riaz, Hyungji Kim, Mihai Surdeanu · 7 de agosto de 2026

  Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed \textbf{P}ositional \textbf{E}mbeddings (\textbf{SiPE}), which learns a lightweight syntactic prior from dependency pars…

• Analysis of Numerical Localisation in LLM Translations

  Patrizia Kaye · 7 de agosto de 2026

  The work of Tang et. al. (2025) on numerical translation is extended by analysing the capability of five large language models (LLMs) for the localisation of times, numbers, and dates instead of translation. Models were selected that could be loaded onto and run on commodity hardware and a baseline …

• Task-Conditional Flow Matching for Balanced Multilingual Text Embedding Adaptation

  Tirth Bhatt, Naren Kumar S, Mayank Singh · 7 de agosto de 2026

  Multilingual text embedding models are commonly adapted using a single training objective across diverse tasks, despite different tasks requiring fundamentally different optimization strategies. We introduce Task-Conditional Flow Matching (TCFM), a multilingual embedding adaptation framework that se…

• MameLoshnLM: Yiddish Language Model and Evaluation Benchmark

  Uri Katz, Omer Goldman, Tomasz Limisiewicz, Reut Tsarfaty, Noah A. Smith · 7 de agosto de 2026

  We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling. Existing multilin…

• Where Privacy Risk Lives in English-Source Multilingual RAG: A Stage-Decomposed Audit Across Five Query Languages

  Yanhang Li, Zhichao Fan, Zexin Zhuang · 7 de agosto de 2026

  A common assumption holds that switching to a non-English language makes a multilingual RAG system easier to attack for personal information. We test this on an English-source synthetic-PII corpus with five query languages and a two-stage defence (LLM input judge + regex output filter), in a pipelin…

• On-Policy Delta Distillation for Multilingual Math Reasoning

  Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han · 7 de agosto de 2026

  On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in Engl…

• Unforgettable Generalization in Language Models

  Eric Zhang, Leshem Choshen, Jacob Andreas · 6 de agosto de 2026

  When language models (LMs) are trained to forget (or "unlearn'') a skill, how precisely does their behavior change? We study the behavior of transformer LMs in which tasks have been forgotten via fine-tuning on randomized labels. Such LMs learn to generate near-random predictions for individual exam…

• BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning

  Sajib Hossain, Md Kamrus Samad, Anan Ghosh, Labib Imam Chowdhury, Nabeel Mohammed · 6 de agosto de 2026

  Deep neural networks have shown impressive success in NLP tasks owing to their complex structure and huge number of edges. Achieving state-of-the-art performance in natural language processing with a large pre-trained model such as BERT is expensive and time-consuming, carries a large carbon footpri…

• Mind the Cap: Output-Budget Regimes Change the Measured Multilingual Reasoning Gap

  Ankit Goyal, Jaideep Ray · 6 de agosto de 2026

  Multilingual evaluations report accuracy at a single output-token cap, but languages need different numbers of tokens to express the same content, so the cap is a hidden experimental variable. We test whether the native-vs-translate gap on MGSM (German, Thai, Swahili) is a token-budget artifact for …

• Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

  George Fountzoulas · 6 de agosto de 2026

  Papers 1-2 of the Kathleen series showed that a byte-level, attention-free architecture built from a wavetable encoder and multi-scale reverberant state can match strong baselines on classification at ~450-700K parameters, without pretraining. We ask whether the same ingredients can generate. (1) Sc…

• Omega-S: A Functional Resilience Index for LLM Fine-Tuning

  Alberto Acedo · 5 de agosto de 2026

  Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and …

• M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

  Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y · 5 de agosto de 2026

  Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency. We introduce M-GATE (Multilingual Grammar, Accuracy in T…

• TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

  Bhavin Jawade, Cameron R. Wolfe · 5 de agosto de 2026

  Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements. However, both LLMs and LRMs are computationally expensive to deploy at scale, while small la…

• Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

  Jo-Ku Cheng, Nikolaos Aletras, Marco Valentino · 5 de agosto de 2026

  Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior…

• LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

  Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh, Heng Li · 5 de agosto de 2026

  Large Language Models (LLMs) are integrated into software systems and AI services, making efficient LLM serving a concern for software engineering. Serving LLMs is challenging because inference requires computation, memory, GPU resources, and execution while maintaining latency and throughput. Altho…

• Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension

  Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan, Niranjan Wartikar · 5 de agosto de 2026

  Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token embeddings can strongly affect continued pre-training (CPT) efficiency. We present a systematic study of more than 20 initialization strategies for …

• Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

  Ravi Satya Durga Prasad Yenugula · 5 de agosto de 2026

  Model families train every size from scratch. Can a pretrained large model be converted into a smaller sibling? We characterize the 1.4B->410M conversion in the Pythia family end-to-end: (i) representations align strongly across sizes (ridge R^2=0.84) while parameters align weakly; (ii) dense weight…

• Dense Language Generation Made Simple: Deterministic, Randomized, and Multi-Order Algorithms

  Ziyi Cai, Shuangping Li, Yiheng Shen, Kangning Wang, Peng Zhang · 4 de agosto de 2026

  Language generation in the limit is a theoretical framework for studying how a generator can learn to produce new valid strings from a stream of positive examples. In this model, an adversary chooses an unknown language from a countable family and enumerates its elements in an arbitrary order, while…

• OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset

  Allen Roush, Yusuf Shabazz, Arvind Balaji, Peter Zhang, Stefano Mezza, Markus Zhang, Sanjay Basu, Sriram Vishwanath, Mehdi Fatemi, Ravid Shwartz-Ziv · 4 de agosto de 2026

  We introduce OpenDebateEvidence, a comprehensive dataset for argument mining and summarization sourced from the American Competitive Debate community. This dataset includes over 3.5 million documents with rich metadata, making it one of the most extensive collections of debate evidence. OpenDebateEv…

• Cultural Awareness is Represented but Not Decoded: Tracing Mythological Knowledge across 18 Open-Source LLMs

  Iaroslav Chelombitko, Ekaterina Chelombitko, Mika H\"am\"al\"ainen · 4 de agosto de 2026

  Open-source LLMs reliably name Zeus, Jupiter, and Thor, but recover their counterparts in less-represented traditions like Finnish, Slavic, Egyptian, or Chinese mythology far less consistently. We ask where inside the model this cultural default is produced. On a parallel cross-cultural substrate of…

• Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+

  Sherzod Hakimov, Karl Osswald, Jelle Psurek, Eszter Bukovszky, A. Altar L\"user, David Schlangen · 4 de agosto de 2026

  We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and …

• Not the Dimension, the Norm: What Matters in Gradient-Free Weight Perturbation of Language Models

  Taeyeong Kim, Ahhyun Kim, TaeHyeon Kim, Unggi Lee · 4 de agosto de 2026

  Adapting a language model to a task no longer requires training all of its weights, and a line of parameter-efficient methods has driven the trainable count from billions down to a handful of scalars. Gradient-free adaptation, which samples random weight perturbations and keeps the ones that score w…

• Native Multilingual Chain-of-Thought Reasoning in Low-Resource Southeast Asian Languages

  Sean Gip Lim, William Chandra Tjhi, Hai Leong Chieu · 4 de agosto de 2026

  Large Language Models have achieved substantial progress in reasoning capabilities. Yet in low-resource native settings, many suffer from cross-lingual collapse, reverting to English during intermediate steps that require complex logical reasoning. This presents a cold-start bottleneck for policy op…