Natural Language Processing Techniques

Neueste Paper

• Adaptive Generation of Bias-Eliciting Questions for LLMs

  Robin Staab, Jasper Dekoninck, Maximilian Baader, Martin Vechev · 10. Juli 2026

  Large language models (LLMs) are now widely deployed in user-facing applications, reaching hundreds of millions of users worldwide. Despite their widespread adoption, growing reliance on their outputs raises significant concerns, particularly as users may be exposed to model-inherent biases that dis…

• A Practical Investigation of Training-free Relaxed Speculative Decoding

  Guoxuan Xia, Luka Ribar, Paul Balanca · 10. Juli 2026

  Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recen…

• Holographic Neural PCFG for Unsupervised Parsing

  Ryosuke Yamaki, Daichi Mochihashi, Nobutaka Shimada, Tadahiro Taniguchi · 10. Juli 2026

  Unsupervised constituency parsing aims to accurately induce latent tree structures from raw text alone. Recent neural parameterizations of PCFGs achieve strong performance in both supervised and unsupervised parsing, yet rely on high-capacity black-box networks for rule scoring -- as exemplified by …

• Uncertainty-gated selection for block-sparse attention

  Thomas Rossi · 10. Juli 2026

  Block-sparse attention scales long-context language models by replacing the O(N^2) softmax with a per-query top-k selection over key blocks. This cutoff is myopic: when the k-th and (k+1)-th blocks are nearly tied in score, the selector commits without spending extra budget, and a dropped block carr…

• Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation

  Didula Samaraweera, Anjana Supun, Srinath Perera · 10. Juli 2026

  Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia. Improving Small Language Models~(SLMs) for these languages faces a trilemma: Supervised Fine-Tuning~(S…

• Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

  Yufeng Wang, Lu Wei, Haibin Ling · 9. Juli 2026

  Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit \emph{reference anchors}. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor r…

• MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese

  Tardelli Ronan Coelho Stekel · 9. Juli 2026

  Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce MTEB-BR, a benchmark of 22 native Brazilian-Portuguese tasks across seven categor…

• Trees from Marginals: Autoregressive drafting with factorized priors

  Yuma Oda, Ryan Mathieu, Roman Knyazhitskiy, Artur Chakhvadze · 9. Juli 2026

  Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence …

• TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models

  Yutang Ma, Kecheng Huang, Xikun Jiang, Zili Shao · 9. Juli 2026

  Large Language Models (LLMs) store factual knowledge and domain-specific patterns implicitly in dense Transformer parameters, making knowledge expansion costly through pretraining, fine-tuning, retrieval augmentation, or longer contexts. Engram-style memory offers a compact hidden-state injection pa…

• FPTQuant: Function-Preserving Transforms for LLM Quantization

  Boris van Breugel, Yelysei Bondarenko, Paul Whatmough, Markus Nagel · 9. Juli 2026

  Large language models (LLMs) require substantial compute, and thus energy, at inference time. While quantizing weights and activations is effective at improving efficiency, naive quantization of LLMs can significantly degrade performance due to large magnitude outliers. This paper describes FPTQuant…

• FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

  Athanasios Zeris · 9. Juli 2026

  FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragraph scale achieves val…

• evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

  Shreyas K Chandrahas · 7. Juli 2026

  The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from…

• MTEB-PT: A Text Embedding Benchmark for Brazilian Portuguese

  Tardelli Ronan Coelho Stekel · 7. Juli 2026

  Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce MTEB-PT, a benchmark of 22 native Brazilian-Portuguese tasks across seven categor…

• How Much of the Routing Gap Is Real? Decomposing the Router-to-Oracle Gap into Reproducible Specialist Advantage and Single-Draw Label Noise

  Teng-Ruei Chen · 7. Juli 2026

  Routing among large language models (LLMs) promises better quality at lower cost, motivated by the reported gap between learned routers and a per-instance oracle. But that oracle is computed from a single correctness label per (query, model), so under stochastic decoding it is one Bernoulli draw, no…

• Legible-by-Construction: Attention and End-to-End Transformers

  Mark Oskin · 7. Juli 2026

  A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other…

• LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

  Zhuowen Liu, Longkun Hao, Shiyu Feng, Xiaowen Chang, Ruiqun Li, Changqun Li · 7. Juli 2026

  The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. Howeve…

• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

  Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang · 7. Juli 2026

  Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification. While recent parallel drafters efficiently propose long token sequences in a single forward pass, they suffer from rapid acceptance decay due to a lack of inter-token depend…

• A Failure-Mode Benchmark for Polymorphic Sybil Poisoning in RAG

  Donghyun Lee (Dongguk University), Juntae Kim (Dongguk University) · 7. Juli 2026

  We release a benchmark and failure-mode-aware evaluation framework for grounded QA under coordinated retrieval poisoning. The framework partitions reader outputs into four mutually exclusive categories (\emph{gold}, \emph{hijack}, \emph{abstention}, \emph{drift}), with instance-level paired clean-to…

• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization

  Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich · 7. Juli 2026

  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disp…

• SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication

  Mingkai Zheng, Junlin Chen, Haotian Xie, Zhao Zhang · 3. Juli 2026

  Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale. Existing communication-reduction meth…

• LLMs in the Real World: Evaluating "AI" in Emergency Contexts

  Sara Court, Lara Downing, Micha Elsner · 2. Juli 2026

  This paper offers a call to action. We urge our colleagues in the research community to play a greater role in the articulation of our findings to the public. To illustrate the stakes we present a case study on the initial stages of an LLM-based machine translation application's deployment in a real…

• Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization

  Fei Wang, Chao Xue, Taoran Liu, Li Shen, Ye Liu, ChangXing Ding · 2. Juli 2026

  Mixed-precision quantization (MPQ) has become a key technique for deploying large language models under stringent memory and compute constraints. We first identify a phenomenon that we term the Perplexity Illusion: layers ranked as important by perplexity-based sensitivity show little rank correlati…

• QuasiMoTTo: Quasi-Monte Carlo Test-Time Scaling

  Michael Y. Li, Anthony Zhan, Kanishk Gandhi, Noah D. Goodman, Emily B. Fox · 2. Juli 2026

  Scaling inference compute, by generating many parallel attempts per problem, is a costly but reliable lever for improving language model capabilities. By default these attempts are generated independently, wasting inference compute on redundant solutions. This waste seems unavoidable. After all, ind…

• FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data

  Tao Feng, Haozhen Zhang, Zijie Lei, Pengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jiaxuan You · 2. Juli 2026

  The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks. This diversity drives researchers to employ multiple LLMs in practice, leaving behind valuable multi-LLM log data. This naturally leads to the question of whether such …

• Generated Contents Enrichment

  Mahdi Naseri, Jiayan Qiu, Zhou Wang · 2. Juli 2026

  We study Generated Contents Enrichment (GCE), a conditional image-generation task in which a sparse scene description is first enriched through an explicit scene representation and then rendered into semantically richer visual content. Conventional image-generation systems can produce visually reali…