Natural Language Processing Techniques

Neueste Paper

• Unraveling Syntax: How Language Models Learn Context-Free Grammars

  Laura Ying Schulz, Daniel Mitropolsky, Tomaso Poggio · 2. März 2026

  While large models achieve impressive results, their learning dynamics are far from understood. Many domains of interest, such as natural language syntax, coding languages, arithmetic problems, are captured by context-free grammars (CFGs). In this work, we extend prior work on neural language modeli…

• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes

  Changsheng Zhao, Ernie Chang, Zechun Liu, Chia-Jung Chang, Wei Wen, Chen Lai, Sheng Cao, Yuandong Tian, Raghuraman Krishnamoorthi, Yangyang Shi, Vikas Chandra · 2. März 2026

  The paradigm shift in large language models (LLMs) from instinctive responses to chain-of-thought (CoT) reasoning has fueled two prevailing assumptions: (1) reasoning capabilities only emerge in sufficiently large models, and (2) such capabilities require training on massive datasets. While the firs…

• Task-Centric Acceleration of Small-Language Models

  Dor Tsur, Sharon Adar, Ran Levy · 2. März 2026

  Small language models (SLMs) have emerged as efficient alternatives to large language models for task-specific applications. However, they are often employed in high-volume, low-latency settings, where efficiency is crucial. We propose TASC, Task-Adaptive Sequence Compression, a framework for SLM ac…

• FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuning

  Chaitali Bhattacharyya, Hyunsei Lee, Junyoung Lee, Shinhyoung Jang, Il hong Suh, Yeseong Kim · 2. März 2026

  Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA, llama} have served as starting points for domain…

• Preference Packing: Efficient Preference Optimization for Large Language Models

  Jaekyung Cho · 2. März 2026

  Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preferenc…

• Transformers converge to invariant algorithmic cores

  Joshua S. Schiffman · 27. Februar 2026

  Large language models exhibit sophisticated capabilities, yet understanding how they work internally remains a central challenge. A fundamental obstacle is that training selects for behavior, not circuitry, so many weight configurations can implement the same function. Which internal structures refl…

• Contextual Memory Virtualisation: DAG-Based State Management and Structurally Lossless Trimming for LLM Agents

  Cosmo Santoni · 27. Februar 2026

  As large language models engage in extended reasoning tasks, they accumulate significant state -- architectural mappings, trade-off decisions, codebase conventions -- within the context window. This understanding is lost when sessions reach context limits and undergo lossy compaction. We propose Con…

• OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data

  Yan Zhao, Zhengxue Cheng, Junxuan Zhang, Dajiang Zhou, Qunshan Gu, Qi Wang, Li Song · 27. Februar 2026

  Lossless compression is essential for efficient data storage and transmission. Although learning-based lossless compressors achieve strong results, most of them are designed for a single modality, leading to redundant compressor deployments in multi-modal settings. Designing a unified multi-modal co…

• Toward Automated Validation of Language Model Synthesized Test Cases using Semantic Entropy

  Hamed Taherkhani, Jiho Shin, Muhammad Ammar Tahir, Md Rakib Hossain Misu, Vineet Sunil Gattani, Hadi Hemmati · 27. Februar 2026

  Modern Large Language Model (LLM)-based programming agents often rely on test execution feedback to refine their generated code. These tests are synthetically generated by LLMs. However, LLMs may produce invalid or hallucinated test cases, which can mislead feedback loops and degrade the performance…

• Decoding Translation-Related Functional Sequences in 5'UTRs Using Interpretable Deep Learning Models

  Yuxi Lin, Yaxue Fang, Zehong Zhang, Zhouwu Liu, Siyun Zhong, Zhongfang Wang, Fulong Yu · 27. Februar 2026

  Understanding how 5' untranslated regions (5'UTRs) regulate mRNA translation is critical for controlling protein expression and designing effective therapeutic mRNAs. While recent deep learning models have shown promise in predicting translational efficiency from 5'UTR sequences, most are constraine…

• From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation

  Linus Bantel, Moritz Strack, Alexander Strack, Dirk Pfl\"uger · 27. Februar 2026

  Large Language Models (LLM) show strong abilities in code generation, but their skill in creating efficient parallel programs is less studied. This paper explores how LLMs generate task-based parallel code from three kinds of input prompts: natural language problem descriptions, sequential reference…

• LLM4AD: A Platform for Algorithm Design with Large Language Model

  Fei Liu, Rui Zhang, Zhuoliang Xie, Rui Sun, Kai Li, Qinglong Hu, Ping Guo, Xi Lin, Xialiang Tong, Mingxuan Yuan, Zhenkun Wang, Zhichao Lu, Qingfu Zhang · 27. Februar 2026

  We introduce LLM4AD, a unified Python platform for algorithm design (AD) with large language models (LLMs). LLM4AD is a generic framework with modularized blocks for search methods, algorithm design tasks, and LLM interface. The platform integrates numerous key methods and supports a wide range of a…

• Mitigating Legibility Tax with Decoupled Prover-Verifier Games

  Yegon Kim, Juho Lee · 27. Februar 2026

  As large language models become increasingly capable, it is critical that their outputs can be easily checked by less capable systems. Prover-verifier games can be used to improve checkability of model outputs, but display a degradation in accuracy compared to a baseline trained only to maximize cor…

• Cost-of-Pass: An Economic Framework for Evaluating Language Models

  Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou · 27. Februar 2026

  Widespread adoption of AI systems hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics accounting for both performance and costs. Building on production theory, we develop an economically grounded framework to evaluate lan…

• Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

  Hanna Yukhymenko, Anton Alexandrov, Martin Vechev · 26. Februar 2026

  The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer from semantic drift and context loss, which can lead to misleading performance metrics. In this work, we present a full…

• DOTResize: Reducing LLM Width via Discrete Optimal Transport-based Neuron Merging

  Neha Verma, Kenton Murray, Kevin Duh · 26. Februar 2026

  Structured pruning methods designed for Large Language Models (LLMs) generally focus on identifying and removing the least important components to optimize model size. However, in this work, we question this prevalent approach by instead exploring how to recombine information from structures designa…

• Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

  Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, Lifeng Jin · 26. Februar 2026

  Reinforcement fine-tuning (RFT) often suffers from reward over-optimization, where a policy model hacks the reward signals to achieve high scores while producing low-quality outputs. Our theoretical analysis shows that the key lies in reward misspecification at the high-reward tail: the inability to…

• Latent Context Compilation: Distilling Long Context into Compact Portable Memory

  Zeju Li, Yizhou Zhou, Qiang Xu · 26. Februar 2026

  Efficient long-context LLM deployment is stalled by a dichotomy between amortized compression, which struggles with out-of-distribution generalization, and Test-Time Training, which incurs prohibitive synthetic data costs and requires modifying model weights, creating stateful parameters that compli…

• BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMs

  Junxiao Yang, Jinzhe Tu, Haoran Liu, Xiaoce Wang, Chujie Zheng, Zhexin Zhang, Shiyao Cui, Caishun Chen, Tiantian He, Hongning Wang, Yew-Soon Ong, Minlie Huang · 26. Februar 2026

  Recent advances in Large Reasoning Models (LRMs) have shown impressive capabilities in mathematical and logical reasoning. However, current LRMs rarely admit ignorance or respond with "I don't know". Instead, they often produce incorrect answers while showing undue confidence, raising concerns about…

• Slm-mux: Orchestrating small language models for reasoning

  Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie, Tushar Krishna, Vijay Janapa Reddi, Yilun Du · 26. Februar 2026

  With the rapid development of language models, the number of small language models (SLMs) has grown significantly. Although they do not achieve state-of-the-art accuracy, they are more efficient and often excel at specific tasks. This raises a natural question: can multiple SLMs be orchestrated into…

• LLMs Process Lists With General Filter Heads

  Arnab Sen Sharma, Giordano Rogers, Natalie Shapira, David Bau · 25. Februar 2026

  We investigate the mechanisms underlying a range of list-processing tasks in LLMs, and we find that LLMs have learned to encode a compact, causal representation of a general filtering operation that mirrors the generic "filter" function of functional programming. Using causal mediation analysis on a…

• Augmenting Lateral Thinking in Language Models with Humor and Riddle Data for the BRAINTEASER Task

  Mina Ghashami, Soumya Smruti Mishra · 25. Februar 2026

  The SemEval 2024 BRAINTEASER task challenges language models to perform lateral thinking -- a form of creative, non-linear reasoning that remains underexplored in NLP. The task comprises two subtasks, Sentence Puzzle and Word Puzzle, requiring models to defy conventional commonsense associations. We…

• Programming by Backprop: An Instruction is Worth 100 Examples When Finetuning LLMs

  Jonathan Cook, Silvia Sapora, Arash Ahmadian, Akbir Khan, Tim Rocktaschel, Jakob Foerster, Laura Ruis · 25. Februar 2026

  Large language models (LLMs) are typically trained to acquire behaviours from demonstrations or experience, yet much of their training data is declarative: instructions, rules, and descriptions that specify behaviours without showing how to execute them. We introduce Programming by Backprop (PBB): a…

• QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation

  Yaoyu Zhu, Di Huang, Hanqi Lyu, Xiaoyun Zhang, Chongxiao Li, Wenxuan Shi, Yutong Wu, Jianan Mu, Jinghua Wang, Yang Zhao, Pengwei Jin, Shuyao Cheng, Shengwen Liang, Xishan Zhang, Rui Zhang, Zidong Du, Qi Guo, Xing Hu, Yunji Chen · 24. Februar 2026

  Large language models (LLMs) trained via reinforcement learning with verifiable reward (RLVR) have achieved breakthroughs on tasks with explicit, automatable verification, such as software programming and mathematical problems. Extending RLVR to electronic design automation (EDA), especially automat…

• Foundations of Top-$k$ Decoding For Language Models

  Georgy Noarov, Soham Mallick, Tao Wang, Sunay Joshi, Yan Sun, Yangxinyu Xie, Mengxin Yu, Edgar Dobriban · 24. Februar 2026

  Top-$k$ decoding is a widely used method for sampling from LLMs: at each token, only the largest $k$ next-token-probabilities are kept, and the next token is sampled after re-normalizing them to sum to unity. Top-$k$ and other sampling methods are motivated by the intuition that true next-token dist…