Natural Language Processing Techniques

Latest papers

• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes

  Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Yemuk Choi, Kyubeen Han, Seokhee Hong, Junwon Hwang, Taewan Hwang, Joonwon Jang, Hyojin Jeon, Kijeong Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Euisoon Kim, Hyosang Kim, Jihoon Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayoung Lee, Gwangho Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Sangha Park, Young Min Paik, Yongmin Park, Youngyong Park, Sanghyun Seo, Sihoon Yang, Heuiyeen Yeen, Sihyuk Yi, Hyeongu Yun · 5 January 2026

  This technical report introduces EXAONE 4.0, which integrates a Non-reasoning mode and a Reasoning mode to achieve both the excellent usability of EXAONE 3.5 and the advanced reasoning abilities of EXAONE Deep. To pave the way for the agentic AI era, EXAONE 4.0 incorporates essential features such a…

• Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis

  Sonia Cromp, Satya Sai Srinath Namburi GNVV, Mohammed Alkhudhayri, Catherine Cao, Samuel Guo, Nicholas Roberts, Frederic Sala · 5 January 2026

  While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transforme…

• Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluators

  Zhengyu Hu, Jieyu Zhang, Zhihan Xiong, Alexander Ratner, Kaize Ding, Ranjay Krishna · 5 January 2026

  Despite the remarkable success of Large Language Models (LLMs), evaluating their outputs' quality regarding preference remains a critical challenge. While existing works usually leverage a strong LLM as the judge for comparing LLMs' response pairwisely, such a single-evaluator approach is vulnerable…

• The Curse of Depth in Large Language Models

  Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu · 5 January 2026

  In this paper, we introduce the Curse of Depth, a concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) where nearly half of the layers are less effective than expected. We first confirm the wide existence of this phenomenon across the most po…

• JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation

  Leonard Lin (Shisa.AI), Adam Lensenmayer (Shisa.AI) · 5 January 2026

  We introduce JP-TL-Bench, a lightweight, open benchmark designed to guide the iterative development of Japanese-English translation systems. In this context, the challenge is often "which of these two good translations is better?" rather than "is this translation acceptable?" This distinction matter…

• From Transformers to LLMs: A Systematic Survey of Efficiency Considerations in NLP

  Wazib Ansar, Saptarsi Goswami, Amlan Chakrabarti · 5 January 2026

  The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy con…

• Fast-weight Product Key Memory

  Tianyu Zhao, Llion Jones · 5 January 2026

  Sequence modeling layers in modern language models typically face a trade-off between storage capacity and computational efficiency. While Softmax attention offers unbounded storage at prohibitive quadratic costs, linear variants provide efficiency but suffer from limited, fixed-size storage. We pro…

• Large Multimodal Models for Low-Resource Languages: A Survey

  Marian Lupascu, Ana-Cristina Rogoz, Mihai Sorin Stupariu, Radu Tudor Ionescu · 1 January 2026

  In this survey, we systematically analyze techniques used to adapt large multimodal models (LMMs) for low-resource (LR) languages, examining approaches ranging from visual enhancement and data creation to cross-modal transfer and fusion strategies. Through a comprehensive analysis of 117 studies acr…

• MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining

  Zhixun Chen, Ping Guo, Wenhan Han, Yifan Zhang, Binbin Liu, Haobin Lin, Fengze Liu, Yan Zhao, Bingni Zhang, Taifeng Wang, Yin Zheng, Meng Fang · 1 January 2026

  Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a single rater for 17 target languages. MuR…

• MangaVQA and MangaLMM: A Benchmark and Specialized Model for Multimodal Manga Understanding

  Jeonghun Baek, Kazuki Egashira, Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Hikaru Ikuta, Kiyoharu Aizawa · 1 January 2026

  Manga, or Japanese comics, is a richly multimodal narrative form that blends images and text in complex ways. Teaching large multimodal models (LMMs) to understand such narratives at a human-like level could help manga creators reflect on and refine their stories. To this end, we introduce two bench…

• Joint Selection for Large-Scale Pre-Training Data via Policy Gradient-based Mask Learning

  Ziqing Fan, Yuqiao Xian, Yan Sun, Li Shen · 1 January 2026

  A fine-grained data recipe is crucial for pre-training large language models, as it can significantly enhance training efficiency and model performance. One important ingredient in the recipe is to select samples based on scores produced by defined rules, LLM judgment, or statistical information in …

• Natural Language Processing for Tigrinya: Current State and Future Directions

  Fitsum Gaim, Jong C. Park · 1 January 2026

  Despite being spoken by millions of people, Tigrinya remains severely underrepresented in Natural Language Processing (NLP) research. This work presents a comprehensive survey of NLP research for Tigrinya, analyzing over 50 studies from 2011 to 2025. We systematically review the current state of com…

• MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs

  Wenqian Ye, Bohan Liu, Guangtao Zheng, Di Wang, Yunsheng Ma, Xu Cao, Bolin Lai, James M. Rehg, Aidong Zhang · 1 January 2026

  Spurious bias, a tendency to exploit spurious correlations between superficial input attributes and prediction targets, has revealed a severe robustness pitfall in classical machine learning problems. Multimodal Large Language Models (MLLMs), which leverage pretrained vision and language models, hav…

• Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

  Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, Dacheng Tao · 1 January 2026

  Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available…

• LTLBench: Towards Benchmarks for Evaluating Temporal Logic Reasoning in Large Language Models

  Weizhi Tang, Kwabena Nuamah, Vaishak Belle · 1 January 2026

  Temporal Reasoning (TR) is a critical ability for LLMs to understand and reason over temporal information and relationships between events. To study the TR ability in LLMs, prior works provide different ways for evaluating various aspects of TR ability. In this work, we propose an alternative perspe…

• Adaptive Dependency-aware Prompt Optimization Framework for Multi-Step LLM Pipeline

  Minjun Zhao, Xinyu Zhang, Shuai Zhang, Deyang Li, Ruifeng Shi · 1 January 2026

  Multi-step LLM pipelines invoke large language models multiple times in a structured sequence and can effectively solve complex tasks, but their performance heavily depends on the prompts used at each step. Jointly optimizing these prompts is difficult due to missing step-level supervision and inter…

• ICONS: Influence Consensus for Vision-Language Data Selection

  Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, Olga Russakovsky · 30 December 2025

  Training vision-language models via instruction tuning relies on large data mixtures spanning diverse tasks and domains, yet these mixtures frequently include redundant information that increases computational costs without proportional gains. Existing methods typically rely on task-agnostic heurist…

• The Bayesian Geometry of Transformer Attention

  Naman Aggarwal, Siddhartha R. Dalal, Vishal Misra · 30 December 2025

  Transformers often appear to perform Bayesian reasoning in context, but verifying this rigorously has been impossible: natural data lack analytic posteriors, and large models conflate reasoning with memorization. We address this by constructing \emph{Bayesian wind tunnels} -- controlled environments…

• Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression

  Cheng Yuan, Jiawei Shao, Chi Zhang, Xuelong Li · 30 December 2025

  Recent years have witnessed the rapid advancements of large language models (LLMs) and their expanding applications, leading to soaring demands for computational resources. The widespread adoption of test-time scaling further aggravates the tension between model capability and resource consumption, …

• The Heap: A Contamination-Free Multilingual Code Dataset for Evaluating Large Language Models

  Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen, Maliheh Izadi · 30 December 2025

  The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without su…

• TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs

  Sibo Xiao, Jinyuan Fu, Zhongle Xie, Lidan Shou · 30 December 2025

  Accelerating the inference of large language models (LLMs) has been a critical challenge in generative AI. Speculative decoding (SD) substantially improves LLM inference efficiency. However, its utility is limited by a fundamental constraint: the draft and target models must share the same vocabular…

• MatKV: Trading Compute for Flash Storage in LLM Inference

  Kun-Woo Shin (Seoul National University, Korea), Jay H. Park (Samsung Electronics, Korea), Moonwook Oh (Samsung Electronics, Korea), Yohan Jo (Seoul National University, Korea), Jaeyoung Do (Seoul National University, Korea), Sang-Won Lee (Seoul National University, Korea) · 30 December 2025

  We observe two major trends in LLM-based generative AI: (1) inference is becoming the dominant factor in terms of cost and power consumption, surpassing training, and (2) retrieval augmented generation (RAG) is becoming prevalent. When processing long inputs in RAG, the prefill phase of computing th…

• Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL

  Rihong Qiu, Zhibang Yang, Xinke Jiang, Weibin Liao, Xin Gao, Xu Chu, Junfeng Zhao, Yasha Wang · 30 December 2025

  Text-to-SQL translates natural language questions into SQL statements grounded in a target database schema. Ensuring the reliability and executability of such systems requires validating generated SQL, but most existing approaches focus only on syntactic correctness, with few addressing semantic val…

• Generative Language Models on Nucleotide Sequences of Human Genes

  Musa Nuri Ihtiyar, Arzucan Ozgur · 29 December 2025

  Language models, especially transformer-based ones, have achieved colossal success in NLP. To be precise, studies like BERT for NLU and works like GPT-3 for NLG are very important. If we consider DNA sequences as a text written with an alphabet of four letters representing the nucleotides, they are …

• Parallel Token Prediction for Language Models

  Felix Draxler, Justus Will, Farrin Marouf Sofian, Theofanis Karaletsos, Sameer Singh, Stephan Mandt · 25 December 2025

  We propose Parallel Token Prediction (PTP), a universal framework for parallel sequence generation in language models. PTP jointly predicts multiple dependent tokens in a single transformer call by incorporating the sampling procedure into the model. This reduces the latency bottleneck of autoregres…