Speech Recognition and Synthesis

Latest papers

• Improving Direct Persian-English Speech-to-Speech Translation with Discrete Units and Synthetic Parallel Data

  Sina Rashidi, Hossein Sameti · 18 November 2025

  Direct speech-to-speech translation (S2ST), in which all components are trained jointly, is an attractive alternative to cascaded systems because it offers a simpler pipeline and lower inference latency. However, direct S2ST models require large amounts of parallel speech data in the source and targ…

• BanglaTalk: Towards Real-Time Speech Assistance for Bengali Regional Dialects

  Jakir Hasan, Shubhashis Roy Dipta · 17 November 2025

  Real-time speech assistants are becoming increasingly popular for ensuring improved accessibility to information. Bengali, being a low-resource language with a high regional dialectal diversity, has seen limited progress in developing such systems. Existing systems are not optimized for real-time us…

• Sparse Tuning Enhances Plasticity in PTM-based Continual Learning

  Huan Zhang, Shenghua Fan, Shuyu Dong, Yujin Zheng, Dingwen Wang, Fan Lyu · 17 November 2025

  Continual Learning with Pre-trained Models holds great promise for efficient adaptation across sequential tasks. However, most existing approaches freeze PTMs and rely on auxiliary modules like prompts or adapters, limiting model plasticity and leading to suboptimal generalization when facing signif…

• Know Your Limits: Entropy Estimation Modeling for Compression and Generalization

  Benjamin L. Badger, Matthew Neligeorge · 14 November 2025

  Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a lower bound to language compression. The most efficient language compression algorithms today are causal (next token pred…

• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism

  Zedong Liu, Shenggan Cheng, Guangming Tan, Yang You, Dingwen Tao · 12 November 2025

  Multimodal large language models (MLLMs) extend LLMs to handle images, videos, and audio by incorporating feature extractors and projection modules. However, these additional components -- combined with complex inference pipelines and heterogeneous workloads -- introduce significant inference overhe…

• SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognition

  Jiaqi Wang, Liutao Yu, Xiongri Shen, Sihang Guo, Chenlin Zhou, Leilei Zhao, Yi Zhong, Zhengyu Ma, Zhiguo Zhang · 12 November 2025

  Spiking neural networks (SNNs) offer a promising path toward energy-efficient speech command recognition (SCR) by leveraging their event-driven processing paradigm. However, existing SNN-based SCR methods often struggle to capture rich temporal dependencies and contextual information from speech due…

• Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment

  Seong-Gyun Leem, Daniel Fulford, Jukka-Pekka Onnela, David Gard, Carlos Busso · 11 November 2025

  Speech emotion recognition (SER) systems often struggle in real-world environments, where ambient noise severely degrades their performance. This paper explores a novel approach that exploits prior knowledge of testing environments to maximize SER performance under noisy conditions. To address this …

• E2E-VGuard: Adversarial Prevention for Production LLM-based End-To-End Speech Synthesis

  Zhisheng Zhang, Derui Wang, Yifan Mi, Zhiyong Wu, Jie Gao, Yuxin Cao, Kai Ye, Minhui Xue, Jie Hao · 11 November 2025

  Recent advancements in speech synthesis technology have enriched our daily lives, with high-quality and human-like audio widely adopted across real-world applications. However, malicious exploitation like voice-cloning fraud poses severe security risks. Existing defense techniques struggle to addres…

• Factual and Musical Evaluation Metrics for Music Language Models

  Daniel Chenyu Lin, Michael Freeman, John Thickstun · 11 November 2025

  Music language models (Music LMs), like vision language models, leverage multimodal representations to answer natural language queries about musical audio recordings. Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct. Spec…

• Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition

  Zijin Gu, Tatiana Likhomanenko, Navdeep Jaitly · 6 November 2025

  Mixture-of-experts (MoE) architectures have expanded from language modeling to automatic speech recognition (ASR). Traditional MoE methods, such as the Switch Transformer, route experts independently within each layer. Our analysis reveals that routers in most layers make expert choices that are not…

• PolyNorm: Few-Shot LLM-Based Text Normalization for Text-to-Speech

  Michel Wong, Ali Alshehri, Sophia Kao, Haotian He · 6 November 2025

  Text Normalization (TN) is a key preprocessing step in Text-to-Speech (TTS) systems, converting written forms into their canonical spoken equivalents. Traditional TN systems can exhibit high accuracy, but involve substantial engineering effort, are difficult to scale, and pose challenges to language…

• Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models

  Jucheng Shen, Yeonju Ro · 5 November 2025

  Masked diffusion language models (MDLMs) are becoming competitive with their autoregressive counterparts but typically decode with fixed steps and sequential unmasking. To accelerate decoding, recent work such as Fast-dLLM enables parallel decoding via a static global confidence threshold, yet we ob…

• NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion

  Zongyang Du, Shreeram Suresh Chandra, Ismail Rasim Ulgen, Aurosweta Mahapatra, Ali N. Salman, Carlos Busso, Berrak Sisman · 4 November 2025

  Everyday speech conveys far more than words, it reflects who we are, how we feel, and the circumstances surrounding our interactions. Yet, most existing speech datasets are acted, limited in scale, and fail to capture the expressive richness of real-life communication. With the rise of large neural …

• PoLAR: Polar-Decomposed Low-Rank Adapter Representation

  Kai Lion, Liang Zhang, Bingcong Li, Niao He · 3 November 2025

  We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the pola…

• Towards a Method for Synthetic Generation of PWA Transcripts

  Jason M. Pittman, Anton Phillips Jr., Yesenia Medina-Santos, Brielle C. Stark · 30 October 2025

  In aphasia research, Speech-Language Pathologists (SLPs) devote extensive time to manually coding speech samples using Correct Information Units (CIUs), a measure of how informative an individual sample of speech is. Developing automated systems to recognize aphasic language is limited by data scarc…

• ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources

  Jason Wu, Yuyang Yuan, Kang Yang, Lance Kaplan, Mani Srivastava · 29 October 2025

  Multimodal deep learning systems are deployed in dynamic scenarios due to the robustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute resource availability (due to multi-tenancy, device heterogeneity, etc.) and fluctuating quality of inputs (from sensor f…

• SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training

  Yehonathan Refael, Guy Smorodinsky, Tom Tirer, Ofir Lindenbaum · 28 October 2025

  Low-rank gradient-based optimization methods have significantly improved memory efficiency during the training of large language models (LLMs), enabling operations within constrained hardware without sacrificing performance. However, these methods primarily emphasize memory savings, often overlookin…

• LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models

  Shubhang Bhatnagar, Andy Xu, Kar-Han Tan, Narendra Ahuja · 28 October 2025

  Large Language Models (LLMs) with multimodal capabilities have revolutionized vision-language tasks, but their deployment often requires huge memory and computational resources. While post-training quantization (PTQ) has successfully compressed language models to as low as 1-bit precision without si…

• OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

  Chen Wang, Tianyu Peng, Wen Yang, Yinan Bai, Guangfu Wang, Jun Lin, Lanpeng Jia, Lingxiang Wu, Jinqiao Wang, Chengqing Zong, Jiajun Zhang · 28 October 2025

  Empathetic interaction is a cornerstone of human-machine communication, due to the need for understanding speech enriched with paralinguistic cues and generating emotional and expressive responses. However, the most powerful empathetic LSLMs are increasingly closed off, leaving the crucial details a…

• Nes2Net: A Lightweight Nested Architecture for Foundation Model Driven Speech Anti-spoofing

  Tianchi Liu, Duc-Tuan Truong, Rohan Kumar Das, Kong Aik Lee, Haizhou Li · 28 October 2025

  Speech foundation models have significantly advanced various speech-related tasks by providing exceptional representation capabilities. However, their high-dimensional output features often create a mismatch with downstream task models, which typically require lower-dimensional inputs. A common solu…

• Arabic Little STT: Arabic Children Speech Recognition Dataset

  Mouhand Alkadri, Dania Desouki, Khloud Al Jallad · 28 October 2025

  The performance of Artificial Intelligence (AI) systems fundamentally depends on high-quality training data. However, low-resource languages like Arabic suffer from severe data scarcity. Moreover, the absence of child-specific speech corpora is an essential gap that poses significant challenges. To …

• Adaptive Non-uniform Timestep Sampling for Accelerating Diffusion Model Training

  Myunsoo Kim, Donghyeon Ki, Seong-Woong Shim, Byung-Jun Lee · 27 October 2025

• Data-Centric Lessons To Improve Speech-Language Pretraining

  Vishaal Udandarao, Zhiyun Lu, Xuankai Chang, Yongqiang Wang, Violet Z. Yao, Albin Madapally Jose, Fartash Faghri, Josh Gardner, Chung-Cheng Chiu · 27 October 2025

• Elementary, My Dear Watson: Non-Invasive Neural Keyword Spotting in the LibriBrain Dataset

  Gereon Elvers, Gilad Landau, Oiwi Parker Jones · 27 October 2025

• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning

  Yeongbin Seo, Dongha Lee, Jaehyung Kim, Jinyoung Yeo · 27 October 2025