Speech Recognition and Synthesis

Latest papers

• On the Quantization Robustness of Diffusion Language Models in Coding Benchmarks

  Aarav Gupta, Gururaj Deshpande, Chandreyi Chakraborty · 23 April 2026

  Auto-regressive Large Language Models (LLMs) achieve strong performance on coding tasks, but incur high memory and inference costs. Diffusion-based language models (d-LLMs) offer bounded inference cost via iterative denoising, but their behavior under post-training quantization (PTQ) has been sparse…

• Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model

  Shuhai Peng, Hui Lu, Jinjiang Liu, Liyang Chen, Guiping Zhong, Jiakui Li, Huimeng Wang, Haiyun Li, Liang Cao, Shiyin Kang, Zhiyong Wu · 22 April 2026

  While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications. Direct adaptation to streaming scenarios often leads to catastrophic inference performance degradation due to the severe mism…

• Tadabur: A Large-Scale Quran Audio Dataset

  Faisal Alherran · 22 April 2026

  Despite growing interest in Quranic data research, existing Quran datasets remain limited in both scale and diversity. To address this gap, we present Tadabur, a large-scale Quran audio dataset. Tadabur comprises more than 1400+ hours of recitation audio from over 600 distinct reciters, providing su…

• UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction

  Yadong Li, Guoxin Wu, Haiping Hou, Biye Li · 22 April 2026

  Full-duplex speech interaction, as the most natural and intuitive mode of human communication, is driving artificial intelligence toward more human-like conversational systems. Traditional cascaded speech processing pipelines suffer from critical limitations, including accumulated latency, informati…

• Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization

  Andrei Andrusenko, Vladimir Bataev, Lilit Grigoryan, Nune Tadevosyan, Vitaly Lavrukhin, Boris Ginsburg · 22 April 2026

  Unification of automatic speech recognition (ASR) systems reduces development and maintenance costs, but training a single model to perform well in both offline and low-latency streaming settings remains challenging. We present a Unified ASR framework for Transducer (RNNT) training that supports bot…

• Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation

  Habibeh Naderi, Behrouz Haji Soleimani, Stan Matwin · 21 April 2026

  We introduce CoCo-LoRA, a multimodal, uncertainty-aware parameter-efficient fine-tuning method for text prediction tasks accompanied by audio context. Existing PEFT approaches such as LoRA are efficient but typically deterministic, while recent Bayesian low-rank adapters model uncertainty in a light…

• Cross-Modal Bayesian Low-Rank Adaptation for Uncertainty-Aware Multimodal Learning

  Habibeh Naderi, Behrouz Haji Soleimani, Stan Matwin · 21 April 2026

  Large pre-trained language models are increasingly adapted to downstream tasks using parameter-efficient fine-tuning (PEFT), but existing PEFT methods are typically deterministic and unimodal, making them poorly suited for low-resource multimodal settings where predictive uncertainty and cross-modal…

• Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization

  Habibeh Naderi, Behrouz Haji Soleimani, Stan Matwin · 20 April 2026

  We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained sp…

• AST: Adaptive, Seamless, and Training-Free Precise Speech Editing

  Sihan Lv, Yechen Jin, Zhen Li, Jintao Chen, Jinshan Zhang, Ying Li, Jianwei Yin, Meng Xi · 20 April 2026

  Text-based speech editing aims to modify specific segments while preserving speaker identity and acoustic context. Existing methods rely on task-specific training, which incurs high data costs and struggles with temporal fidelity in unedited regions. Meanwhile, adapting Text-to-Speech (TTS) models o…

• HARNESS: Lightweight Distilled Arabic Speech Foundation Models

  Vrunda N. Sukhadia, Shammur Absar Chowdhury · 17 April 2026

  Large self-supervised speech (SSL) models achieve strong downstream performance, but their size limits deployment in resource-constrained settings. We present HArnESS, an Arabic-centric self-supervised speech model family trained from scratch with iterative self-distillation, together with lightweig…

• Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference

  Nenad Banfic, David Fan, Kunal Vaishnavi, Sam Kemp, Sunghoon Choi, Rui Ren, Sayan Shaw, Meng Tang · 17 April 2026

  Deploying high-quality automatic speech recognition (ASR) on edge devices requires models that jointly optimize accuracy, latency, and memory footprint while operating entirely on CPU without GPU acceleration. We conduct a systematic empirical study of state-of-the-art ASR architectures, encompassin…

• Diffusion Language Models for Speech Recognition

  Davyd Naveriani, Albert Zeyer, Ralf Schl\"uter, Hermann Ney · 16 April 2026

  Diffusion language models have recently emerged as a leading alternative to standard language models, due to their ability for bidirectional attention and parallel text generation. In this work, we explore variants for their use in speech recognition. Specifically, we introduce a comprehensive guide…

• Giving Voice to the Constitution: Low-Resource Text-to-Speech for Quechua and Spanish Using a Bilingual Legal Corpus

  John E. Ortega, Rodolfo Zevallos, Fabricio Carraro · 16 April 2026

  We present a unified pipeline for synthesizing high-quality Quechua and Spanish speech for the Peruvian Constitution using three state-of-the-art text-to-speech (TTS) architectures: XTTS v2, F5-TTS, and DiFlow-TTS. Our models are trained on independent Spanish and Quechua speech datasets with hetero…

• X-VC: Zero-shot Streaming Voice Conversion in Codec Space

  Qixi Zheng, Yuxiang Zhao, Tianrui Wang, Wenxi Chen, Kele Xu, Yikang Li, Qinyuan Chen, Xipeng Qiu, Kai Yu, Xie Chen · 15 April 2026

  Zero-shot voice conversion (VC) aims to convert a source utterance into the voice of an unseen target speaker while preserving its linguistic content. Although recent systems have improved conversion quality, building zero-shot VC systems for interactive scenarios remains challenging because high-fi…

• CoMelSinger: Discrete Token-Based Zero-Shot Singing Synthesis With Structured Melody Control and Guidance

  Junchuan Zhao, Wei Zeng, Tianle Lyu, Ye Wang · 14 April 2026

  Singing Voice Synthesis (SVS) aims to generate expressive vocal performances from structured musical inputs such as lyrics and pitch sequences. While recent progress in discrete codec-based speech synthesis has enabled zero-shot generation via in-context learning, directly extending these techniques…

• Regularized Entropy Information Adaptation with Temporal-Awareness Networks for Simultaneous Speech Translation

  Joseph Liu, Nameer Hirschkind, Xiao Yu, Mahesh Kumar Nandwana · 14 April 2026

  Simultaneous Speech Translation (SimulST) requires balancing high translation quality with low latency. Recent work introduced REINA, a method that trains a Read/Write policy based on estimating the information gain of reading more audio. However, we find that information-based policies often lack t…

• Real-Time Voicemail Detection in Telephony Audio Using Temporal Speech Activity Features

  Kumar Saurav · 14 April 2026

  Outbound AI calling systems must distinguish voicemail greetings from live human answers in real time to avoid wasted agent interactions and dropped calls. We present a lightweight approach that extracts 15 temporal features from the speech activity pattern of a pre-trained neural voice activity det…

• AudioGuard: Toward Comprehensive Audio Safety Protection Across Diverse Threat Models

  Mintong Kang, Chen Fang, Bo Li · 13 April 2026

  Audio has rapidly become a primary interface for foundation models, powering real-time voice assistants. Ensuring safety in audio systems is inherently more complex than just "unsafe text spoken aloud": real-world risks can hinge on audio-native harmful sound events, speaker attributes (e.g., child …

• Practical Bayesian Inference for Speech SNNs: Uncertainty and Loss-Landscape Smoothing

  Yesmine Abdennadher, Philip N. Garner · 13 April 2026

  Spiking Neural Networks (SNNs) are naturally suited for speech processing tasks due to their specific dynamics, which allows them to handle temporal data. However, the threshold-based generation of spikes in SNNs intuitively causes an angular or irregular predictive landscape. We explore the effect …

• Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

  Peng Wang (X-LANCE Lab, Shanghai Jiao Tong University), Yanqiao Zhu (X-LANCE Lab, Shanghai Jiao Tong University), Zixuan Jiang (X-LANCE Lab, Shanghai Jiao Tong University), Qinyuan Chen (School of Computer Science, Fudan University), Xingjian Zhao (School of Computer Science, Fudan University), Xipeng Qiu (School of Computer Science, Fudan University), Wupeng Wang (Tongyi Fun Team, Alibaba Group), Zhifu Gao (Tongyi Fun Team, Alibaba Group), Xiangang Li (Tongyi Fun Team, Alibaba Group), Kai Yu (X-LANCE Lab, Shanghai Jiao Tong University), Xie Chen (X-LANCE Lab, Shanghai Jiao Tong University) · 13 April 2026

  Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all wo…

• DDSP-QbE++: Improving Speech Quality for Speech Anonymisation for Atypical Speech

  Suhita Ghosh, Yamini Sinha, Sebastian Stober · 13 April 2026

  Differentiable Digital Signal Processing (DDSP) pipelines for voice conversion rely on subtractive synthesis, where a periodic excitation signal is shaped by a learned spectral envelope to reconstruct the target voice. In DDSP-QbE, the excitation is generated via phase accumulation, producing a sawt…

• WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models

  Hanna Lee, Tan Dat Nguyen, Jaehoon Kang, Kyuhong Shim · 13 April 2026

  Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adap…

• A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech

  Jia-Hong Huang, Seulgi Kim, Yi Chieh Liu, Yixian Shen, Hongyi Zhu, Prayag Tiwari, Stevan Rudinac, Evangelos Kanoulas · 10 April 2026

  Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phenomenon undermines the coherence of synthetic speech, especially…

• In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads

  Charlotte Pouw, Hosein Mohebbi, Afra Alishahi, Willem Zuidema · 10 April 2026

  In-Context Learning (ICL) has been extensively studied in text-only Language Models, but remains largely unexplored in the speech domain. Here, we investigate how linguistic and acoustic features affect ICL in Speech Language Models. We focus on the Text-to-Speech (TTS) task, which allows us to anal…

• Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

  Bajian Xiang, Tingwei Guo, Xuan Chen, Yang Han · 10 April 2026

  Large Speech Language Models (LSLMs) typically operate at high token rates (tokens/s) to ensure acoustic fidelity, yet this results in sequence lengths that far exceed the underlying semantic content, incurring prohibitive inference costs. In this paper, we empirically revisit the necessity of such …