Speech Recognition and Synthesis

Latest papers

• Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization

  Mohammed Aman Bhuiyan, Md Sazzad Hossain Adib, Samiul Basir Bhuiyan, Amit Chakraborty, Aritra Islam Saswato, Ahmed Faizul Haque Dhrubo, Mohammad Ashrafuzzaman Khan · 12 May 2026

  Automatic Speech Recognition (ASR) and speaker diarization in Bangla remain challenging due to long form recordings, diverse acoustic conditions, and significant speaker variability. This work addresses these two core tasks in Bangla spoken language understanding by developing robust systems for lon…

• Kinetic-Optimal Scheduling with Moment Correction for Metric-Induced Discrete Flow Matching in Zero-Shot Text-to-Speech

  Dong Yang, Yiyi Cai, Haoyu Zhang, Yuki Saito, Hiroshi Saruwatari · 12 May 2026

  Metric-induced discrete flow matching (MI-DFM) exploits token-latent geometry for discrete generation, but its practical use is limited by two issues: heuristic schedulers requiring hyperparameter search, and finite-step path-tracking error from its first-order continuous-time Markov chain (CTMC) so…

• WorldSpeech: A Multilingual Speech Corpus from Around the World

  Antonis Asonitis, Luca A. Lanzend\"orfer, Fr\'ed\'eric Berdoz, Roger Wattenhofer · 12 May 2026

  Automatic speech recognition (ASR) performs well for high-resource languages with abundant paired audio-transcript data, but its accuracy degrades sharply for most languages due to limited publicly available aligned data. To this end, we introduce WorldSpeech, a 24 kHz multilingual speech corpus com…

• Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)

  Cyril Allauzen, Tom Bagby, Georg Heigold, Ehsan Variani, Ke Wu · 7 May 2026

  The Massive Sound Embedding Benchmark (MSEB) has emerged as a standard for evaluating the functional breadth of audio models. While initial baselines focused on specialized encoders, the shift toward "audio-native" Large Language Models (LLMs) suggests a new paradigm where a single multimodal backbo…

• JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions

  Leying Zhang, Bowen Shi, Haibin Wu, Bach Viet Do, Yanmin Qian · 7 May 2026

  The rapid advancement of generative audio models has outpaced the development of robust evaluation methodologies. Existing objective metrics and general multimodal large language models (MLLMs) often struggle with domain generalization, zero-shot capabilities, and instructional flexibility. To addre…

• Contrastive Regularization for Accent-Robust ASR

  Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham, Sameer Alam · 6 May 2026

  ASR systems based on self-supervised acoustic pretraining and CTC fine-tuning achieve strong performance on native speech but remain sensitive to accent variability. We investigate supervised contrastive learning (SupCon) as a lightweight, accent-invariant auxiliary objective for CTC fine-tuning. An…

• Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detection

  Joydeep Chandra · 6 May 2026

  Continuous monitoring of bipolar disorder agitation via voice biomarkers requires disentangling stable speaker traits from volatile affective states on resource-constrained edge devices. We introduce MP-IB, the first framework to treat mixed-precision quantization as an information bottleneck for cl…

• BaldWhisper: Faster Whisper with Head Shearing and Layer Merging

  Yaya Sy, Christophe Cerisara, Irina Illina · 6 May 2026

  Pruning large pre-trained transformers in a data-scarce scenario is challenging, as it often requires massive retraining data to recover performance. For instance, Distill-Whisper prunes Whisper by 40 and retrains on 21,000 hours of speech, far beyond what is available for most languages. Can Whispe…

• Dimensionality-Aware Anomaly Detection in Learned Representations of Self-Supervised Speech Models

  Sandra Arcos-Holzinger, Sarah M. Erfani, James Bailey, Sanjeev Khudanpur · 5 May 2026

  Self-supervised speech models (S3Ms) achieve strong downstream performance, yet their learned representations remain poorly understood under natural and adversarial perturbations. Prior studies rely on representation similarity or global dimensionality, offering limited visibility into local geometr…

• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment

  Woody Haosheng Gan, William Held, Diyi Yang · 4 May 2026

  The rapid proliferation of large audio models (LAMs) demands efficient approaches for model comparison, yet comprehensive benchmarks are costly. To fill this gap, we investigate whether minimal subsets can reliably evaluate LAMs while reducing costs and data redundancy. Analyzing 10 subset selection…

• Selective Augmentation: Improving Universal Automatic Phonetic Transcription via G2P Bootstrapping

  Tobias Bystrich, Julia M. Pritzen, Christoph A. Schmidt, Claudia Wich-Reif · 1 May 2026

  In the field of universal automatic phonetic transcription (APT), clean and diverse training transcriptions are required. However, such high-quality data is limited. We propose the bootstrapping approach Selective Augmentation to improve the available training transcriptions by selectively transferr…

• A Toolkit for Detecting Spurious Correlations in Speech Datasets

  Lara Gauder, Pablo Riera, Andrea Slachevsky, Gonzalo Forno, Adolfo M. Garc\'ia, Luciana Ferrer · 30 April 2026

  We introduce a toolkit for uncovering spurious correlations between recording characteristics and target class in speech datasets. Spurious correlations may arise due to heterogeneous recording conditions, a common scenario for health-related datasets. When present both in the training and test data…

• DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

  Ismail Rasim Ulgen, Zexin Cai, Nicholas Andrews, Philipp Koehn, Berrak Sisman · 30 April 2026

  To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility-privacy trade-off, operatin…

• Text-Utilization for Encoder-dominated Speech Recognition Models

  Albert Zeyer, Tim Posielek, Ralf Schl\"uter, Hermann Ney · 30 April 2026

  This paper investigates efficient methods for utilizing text-only data to improve speech recognition, focusing on encoder-dominated models that facilitate faster recognition. We provide a comprehensive comparison of techniques to integrate text-only data, including modality matching and dynamic down…

• RAS: a Reliability Oriented Metric for Automatic Speech Recognition

  Wenbin Huang, Yuhang Qiu, Bohan Li, Yiwei Guo, Jing Peng, Hankun Wang, Xie Chen, Kai Yu · 29 April 2026

  Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcriptio…

• Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech

  Rikuto Kotoge, Yuichi Sasaki · 29 April 2026

  Aligning text-to-speech (TTS) system outputs with human feedback through preference optimization has been shown to effectively improve the robustness and naturalness of language model-based TTS models. Current approaches primarily require paired desirable and undesirable samples at the utterance lev…

• Scaling Properties of Continuous Diffusion Spoken Language Models

  Jason Ramapuram, Eeshan Gunesh Dhekane, Amitis Shidani, Dan Busbridge, Bogdan Mazoure, Zijin Gu, Russ Webb, Tatiana Likhomanenko, Navdeep Jaitly · 28 April 2026

  Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore wheth…

• In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

  Xulin Fan, Vishal Sunder, Samuel Thomas, Mark Hasegawa-Johnson, Brian Kingsbury, George Saon · 28 April 2026

  Recent advances in speech-aware language models have coupled strong acoustic encoders with large language models, enabling systems that move beyond transcription to produce richer outputs. Among these, word-level timestamp prediction is critical for applications such as captioning, media search, and…

• FMSD-TTS: Few-shot Multi-Speaker Multi-Dialect Text-to-Speech Synthesis for \"U-Tsang, Amdo and Kham Speech Dataset Generation

  Yutong Liu, Ziyue Zhang, Ban Ma-bao, Yuqing Cai, Yongbin Yu, Renzeng Duojie, Xiangxiang Wang, Fan Gao, Cheng Huang, Nyima Tashi · 27 April 2026

  Tibetan is a low-resource language with minimal parallel speech corpora spanning its three major dialects-\"U-Tsang, Amdo, and Kham-limiting progress in speech modeling. To address this issue, we propose FMSD-TTS, a few-shot, multi-speaker, multi-dialect text-to-speech framework that synthesizes par…

• KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness

  Jinyoung Kim, Hyeongsoo Lim, Eunseo Seo, Minho Jang, Keunwoo Choi, Seungyoun Shin, Ji Won Yoon · 24 April 2026

  Recent advances in large audio language models (LALMs) have enabled multilingual speech understanding. However, benchmarks for evaluating LALMs remain scarce for non-English languages, with Korean being one such underexplored case. In this paper, we introduce KoALa-Bench, a comprehensive benchmark f…

• Enhancing ASR Performance in the Medical Domain for Dravidian Languages

  Sri Charan Devarakonda, Ravi Sastry Kolluru, Manjula Sri Rayudu, Rashmi Kapoor, Madhu G, Anil Kumar Vuppala · 24 April 2026

  Automatic Speech Recognition (ASR) for low-resource Dravidian languages like Telugu and Kannada faces significant challenges in specialized medical domains due to limited annotated data and morphological complexity. This work proposes a novel confidence-aware training framework that integrates real …

• Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speech

  Gus Lathouwers, Lingyun Gao, Catia Cucchiarini, Helmer Strik · 24 April 2026

  Automatic Speech Recognition (ASR) is increasingly used in applications involving child speech, such as language learning and literacy acquisition. However, the effectiveness of such applications is limited by high ASR error rates. The negative effects can be mitigated by identifying in advance whic…

• Enhancing Speaker Verification with Whispered Speech via Post-Processing

  Magdalena Go{\l}\k{e}biowska, Piotr Syga · 24 April 2026

  Speaker verification is a task of confirming an individual's identity through the analysis of their voice. Whispered speech differs from phonated speech in acoustic characteristics, which degrades the performance of speaker verification systems in real-life scenarios, including avoiding fully phonat…

• SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation

  Ruohan Liu, Shukang Yin, Tao Wang, Dong Zhang, Weiji Zhuang, Shuhuai Ren, Ran He, Caifeng Shan, Chaoyou Fu · 24 April 2026

  Paralinguistic cues are essential for natural human-computer interaction, yet their evaluation in Large Audio-Language Models (LALMs) remains limited by coarse feature coverage and the inherent subjectivity of assessment. To address these challenges, we introduce SpeechParaling-Bench, a comprehensiv…

• Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis

  Zehua Pei, Hui-Ling Zhen, Lancheng Zou, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu · 24 April 2026

  Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational resources. While Mixture-of-Experts (MoE) architectures can reduce this cost through sparse activation, restructuring exist…