Speech Recognition and Synthesis

Latest papers

• UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling

  Chuanbo Zhu, Wuyou Zhou, Rongxiu Zhong, Shilei Zhang, Kun Qian, Yike Guo, Wei Xue · 1 July 2026

  Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion editing as separate tasks, limiting both editing granularity and flexibility…

• Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection

  Olivier Jiyoun Jung, Jonghyeon Park, Myungwoo Oh · 1 July 2026

  Early detection of dementia through speech analysis offers a non-invasive screening alternative, but capturing both acoustic and linguistic biomarkers remains challenging. We propose a multimodal framework leveraging Whisper for dual-purpose extraction: acoustic representations from encoder outputs …

• Sparsity-Inducing Divergence Losses for Biometric Verification

  Dimitrios Koutsianos, Ladislav Mo\v{s}ner, Yannis Panagakis, Themos Stafylakis · 1 July 2026

  Performance in face and speaker verification is largely driven by margin-penalty softmax losses such as CosFace and ArcFace. Recently introduced $\alpha$-divergence loss functions offer a compelling alternative, particularly due to their ability to induce sparse solutions (when $\alpha>1$). However,…

• Linguistic Bias Mitigation for Spoofing Detection via Gradient Reversal and A Variational Information Bottleneck

  Anh-Tuan Dao, Driss Matrouf, Mickael Rouvier, Nicholas Evans · 1 July 2026

  Rapid advancements in generative speech technology have compromised the reliability of voice biometrics. While current spoofing detectors excel when assessed under in-domain conditions, generalisation to out-of-domain settings is often poor. We show that this can be due to linguistic bias. A relianc…

• AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

  Chuxiao Zuo, Yao Zhu, Minqiang Xu, Manhong Wang, Yunke Zhang, Fei Huang · 30 June 2026

  Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions. In practical scenarios, background multi-speaker conversations, ambient noise, and overlapping speech further degrade identifi…

• OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL

  Karl El Hajal, Mathew Magimai. -Doss · 30 June 2026

  We propose Online Latent prediction with Invariant Views and rEconstruction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unifie…

• LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features

  Jonghyeon Park, Olivier Jiyoun Jung, Myungwoo Oh · 30 June 2026

  Early detection of dementia enables timely intervention, and reflecting cognitive impairment, spontaneous speech offers a non-invasive screening modality. Conventional approaches often focus on a single representational dimension -- such as acoustic descriptors, pause modeling, automatic speech reco…

• Geometrically Principled Randomized Optimization for Efficient LLM Training

  Sahar Rajabi, Nayeema Nonta, Sirisha Rambhatla · 30 June 2026

  Low-rank gradient optimization for large language models is currently divided into two categories: structured methods that rigorously identify subspaces, and randomized approaches employed primarily for computational efficiency. In this work, we question the intuition behind why random projections a…

• What Was That Again? Certified Robustness for Automatic Speech Recognition

  Andrew C. Cullen, Neil Marchant, Jiani Xie, Paul Montague, Benjamin I. P. Rubinstein · 29 June 2026

  Automatic Speech Recognition systems are notoriously both sensitive to adversarial and benign perturbations. While this has been repeatedly demonstrated using reference datasets, detecting such behaviors in deployed systems is incredibly challenging, due to the absence of oracle knowledge of the tru…

• DG^VoiC: Speaker Clustering for Fraud Investigation under Real Call-Centre Conditions

  Muhammad Shakeel Akram, Amal Htait, Abdul Hamid Sadka, Emma Meisingseth, Karishma Jaitly · 29 June 2026

  Insurance fraud remains costly and operationally difficult, particularly in call-centre workflows where many customer interactions begin at FNOL. While recent fraud detection methods mainly rely on structured data, text, or images, repeated speaker identity across calls remains underused as an inves…

• HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

  Artem Ploujnikov, Francesco Verdini, Samir Sadok, Mirco Ravanelli · 29 June 2026

  Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs). However, numerous studies report performance degradation on various downstream tasks due to information loss during discret…

• Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings

  Zahra Omidi, John H. L. Hansen · 29 June 2026

  The variations in vocal effort range (e.g. whisper, soft, neutral, loud, shout) alter production and speech acoustics, reducing intelligibility and limiting the robustness of any subsequent speech technology. Classification is challenging since effort lies on a continuum, adjacent categories are eas…

• Tuning Language Models by Mixture-of-Depths Ensemble

  Haoyan Luo, Lucia Specia · 26 June 2026

  Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for finetuning and final-layer representations for predictions, potentially overlooking the predictive power embedded in late layers. Interpretability tools such as the logit lens show that late-layer representatio…

• Neural Speaker Diarization via Multilingual Training: Evaluation on Low-Resource Nepali-Hindi Speech

  Samip Neupane, Sandesh Pokhrel, Sandesh Pyakurel, Basanta Joshi · 26 June 2026

  Speaker diarization, the task of determining "who spoke when" in a multi-speaker recording, is a critical component in applications such as meeting transcription, accessibility tools, and multilingual information retrieval. While end-to-end neural diarization systems have achieved strong performance…

• VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

  Tianxin Xie, Chenxing Li, Dong Yu, Li Liu · 26 June 2026

  Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.g., crosstalk, dialects). Moreover, fine-tuning pretrained models requires large, high-quality datasets, limiting rapid …

• Phoneme-Level Mispronunciation Screening in Polish-Speaking Children with an Explainable Assistant

  Milosz Dudek, Daria Hemmerling, Kamil Kwarciak, Maciej Stroinski, Maria Pensko, Mateusz Kowalewski, Leonid Pavlovskyi, Sebastian Jurczak, Anna-Mariia Vitkovska, Zuzanna Miodonska, Natalia Mocko, Michal Krecichwost · 25 June 2026

  Early identification of speech sound errors in children is often limited by access to specialists, motivating lightweight screening tools that can operate outside the clinic. We present a screening pipeline for Polish-speaking children focused on sibilant substitutions, coupling a wav2vec2-based CTC…

• STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity

  Sitong Cheng, Weizhen Bian, Songjun Cao, Jin Li, Bei Liu, Chunyang Jiang, Yike Zhang, Weihao Wu, Yiming Li, Chi-Min Chan, Long Ma, Wei Xue · 25 June 2026

  Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful an…

• Error-Aware TF-IDF Retrieval-Augmented Generation for ASR Error Correction

  Mohammad Aref Jafari-Raddani · 25 June 2026

  End-to-end automatic speech recognition systems frequently hallucinate rare entities and domain-specific terms, especially in low-resource languages. While retrieval-augmented generation frameworks can mitigate these errors using large language models, current architectures face significant challeng…

• CrossAccent-TTS: Cross-Lingual Accent-Intensity Controllable Text-to-Speech via Disentangled Speaker and Accent Representations

  Ram Annamdevula, Ankit Tatawat, Ashishkumar P. Gudmalwar, Nirmesh J. Shah, Pankaj Wasnik · 25 June 2026

  Accent conversion and controllability remain fundamental challenges in cross-lingual text-to-speech (TTS), particularly for low-resource and phonetically diverse Indic languages. While recent large language model (LLM)-based TTS systems exhibit strong cross-lingual generalization, they provide limit…

• ZONOS2 Technical Report

  Gabriel Clark, Sofian Mejjoute, Mohamed Osman, George Close, Beren Millidge · 24 June 2026

  We present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice cloning fidelity. We improve upon Zonos-v0.1 across scale, data, and training recipe. We scale the model from 1.6B to 8B total parameters (900M active) with a novel mixture-of-experts (MoE) ba…

• Data Scale, Not Latency, Shapes Cross-Lingual Encoder Transfer in Streaming ASR

  Nenad Banfic · 24 June 2026

  Adapting a streaming speech recognition model to a new language requires choosing between two plausible warm starts: a multilingual (ML) encoder or an English-only (EN) encoder. The common intuition is that the multilingual encoder should help most at low data, but it is unclear how long that advant…

• How Well Do Self-Supervised Speech Models Encode Age and Gender in Children's Speech? A Layer-Wise Analysis Across Multiple Architectures

  Abhijit Sinha, Hemant Kumar Kathania, Mohit Joshi, Harishankar Kumar, Shrikanth Narayanan, Sudarsana Reddy Kadiri · 23 June 2026

  Self-supervised learning (SSL) models have become a central component of modern speech processing systems, as they enable the learning of rich acoustic representations without reliance on labeled data. Despite their success on adult speech, it remains unclear how effectively these models capture spe…

• Bagpiper-TTS: Natural Language Guided Universal Speech Synthesis

  Jinchuan Tian, Haoran Wang, Siddhant Arora, Takashi Maekaku, Keita Goto, Jin Sakuma, Yusuke Shinohara, Chao-Han Huck Yang, Shinji Watanabe · 23 June 2026

  Classical TTS systems typically rely on rigid input formats and predefined metadata slots, limiting their ability to fulfill flexible user requirements. This paper introduces Bagpiper-TTS, a universal speech synthesis system that deals with diverse natural language user requests. Given a natural lan…

• Kiwano: A Cutting-Edge Open-Source Toolkit for Speaker Verification

  Mickael Rouvier, Pierre Michel Bousquet · 23 June 2026

  In this paper, we present Kiwano, an open-source toolkit designed to advance research and evaluation for speaker verification. Kiwano provides a lightweight yet extensible framework built on PyTorch, offering standardized recipes, pretrained models, and integration of several widely used speaker ver…

• BranchShine: Compact Raw-Audio-to-IPA Transcription with a RoPE E-Branchformer Encoder

  Nikhil Navas, Sergio Chevtchenko, Talisson Damiao, Saeed Afshar · 23 June 2026

  Speech-to-IPA transcription is useful when the desired output is pronunciation rather than orthographic text, but competitive multilingual systems are often large and evaluation is sensitive to normalization choices. This paper presents BranchShine, a 33M-parameter raw-audio CTC recognizer with a li…