Speech Recognition and Synthesis

Neueste Paper

• Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generation

  Roy Fejgin, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Ryan Langman, Jaehyeon Kim, Subhankar Ghosh, Shehzeen Hussain, Jason Li · 26. Januar 2026

  Speech generation models based on large language models (LLMs) typically operate on discrete acoustic codes, which differ fundamentally from text tokens due to their multicodebook structure. At each timestep, models must predict N codebook entries jointly, introducing dependencies that challenge sim…

• Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities

  Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini, Helmer Strik · 26. Januar 2026

  An accurate assessment of L2 English pronunciation is crucial for language learning, as it provides personalized feedback and ensures a fair evaluation of individual progress. However, automated scoring remains challenging due to the complexity of sentence-level fluency, prosody, and completeness. T…

• Domain-Incremental Continual Learning for Robust and Efficient Keyword Spotting in Resource Constrained Systems

  Prakash Dhungana, Sayed Ahmad Salehi · 23. Januar 2026

  Keyword Spotting (KWS) systems with small footprint models deployed on edge devices face significant accuracy and robustness challenges due to domain shifts caused by varying noise and recording conditions. To address this, we propose a comprehensive framework for continual learning designed to adap…

• Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition

  Yujian Ma, Xikun Lu, Jinqiu Sang, Xianquan Jiang, Ruizhe Li · 23. Januar 2026

  Large pre-trained speech models such as Whisper offer strong generalization but pose significant challenges for resource-efficient adaptation. Low-Rank Adaptation (LoRA) has become a popular parameter-efficient fine-tuning method, yet its underlying mechanisms in speech tasks remain poorly understoo…

• WavLink: Compact Audio-Text Embeddings with a Global Whisper Token

  Gokul Karthik Kumar, Ludovick Lepauloux, Hakim Hacid · 23. Januar 2026

  Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contrast, audio-text embedding models like CLAP-based models have largely relied on a…

• E-BATS: Efficient Backpropagation-Free Test-Time Adaptation for Speech Foundation Models

  Jiaheng Dong, Hong Jia, Soumyajit Chatterjee, Abhirup Ghosh, James Bailey, Ting Dang · 22. Januar 2026

  Speech Foundation Models encounter significant performance degradation when deployed in real-world scenarios involving acoustic domain shifts, such as background noise and speaker accents. Test-time adaptation (TTA) has recently emerged as a viable strategy to address such domain shifts at inference…

• Mitigating Data Imbalance in Automated Speaking Assessment

  Fong-Chun Tsai, Kuan-Tang Huang, Bi-Cheng Yan, Tien-Hong Lo, Berlin Chen · 22. Januar 2026

  Automated Speaking Assessment (ASA) plays a crucial role in evaluating second-language (L2) learners proficiency. However, ASA models often suffer from class imbalance, leading to biased predictions. To address this, we introduce a novel objective for training ASA models, dubbed the Balancing Logit …

• Prosody-Guided Harmonic Attention for Phase-Coherent Neural Vocoding in the Complex Spectrum

  Mohammed Salah Al-Radhi, Riad Larbi, M\'aty\'as Bartalis, G\'eza N\'emeth · 22. Januar 2026

  Neural vocoders are central to speech synthesis; despite their success, most still suffer from limited prosody modeling and inaccurate phase reconstruction. We propose a vocoder that introduces prosody-guided harmonic attention to enhance voiced segment encoding and directly predicts complex spectra…

• WavLink: Compact Audio--Text Embeddings with a Global Whisper Token

  Gokul Karthik Kumar, Ludovick Lepauloux, Hakim Hacid · 22. Januar 2026

  Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contrast, audio-text embedding models like CLAP-based models have largely relied on a…

• Super Monotonic Alignment Search

  Junhyeok Lee, Hyeongju Kim · 21. Januar 2026

  Monotonic alignment search (MAS), introduced by Glow-TTS, is one of the most popular algorithm in text-to-speech to estimate unknown alignments between text and speech. Since this algorithm needs to search for the most probable alignment with dynamic programming by caching all possible paths, the ti…

• SSVD-O: Parameter-Efficient Fine-Tuning with Structured SVD for Speech Recognition

  Pu Wang, Shinji Watanabe, Hugo Van hamme · 21. Januar 2026

  Parameter-efficient fine-tuning (PEFT) is a scalable approach for adapting large speech foundation models to new domains. While methods such as LoRA and its state-of-the-art variants reduce adaptation costs, they typically allocate parameters uniformly across model subspaces, which limits their effi…

• AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering

  Chun-Yi Kuan, Hung-yi Lee · 21. Januar 2026

  Recent advances in audio-aware large language models have shown strong performance on audio question answering. However, existing benchmarks mainly cover answerable questions and overlook the challenge of unanswerable ones, where no reliable answer can be inferred from the audio. Such cases are comm…

• Improving the Speaker Anonymization Evaluation's Robustness to Target Speakers with Adversarial Learning

  Carlos Franzreb, Arnab Das, Tim Polzehl, Sebastian M\"oller · 21. Januar 2026

  The current privacy evaluation for speaker anonymization often overestimates privacy when a same-gender target selection algorithm (TSA) is used, although this TSA leaks the speaker's gender and should hence be more vulnerable. We hypothesize that this occurs because the evaluation does not account …

• MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting

  Youngmoon Jung, Myunghun Jung, Joon-Young Yang, Yong-Hyeok Lee, Jaeyoung Roh, Hoon-Young Cho · 21. Januar 2026

  Open-vocabulary keyword spotting (KWS) with text-based enrollment has emerged as a flexible alternative to fixed-phrase triggers. Prior utterance-level matching methods, from an embedding-learning standpoint, learn embeddings at a single fixed dimensionality. We depart from this design and propose M…

• Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks

  Shih-Heng Wang, Jiatong Shi, Jinchuan Tian, Haibin Wu, Shinji Watanabe · 21. Januar 2026

  This paper investigates three crucial yet underexplored aspects of the generalization capabilities of neural audio codecs (NACs): (i) whether NACs can generalize to unseen languages during pre-training, (ii) whether speech-only pre-trained NACs can effectively generalize to non-speech applications s…

• Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens

  Ismail Rasim Ulgen, Zongyang Du, Junchen Lu, Philipp Koehn, Berrak Sisman · 21. Januar 2026

  Objective evaluation of synthesized speech is critical for advancing speech generation systems, yet existing metrics for intelligibility and prosody remain limited in scope and weakly correlated with human perception. Word Error Rate (WER) provides only a coarse text-based measure of intelligibility…

• Harmonizing the Arabic Audio Space with Data Scheduling

  Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury · 21. Januar 2026

  Audio large language models (LLMs) enable unified speech understanding and generation, yet their adaptation to linguistically complex, dialect-rich settings remains underexplored. This paper presents the first systematic study of multi-task instruction tuning for an Arabic-centric audio LLM, coverin…

• Performance and Complexity Trade-off Optimization of Speech Models During Training

  Esteban G\'omez, Tom B\"ackstr\"om · 21. Januar 2026

  In speech machine learning, neural network models are typically designed by choosing an architecture with fixed layer sizes and structure. These models are then trained to maximize performance on metrics aligned with the task's objective. While the overall architecture is usually guided by prior kno…

• Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models

  Nikita Kuzmin, Songting Liu, Kong Aik Lee, Eng Siong Chng · 21. Januar 2026

  Protecting speaker identity is crucial for online voice applications, yet streaming speaker anonymization (SA) remains underexplored. Recent research has demonstrated that neural audio codec (NAC) provides superior speaker feature disentanglement and linguistic fidelity. NAC can also be used with ca…

• DAME: Duration-Aware Matryoshka Embedding for Duration-Robust Speaker Verification

  Youngmoon Jung, Joon-Young Yang, Ju-ho Kim, Jaeyoung Roh, Chang Woo Han, Hoon-Young Cho · 21. Januar 2026

  Short-utterance speaker verification remains challenging due to limited speaker-discriminative cues in short speech segments. While existing methods focus on enhancing speaker encoders, the embedding learning strategy still forces a single fixed-dimensional representation reused for utterances of an…

• MOSS Transcribe Diarize Technical Report

  MOSI. AI, :, Donghua Yu, Zhengyuan Lin, Chen Yang, Yiyang Zhang, Hanfu Chen, Jingqi Chen, Ke Chen, Liwei Fan, Yi Jiang, Jie Zhu, Muchen Li, Wenxuan Wang, Yang Wang, Zhe Xu, Yitian Gong, Yuqian Zhang, Wenbo Zhang, Songlin Wang, Zhiyu Wu, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu · 21. Januar 2026

  Speaker-Attributed, Time-Stamped Transcription (SATS) aims to transcribe what is said and to precisely determine the timing of each speaker, which is particularly valuable for meeting transcription. Existing SATS systems rarely adopt an end-to-end formulation and are further constrained by limited c…

• DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

  Hanlin Zhang, Daxin Tan, Dehua Tao, Xiao Chen, Haochen Tan, Yunhe Li, Yuchen Cao, Jianping Wang, Linqi Song · 19. Januar 2026

  Speech tokenizers serve as the cornerstone of discrete Speech Large Language Models (Speech LLMs). Existing tokenizers either prioritize semantic encoding, fuse semantic content with acoustic style inseparably, or achieve incomplete semantic-acoustic disentanglement. To achieve better disentanglemen…

• Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers

  Runyuan Cai, Yu Lin, Yiming Wang, Chunlin Fu, Xiaodong Zeng · 19. Januar 2026

  Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-…

• Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation

  Yibo Wang, Tiansheng Huang, Li Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, Dacheng Tao · 19. Januar 2026

  Harmful fine-tuning attack introduces significant security risks to the fine-tuning services. Main-stream defenses aim to vaccinate the model such that the later harmful fine-tuning attack is less effective. However, our evaluation results show that such defenses are fragile--with a few fine-tuning …

• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study

  Xiaoran Fan, Zhichao Sun, Yangfan Gao, Jingfei Xiong, Hang Yan, Yifei Cao, Jiajun Sun, Shuo Li, Zhihao Zhang, Zhiheng Xi, Yuhao Zhou, Senjie Jin, Changhao Jiang, Junjie Ye, Ming Zhang, Rui Zheng, Zhenhua Han, Yunke Zhang, Demei Yan, Shaokang Dong, Tao Ji, Tao Gui · 19. Januar 2026

  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation. However, challenges remain in achieving effective cross-modal alignment and high-quality speech generation. In this work, we systematically investigate the role of speech tokenizer desi…