AI models that understand and generate human speech are gaining in accuracy, but also in methods to evaluate this accuracy. This week, Speech Recognition and Synthesis dominate with 112 papers published over four weeks, compared to 63 four weeks earlier.
The emergence of the term operating characteristic curve (a tool that measures a model's ability to distinguish correct answers from incorrect ones) in 18 papers shows that researchers are no longer just counting errors. They are also analyzing how these errors are distributed across contexts - for example, a model that recognizes male voices better than female ones, or that stumbles on regional accents.
Three recent papers illustrate this trend:
- Partial AUC Maximization from Positive-unlabeled Data proposes a method to optimize these curves when unlabeled incorrect data is unavailable.
- Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning uses these curves to evaluate targeted knowledge removal in a model without altering its other performances.
- RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications extends the tool to systems that must explain their choices, such as a medical chatbot justifying a diagnosis.
In the same domain, two other recent papers focus on the robustness of models when faced with languages or situations underrepresented in training data:
- How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality tests audio codecs on African dialects, where models lose up to 30% of their accuracy compared to English.
- Code-Switching Spoken Language Identification as Multi-Label Set Prediction addresses the mixing of languages within the same sentence, such as Spanish and English in a conversation.