A Toolkit for Detecting Spurious Correlations in Speech Datasets
Lara Gauder, Pablo Riera, Andrea Slachevsky, Gonzalo Forno, Adolfo M. García + 1 more
TLDR
This toolkit identifies spurious correlations in speech datasets by analyzing non-speech audio, preventing performance overestimation.
Key contributions
- Introduces a toolkit to detect spurious correlations in speech datasets.
- Prevents overestimation of system performance, crucial for high-stakes applications.
- Employs a diagnostic method analyzing non-speech audio for target class detection.
Why it matters
Spurious correlations lead to inflated performance metrics, which is dangerous in critical applications like healthcare. This toolkit provides a vital diagnostic tool to ensure robust and reliable speech systems by identifying these hidden biases. Its public availability promotes better research practices.
Original Abstract
We introduce a toolkit for uncovering spurious correlations between recording characteristics and target class in speech datasets. Spurious correlations may arise due to heterogeneous recording conditions, a common scenario for health-related datasets. When present both in the training and test data, these correlations result in an overestimation of the system performance -- a dangerous situation, specially in high-stakes application where systems are required to satisfy minimum performance requirements. Our toolkit implements a diagnostic method based on the detection of the target class using only the non-speech regions in the audio. Better than chance performance at this task indicates that information about the target class can be extracted from the non-speech regions, flagging the presence of spurious correlations. The toolkit is publicly available for research use.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.