TU Berlin

Quality and Usability Lab2018_04_16_Das

Page Content

to Navigation

There is no English translation for this web page.

Modelling Multiple Annotator Ratings for Inferring Ground Truth in Labelled Speech Datasets

LOCATION:  TEL, Auditorium 3 (20th floor), Ernst-Reuter-Platz 7, 10587 Berlin

Date/Time: 16.04.2018, 14:15-15:00  

SPEAKER: Sarjo Das (TU Berlin)


Recorded speech corpora are often analyzed and labelled by a group of annotators in terms of speech characteristics and quality. Since speaker’s traits are subjective and each of the annotators has different levels of expertise and biases, the resultant annotations are not equally reliable. Therefore, an annotator aware probabilistic model is suggested based on existing studies in multi-annotator learning, that can take into account the annotators’ behavior to provide ground truth labels, which are reliable reflecting the collective wisdom of the annotators. Using supervised algorithms, the performance of the ground truth label obtained from the suggested probabilistic model was compared against other commonly used techniques to get a consensus label, like majority voting (ordinal/categorical labels) or finding median/mean (continuous labels). Unweighted recall and relative absolute error were used as the performance metrics for classification and regression. It was observed that the performance of the ground truth label-based model was almost similar or superior to the other commonly used techniques.





Quick Access

Schnellnavigation zur Seite über Nummerneingabe