TU Berlin

Quality and Usability LabReviewed Conference Papers

Inhalt des Dokuments

zur Navigation

Reviewed Conference Papers

go back to overview

Modeling Worker Performance Based on Intra-rater Reliability in Crowdsourcing : A Case Study of Speech Quality Assessment
Zitatschlüssel zequeirajimenez2019a
Autor Zequeira Jiménez, Rafael and Llagostera, Anna and Naderi, Babak and Möller, Sebastian and Berger, Jens
Buchtitel 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX)
Seiten 1–6
Jahr 2019
ISSN 2372-7179
DOI 10.1109/QoMEX.2019.8743148
Monat jun
Verlag IEEE
Serie QoMEX 2019
Wie herausgegeben Fullpaper
Zusammenfassung Crowdsourcing has become a convenient instrument for addressing subjective user studies to a large amounts of users. Data from crowdsourcing can be corrupted due to users' neglect, and different mechanisms has been proposed to address the users' reliability and to ensure valid experiments' results. Users that are consistent in their answers or present a high intra-rater reliability score, are desired for subjective studies. This work investigates the relationship between the intra-rater reliability and the user performance in the context of a speech quality assessment task. To this end, a crowdsourcing study has been conducted in which users were requested to rate speech stimuli with respect to their overall quality. Ratings were collected on a 5-point scale in accordance with the ITU-T Rec. P.808. The speech stimuli were taken from the database ITU-T Rec. P.501 Annex D, and the results are to be contrasted with ratings collected in a laboratory experiment. Furthermore, a model as a function of intra-rater reliability, root-mean-squared-deviation between the listeners ratings and age, has been built to predict the listener performance. Such a model is intended to provide a measure of how valid the crowdsourcing results are, when there is no laboratory results to compare to.
Download Bibtex Eintrag

go back to overview



Schnellnavigation zur Seite über Nummerneingabe