TU Berlin

Quality and Usability LabReviewed Conference Papers

Inhalt des Dokuments

zur Navigation

Reviewed Conference Papers

go back to overview

Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach
Zitatschlüssel naderi2020a
Autor Naderi, Babak and Hoßfeld, Tobias and Hirth, Matthias and Metzger, Florian and Möller, Sebastian and Zequeira Jiménez, Rafael
Buchtitel 12th International Conference on Quality of Multimedia Experience (QoMEX)
Seiten 1–6
Jahr 2020
ISBN 978-1-7281-5965-2
Monat may
Verlag IEEE
Serie QoMEX
Wie herausgegeben Fullpaper
Zusammenfassung The subjective quality of transmitted speech is traditionally assessed in a controlled laboratory environment according to ITU-T Rec. P.800. In turn, with crowdsourcing, crowdworkers participate in a subjective online experiment using their own listening device, and in their own working environment. Despite such less controllable conditions, the increased use of crowdsourcing micro-task platforms for quality assessment tasks has pushed a high demand for standardized methods, resulting in ITU-T Rec. P.808. This work investigates the impact of the number of judgments on the reliability and the validity of quality ratings collected through crowdsourcing-based speech quality assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments on different platforms were conducted to evaluate the overall quality of three different speech datasets, using the Absolute Category Rating procedure. For each dataset, the Mean Opinion Scores (MOS) are calculated using differing numbers of crowdsourcing judgements. Then the results are compared to MOS values collected in a standard laboratory experiment, to assess the validity of crowdsourcing approach as a function of number of votes. In addition, the reliability of the average scores is analyzed by checking inter-rater reliability, gain in certainty, and the confidence of the MOS. The results provide a suggestion on the required number of votes per condition, and allow to model its impact on validity and reliability.
Link zur Publikation Download Bibtex Eintrag

go back to overview



Schnellnavigation zur Seite über Nummerneingabe