direkt zum Inhalt springen

direkt zum Hauptnavigationsmenü

Sie sind hier

TU Berlin

Page Content

Reviewed Conference Papers

go back to overview

Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach
Citation key naderi2020a
Author Naderi, Babak and Hoßfeld, Tobias and Hirth, Matthias and Metzger, Florian and Möller, Sebastian and Zequeira Jiménez, Rafael
Title of Book 12th International Conference on Quality of Multimedia Experience (QoMEX)
Pages 1–6
Year 2020
ISBN 978-1-7281-5965-2
Month may
Publisher IEEE
Series QoMEX
How Published Fullpaper
Abstract The subjective quality of transmitted speech is traditionally assessed in a controlled laboratory environment according to ITU-T Rec. P.800. In turn, with crowdsourcing, crowdworkers participate in a subjective online experiment using their own listening device, and in their own working environment. Despite such less controllable conditions, the increased use of crowdsourcing micro-task platforms for quality assessment tasks has pushed a high demand for standardized methods, resulting in ITU-T Rec. P.808. This work investigates the impact of the number of judgments on the reliability and the validity of quality ratings collected through crowdsourcing-based speech quality assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments on different platforms were conducted to evaluate the overall quality of three different speech datasets, using the Absolute Category Rating procedure. For each dataset, the Mean Opinion Scores (MOS) are calculated using differing numbers of crowdsourcing judgements. Then the results are compared to MOS values collected in a standard laboratory experiment, to assess the validity of crowdsourcing approach as a function of number of votes. In addition, the reliability of the average scores is analyzed by checking inter-rater reliability, gain in certainty, and the confidence of the MOS. The results provide a suggestion on the required number of votes per condition, and allow to model its impact on validity and reliability.
Link to publication Download Bibtex entry

go back to overview

Zusatzinformationen / Extras

Quick Access:

Schnellnavigation zur Seite über Nummerneingabe