direkt zum Inhalt springen

direkt zum Hauptnavigationsmenü

Sie sind hier

TU Berlin

Page Content

Subject-based Usability Evaluation Methods - Publications

go back to project page

Evaluating Spoken Dialogue Systems According to De-Facto Standards: A Case Study
Citation key moeller2007k
Author Möller, Sebastian and Smeele, Paula and Boland, Heleen and Krebber, Jan
Pages 26–53
Year 2007
ISSN 0885-2308
Address Berlin, Germany
Journal Computer Speech and Language
Volume 21
Month nov
Abstract In the present paper, we investigate the validity and reliability of de-facto evaluation standards, defined for measuring or predicting the quality of the interaction with spoken dialogue systems. Two experiments have been carried out with a dialogue system for controlling domestic devices. During these experiments, subjective judgments of quality have been collected by two questionnaire methods (ITU-T Rec. P.851 and SASSI), and parameters describing the interaction have been logged and annotated. Both metrics served the derivation of prediction models according to the PARADISE approach. Although the limited database allows only tentative conclusions to be drawn, the results suggest that both questionnaire methods provide valid measurements of a large number of different quality aspects; most of the perceptive dimensions underlying the subjective judgments can also be measured with a high reliability. The extracted parameters mainly describe quality aspects which are directly linked to the system, environmental and task characteristics. Used as an input to prediction models, the parameters provide helpful information for system design and optimization, but not general predictions of system usability and acceptability.
Link to publication Download Bibtex entry

go back to project page

Zusatzinformationen / Extras

Quick Access:

Schnellnavigation zur Seite über Nummerneingabe

Auxiliary Functions