mmEVAL

Quality Assessment for Multimodal Human-Computer Interaction

• Multimodal dialogue systems aim to provide a more intuitive, natural and robust interaction between human users and computers than unimodal ones. However, despite an emerging interest in multimodal interfaces, little is known about the relevant properties determining the quality of such systems.
  • Different quality aspects have to be identified and classified, in order to establish reliable, valid and standardized evaluation methods and prediction models.
  • Appropriate evaluation methods for those quality aspects have to be developed and tested.
  • Effects of modality selection and combination as well as effects of the modality of the users’ input on recognition quality have to be investigated.
  • Experimental data has to be correlated to parameters describing multimodal dialogue systems and the interaction of users with such systems to be able to predict the usability and acceptability attributed to a system.

Cooperation

• We are associated with the Australian national research project “From Talking Heads to Thinking Heads [2]” (see also our local project page [3]), which main objective is to explore talking head user interfaces.
• mmEval is closely cooperating with “Automatic Usability Engineering Methods [4]” and “Subject-based Usability Evaluation Methods [5]”.

Outcome

• A taxonomy of quality aspects for multimodal dialogue systems.
• Insight into interrelationships of such quality aspects.
• Experimental results for multimodal user input and system outputin relation to overall subjective quality.
• Identifying influences of interaction, attention and system parameters on user ratings.
• A definition of a set of multimodal interaction parameters.
• A prediction model for evaluation results on the basis of dialogue and system parameters.