TU Berlin

Quality and Usability LabReviewed Conference Papers

Inhalt des Dokuments

zur Navigation

Reviewed Conference Papers

go back to overview

Crowdsourcing versus the laboratory: towards crowd-based linguistic text quality assessment of query-based extractive summarization
Zitatschlüssel iskender2020a
Autor Iskender, Neslihan and Polzehl, Tim and Möller, Sebastian
Buchtitel Proceedings of the Conference on Digital Curation Technologies (Qurator 2020)
Seiten 1–16
Jahr 2020
Adresse Berlin, Germany
Monat jan
Notiz online
Verlag CEUR
Wie herausgegeben Fullpaper
Zusammenfassung Curating text manually in order to improve the quality of automatic natural language processing tools can become very time consuming and expensive. Especially, in the case of query-based extractive online forum summarization, curating complex information spread along multiple posts from multiple forum members to create a short meta-summary that answers a given query is a very challenging task. To overcome this challenge, we explore the applicability of microtask crowdsourcing as a fast and cheap alternative for query-based extractive text summarization of online forum discussions. We measure the linguistic quality of crowd-based forum summarizations, which is usually conducted in a traditional laboratory environment with the help of experts, via comparative crowdsourcing and laboratory experiments. To our knowledge, no other study considered query-based extractive text summarization and summary quality evaluation as an application area of the microtask crowdsourcing. By conducting experiments both in crowdsourcing and laboratory environments, and comparing the results of linguistic quality judgments, we found out that microtask crowdsourcing shows high applicability for determining the factors overall quality, grammaticality, non-redundancy, referential clarity, focus, and structure & coherence. Further, our comparison of these findings with a preliminary and initial set of expert annotations suggest that the crowd assessments can reach comparable results to experts specifically when determining factors such as overall quality and structure & coherence mean values. Eventually, preliminary analyses reveal a high correlation between the crowd and expert ratings when assessing low-quality summaries.
Link zur Publikation Link zur Originalpublikation Download Bibtex Eintrag

go back to overview



Schnellnavigation zur Seite über Nummerneingabe