TU Berlin

Quality and Usability LabReviewed Conference Papers

Page Content

to Navigation

Reviewed Conference Papers

go back to overview

Crowdsourcing versus the laboratory: towards crowd-based linguistic text quality assessment of query-based extractive summarization
Citation key iskender2020a
Author Iskender, Neslihan and Polzehl, Tim and Möller, Sebastian
Title of Book Proceedings of the Conference on Digital Curation Technologies (Qurator 2020)
Pages 1–16
Year 2020
Address Berlin, Germany
Month jan
Note online
Publisher CEUR
How Published Fullpaper
Abstract Curating text manually in order to improve the quality of automatic natural language processing tools can become very time consuming and expensive. Especially, in the case of query-based extractive online forum summarization, curating complex information spread along multiple posts from multiple forum members to create a short meta-summary that answers a given query is a very challenging task. To overcome this challenge, we explore the applicability of microtask crowdsourcing as a fast and cheap alternative for query-based extractive text summarization of online forum discussions. We measure the linguistic quality of crowd-based forum summarizations, which is usually conducted in a traditional laboratory environment with the help of experts, via comparative crowdsourcing and laboratory experiments. To our knowledge, no other study considered query-based extractive text summarization and summary quality evaluation as an application area of the microtask crowdsourcing. By conducting experiments both in crowdsourcing and laboratory environments, and comparing the results of linguistic quality judgments, we found out that microtask crowdsourcing shows high applicability for determining the factors overall quality, grammaticality, non-redundancy, referential clarity, focus, and structure & coherence. Further, our comparison of these findings with a preliminary and initial set of expert annotations suggest that the crowd assessments can reach comparable results to experts specifically when determining factors such as overall quality and structure & coherence mean values. Eventually, preliminary analyses reveal a high correlation between the crowd and expert ratings when assessing low-quality summaries.
Link to publication Link to original publication Download Bibtex entry

go back to overview


Quick Access

Schnellnavigation zur Seite über Nummerneingabe