Critical issues presentations/WikiLyzer: Visual analytics for automatic quality assessment of user-generated content on the English Wikipedia

From Wikimania 2016 • Esino Lario, Italy
Submission no. 146
Title of the submission

WikiLyzer: Visual analytics for automatic quality assessment of user-generated content on the English Wikipedia

Author of the submission
  • David Strohmaier
  • Cecilia di Sciascio
  • Eduardo Veas
Country of origin

Austria; Austria; Austria

Topics

Projects, Research, Technical

Keywords
  • Wikipedia
  • Visual Analytics
  • Automatic Quality Assessment
  • User-Generated Content
  • WikiLyzer
  • Quality Analyzer
  • Quality Assisted Editor
Abstract

Wikipedia is a major source of information in the web. It consists of user-generated content and has about 12 million edits/contributions per month. One of the keys to its success -- the user-generated content -- is also a hindrance to its growth and quality: in the context of user-generated content, contributions can be of poor quality because everyone, even anonymous users, can participate. The criteria defined for high-quality articles by Wikipedia community (featured article criteria) is also based on community review. Yet, reviewing all contributions and identifying featured articles is a long-winded process. In 2014, 269000 new articles were created and only 602 peer-reviews were performed, therefore only 581 new featured article candidates were nominated. The amount of new featured articles in the year 2014 was 298. Thus, a lot of non-featured articles are yet to be reviewed, because the amount of data is simply too big to be reviewed with human power.

Related work has shown that it is possible to automatically measure the quality of Wikipedia articles, in order to detect articles that would likely to meet these high-quality standards. Yet, despite all these quality measures, it is difficult to identify what would improve an article.

Therefore, we have developed the WikiLyzer-toolkit (see [1]). It contains two web-based interactive graphic tools made for identifying potential featured articles and assisting authors to improve their quality:

i) The Quality Analyzer (QA) (see [2]). Leveraging state-of-the-art quality measure methods and machine learning approaches, it supports Wikipedians in finding high quality articles based on predefined keywords. Furthermore, it contains an advanced mode for creating new quality measure methods and comparing them with state-of-the-art ones.

ii) The Quality Assisted Editor (QAE) (see [3]) to view which parts of the article should be improved in order to reach a higher overall article quality.

Two experts on Wikipedia quality measures tried the Quality Analyzer. They stated that creating a new quality measure method is most of the time a trial-and-error process, and that the Quality Analyzer is a great tool, because it enables them to try interactively: after each change it provides feedback concerning the quality of the new method compared with the state-of-the-art, showing F1-score, Precision and Recall. Experts also assessed that the QA reduces workload, because they do not have to prepare their own datasets or develop interfaces to Wikipedia. Building on these results, we analyzed methods for finer classification of articles following Wikipedia guidelines (FA, A, GA, B+, B, C, Start, Stub). We tried to find potential featured articles (A-, GA-, B+-, B-class articles) by combining state-of-the-art quality measure methods and decision tree learning and obtained that it is possible to identify potential featured articles from a random dataset.

A user study with the Quality Assisted Editor, conducted with 24 participants, showed that the editor helps participants detect flaws in individual sections, as well as identify (potential) featured and non-featured articles. The subjective workload while performing with the Quality Assisted Editor was perceived significantly lower compared with a benchmark tool (see [4]). The Quality Assisted Editor was rated as an excellent tool in a System Usability Scale . Basing on our results we created a gadget (see [5]) that adds quality information to the currently viewed article. It colors the heading of the article in order to visualize the overall quality score. Furthermore, after each section-heading the gadget adds a quality table that contains a list of quality indicators, and thus suggestions on how to improve a specific part of the article.

[1]www.david-strohmaier.com/wikilyzer

[2]tools.wmflabs.org/quality-analyzer/qualityanalyzer.php

[3]http://tools.wmflabs.org/quality-assisted-editor/qualityassistededitor.php

[4]https://en.wikipedia.org/wiki/Wikipedia:Metadata_gadget

[5]https://en.wikipedia.org/wiki/Wikipedia:Gadget

Result

Not accepted

Retrieved from "https://wikimania2016.wikimedia.org/w/index.php?title=Critical_issues_presentations/WikiLyzer:_Visual_analytics_for_automatic_quality_assessment_of_user-generated_content_on_the_English_Wikipedia&oldid=22741"