The Cropland Capture Game: Good Annotators Versus Vote Aggregation Methods

  • Artem Baklanov
  • Steffen Fritz
  • Michael Khachay
  • Oleg Nurmukhametov
  • Linda See
Conference paper

DOI: 10.1007/978-3-319-38884-7_13

Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 453)
Cite this paper as:
Baklanov A., Fritz S., Khachay M., Nurmukhametov O., See L. (2016) The Cropland Capture Game: Good Annotators Versus Vote Aggregation Methods. In: Nguyen T., van Do T., An Le Thi H., Nguyen N. (eds) Advanced Computational Methods for Knowledge Engineering. Advances in Intelligent Systems and Computing, vol 453. Springer, Cham

Abstract

The Cropland Capture game, which is a recently developed Geo-Wiki game, aims to map cultivated lands using around 17,000 satellite images from the Earth’s surface. Using a perceptual hash and blur detection algorithm, we improve the quality of the Cropland Capture game’s dataset. We then benchmark state-of-the-art algorithms for an aggregation of votes using results of well-known machine learning algorithms as a baseline. We demonstrate that volunteer-image assignment is highly irregular and only good annotators are presented (there are no spammers and malicious voters). We conjecture that the last fact is the main reason for surprisingly similar accuracy levels across all examined algorithms. Finally, we increase the estimated consistency with expert opinion from 77 to 91 % and up to 96 % if we restrict our attention to images with more than 9 votes.

Keywords

Crowdsourcing Image processing Votes aggregation 

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Artem Baklanov
    • 1
    • 2
    • 3
  • Steffen Fritz
    • 1
  • Michael Khachay
    • 2
    • 3
  • Oleg Nurmukhametov
    • 2
  • Linda See
    • 1
  1. 1.International Institute for Applied Systems Analysis (IIASA)LaxenburgAustria
  2. 2.Krasovsky Institute of Mathematics and MechanicsEkaterinburgRussia
  3. 3.Ural Federal UniversityEkaterinburgRussia

Personalised recommendations