The Cropland Capture Game: Good Annotators Versus Vote Aggregation Methods
- Cite this paper as:
- Baklanov A., Fritz S., Khachay M., Nurmukhametov O., See L. (2016) The Cropland Capture Game: Good Annotators Versus Vote Aggregation Methods. In: Nguyen T., van Do T., An Le Thi H., Nguyen N. (eds) Advanced Computational Methods for Knowledge Engineering. Advances in Intelligent Systems and Computing, vol 453. Springer, Cham
The Cropland Capture game, which is a recently developed Geo-Wiki game, aims to map cultivated lands using around 17,000 satellite images from the Earth’s surface. Using a perceptual hash and blur detection algorithm, we improve the quality of the Cropland Capture game’s dataset. We then benchmark state-of-the-art algorithms for an aggregation of votes using results of well-known machine learning algorithms as a baseline. We demonstrate that volunteer-image assignment is highly irregular and only good annotators are presented (there are no spammers and malicious voters). We conjecture that the last fact is the main reason for surprisingly similar accuracy levels across all examined algorithms. Finally, we increase the estimated consistency with expert opinion from 77 to 91 % and up to 96 % if we restrict our attention to images with more than 9 votes.