Multimedia Tools and Applications

, Volume 75, Issue 23, pp 15901–15928 | Cite as

Assessment of crowdsourcing and gamification loss in user-assisted object segmentation

  • Axel Carlier
  • Amaia Salvador
  • Ferran Cabezas
  • Xavier Giro-i-Nieto
  • Vincent Charvillat
  • Oge Marques


There has been a growing interest in applying human computation – particularly crowdsourcing techniques – to assist in the solution of multimedia, image processing, and computer vision problems which are still too difficult to solve using fully automatic algorithms, and yet relatively easy for humans. In this paper we focus on a specific problem – object segmentation within color images – and compare different solutions which combine color image segmentation algorithms with human efforts, either in the form of an explicit interactive segmentation task or through an implicit collection of valuable human traces with a game. We use Click’n’Cut, a friendly, web-based, interactive segmentation tool that allows segmentation tasks to be assigned to many users, and Ask’nSeek, a game with a purpose designed for object detection and segmentation. The two main contributions of this paper are: (i) We use the results of Click’n’Cut campaigns with different groups of users to examine and quantify the crowdsourcing loss incurred when an interactive segmentation task is assigned to paid crowd-workers, comparing their results to the ones obtained when computer vision experts are asked to perform the same tasks. (ii) Since interactive segmentation tasks are inherently tedious and prone to fatigue, we compare the quality of the results obtained with Click’n’Cut with the ones obtained using a (fun, interactive, and potentially less tedious) game designed for the same purpose. We call this contribution the assessment of the gamification loss, since it refers to how much quality of segmentation results may be lost when we switch to a game-based approach to the same task. We demonstrate that the crowdsourcing loss is significant when using all the data points from workers, but decreases substantially (and becomes comparable to the quality of expert users performing similar tasks) after performing a modest amount of data analysis and filtering out of users whose data are clearly not useful. We also show that – on the other hand – the gamification loss is significantly more severe: the quality of the results drops roughly by half when switching from a focused (yet tedious) task to a more fun and relaxed game environment.


GWAP Crowdsourcing Serious games Object detection Object segmentation 



This work has been developed in the framework of the project TEC2013-43935-R, financed by the Spanish Ministerio de Economa y Competitividad and the European Regional Development Fund (ERDF).


  1. 1.
    Adamek T (2006) Using contour information and segmentation for object registration, modeling and retrieval. Ph.D. dissertation, Dublin City UniversityGoogle Scholar
  2. 2.
    Arbelaez P, Cohen L (2008) Constrained image segmentation from hierarchical boundaries. In: CVPRGoogle Scholar
  3. 3.
    Arbeláez P, Pont-Tuset J, Barron JT, Marques F, Malik J (2014) Multiscale combinatorial grouping. In: CVPRGoogle Scholar
  4. 4.
    Batra D, Kowdle A, Parikh D, Luo J, Chen T (2010) icoseg: Interactive co-segmentation with intelligent scribble guidance. In: Proceedings of CVPR’10, pp 3169–3176Google Scholar
  5. 5.
    Bell S, Upchurch P, Snavely N, Bala K (2013) Opensurfaces: A richly annotated catalog of surface appearance. ACM TOG 32(4)Google Scholar
  6. 6.
    Boykov Y, Jolly M-P (2001) Interactive graph cuts for optimal boundary map; region segmentation of objects in n-d images. In: ICCVGoogle Scholar
  7. 7.
    Cabezas F, Carlier A, Salvador A, Giró-i Nieto X, Charvillat V (2015) Quality control in crowdsourced object segmentation. arXiv:1505.00145
  8. 8.
    Carlier A, Marques O, Charvillat V (2012) Ask’nseek: A new game for object detection and labeling. In: Computer Vision–ECCV 2012. Workshops and Demonstrations. Springer, pp 249–258Google Scholar
  9. 9.
    Carlier A, Charvillat V, Salvador A, Giro-i Nieto X, Marques O (2014) Click’n’cut: Crowdsourced interactive segmentation with object candidates. In: Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia, ser. CrowdMM ’14. New York, NY, USA: ACM, pp 53–56. [Online]. Available: doi: 10.1145/2660114.2660125
  10. 10.
    Carreira J, Sminchisescu C (2010) Constrained parametric min-cuts for automatic object segmentation. In: CVPRGoogle Scholar
  11. 11.
    Chen L-C, Fidler S, Yuille AL, Urtasun R (2014) Beat the mturkers: Automatic image labeling from weak 3d supervision. In: CVPRGoogle Scholar
  12. 12.
    Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338CrossRefGoogle Scholar
  13. 13.
    Fathi A, Balcan MF, Ren X, Rehg JM (2011) Combining self training and active learning for video segmentation. In: Hoey J, McKenna S, Trucco E (eds) Proceedings of the British Machine Vision Conference (BMVC 2011), vol 29, pp 78–1Google Scholar
  14. 14.
    Giró-i Nieto X, Martos M, Mohedano E, Pont-Tuset J (2014) From global image annotation to interactive object segmentation. MTAP 70(1)Google Scholar
  15. 15.
    Jain SD, Grauman K (2013) Predicting sufficient annotation strength for interactive foreground segmentation. In: Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, pp 1313–1320Google Scholar
  16. 16.
    Lee HS, Kim J, Park SJ, Kim J (2014) Interactive segmentation as supervised classification with superpixels. In: WCVPR 2014-W. on Computer Vision and Human ComputationGoogle Scholar
  17. 17.
    Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. CoRRGoogle Scholar
  18. 18.
    Liu T, Yuan Z, Sun J, Wang J, Zheng N, Tang X, Shum H-Y (2011) Learning to detect a salient object. PAMI 33(2)Google Scholar
  19. 19.
    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3431–3440Google Scholar
  20. 20.
    Lux M, Müller A, Guggenberger M (2012) Finding image regions with human computation and games with a purpose. In: AIIDEGoogle Scholar
  21. 21.
    Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCVGoogle Scholar
  22. 22.
    McGuinness K, O’Connor N (2013) Improved graph cut segmentation by learning a contrast model on the fly. In: ICIPGoogle Scholar
  23. 23.
    McGuinness K, O’connor NE (2010) A comparative evaluation of interactive segmentation algorithms. Pattern Recogn 43(2):434–444CrossRefzbMATHGoogle Scholar
  24. 24.
    Noma A, Graciano ABV, Cesar Jr RM, Consularo LA, Bloch I (2012) Interactive image segmentation by matching attributed relational graphs. Pattern Recogn 45(3)Google Scholar
  25. 25.
    Oleson D, Sorokin A, Laughlin GP, Hester V, Le J, Biewald L (2011) Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human Computation 11:11Google Scholar
  26. 26.
    Pinheiro PO, Collobert R (2015) From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1713–1721Google Scholar
  27. 27.
    Rother C, Kolmogorov V, Blake A (2004) “grabcut”: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3)Google Scholar
  28. 28.
    Rupprecht C, Peter L, Navab N (2015) Image segmentation in twenty questions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3314–3322Google Scholar
  29. 29.
    Russakovsky O, Bearman AL, Ferrari V, Li F-F (2015) What’s the point: Semantic segmentation with point supervision. arXiv:1506.02106
  30. 30.
    Russell BC, Torralba A, Murphy KP, Freeman TW (2008) Labelme: A database and web-based tool for image annotation. IJCV 77(1-3)Google Scholar
  31. 31.
    Salembier P, Garrido L (2000) Binary partition tree as an efficient representation for image processing, segmentation, and information retrieval. IEEE Trans Image Process 9(4)Google Scholar
  32. 32.
    Salvador A, Carlier A, Giro-i Nieto X, Marques O, Charvillat V (2013) Crowdsourced object segmentation with a game. In: ACM CrowdMMGoogle Scholar
  33. 33.
    Steggink J, Snoek C (2011) Adding semantics to image-region annotations with the name-it-game. Multimedia Systems:17Google Scholar
  34. 34.
    Sun Y, CHen Y, Wang W, Tang X (2014) Deep learning face representation by joint identification-verification. In: Proceedings of Neural Information Processing Systems Conference (NIPS)Google Scholar
  35. 35.
    von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: ACM CHIGoogle Scholar
  36. 36.
    von Ahn L, Liu R, Blum M (2006) Peekaboom: a game for locating objects in images. In: ACM CHIGoogle Scholar
  37. 37.
    Wang J, Bhat P, Colburn RA, Agrawala M, Cohen MF (2005) Interactive video cutout. ACM Trans. Graph. 24(3)Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Axel Carlier
    • 1
  • Amaia Salvador
    • 2
  • Ferran Cabezas
    • 2
  • Xavier Giro-i-Nieto
    • 2
  • Vincent Charvillat
    • 1
  • Oge Marques
    • 3
  1. 1.IRIT-ENSEEIHTUniversity of ToulouseToulouseFrance
  2. 2.Universitat Politecnica de Catalunya (UPC)BarcelonaSpain
  3. 3.Florida Atlantic University (FAU)Boca RatonUSA

Personalised recommendations