Skip to main content

Efficient Crowdsourcing for Semantic Segmentation Considering Human Cognitive Characteristics

  • 1241 Accesses

Part of the Communications in Computer and Information Science book series (CCIS,volume 1580)


Datasets of annotated images for the machine learning of semantic segmentation can be built using crowdsourcing; however the quality is not always at a certain level. To improve the quality of the task results, we propose a task design in which categories are grouped for annotation into several sets, the sets are distributed to workers, and the results are aggregated to acquire the annotated images. We expected that the optimal set size would be 4 to 8 categories, based on the cognitive psychological knowledge that the human working memory capacity is 4 to 8 items. The evaluation experiment demonstrated that reducing the number of categories assigned to a worker improves the quality, whereas it increases the number of workers who are engaged in the annotation of an image and elongates the total time. Owing to this trade-off between the quality and efficiency, 4 to 7 categories for a worker is suggested to be optimal in terms of the quality and efficiency.


  • Crowdsourcing
  • Annotation
  • Working memory

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. Berggren, N., Eimer, M.: Attentional access to multiple target objects in visual search. J. Cogn. Neurosci. 32(2), 283–300 (2020)

    CrossRef  Google Scholar 

  2. Brener, R.: An experimental investigation of memory span. J. Exp. Psychol. 26(5), 467 (1940)

    CrossRef  Google Scholar 

  3. Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213–3223. IEEE Computer Society, Los Alamitos (2016)

    Google Scholar 

  4. Lin, D., Dai, J., Jia, J., He, K., Sun, J.: Scribblesup: scribble-supervised convolutional networks for semantic segmentation (2016)

    Google Scholar 

  5. Lin, H., Upchurch, P., Bala, K.: Block annotation: better image annotation with sub-image decomposition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5290–5300 (2019)

    Google Scholar 

  6. Luck, S.J., Vogel, E.K.: Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn. Sci. 17(8), 391–400 (2013)

    CrossRef  Google Scholar 

  7. Neisser, U.: Cognitive Psychology. Century Psychology Series Prentice-Hall, Appleton-Century-Crofts (1967)

    Google Scholar 

  8. Repovš, G., Baddeley, A.: The multi-component model of working memory: explorations in experimental cognitive psychology. Neuroscience 139(1), 5–21 (2006)

    CrossRef  Google Scholar 

  9. Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cogn. Psychol. 12(1), 97–136 (1980)

    CrossRef  Google Scholar 

  10. Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5122–5130 (2017)

    Google Scholar 

Download references


This work was supported by JST CREST Grant Number JPMJCR16E3 including AIP challenge, Japan and JSPS KAKENHI Grant Number JP21K12602.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Masaki Kobayashi .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kobayashi, M., Morita, H., Morishima, A. (2022). Efficient Crowdsourcing for Semantic Segmentation Considering Human Cognitive Characteristics. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2022 Posters. HCII 2022. Communications in Computer and Information Science, vol 1580. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06416-6

  • Online ISBN: 978-3-031-06417-3

  • eBook Packages: Computer ScienceComputer Science (R0)