Datasets of annotated images for the machine learning of semantic segmentation can be built using crowdsourcing; however the quality is not always at a certain level. To improve the quality of the task results, we propose a task design in which categories are grouped for annotation into several sets, the sets are distributed to workers, and the results are aggregated to acquire the annotated images. We expected that the optimal set size would be 4 to 8 categories, based on the cognitive psychological knowledge that the human working memory capacity is 4 to 8 items. The evaluation experiment demonstrated that reducing the number of categories assigned to a worker improves the quality, whereas it increases the number of workers who are engaged in the annotation of an image and elongates the total time. Owing to this trade-off between the quality and efficiency, 4 to 7 categories for a worker is suggested to be optimal in terms of the quality and efficiency.
- Working memory
This is a preview of subscription content, access via your institution.
Tax calculation will be finalised at checkout
Purchases are for personal use onlyLearn about institutional subscriptions
Berggren, N., Eimer, M.: Attentional access to multiple target objects in visual search. J. Cogn. Neurosci. 32(2), 283–300 (2020)
Brener, R.: An experimental investigation of memory span. J. Exp. Psychol. 26(5), 467 (1940)
Cordts, M., et al.: The cityscapes dataset for semantic urban scene understanding. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213–3223. IEEE Computer Society, Los Alamitos (2016)
Lin, D., Dai, J., Jia, J., He, K., Sun, J.: Scribblesup: scribble-supervised convolutional networks for semantic segmentation (2016)
Lin, H., Upchurch, P., Bala, K.: Block annotation: better image annotation with sub-image decomposition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5290–5300 (2019)
Luck, S.J., Vogel, E.K.: Visual working memory capacity: from psychophysics and neurobiology to individual differences. Trends Cogn. Sci. 17(8), 391–400 (2013)
Neisser, U.: Cognitive Psychology. Century Psychology Series Prentice-Hall, Appleton-Century-Crofts (1967)
Repovš, G., Baddeley, A.: The multi-component model of working memory: explorations in experimental cognitive psychology. Neuroscience 139(1), 5–21 (2006)
Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cogn. Psychol. 12(1), 97–136 (1980)
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5122–5130 (2017)
This work was supported by JST CREST Grant Number JPMJCR16E3 including AIP challenge, Japan and JSPS KAKENHI Grant Number JP21K12602.
Editors and Affiliations
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kobayashi, M., Morita, H., Morishima, A. (2022). Efficient Crowdsourcing for Semantic Segmentation Considering Human Cognitive Characteristics. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2022 Posters. HCII 2022. Communications in Computer and Information Science, vol 1580. Springer, Cham. https://doi.org/10.1007/978-3-031-06417-3_41
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-06416-6
Online ISBN: 978-3-031-06417-3