Quality Control for Crowdsourced Multi-label Classification Using RAkEL

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10634)


The quality of labels is one of the major issues in crowdsourced labeling tasks. A convenient method for ensuring the quality of labels is to assign the same labeling task to multiple workers and aggregate the labels. Several statistical aggregation methods for single-label classification tasks have been proposed; however, for multi-label classification tasks has not been well studied. Although the existing aggregation methods for single-label classification tasks can be applied to the multi-label classification tasks, they are not designed to incorporate relationships among classes, or they require large computation time. To address these issues, we propose to use RAndom k-labELsets (RAkEL). By incorporating an existing aggregation method for single-label classification tasks into RAkEL, we propose a novel quality control method for crowdsourced multi-label classification. We demonstrate that our method achieves better quality than the existing methods with real data especially when spammers are included in the worker pool.


Crowdsourcing Multi-label classification RAkEL 



We thank Lei Duan and Satoshi Oyama for sharing the datasets used in [4, 5].


  1. 1.
    Bragg, J., Mausam, Weld, D.S.: Crowdsourcing multi-label classification for taxonomy creation. In: HCOMP (2013)Google Scholar
  2. 2.
    Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 28, 20–28 (1979)CrossRefGoogle Scholar
  3. 3.
    Demartini, G., Difallah, D.E., Cudré-Mauroux, P.: Large-scale linked data integration using probabilistic reasoning and crowdsourcing. VLDB J. 22, 665–687 (2013)CrossRefGoogle Scholar
  4. 4.
    Duan, L., Oyama, S., Kurihara, M., Sato, H.: Crowdsourced semantic matching of multi-label annotations. In: Proceedings of the 24th International Conference on Artificial Intelligence, pp. 3483–3489 (2015)Google Scholar
  5. 5.
    Duan, L., Oyama, S., Sato, H., Kurihara, M.: Separate or joint? Estimation of multiple labels from crowdsourced annotations. Expert Syst. Appl. 41(13), 5723–5732 (2014)CrossRefGoogle Scholar
  6. 6.
    Ekman, P.: An argument for basic emotions. Cogn. Emotion 6(3–4), 169–200 (1992)CrossRefGoogle Scholar
  7. 7.
    Godbole, S., Sarawagi, S.: Discriminative methods for multi-labeled classification. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS, vol. 3056, pp. 22–30. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24775-3_5 CrossRefGoogle Scholar
  8. 8.
    Nakamura, A.: Kanjo Hyogen Jiten [Dictionary of Emotive Expressions]. Tokyodo (1993)Google Scholar
  9. 9.
    Oyama, S., Baba, Y., Sakurai, Y., Kashima, H.: Accurate integration of crowdsourced labels using workers’ self-reported confidence scores. In: Proceedings of the 23rd International Joint Conference on Artificial Intelligence, pp. 2554–2560 (2013)Google Scholar
  10. 10.
    Tsoumakas, G., Katakis, I.: Multi-label classification: an overview. Int. J. Data Warehouse. Min. 3(3), 1–11 (2007)CrossRefGoogle Scholar
  11. 11.
    Tsoumakas, G., Katakis, I., Vlahavas, I.: Random k-labelsets for multi-label classification. IEEE Trans. Knowl. Data Eng. 23(7), 1079–1089 (2011)CrossRefGoogle Scholar
  12. 12.
    Welinder, P., Branson, S., Perona, P., Belongie, S.J.: The multidimensional wisdom of crowds. In: Advances in Neural Information Processing Systems, vol. 23, pp. 2424–2432 (2010)Google Scholar
  13. 13.
    Whitehill, J., Fan Wu, T., Bergsma, J., Movellan, J.R., Ruvolo, P.L.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Advances in Neural Information Processing Systems, vol. 22, pp. 2035–2043 (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Intelligence Science and Technology, Graduate School of InformaticsKyoto UniversityKyotoJapan
  2. 2.RIKEN Center for Advanced Intelligence ProjectTokyoJapan

Personalised recommendations