Skip to main content
SpringerLink
Log in

Crowd Label Aggregation Under a Belief Function Framework

  • Conference paper
  • First Online:
Knowledge Science, Engineering and Management (KSEM 2016)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9983))

Abstract

Crowdsourcing emerged as an efficient human-powered concept to tackle the problem of labeling complex tasks that computer programs still cannot solve. Amazon’s Mechanical Turk is one of the most popular platforms that allows to gather labels from human workers. These labels are then aggregated in order to estimate the true labels. Considering that not all labelers are experts, their answers may be imperfect and consequently unreliable. In this paper, we propose a novel label aggregation method based on the belief function theory. The proposed method grants a strong framework that does not only allow to reliably aggregate imperfect labels but also to integrate labelers expertise for more accurate results. To demonstrate the effectiveness of the proposed method, experiments are conducted on real datasets. The results show that our method is a promising solution in the crowd labeling domain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Howe, J.: The rise of crowdsourcing. Wired Mag. 14(6), 1–4 (2006)

    MathSciNet  Google Scholar 

  2. Howe, J.: Crowdsourcing: How the Power of the Crowd is Driving the Future of Business. Random House, New York (2008)

    Google Scholar 

  3. Dempster, A.P.: Upper and lower probabilities induced by a multivalued mapping. Ann. Math. Stat. 38, 325–339 (1967)

    Article  MathSciNet  MATH  Google Scholar 

  4. Shafer, G.: A Mathematical Theory of Evidence, vol. 1. Princeton University Press, Princeton (1976)

    MATH  Google Scholar 

  5. Khattak, F.K., et al.: Quality control of crowd labeling through expert evaluation. In: Proceedings of the Neural Information Processing Systems 2nd Workshop on Computational Social Science and the Wisdom of Crowds (2011)

    Google Scholar 

  6. Jousselme, A.-L., et al.: A new distance between two bodies of evidence. Inf. Fusion 2(2), 91–101 (2001)

    Article  Google Scholar 

  7. Lefèvre, E., Elouedi, Z.: How to preserve the confict as an alarm in the combination of belief functions? Decis. Support Syst. 56, 326–333 (2013)

    Article  Google Scholar 

  8. Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990)

    Article  Google Scholar 

  9. Quoc Viet Hung, N., Tam, N.T., Tran, L.N., Aberer, K.: An evaluation of aggregation techniques in crowdsourcing. In: Lin, X., Manolopoulos, Y., Srivastava, D., Huang, G. (eds.) WISE 2013. LNCS, vol. 8181, pp. 1–15. Springer, Heidelberg (2013). doi:10.1007/978-3-642-41154-0_1

    Chapter  Google Scholar 

  10. Kuncheva, L., et al.: Limits on the majority vote accuracy in classifier fusion. Pattern Anal. Appl. 6, 22–31 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  11. Lee, K., et al.: The social honeypot project: protecting online communities from spammers. In: International World Wide Web Conference, pp. 1139–1140 (2010)

    Google Scholar 

  12. Karger, D., et al.: Iterative learning for reliable crowdsourcing systems. In: Neural Information Processing Systems, pp. 1953–1961 (2011)

    Google Scholar 

  13. Quinn, A.J., et al.: Human computation: a survey and taxonomy of a growing field. In: Conference on Human Factors in Computing Systems, pp. 1403–1412 (2011)

    Google Scholar 

  14. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 28, 20–28 (2010)

    Article  Google Scholar 

  15. Smets, P.: The transferable belief model for quantified belief representation. In: Smets, P. (ed.) Quantified Representation of Uncertainty and Imprecision, pp. 267–301. Springer, Dordrecht (1998)

    Chapter  Google Scholar 

  16. Trabelsi, A., Lefèvre, E., Elouedi, Z.: Belief function combination: comparative study within the classifier fusion framework. In: Gaber, T., Hassanien, A.E., El-Bendary, N., Dey, N. (eds.) The 1st International Conference on Advanced Intelligent System and Informatics (AISI 2015), November 28-30, 2015, Beni Suef, Egypt. Advances in Intelligent Systems and Computing, pp. 425–435. Springer, Cham (2016)

    Google Scholar 

  17. Whitehill, J., et al.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Neural Information Processing Systems, pp. 2035–2043 (2009)

    Google Scholar 

  18. Snow, R., et al.: Cheap and fast but is it good? Evaluation non-expert annotations for natural language tasks. In: The Conference on Empirical Methods in Natural Languages Processing, pp. 254–263 (2008)

    Google Scholar 

  19. Ben Rjab, A., Kharoune, M., Miklos, Z., Martin, A., Ben Yaghlane, B.: Characterization of experts in crowdsourcing platforms. In: 24ème Conférence sur la Logique Floue et ses Applications, Poitiers, France, November 2015

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LARODEC, Institut Supérieur de Gestion de Tunis, Université de Tunis, Tunis, Tunisia

    Lina Abassi & Imen Boukhris

Authors
  1. Lina Abassi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Imen Boukhris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lina Abassi .

Editor information

Editors and Affiliations

  1. University of Passau, Passau, Germany

    Franz Lehner

  2. University of Passau , Passau, Germany

    Nora Fteimi

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Abassi, L., Boukhris, I. (2016). Crowd Label Aggregation Under a Belief Function Framework. In: Lehner, F., Fteimi, N. (eds) Knowledge Science, Engineering and Management. KSEM 2016. Lecture Notes in Computer Science(), vol 9983. Springer, Cham. https://doi.org/10.1007/978-3-319-47650-6_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47650-6_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47649-0

  • Online ISBN: 978-3-319-47650-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation