Abstract
Crowdsourcing has become a phenomenon of increasing interest in several research fields such as artificial intelligence. It typically uses human cognitive ability in order to effectively solve tasks that can hardly be addressed by automated computation. The major problem however is that so far studies could not completely control the quality of obtained data since contributors are uncertainly reliable. In this work, we propose an approach that aggregates labels using the belief function theory under the assumption that these labels could be partial hence imprecise. Simulated data demonstrate that our method produces more reliable aggregation results.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Zheng, Y., Wang, J., Li, G., Feng, J.: QASCA: a quality-aware task assignment system for crowdsourcing applications. In: International Conference on Management of Data, pp. 1031–1046 (2015)
Yan, T., Kumar, V., Ganesan, D.: Designing games with a purpose. Commun. ACM 51(8), 58–67 (2008)
Snow, R., O’Connor, B., Jurafsky, D., Ng, A.Y.: Cheap and fast but is it good? Evaluation non-expert annotations for natural language tasks. In: The Conference on Empirical Methods in Natural Languages Processing, pp. 254–263 (2008)
Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: International Conference on Knowledge Discovery and Data Mining, pp. 614–622 (2008)
Shafer, G.: A Mathematical Theory of Evidence, vol. 1. Princeton University Press, Princeton (1976)
Dempster, A.P.: Upper and lower probabilities induced by a multivalued mapping. In: The Annals of Mathematical Statistics, pp. 325–339 (1967)
Jousselme, A.-L., Grenier, D., Bossé, É.: A new distance between two bodies of evidence. In: Information Fusion, pp. 91–101 (2001)
Lefèvre, E., Elouedi, Z.: How to preserve the confict as an alarm in the combination of belief functions? Decis. Support Syst. 56, 326–333 (2013)
Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990)
Raykar, V.C., Yu, S.: Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res. 13, 491–518 (2012)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. Appl. Stat. 28, 20–28 (2010)
Khattak, F.K., Salleb, A.: Quality control of crowd labeling through expert evaluation. In: The Neural Information Processing Systems 2nd Workshop on Computational Social Science and the Wisdom of Crowds, pp. 27–29 (2011)
Smets, P., Mamdani, A., Dubois, D., Prade, H.: Non Standard Logics for Automated Reasoning, pp. 253–286. Academic Press, London (1988)
Ben Rjab, A., Kharoune, M., Miklos, Z., Martin, A.: Characterization of experts in crowdsourcing platforms. In: International Conference on BELIEF 2016, pp. 97–104 (2016)
Watanabe, M., Yamaguchi, K.: The EM Algorithm and Related Statistical Models, 250 p. CRC Press, Boca Raton (2003)
Whitehill, J., Wu, T., Bergsma, J., Movellan, J.R., Ruvolo, P.L.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Neural Information Processing Systems, pp. 2035–2043 (2009)
Abassi, L., Boukhris, I.: Crowd label aggregation under a belief function framework. In: International Conference on Knowledge Science, Engineering and Management, pp. 185–196. Springer (2016)
Abassi, L., Boukhris, I.: A gold standards-based crowd label aggregation within the belief function theory. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pp. 97–106. Springer (2017)
Abassi, L., Boukhris, I.: Iterative aggregation of crowdsourced tasks within the belief function theory. In: European Conference on Symbolic and Quantitative Approaches to Reasoning and Uncertainty, pp. 159–168. Springer (2017)
Abassi, L., Boukhris, I.: A worker clustering-based approach of label aggregation under the belief function theory. In: Applied Intelligence, pp. 1573–7497 (2018)
Florentin, S., Arnaud, M., Christophe, O.: Contradiction measures and specificity degrees of basic belief assignments. In: 14th International Conference on Information Fusion, pp. 1–8 (2011)
Kuncheva, L., et al.: Limits on the majority vote accuracy in classifier fusion. Pattern Anal. Appl. 6, 22–31 (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Abassi, L., Boukhris, I. (2020). Imprecise Label Aggregation Approach Under the Belief Function Theory. In: Abraham, A., Cherukuri, A., Melin, P., Gandhi, N. (eds) Intelligent Systems Design and Applications. ISDA 2018 2018. Advances in Intelligent Systems and Computing, vol 941. Springer, Cham. https://doi.org/10.1007/978-3-030-16660-1_59
Download citation
DOI: https://doi.org/10.1007/978-3-030-16660-1_59
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-16659-5
Online ISBN: 978-3-030-16660-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)