Reverse-auction-based crowdsourced labeling for active learning

  • Hai Tang
  • Mingjun XiaoEmail author
  • Guoju Gao
  • Hui Zhao
Part of the following topical collections:
  1. Special Issue on Trust, Privacy, and Security in Crowdsourcing Computing


In the past few years, Machine Learning (ML) has aroused great interest in both academic and industrial societies. ML is booming because of its huge application potential in many areas, such as facial recognition, natural language processing, self-driving car, and so on. Nevertheless, one of the key problems is the scarcity of labeled data. Fortunately, mobile crowdsourcing makes it possible to recruit mobile workers to label large-scale data by offering them small payments. In this paper, we use crowdsourcing to tackle the scarcity of training data in active learning, and then propose an approximately truthful, individually rational, privacy-preserving incentive mechanism with a guaranteed approximate performance, based on the single-minded reverse auction for data labeling in active learning. Different from prior works, we take crowd workers’ reliability into consideration when selecting data to be labeled which can improve the labeling quality and the model performance. In addition, we employ differential privacy to preserve workers’ bid privacy because a worker’s bid usually contains sensitive information. The simulation results demonstrate that the learning model is much accurate compared with the traditional active learning without the consideration of reliability in the case of the same number of iterations.


Active learning Differential privacy Incentive mechanism Mobile crowdsourcing 



This research was supported by the National Natural Science Foundation of China (NSFC) (Grant No. 61872330, 61572457, 61379132, U1709217), the NSF of Jiangsu Province in China the NSF of Jiangsu Province in China (Grant No. BK20191194, BK20131174, BK2009150) and Anhui Initiative in Quantum Information Technologies (Grant No. AHY150300).


  1. 1.
    Bhagyashree, S.I.R., Nagaraj, K., Prince, M., Fall, C.H.D., Krishna, M.: Diagnosis of dementia by machine learning methods in epidemiological studies: A pilot exploratory study from South India. Soc. Psychiatry Psychiatr. Epidemiol. 53(1), 77–86 (2018)CrossRefGoogle Scholar
  2. 2.
    Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)Google Scholar
  3. 3.
    Dheeru, D., Karra Taniskidou, E.: UCI machine learning repository. (2017)
  4. 4.
    Eshtay, M., Faris, H., Obeid, N.: Improving extreme learning machine by competitive swarm optimization and its application for medical diagnosis problems. Expert Syst. Appl. 104(3), 134–152 (2018)CrossRefGoogle Scholar
  5. 5.
    Fang, M., Yin, J., Tao, D.: Active learning for crowdsourcing using knowledge transfer. In: Proceedings of the Twenty-Eight AAAI Conference on Artificial Intelligence (2014)Google Scholar
  6. 6.
    Gao, R., Zhao, M., Ye, T., Ye, F., Wang, Y., Bian, K., Wang, T., Li, X.: Jigsaw: Indoor floor plan reconstruction via mobile crowdsensing. In: MobiCom (2014)Google Scholar
  7. 7.
    Gao, G., Xiao, M., Wu, J., Huang, L., Hu, C.: Truthful incentive mechanism for nondeterministic crowdsensing with vehicles. IEEE Trans. Mob. Comput. 17(12), 2982–2997 (2018)CrossRefGoogle Scholar
  8. 8.
    He, R., Wu, X., Sun, Z., Tan, T.: Wasserstein cnn: Learning invariant features for nir-vis face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1761–1773 (2019)CrossRefGoogle Scholar
  9. 9.
    Hu, S., Su, L., Liu, H., Wang, H., Abdelzaher, T.F.: Smartroad: Smartphone-based crowd sensing for traffic regulator detection and identification. ACM Trans. Sensor Netw. 11(4), 55:1–55:27 (2015)CrossRefGoogle Scholar
  10. 10.
    Jin, H., Su, L., Chen, D., Nahrstedt, K., Xu, J.: Quality of information aware incentive mechanisms for mobile crowd sensing systems. In: MobiHoc (2015)Google Scholar
  11. 11.
    Jin, H., Su, L., Ding, B., Nahrstedt, K., Borisov, N.: Enabling privacy-preserving incentives for mobile crowd sensing systems. In: ICDCS (2016)Google Scholar
  12. 12.
    Jin, H., Su, L., Xiao, H., Nahrstedt, K.: Inception: Incentivizing privacy-preserving data aggregation for mobile crowd sensing systems. In: MobiHoc (2016)Google Scholar
  13. 13.
    Kajino, H., Baba, Y., Kashima, H.: Instance-privacy preserving crowdsourcing. In: HCOMP (2014)Google Scholar
  14. 14.
    Li, Q., Cao, G.: Providing efficient privacy-aware incentives for mobile sensing. In: ICDCS (2014)Google Scholar
  15. 15.
    Li, Q., Li, Y., Cao, J., Zhao, B., Fan, W., Han, J.: Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In: SIGMOD (2014)Google Scholar
  16. 16.
    Li, G., Zheng, Y., Fan, J., Wang, J., Cheng, R.: Crowdsourced data management: Overview and challenges. In: Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD ’17, pp 1711–1716 (2017)Google Scholar
  17. 17.
    Li, Y., Williams, S., Moran, B., Kealy, A.: A probabilistic indoor localization system for heterogeneous devices. IEEE Sensors J. 10(5), 1–1 (2019)Google Scholar
  18. 18.
    Li, Y., Wu, X., Kittler, J.: L1-(2d)2pcanet: A deep learning network for face recognition. J. Electron. Imaging. 28(2), 1–9 (2019)Google Scholar
  19. 19.
    Mcsherry, F., Talwar, K.: Mechanism design via differential privacy. In: FOCS (2007)Google Scholar
  20. 20.
    Mozafari, B., Sarkar, P., Franklin, J., Jordan, I., Madden, S.: Scaling up crowd-sourcing to very large datasets:a case for active learning. In: VLDB (2014)Google Scholar
  21. 21.
    Nguyen, A.T., Wallace, C.B., Lease, M.: Combining crowd and expert labels using decision theoretic active learning. In: Third AAAI Conference on Human Computation and Crowdsourcing (2015)Google Scholar
  22. 22.
    Oleson, D., Sorokin, A., Laughlin, G.P., Hester, V., Le, J., Biewald, L.: Programmatic gold:targeted and scalable quality assurance in crowdsourcing. In: HCOMP (2011)Google Scholar
  23. 23.
    Samant, P., Agarwal, R.: Machine learning techniques for medical diagnosis of diabetes using iris images. Comput. Methods Programs Biomed. 157(1), 121–128 (2018)CrossRefGoogle Scholar
  24. 24.
    Settles, B.: Active Learning Synthesisi Lectures on Artificial Intelligence and Machine Learning. Morgan Claypool Publishers (2012)Google Scholar
  25. 25.
    Shao, H.: Active learning for text mining from crowds. In: IEA/AIE (2017)Google Scholar
  26. 26.
    Wang, W., Guo, X.Y., Li, S.Y., Jiang, Y., Zhou, Z.H.: Obtaining high-quality label by distinguishing between easy and hard items in crowdsourcing. In: IJCAI (2017)Google Scholar
  27. 27.
    Wang, W., Guo, X.Y., Li, S.Y., Jiang, Y., Zhou, Z.H.: Obtaining high-quality label by distinguishing between easy and hard items in crowdsourcing. In: Proceedings of Twenty-Sixth International Joint Conference on Artificial Intelligence (2017)Google Scholar
  28. 28.
    Xiao, M., Wu, J., Huang, L., Cheng, R., Wang, Y.: Online task assignment for crowdsensing in predictable mobile social networks. IEEE Trans. Mob. Comput. 16(8), 2306–2320 (2017)CrossRefGoogle Scholar
  29. 29.
    Xiao, M., Wu, J., Zhang, S., Yu, J.: Secret-sharing-based secure user recruitment protocol for mobile crowdsensing. In: IEEE INFOCOM 2017 - IEEE Conference on Computer Communications, pp 1–9 (2017)Google Scholar
  30. 30.
    Xiao, M., Ma, K., Liu, A., Zhao, H., Li, Z., Zheng, K., Zhou, X.: Sra: Secure reverse auction for task assignment in spatial crowdsourcing. IEEE Trans. Knowl. Data Eng., 1–1 (2019)Google Scholar
  31. 31.
    Yang, D., Xue, G., Fang, X., Tang, J.: Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing. Mobicom (2012)Google Scholar
  32. 32.
    Yang, J., Drake, T., Damianou, A., Maarek, Y.: Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa. In: Proceedings of the 2018 World Wide Web Conference, vol. 10, pp 23–32 (2018)Google Scholar
  33. 33.
    Yu, T., Gui, L., Yu, T., Wang, J.: Walrasian equilibrium-based incentive scheme for mobile crowdsourcing fingerprint localization. Sensors, 19(12) (2019)CrossRefGoogle Scholar
  34. 34.
    Zhong, J., Tang, K., Zhou, Z.H.: Active learning from crowds with unsure option. In: IJCAI (2015)Google Scholar
  35. 35.
    Zhong, J., Tang, K., Zhou, Z.H.: Active learning from crowds with unsure option. In: Proceedings of Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)Google Scholar
  36. 36.
    Zhou, X., Chen, T., Guo, D., Teng, X., Yuan, B.: From one to crowd: A survey on crowdsourcing-based wireless indoor localization. Front. Comput. Sci. 12 (3), 423–450 (2018)CrossRefGoogle Scholar
  37. 37.
    Zhuo, G., Jia, Q., Guo, L., Li, M., Li, P.: Privacy-preserving verifiable data aggregation and analysis for cloud-assisted mobile crowdsourcing. In: IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications, pp 1–9 (2016)Google Scholar
  38. 38.
    Zhu, R., Li, Z., Wu, F., Shin, K., Chen, G.: Differentially private spectrum auction with approximate revenue maximization. In: MOBIHOC (2014)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Computer Science and Technology/Suzhou Institute for Advanced StudyUniversity of Science and Technology of ChinaHefeiChina

Personalised recommendations