Skip to main content
SpringerLink
Log in

Budget-constraint mechanism for incremental multi-labeling crowdsensing

  • Published:
Telecommunication Systems Aims and scope Submit manuscript

Abstract

Machine learning techniques require an enormous amount of high-quality data labeling for more naturally simulating human comprehension. Recently, mobile crowdsensing, as a new paradigm, makes it possible that a large number of instances can be often quickly labeled at low cost. Existing works only focus on the single labeling for supervised learning problems of traditional machine learning, where one instance associates with only label. However, in many real world applications, an instance may have more than one label. To the end, in this paper, we explore an incremental multi-labeling issue by incentivizing crowd users to label instances under the budget constraint, where each instance is composed of multiple labels. Considering both uncertainty and diversity of the number of each instance’s labels, this paper proposes two mechanisms for incremental multi-labeling crowdsensing by introducing both uncertainty and diversity. Through extensive simulations, we validate their theoretical properties and evaluate the performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://mulan.sourceforge.net/datasets.html.

References

  1. Badanidiyuru, A., Kleinberg, R., & Singer, Y. (2012). Learning on a budget: Posted price mechanisms for online procurement. In Proceedings of the 13th ACM conference on electronic commerce, pp. 128–145.

  2. Kapoor, A., Horvitz, E., & Basu, S. (2007). Selective supervision: Guiding supervised learning with decision-theoretic active learning. In Proceedings of international joint conference on artificial intelligence (IJCAI), pp. 877–882.

  3. Krause, A., & Guestrin, C. (2007). Near-optimal observation selection using submodular function. In Proceedings of AAAI, pp. 1650–1654.

  4. Settles, B., & Craven, M. (2008). An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), pp. 1069–1078.

  5. Yang, B., Sun, J. T., Wang, T., & Chen, Z. (2009). Effective multilabel active learning for text classification. In Proceedings of KDD, pp. 917–926.

  6. Baldridge, J., & Osborne, M. (2004). Active learning and the total cost of annotation. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), pp. 9–16.

  7. Chávez-Martínez, G., Ruiz-Correa, S., & Gatica-Perez, D. (2015). Happy and agreeable? Multi-label classification of impressions in social video. In Proceedings of the 14th international conference on mobile and ubiquitous multimedia, ACM, pp. 109–120.

  8. Culotta, A., & McCallum, A. (2004). Reducing labeling effort for structured prediction tasks. In Proceedings of the national conference on artificial intelligence (AAAI), pp. 746–751.

  9. Duan, L., Oyama, S., Sato, H., & Kurihara, M. (2014). Separate or joint? Estimation of multiple labels from crowdsourced annotations. Expert Systems with Applications, 41(13), 5723–5732.

    Article  Google Scholar 

  10. Yang, D., Xue, G., Fang, X., & Tang, J. (2012). Crowdsourcing to smartphones: Incentive mechanism design for mobile phone sensing. In Proceedings of ACM MobiCom.

  11. Zhou, D., Liu, Q., Platt, J. C., Meek, C., & Shah, N. B. (2016). Regularized minimax conditional entropy for crowdsourcing. arXiv, pp. 1–31.

  12. Zhou, D., Liu, Q., Platt, J. C., Meek, C., & Shah, N. B. (2015). Regularized minimax conditional entropy for crowdsourcing. arXiv preprint arXiv:1503.07240.

  13. Qi, G. J., Hua, X. S., Rui, Y., Tang, J., & Zhang, H. J. (2008). Twodimensional active learning for image classification. In Proceedings of CVPR, pp. 1–8.

  14. Gan, X., Wang, X., Niu, W., Hang, G., Tian, X., Wang, X., & Xu, J. J. (2017). Incentivize multi-class crowd labeling under budget constraint. IEEE Journal on Selected Areas in Communications. doi:10.1109/JSAC.2017.2680838.

  15. Lee, J. S., & Hoh, B. (2010). Dynamic pricing incentive for participatory sensing. Pervasive and Mobile Computing, 6(6), 693–708.

    Article  Google Scholar 

  16. Karger, D. R., Oh, S., & Shah, D. (2013). Efficient crowdsourcing for multi-class labeling. ACM SIGMETRICS Performance Evaluation Review, 41(1), 81–92.

    Article  Google Scholar 

  17. Krishna, R. A., Hata, K., Chen, S., Kravitz, J., Shamma, D. A., Fei-Fei, L., & Bernstein, M. S. (2016). Embracing error to enable rapid crowdsourcing. In Proceedings of the 2016 CHI conference on human factors in computing systems, ACM, pp. 3167–3179.

  18. Duan, L., Kubo, T., Sugiyama, K., Huang, J., Hasegawa, T., & Walrand, J. (2012). Incentive mechanisms for smartphone collaboration in data acquisition and distributed computing. In Proceedings of IEEE INFOCOM, pp. 1701–1709.

  19. Jaimes, L. G., Vergara-Laurens, I., & Labrador, M. A. (2012). A location-based incentive mechanism for participatory sensing systems with budget constraints. In 2012 IEEE international conference on pervasive computing and communications (PerCom), IEEE, pp. 103–108.

  20. Li, X., & Guo, Y. (2013). Active learning with multi-label SVM classification. In IJCAI.

  21. Chen, N., Gravin, N., & Lu, P. (2011). On the approximability of budget feasible mechanisms. In Proceedings of the twenty-second annual ACM-SIAM symposium on discrete algorithms, pp. 685–699.

  22. Zhang, Q., Wen, Y., Tian, X., Gan, X., & Wang, X. (2015). Incentivize crowd labeling under budget constraint. In Proceedings of IEEE INFOCOM.

  23. Raykar, V., & Agrawal, P. (2014). Sequential crowdsourced labeling as an epsilon-greedy exploration in a Markov decision process. In Artificial intelligence and statistics, pp. 832–840.

  24. Chakraborty, S., Balasubramanian, V., & Panchanathan, S. (2011). Optimal batch selection for active learning in multi-label classification. In Proceedings of the 19th ACM international conference on multimedia, pp. 1413–1416.

  25. Huang, S. J., & Zhou, Z. H. (2013). Active query driven by uncertainty and diversity for incremental multi-label learning. In Proceedings of IEEE 13th international conference on data mining, pp. 1079–1084.

  26. Reddy, S., Estrin, D., & Srivastava, M. (2010). Recruitment framework for participatory sensing data collections. In Pervasive computing, Springer, pp. 138–155.

  27. Singer, Y., & Mittal, M. (2013). Pricing mechanisms for crowdsourcing markets. In Proceedings of the 22nd international conference on World Wide Web, International World Wide Web Conferences Steering Committee, pp. 1157–1166.

  28. Singer, Y., & Mittal, M. (2011). Pricing tasks in online labor markets. In Human computation.

  29. Singla, A., & Krause, A. (2013). Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proceedings of ACM WWW, pp. 1167–1177.

  30. Sun, J. (2016). Marginal quality-based long-term incentive mechanisms for crowd sensing. International Journal of Communication Systems, 29(5), 942–958.

    Article  Google Scholar 

  31. Li, X., Wang, L., & Sung, E. (2004). Multilabel SVM active learning for image classification. In Proceedings of ICIP, pp. 2207–2210.

  32. Singer, Y. (2010). Budget feasible mechanisms. IEEE Foundations of Computer Science (FOCS), 2010, 765–774.

    Google Scholar 

  33. Zhang, J., Wu, X., & Sheng, V. S. (2016). Learning from crowdsourced labeled data: A survey. Artificial Intelligence Review, 46(4), 543–576.

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China under Grant No. 61375021.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Jiajun Sun & Ningzhong Liu

  2. Shanghai Jiao Tong University, Shanghai, 200030, China

    Dianliang Wu

Authors
  1. Jiajun Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ningzhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dianliang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ningzhong Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, J., Liu, N. & Wu, D. Budget-constraint mechanism for incremental multi-labeling crowdsensing. Telecommun Syst 67, 297–307 (2018). https://doi.org/10.1007/s11235-017-0339-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11235-017-0339-7

Keywords

Search

Navigation