Millionaire: a hint-guided approach for crowdsourcing

  • Bo Han
  • Quanming Yao
  • Yuangang Pan
  • Ivor W. TsangEmail author
  • Xiaokui Xiao
  • Qiang Yang
  • Masashi Sugiyama
Part of the following topical collections:
  1. Special Issue of the ACML 2018 Journal Track


Modern machine learning is migrating to the era of complex models, which requires a plethora of well-annotated data. While crowdsourcing is a promising tool to achieve this goal, existing crowdsourcing approaches barely acquire a sufficient amount of high-quality labels. In this paper, motivated by the “Guess-with-Hints” answer strategy from the Millionaire game show, we introduce the hint-guided approach into crowdsourcing to deal with this challenge. Our approach encourages workers to get help from hints when they are unsure of questions. Specifically, we propose a hybrid-stage setting, consisting of the main stage and the hint stage. When workers face any uncertain question on the main stage, they are allowed to enter the hint stage and look up hints before making any answer. A unique payment mechanism that meets two important design principles for crowdsourcing is developed. Besides, the proposed mechanism further encourages high-quality workers less using hints, which helps identify and assigns larger possible payment to them. Experiments are performed on Amazon Mechanical Turk, which show that our approach ensures a sufficient number of high-quality labels with low expenditure and detects high-quality workers.


Game theory Computational modeling Crowdsourcing Quality control Human factors 



Ivor W. Tsang was partially supported by ARC FT130100746, LP150100671 and DP180100106. Masashi Sugiyama was supported by the International Research Center for Neurointelligence (WPI-IRCN) at The University of Tokyo Institutes for Advanced Study. Xiaokui Xiao was partially supported by MOE, Singapore under grant MOE2015-T2-2-069, and by NUS, Singapore under an SUG. Bo Han and Ivor W. Tsang would like to thank Yao-Xiang Ding, Dengyong Zhou and Jun Zhu for helpful comments and discussions.


  1. Bi, W., Wang, L., Kwok, J., & Tu, Z. (2014). Learning to predict from crowdsourced data. In Conference on uncertainty in artificial intelligence (pp. 82–91).Google Scholar
  2. Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., et al. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.
  3. Buhrmester, M., Kwang, T., & Gosling, S. (2011). Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.CrossRefGoogle Scholar
  4. Chen, X., Lin, Q., & Zhou, D. (2013). Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In International conference on machine learning (pp. 64–72).Google Scholar
  5. Chen, Y., Chong, S., Kash, I., Moran, T., & Vadhan, S. (2016). Truthful mechanisms for agents that value privacy. ACM Transactions on Economics and Computation, 4(3), 13.MathSciNetCrossRefGoogle Scholar
  6. Difallah, D., Demartini, G., & Cudré-Mauroux, P. (2012). Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch (pp. 26–30).Google Scholar
  7. Ding, Y. X., & Zhou, Z. H. (2017). Crowdsourcing with unsure option. Machine Learning, 107, 749–766.MathSciNetCrossRefGoogle Scholar
  8. Fan, J., Li, G., Ooi, B., Tan, K., & Feng, J. (2015). iCrowd: An adaptive crowdsourcing framework. In ACM special interest group on management of data (pp. 1015–1030).Google Scholar
  9. Goel, G., Nikzad, A., & Singla, A. (2014). Mechanism design for crowdsourcing markets with heterogeneous tasks. In AAAI conference on human computation and crowdsourcing.Google Scholar
  10. Han, B., Pan, Y., & Tsang, I. (2017). Robust Plackett–Luce model for k-ary crowdsourced preferences. Machine Learning, 107, 675–702.MathSciNetCrossRefGoogle Scholar
  11. Han, B., Tsang, I., & Chen, L. (2016). On the convergence of a family of robust losses for stochastic gradient descent. In Joint European conference on machine learning and knowledge discovery in databases (pp. 665–680). Springer.Google Scholar
  12. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., et al. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6), 82–97.CrossRefGoogle Scholar
  13. Ho, C., Slivkins, A., Suri, S., & Vaughan, J. (2015). Incentivizing high quality crowdwork. In World Wide Web (pp. 419–429).Google Scholar
  14. Hu, H., Zheng, Y., Bao, Z., Li, G., Feng, J., & Cheng, R. (2016). Crowdsourced POI labelling: Location-aware result inference and task assignment. In International conference on data engineering (pp. 61–72).Google Scholar
  15. Ipeirotis, P., Provost, F., & Wang, J. (2010). Quality management on Amazon Mechanical Turk. In ACM SIGKDD conference on knowledge discovery and data mining workshop (pp. 64–67).Google Scholar
  16. Joglekar, M., Garcia-Molina, H., & Parameswaran, A. (2013). Evaluating the crowd with confidence. In ACM SIGKDD international conference on knowledge discovery and data mining (pp. 686–694).Google Scholar
  17. Kajino, H., Tsuboi, Y., & Kashima, H. (2012). A convex formulation for learning from crowds. Transactions of the Japanese Society for Artificial Intelligence, 27(3), 133–142.CrossRefGoogle Scholar
  18. Karger, D., Oh, S., & Shah, D. (2011). Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems (pp. 1953–1961).Google Scholar
  19. Koedinger, K., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264.CrossRefGoogle Scholar
  20. Lambert, N., Langford, J., Vaughan, J., Chen, Y., Reeves, D., Shoham, Y., et al. (2015). An axiomatic characterization of wagering mechanisms. Journal of Economic Theory, 156, 389–416.MathSciNetCrossRefGoogle Scholar
  21. Li, G., Chai, C., Fan, J., Weng, X., Li, J., Zheng, Y., et al. (2017a). CDB: Optimizing queries with crowd-based selections and joins. In ACM SIGMOD international conference on management of data (pp. 1463–1478).Google Scholar
  22. Li, G., Wang, J., Zheng, Y., & Franklin, M. J. (2016). Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9), 2296–2319.CrossRefGoogle Scholar
  23. Li, G., Zheng, Y., Fan, J., Wang, J., & Cheng, R. (2017b). Crowdsourced data management: Overview and challenges. In ACM SIGMOD international conference on management of data (pp. 1711–1716).Google Scholar
  24. Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior Research Methods, 47(2), 519–528.CrossRefGoogle Scholar
  25. Liu, Q., Peng, J., & Ihler, A. (2012). Variational inference for crowdsourcing. In Advances in neural information processing systems (pp. 692–700).Google Scholar
  26. Natarajan, N., Dhillon, I., Ravikumar, P., & Tewari, A. (2013). Learning with noisy labels. In Advances in neural information processing systems (pp. 1196–1204).Google Scholar
  27. Nisan, N., Roughgarden, T., Tardos, E., & Vazirani, V. (2007). Algorithmic game theory (Vol. 1). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  28. Patrini, G., Rozza, A., Menon, A., Nock, R., & Qu, L. (2017). Making deep neural networks robust to label noise: A loss correction approach. In IEEE conference on computer vision and pattern recognition.Google Scholar
  29. Pennock, D., Syrgkanis, V., & Vaughan, J. (2016). Bounded rationality in wagering mechanisms. In Conference on uncertainty in artificial intelligence.Google Scholar
  30. Raykar, V., & Yu, S. (2012). Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research, 13, 491–518.MathSciNetzbMATHGoogle Scholar
  31. Raykar, V., Yu, S., Zhao, L., Valadez, G., Florin, C., Bogoni, L., et al. (2010). Learning from crowds. Journal of Machine Learning Research, 11, 1297–1322.MathSciNetGoogle Scholar
  32. Rodrigues, F., Pereira, F., & Ribeiro, B. (2014). Sequence labeling with multiple annotators. Machine Learning, 95(2), 165–181.MathSciNetCrossRefGoogle Scholar
  33. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.MathSciNetCrossRefGoogle Scholar
  34. Shah, N., & Zhou, D. (2015). Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In Advances in neural information processing systems (pp. 1–9).Google Scholar
  35. Shah, N., & Zhou, D. (2016). No Oops, You Wont Do it again: Mechanisms for self-correction in crowdsourcing. In International conference on machine learning.Google Scholar
  36. Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489.CrossRefGoogle Scholar
  37. Singla, A., & Krause, A. (2013). Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In World Wide Web (pp. 1167–1178).Google Scholar
  38. Singla, A., Santoni, M., Bartók, G., Mukerji, P., Meenen, M., & Krause, A. (2015). Incentivizing users for balancing bike sharing systems. In Association for the advancement of artificial intelligence (pp. 723–729).Google Scholar
  39. Smith, J. (2007). Qualitative psychology: A practical guide to research methods. Thousand Oaks: Sage.Google Scholar
  40. Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., et al. (2015). A neural network approach to context-sensitive generation of conversational responses. In Conference of the association for computational linguistics (pp. 196–205).Google Scholar
  41. Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., & Fergus, R. (2015). Training convolutional networks with noisy labels. In International conference on learning representations workshop.Google Scholar
  42. Tian, T., & Zhu, J. (2015). Max-margin majority voting for learning from crowds. In Advances in neural information processing systems (pp. 1621–1629).Google Scholar
  43. Vuurens, J., Vries, A., & Eickhoff, C. (2011). How much spam can you take? An analysis of crowdsourcing results to increase accuracy. In ACM special interest group on information retrieval workshop (pp. 21–26).Google Scholar
  44. Wais, P., Lingamneni, S., Cook, D., Fennell, J., Goldenberg, B., Lubarov, D., Marin, D., & Simons, H. (2010). Towards building a high-quality workforce with Mechanical Turk. In Advances in neural information processing systems workshop.Google Scholar
  45. Wang, L., & Zhou, ZH. (2016). Cost-saving effect of crowdsourcing learning. In International joint conference on artificial intelligence (pp. 2111–2117).Google Scholar
  46. Wang, W., Guo, X. Y., Li, S. Y., Jiang, Y., & Zhou, ZH. (2017). Obtaining high-quality label by distinguishing between easy and hard items in crowdsourcing. In International joint conference on artificial intelligence (pp. 2964–2970).Google Scholar
  47. Yan, Y., Rosales, R., Fung, G., & Dy, J. (2011). Active learning from crowds. International Conference on Machine Learning, 11, 1161–1168.Google Scholar
  48. Yan, Y., Rosales, R., Fung, G., Schmidt, M., Hermosillo, G., Moy, L., & Dy, J. (2010). Modeling annotator expertise: Learning when everybody knows a bit of something. In International conference on artificial intelligence and statistics (pp. 932–939).Google Scholar
  49. Yan, Y., Rosales, R., Fung, G., Subramanian, R., & Dy, J. (2014). Learning from multiple annotators with varying expertise. Machine Learning, 95(3), 291–327.MathSciNetCrossRefGoogle Scholar
  50. Yu, X., Liu, T., Gong, M., & Tao, D. (2017a). Learning with biased complementary labels. arXiv preprint arXiv:1711.09535.
  51. Yu, X., Liu, T., Gong, M., Zhang, K., & Tao, D. (2017b). Transfer learning with label noise. arXiv preprint arXiv:1707.09724.
  52. Zhang, J., Wu, X. D., & Sheng, V. (2016a). Learning from crowdsourced labeled data: A survey. Artificial Intelligence Review, 46(4), 543–576.CrossRefGoogle Scholar
  53. Zhang, Y., Chen, X., Zhou, D., & Jordan, M. (2016b). Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. Journal of Machine Learning Research, 17(102), 1–44.MathSciNetzbMATHGoogle Scholar
  54. Zheng, Y., Cheng, R., Maniu, S., & Mo, L. (2015a). On optimality of jury selection in crowdsourcing. In International conference on extending database technology.Google Scholar
  55. Zheng, Y., Li, G., & Cheng, R. (2016). Docs: A domain-aware crowdsourcing system using knowledge bases. Very Large Data Base Endowment, 10(4), 361–372.Google Scholar
  56. Zheng, Y., Li, G., Li, Y., Shan, C., & Cheng, R. (2017). Truth inference in crowdsourcing: Is the problem solved? Very Large Data Base Endowment, 10(5), 541–552.Google Scholar
  57. Zheng, Y., Wang, J., Li, G., Cheng, R., & Feng, J. (2015b). QASCA: A quality-aware task assignment system for crowdsourcing applications. In ACM special interest group on management of data (pp. 1031–1046).Google Scholar
  58. Zhong, J. H., Tang, K., & Zhou, Z. H. (2015). Active learning from crowds with unsure option. In International joint conference on artificial intelligence (pp. 1061–1068).Google Scholar
  59. Zhou, D., Basu, S., Mao, Y., & Platt, J. (2012). Learning from the wisdom of crowds by minimax entropy. In Advances in neural information processing systems (pp. 2195–2203).Google Scholar
  60. Zhou, D., Liu, Q., Platt, J., & Meek, C. (2014). Aggregating ordinal labels from crowds by minimax conditional entropy. In International conference on machine learning (pp. 262–270).Google Scholar

Copyright information

© The Author(s) 2018

Authors and Affiliations

  1. 1.Centre for Artificial Intelligence (CAI)University of Technology SydneySydneyAustralia
  2. 2.Center for Advanced Intelligence ProjectRIKENTokyoJapan
  3. 3.4Paradigm Inc.BeijingChina
  4. 4.Department of Computer ScienceNational University of SingaporeSingaporeSingapore
  5. 5.Department of Computer Science and EngineeringHong Kong University of Science and TechnologyClear Water BayHong Kong
  6. 6.Graduate School of Frontier SciencesThe University of TokyoTokyoJapan

Personalised recommendations