World Wide Web

, Volume 20, Issue 6, pp 1211–1235 | Cite as

A context-aware approach for trustworthy worker selection in social crowd

  • Yang Zhao
  • Guanfeng LiuEmail author
  • Kai Zheng
  • An Liu
  • Zhixu Li
  • Xiaofang Zhou


Crowdsourcing applications like Amazon Mechanical Turk (AMT) make it possible to address many difficult tasks (e.g., image tagging and sentiment analysis) on the internet and make full use of the wisdom of crowd, where worker quality is one of the most crucial issues for the task owners. Thus, a challenging problem is how to effectively and efficiently select the high quality workers, so that the tasks online can be accomplished successfully under a certain budget. The existing methods on the crowd worker selection problem mainly based on the quality measurement of the crowd workers, those who have to register on the crowdsourcing platforms. With the connect of the OSNs and the crowdsourcing applications, the social contexts like social relationships and social trust between participants and social positions of participants can assist requestors to select one or a group of trustworthy crowdsourcing workers. In this paper, we first present a contextual social network structure and a concept of Strong Social Component (SSC), which emblems a group of workers who have high social contexts values. Then, we propose a novel index for SSC, and a new efficient and effective algorithm C-AWSA to find trustworthy workers, who can complete the tasks with high quality. The results of our experiments conducted on four real OSN datasets illustrate that the superiority of our method in trustworthy worker selection.


Crowdsourcing Contextual social network Strong social component Trustworthy worker 



This work was partially supported by Natural Science Foundation of China (Grant Nos. 61303019, 61572336, 61532018, 61402313, 61502324), Doctoral Fund of Ministry of Education of China (20133201120012), Postdoctoral Science Foundation of China (2015M571805, 2016T90492), Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China, and Open Foundation of State Key Laboratory of Networking and Switching Technology (Beijing University of Posts and Telecommunications (SKLNST-2016-2-02).


  1. 1.
    Adler, P.S.: Market, hierarchy, and trust: the knowledge economy and the future of capitalism. Organ. Sci. 12(2), 215–234 (2001)CrossRefGoogle Scholar
  2. 2.
    Berger, P.L., Luckmann, T.: The social construction of reality: A treatise in the sociology of knowledge, Penguin (1991). no. 10Google Scholar
  3. 3.
    Biggs, N., Lloyd, E., Wilson, R.: Graph Theory. Oxford University Press (1986)Google Scholar
  4. 4.
    Brabham, D.C.: Crowdsourcing as a model for problem solving an introduction and cases. Convergence: the International Journal of Research Into New Media Technologies 14(1), 75–90 (2008)CrossRefGoogle Scholar
  5. 5.
    Cao, C.C., She, J., Tong, Y., Chen, L.: Whom to ask?: jury selection for decision making tasks on micro-blog services. Proceedings of the VLDB Endowment 5 (11), 1495–1506 (2012)CrossRefGoogle Scholar
  6. 6.
    Cao, C.C., Tong, Y., Chen, L., Jagadish, H.: Wisemarket: a new paradigm for managing wisdom of online social users. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 455–463. ACM (2013)Google Scholar
  7. 7.
    Carpenter, B.: A hierarchical bayesian model of crowdsourced relevance coding. In: TREC (2011)Google Scholar
  8. 8.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. J. R. Stat. Soc. Ser. B Methodol., 1–38 (1977)Google Scholar
  9. 9.
    Feng, A., Franklin, M., Kossmann, D., Kraska, T., Madden, S. R., Ramesh, S., Wang, A., Xin, R. : Crowddb: Query processing with the vldb crowd (2011)Google Scholar
  10. 10.
    Forlines, C., Miller, S., Guelcher, L., Bruzzi, R.: Crowdsourcing the future: predictions made with a social network. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pp 3655–3664. ACM (2014)Google Scholar
  11. 11.
    Gao, J., Liu, X., Ooi, B.C., Wang, H., Chen, G.: An online cost sensitive decision-making method in crowdsourcing systems. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp 217–228. ACM (2013)Google Scholar
  12. 12.
    Gentle, J.E., Härdle, W.K., Mori, Y.: Handbook of computational statistics: concepts and methods. Springer Science & Business Media (2012)Google Scholar
  13. 13.
    Gupta, M.R., Chen, Y.: Theory and use of the EM algorithm. Now Publishers Inc (2011)Google Scholar
  14. 14.
    Hirth, M., Scheuring, S., Hossfeld, T., Schwartz, C., Tran-Gia, P.: Predicting result quality in crowdsourcing using application layer monitoring. In: 2014 IEEE Fifth International Conference on Communications and Electronics (ICCE), pp 510–515. IEEE (2014)Google Scholar
  15. 15.
    Howe, J.: The rise of crowdsourcing. Wired Mag. 14(6), 1–4 (2006)Google Scholar
  16. 16.
    Hung, N.Q.V., Tam, N.T., Miklós, Z., Aberer, K.: On leveraging crowdsourcing techniques for schema matching networks. In: Database Systems for Advanced Applications, pp 139–154. Springer (2013)Google Scholar
  17. 17.
    Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp 64–67. ACM (2010)Google Scholar
  18. 18.
    Kelley, H.H., Berscheid, E., Christensen, A., Harvey, J.H., Huston, T.L., Levinger, G., McClintock, E., Peplau, L.A., Peterson, D.R.: Analyzing close relationships. Close Relationships 20, 67 (1983)Google Scholar
  19. 19.
    Korkmaz, T., Krunz, M.: Multi-constrained optimal path selection. In: INFOCOM 2001. Twentieth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings, vol. 2, pp 834–843. IEEE (2001)Google Scholar
  20. 20.
    Li, H., Liu, Q.: Cheaper and better: Selecting good workers for crowdsourcing (2015). arXiv preprint arXiv:1502.00725
  21. 21.
    Li, L., Wang, Y., Lim, E.-P.: Trust-oriented composite service selection and discovery. In: Service-Oriented Computing, pp 50–67. Springer (2009)Google Scholar
  22. 22.
    Li, H., Zhao, B., Fuxman, A.: The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing. In: Proceedings of the 23rd international conference on World Wide Web. International World Wide Web Conferences Steering Committee, pp 165–176 (2014)Google Scholar
  23. 23.
    Liu, G., Wang, Y., Orgun, M.A., Lim, E.-P.: A heuristic algorithm for trust-oriented service provider selection in complex social networks. In: 2010 IEEE International Conference on Services Computing (SCC), pp 130–137. IEEE (2010)Google Scholar
  24. 24.
    Liu, G., Wang, Y., Orgun, M.A., et al.: Optimal social trust path selection in complex social networks. In: AAAI, vol. 10, pp 1397–1398 (2010)Google Scholar
  25. 25.
    Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: Cdas: a crowdsourcing data analytics system. Proceedings of the VLDB Endowment 5(10), 1040–1051 (2012)Google Scholar
  26. 26.
    Liu, Q., Ihler, A.T., Steyvers, M.: Scoring workers in crowdsourcing: How many control questions are enough?. In: Advances in Neural Information Processing Systems, pp 1914–1922 (2013)Google Scholar
  27. 27.
    Liu, G., Liu, A., Wang, Y., Li, L.: An efficient multiple trust paths finding algorithm for trustworthy service provider selection in real-time online social network environments. In: 2014 IEEE International Conference on Web Services (ICWS), pp 121–128. IEEE (2014)Google Scholar
  28. 28.
    Malone, T.W., Laubacher, R., Dellarocas, C.: Harnessing crowds: mapping the genome of collective intelligence (2009)Google Scholar
  29. 29.
    Mansell, R., Collins, B.: Trust and crime in information societies. Edward Elgar Publishing (2005)Google Scholar
  30. 30.
    Marcus, A., Wu, E., Karger, D., Madden, S., Miller, R.: Human-powered sorts and joins. Proceedings of the VLDB Endowment 5(1), 13–24 (2011)CrossRefGoogle Scholar
  31. 31.
    Miller, R.: Intimate relationships. McGraw-Hill Higher Education (2007)Google Scholar
  32. 32.
    Mislove, A., Marcon, M., Gummadi, P.K., Druschel, P., Bhattacharjee, B.: Measurement and analysis of online social networks. In: Internet Measurement Conference, pp 29–42 (2007)Google Scholar
  33. 33.
    Parameswaran, A.G., Garcia-Molina, H., Park, H., Polyzotis, N., Ramesh, A., Widom, J.: Crowdscreen: Algorithms for filtering data with humans. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp 361–372. ACM (2012)Google Scholar
  34. 34.
    Park, H., Garcia-Molina, H., Pang, R., Polyzotis, N., Parameswaran, A., Widom, J.: Deco: A system for declarative crowdsourcing. Proceedings of the VLDB Endowment 5(12), 1990–1993 (2012)CrossRefGoogle Scholar
  35. 35.
    Sun, Z., Wang, H., Wang, H., Shao, B., Li, J.: Efficient subgraph matching on billion node graphs. In: VLDB’12, pp 788–799Google Scholar
  36. 36.
    Tang, J., Zhang, J., Yan, L., Li, J., Zhang, L., Su, Z.: Arnetminer: Extraction and mining of academic social networks. In: KDD’08, pp 990–998Google Scholar
  37. 37.
    Von Ahn, L., Maurer, B., McMillen, C., Abraham, D., Blum, M.: Recaptcha: Human-based character recognition via Web security measures. Science 321 (5895), 1465–1468 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Wang, Y., Varadharajan, V.: Role-based recommendation and trust evaluation. In: E-Commerce Technology and the 4th IEEE International Conference on Enterprise Computing, E-Commerce, and E-Services, 2007. The 9th IEEE International Conference on CEC/EEE 2007, pp 278–288. IEEE (2007)Google Scholar
  39. 39.
    Wang, J., Kraska, T., Franklin, M.J., Feng, J.: Crowder: Crowdsourcing entity resolution. Proceedings of the VLDB Endowment 5(11), 1483–1494 (2012)CrossRefGoogle Scholar
  40. 40.
    Whang, S.E., Lofgren, P., Garcia-Molina, H.: Question selection for crowd entity resolution. Proceedings of the VLDB Endowment 6(6), 349–360 (2013)CrossRefGoogle Scholar
  41. 41.
    Wang, J., Li, G., Kraska, T., Franklin, M.J., Feng, J.: Leveraging transitive relations for crowdsourced joins. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp 229–240. ACM (2013)Google Scholar
  42. 42.
    Yang, X.S., Cheng, R., Mo, L., Kao, B., Cheung, D.W.: On incentive-based tagging. In: 2013 IEEE 29th International Conference on Data Engineering (ICDE), pp 685–696. IEEE (2013)Google Scholar
  43. 43.
    Yang, S., Zhang, J., Chen, I.: Web 2.0 services for identifying communities of practice. In: SCC’07, pp 130–137Google Scholar
  44. 44.
    Ye, B., Wang, Y., Liu, L.: Crowd trust: a context-aware trust model for worker selection in crowdsourcing environments. In: 2015 IEEE International Conference on Web Services (ICWS), pp 121–128. IEEE (2015)Google Scholar
  45. 45.
    Zhang, C.J., Chen, L., Jagadish, H., Cao, C.C.: Reducing uncertainty of schema matching via crowdsourcing. Proceedings of the VLDB Endowment 6(9), 757–768 (2013)CrossRefGoogle Scholar
  46. 46.
    Zhao, Z., Wei, F., Zhou, M., Chen, W., Ng, W.: Crowd-selection query processing in crowdsourcing databases: A task-driven approach. In: International Conference on Extending Database Technology (EDBT 2015), pp. 397–408, Brussels (2015)Google Scholar
  47. 47.
    Zheng, Y., Cheng, R., Maniu, S., Mo, L.: On optimality of jury selection in crowdsourcing. In: 18th International Conference on Extending Database Technology (EDBT 2015), Brussels (2015)Google Scholar
  48. 48.
    Zhu, Y., Qin, L., Xu, J.X., Cheng, H.: Finding top-k similar graphs in graph databases. In: EDBT’12, pp 456–467Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Yang Zhao
    • 1
  • Guanfeng Liu
    • 1
    • 3
    Email author
  • Kai Zheng
    • 1
  • An Liu
    • 1
  • Zhixu Li
    • 1
  • Xiaofang Zhou
    • 1
    • 2
  1. 1.School of Computer Science and TechnologySoochow UniversitySuzhouChina
  2. 2.School of Information Technology and Electrical EngineeringQueensland UniversitySt LuciaAustralia
  3. 3.State Key Laboratory of Networking and Switching Technology (Beijing University of Posts and Telecommunications)BeijingChina

Personalised recommendations