Skip to main content
SpringerLink
Log in

A context-aware approach for trustworthy worker selection in social crowd

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Crowdsourcing applications like Amazon Mechanical Turk (AMT) make it possible to address many difficult tasks (e.g., image tagging and sentiment analysis) on the internet and make full use of the wisdom of crowd, where worker quality is one of the most crucial issues for the task owners. Thus, a challenging problem is how to effectively and efficiently select the high quality workers, so that the tasks online can be accomplished successfully under a certain budget. The existing methods on the crowd worker selection problem mainly based on the quality measurement of the crowd workers, those who have to register on the crowdsourcing platforms. With the connect of the OSNs and the crowdsourcing applications, the social contexts like social relationships and social trust between participants and social positions of participants can assist requestors to select one or a group of trustworthy crowdsourcing workers. In this paper, we first present a contextual social network structure and a concept of Strong Social Component (SSC), which emblems a group of workers who have high social contexts values. Then, we propose a novel index for SSC, and a new efficient and effective algorithm C-AWSA to find trustworthy workers, who can complete the tasks with high quality. The results of our experiments conducted on four real OSN datasets illustrate that the superiority of our method in trustworthy worker selection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15

Similar content being viewed by others

Notes

  1. http://mturk.com.

  2. http://www.crowdflower.com.

  3. http://www.cs.cmu.edu/~enron/

  4. http://www.informatik.uni-trier.de/~ley/db/

References

  1. Adler, P.S.: Market, hierarchy, and trust: the knowledge economy and the future of capitalism. Organ. Sci. 12(2), 215–234 (2001)

    Article  Google Scholar 

  2. Berger, P.L., Luckmann, T.: The social construction of reality: A treatise in the sociology of knowledge, Penguin (1991). no. 10

  3. Biggs, N., Lloyd, E., Wilson, R.: Graph Theory. Oxford University Press (1986)

  4. Brabham, D.C.: Crowdsourcing as a model for problem solving an introduction and cases. Convergence: the International Journal of Research Into New Media Technologies 14(1), 75–90 (2008)

    Article  Google Scholar 

  5. Cao, C.C., She, J., Tong, Y., Chen, L.: Whom to ask?: jury selection for decision making tasks on micro-blog services. Proceedings of the VLDB Endowment 5 (11), 1495–1506 (2012)

    Article  Google Scholar 

  6. Cao, C.C., Tong, Y., Chen, L., Jagadish, H.: Wisemarket: a new paradigm for managing wisdom of online social users. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 455–463. ACM (2013)

  7. Carpenter, B.: A hierarchical bayesian model of crowdsourced relevance coding. In: TREC (2011)

  8. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. J. R. Stat. Soc. Ser. B Methodol., 1–38 (1977)

  9. Feng, A., Franklin, M., Kossmann, D., Kraska, T., Madden, S. R., Ramesh, S., Wang, A., Xin, R. : Crowddb: Query processing with the vldb crowd (2011)

  10. Forlines, C., Miller, S., Guelcher, L., Bruzzi, R.: Crowdsourcing the future: predictions made with a social network. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pp 3655–3664. ACM (2014)

  11. Gao, J., Liu, X., Ooi, B.C., Wang, H., Chen, G.: An online cost sensitive decision-making method in crowdsourcing systems. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp 217–228. ACM (2013)

  12. Gentle, J.E., Härdle, W.K., Mori, Y.: Handbook of computational statistics: concepts and methods. Springer Science & Business Media (2012)

  13. Gupta, M.R., Chen, Y.: Theory and use of the EM algorithm. Now Publishers Inc (2011)

  14. Hirth, M., Scheuring, S., Hossfeld, T., Schwartz, C., Tran-Gia, P.: Predicting result quality in crowdsourcing using application layer monitoring. In: 2014 IEEE Fifth International Conference on Communications and Electronics (ICCE), pp 510–515. IEEE (2014)

  15. Howe, J.: The rise of crowdsourcing. Wired Mag. 14(6), 1–4 (2006)

    Google Scholar 

  16. Hung, N.Q.V., Tam, N.T., Miklós, Z., Aberer, K.: On leveraging crowdsourcing techniques for schema matching networks. In: Database Systems for Advanced Applications, pp 139–154. Springer (2013)

  17. Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp 64–67. ACM (2010)

  18. Kelley, H.H., Berscheid, E., Christensen, A., Harvey, J.H., Huston, T.L., Levinger, G., McClintock, E., Peplau, L.A., Peterson, D.R.: Analyzing close relationships. Close Relationships 20, 67 (1983)

    Google Scholar 

  19. Korkmaz, T., Krunz, M.: Multi-constrained optimal path selection. In: INFOCOM 2001. Twentieth Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings, vol. 2, pp 834–843. IEEE (2001)

  20. Li, H., Liu, Q.: Cheaper and better: Selecting good workers for crowdsourcing (2015). arXiv preprint arXiv:1502.00725

  21. Li, L., Wang, Y., Lim, E.-P.: Trust-oriented composite service selection and discovery. In: Service-Oriented Computing, pp 50–67. Springer (2009)

  22. Li, H., Zhao, B., Fuxman, A.: The wisdom of minority: discovering and targeting the right group of workers for crowdsourcing. In: Proceedings of the 23rd international conference on World Wide Web. International World Wide Web Conferences Steering Committee, pp 165–176 (2014)

  23. Liu, G., Wang, Y., Orgun, M.A., Lim, E.-P.: A heuristic algorithm for trust-oriented service provider selection in complex social networks. In: 2010 IEEE International Conference on Services Computing (SCC), pp 130–137. IEEE (2010)

  24. Liu, G., Wang, Y., Orgun, M.A., et al.: Optimal social trust path selection in complex social networks. In: AAAI, vol. 10, pp 1397–1398 (2010)

  25. Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: Cdas: a crowdsourcing data analytics system. Proceedings of the VLDB Endowment 5(10), 1040–1051 (2012)

  26. Liu, Q., Ihler, A.T., Steyvers, M.: Scoring workers in crowdsourcing: How many control questions are enough?. In: Advances in Neural Information Processing Systems, pp 1914–1922 (2013)

  27. Liu, G., Liu, A., Wang, Y., Li, L.: An efficient multiple trust paths finding algorithm for trustworthy service provider selection in real-time online social network environments. In: 2014 IEEE International Conference on Web Services (ICWS), pp 121–128. IEEE (2014)

  28. Malone, T.W., Laubacher, R., Dellarocas, C.: Harnessing crowds: mapping the genome of collective intelligence (2009)

  29. Mansell, R., Collins, B.: Trust and crime in information societies. Edward Elgar Publishing (2005)

  30. Marcus, A., Wu, E., Karger, D., Madden, S., Miller, R.: Human-powered sorts and joins. Proceedings of the VLDB Endowment 5(1), 13–24 (2011)

    Article  Google Scholar 

  31. Miller, R.: Intimate relationships. McGraw-Hill Higher Education (2007)

  32. Mislove, A., Marcon, M., Gummadi, P.K., Druschel, P., Bhattacharjee, B.: Measurement and analysis of online social networks. In: Internet Measurement Conference, pp 29–42 (2007)

  33. Parameswaran, A.G., Garcia-Molina, H., Park, H., Polyzotis, N., Ramesh, A., Widom, J.: Crowdscreen: Algorithms for filtering data with humans. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp 361–372. ACM (2012)

  34. Park, H., Garcia-Molina, H., Pang, R., Polyzotis, N., Parameswaran, A., Widom, J.: Deco: A system for declarative crowdsourcing. Proceedings of the VLDB Endowment 5(12), 1990–1993 (2012)

    Article  Google Scholar 

  35. Sun, Z., Wang, H., Wang, H., Shao, B., Li, J.: Efficient subgraph matching on billion node graphs. In: VLDB’12, pp 788–799

  36. Tang, J., Zhang, J., Yan, L., Li, J., Zhang, L., Su, Z.: Arnetminer: Extraction and mining of academic social networks. In: KDD’08, pp 990–998

  37. Von Ahn, L., Maurer, B., McMillen, C., Abraham, D., Blum, M.: Recaptcha: Human-based character recognition via Web security measures. Science 321 (5895), 1465–1468 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  38. Wang, Y., Varadharajan, V.: Role-based recommendation and trust evaluation. In: E-Commerce Technology and the 4th IEEE International Conference on Enterprise Computing, E-Commerce, and E-Services, 2007. The 9th IEEE International Conference on CEC/EEE 2007, pp 278–288. IEEE (2007)

  39. Wang, J., Kraska, T., Franklin, M.J., Feng, J.: Crowder: Crowdsourcing entity resolution. Proceedings of the VLDB Endowment 5(11), 1483–1494 (2012)

    Article  Google Scholar 

  40. Whang, S.E., Lofgren, P., Garcia-Molina, H.: Question selection for crowd entity resolution. Proceedings of the VLDB Endowment 6(6), 349–360 (2013)

    Article  Google Scholar 

  41. Wang, J., Li, G., Kraska, T., Franklin, M.J., Feng, J.: Leveraging transitive relations for crowdsourced joins. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp 229–240. ACM (2013)

  42. Yang, X.S., Cheng, R., Mo, L., Kao, B., Cheung, D.W.: On incentive-based tagging. In: 2013 IEEE 29th International Conference on Data Engineering (ICDE), pp 685–696. IEEE (2013)

  43. Yang, S., Zhang, J., Chen, I.: Web 2.0 services for identifying communities of practice. In: SCC’07, pp 130–137

  44. Ye, B., Wang, Y., Liu, L.: Crowd trust: a context-aware trust model for worker selection in crowdsourcing environments. In: 2015 IEEE International Conference on Web Services (ICWS), pp 121–128. IEEE (2015)

  45. Zhang, C.J., Chen, L., Jagadish, H., Cao, C.C.: Reducing uncertainty of schema matching via crowdsourcing. Proceedings of the VLDB Endowment 6(9), 757–768 (2013)

    Article  Google Scholar 

  46. Zhao, Z., Wei, F., Zhou, M., Chen, W., Ng, W.: Crowd-selection query processing in crowdsourcing databases: A task-driven approach. In: International Conference on Extending Database Technology (EDBT 2015), pp. 397–408, Brussels (2015)

  47. Zheng, Y., Cheng, R., Maniu, S., Mo, L.: On optimality of jury selection in crowdsourcing. In: 18th International Conference on Extending Database Technology (EDBT 2015), Brussels (2015)

  48. Zhu, Y., Qin, L., Xu, J.X., Cheng, H.: Finding top-k similar graphs in graph databases. In: EDBT’12, pp 456–467

Download references

Acknowledgments

This work was partially supported by Natural Science Foundation of China (Grant Nos. 61303019, 61572336, 61532018, 61402313, 61502324), Doctoral Fund of Ministry of Education of China (20133201120012), Postdoctoral Science Foundation of China (2015M571805, 2016T90492), Collaborative Innovation Center of Novel Software Technology and Industrialization, Jiangsu, China, and Open Foundation of State Key Laboratory of Networking and Switching Technology (Beijing University of Posts and Telecommunications (SKLNST-2016-2-02).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Soochow University, Suzhou, China

    Yang Zhao, Guanfeng Liu, Kai Zheng, An Liu, Zhixu Li & Xiaofang Zhou

  2. School of Information Technology and Electrical Engineering, Queensland University, St Lucia, QLD, 4072, Australia

    Xiaofang Zhou

  3. State Key Laboratory of Networking and Switching Technology (Beijing University of Posts and Telecommunications), Beijing, China

    Guanfeng Liu

Authors
  1. Yang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guanfeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kai Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. An Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhixu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaofang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanfeng Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y., Liu, G., Zheng, K. et al. A context-aware approach for trustworthy worker selection in social crowd. World Wide Web 20, 1211–1235 (2017). https://doi.org/10.1007/s11280-016-0429-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-016-0429-6

Keywords

Search

Navigation