Abstract
Crowdsourcing continues to gain more momentum as its potential becomes more recognized. Nevertheless, the associated quality aspect remains a valid concern, which introduces uncertainty in the results obtained from the crowd. We identify the different aspects that dynamically affect the overall quality of a crowdsourcing task. Accordingly, we propose a skill ontology-based model that caters for these aspects, as a management technique to be adopted by crowdsourcing platforms. The model maintains a dynamically evolving ontology of skills, with libraries of standardized and personalized assessments for awarding workers skills. Aligning a worker’s set of skills to that required by a task, boosts the ultimate resulting quality. We visualize the model’s components and workflow, and consider how to guard it against malicious or unqualified workers, whose responses introduce this uncertainty and degrade the overall quality.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Surowiecki, J.: The Wisdom of Crowds, p. 336. Anchor, New York (2005)
Howe, J.: The rise of crowdsourcing. North 14(14), 1–5 (2006)
Brabham, D.C.: Crowdsourcing as a model for problem solving: an introduction and cases. Convergence Int J. Res. New Media Technol. 14(1), 75–90 (2008)
Kamps, J., Geva, S., Peters, C., Sakai, T., Trotman, A., Voorhees, E.: Report on the SIGIR 2009 workshop on the future of IR evaluation. ACM SIGIR Forum 43(2), 13 (2009)
Zhu, D., Carterette, B.: An analysis of assessor behavior in crowdsourced preference judgments. In: SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, no. Cse, pp. 17–20 (2010)
Scoring Workers in Crowdsourcing: How Many Control Questions are Enough?.pdf, 2013
Lofi, C., Selke, J., Balke, W.-T.: Information extraction meets crowdsourcing: a promising couple. Datenbank-Spektrum 12(1), 109–120 (2012)
Kuncheva, L.I., Whitaker, C.J., Shipp, C.A., Duin, R.P.W.: Limits on the majority vote accuracy in classifier fusion. Pattern Anal. Appl. 6(1), 22–31 (2003)
Kazai, G.: In search of quality in crowdsourcing for search engine evaluation. SIGIR Forum 44(2), 165–176 (2011)
Mason, W., Watts, D.J.: Financial incentives and the ‘performance of crowds’. ACM SIGKDD Explor. Newslett. 11(2), 100 (2010)
Brabham, D.C.: Moving the crowd at threadless. Inf. Commun. Soc. 13(8), 1122–1145 (2010)
PodCastle: Collaborative training of acoustic models on the basis of wisdom of crowds for podcast transcription, (2009). https://staff.aist.go.jp/m.goto/PAPER/INTERSPEECH2009ogata.pdf
Goto, M., Ogata, J.: Podcastle: recent advances of a spoken document retrieval service improved by anonymous user contributions. In: Proceedings of the 12th Annual Conference of the International Speech Communication Association (Interspeech 2011), pp. 3073–3076 (2011)
Schall, D.: Service-Oriented Crowdsourcing: Architecture, Protocols and Algorithms, p. 105. Springer, New York (2012)
Lai, C.: Endorsements, licensing, and insurance for distributed system services. J. Electron. Publishing 2(1) (1996)
Ludwig, H., Keller, A., Dan, A., King, R.: A service level agreement language for dynamic electronic services. In: Proceedings of 4th IEEE International Workshop on Advanced Issues of E-Commerce and Web-Based Information Systems (WECWIS 2002) (2002)
Sahai, A., Machiraju, V., Anna, D.: Towards automated SLA management for web services (2002). http://www.hpl.hp.com/techreports/2001/HPL-2001-310R1.pdf
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Roy. Stat. Soc.: Ser. C (Appl. Stat.) 28(1), 20–28 (1979)
Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.: Learning from crowds. J. Mach. Learn. Res. 11, 1297–1322 (2010)
Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., Movellan, J.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise. Adv. Neural Inf. Process. Syst. 22(1), 1–9 (2009)
Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64–67. ACM, New York (2010)
Campion, M.A., Fink, A.A., Ruggeberg, B.J., Carr, L., Phillips, G.M., Odman, R.B.: Doing competencies well: best practices in competency modeling. Pers. Psychol. 64(1), 225–262 (2011)
Shippmann, J.S., Ash, R.A., Battista, M., Carr, L., Eyde, L.D., Hesketh, B., Kehoe, J., Pearlman, K., Prien, E.P., Sanchez, J.I.: The practice of competency modeling. Pers. Psychol. 53, 703–740 (2000)
De Coi, J.L., Herder, E., Koesling, A., Lofi, C., Olmedilla, D., Papapetrou, O., Siberski, W.: A model for competence gap analysis. In: WEBIST 2007: Proceedings of the 3rd International Conference on Web Information Systems and Technologies, pp. 304–312 (2007)
Colucci, S., Di Noia, T., Di Sciascio, E., Donini, F.M., Mongiello, M., Mottola, M.: A formal approach to ontology-based semantic match of skills descriptions. J. Univ. Comput. Sci. 9(12), 1437–1454 (2003)
Koeppen, K., Hartig, J., Klieme, E., Leutner, D.: Current issues in competence modeling and assessment. Zeitschrift für Psychologie/J. Psychol. 216(2), 61–73 (2008)
Allahbakhsh, M., Benatallah, B., Ignjatovic, A., Motahari-Nezhad, H.R., Bertino, E., Dustdar, S.: Quality control in crowdsourcing systems: issues and directions. IEEE Internet Comput. 17(2), 76–81 (2013)
Allahbakhsh, M., Ignjatovic, A., Benatallah, B., Foo, N., Beheshti, S.M.R., Bertino, E.: Reputation management in crowdsourcing systems (2012)
Ignjatovic, A., Foo, N., Lee, C.T.: An analytic approach to reputation ranking of participants in online transactions. In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, vol. 1 (2008)
Noorian, Z., Ulieru, M.: The state of the art in trust and reputation systems: a framework for comparison. J. Theor. Appl. Electron. Commer. Res. 5(2), 97–117 (2010)
Liu, X., Song, Y., Liu, S., Wang, H.: Automatic taxonomy construction from keywords. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp. 1433–1441 (2012)
Acknowledgments
We’d like to thank the organizers of NII Shonan 2013 meeting for Intelligent Information Processing - Chances of Crowdsourcing, which spurred this work. We’d also like to thank the reviewers for their careful examination and insightful comments and remarks, which we tried to adopt for improvements.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Maarry, K.E., Balke, WT., Cho, H., Hwang, Sw., Baba, Y. (2014). Skill Ontology-Based Model for Quality Assurance in Crowdsourcing. In: Han, WS., Lee, M., Muliantara, A., Sanjaya, N., Thalheim, B., Zhou, S. (eds) Database Systems for Advanced Applications. DASFAA 2014. Lecture Notes in Computer Science(), vol 8505. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-43984-5_29
Download citation
DOI: https://doi.org/10.1007/978-3-662-43984-5_29
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-43983-8
Online ISBN: 978-3-662-43984-5
eBook Packages: Computer ScienceComputer Science (R0)