Abstract
Crowdsourcing techniques have been shown to provide effective means for solving a variety of ontology engineering problems. Yet, they are mainly being used as external means to ontology engineering, without being closely integrated into the work of ontology engineers. In this paper we investigate how to closely integrate crowdsourcing into ontology engineering practices. Firstly, we show that a set of basic crowdsourcing tasks are used recurrently to solve a range of ontology engineering problems. Secondly, we present the uComp Protégé plugin that facilitates the integration of such typical crowdsourcing tasks into ontology engineering work from within the Protégé ontology editing environment. An evaluation of the plugin in a typical ontology engineering scenario where ontologies are built from automatically learned semantic structures, shows that its use reduces the working times for the ontology engineers 11 times, lowers the overall task costs with 40% to 83% depending on the crowdsourcing settings used and leads to data quality comparable with that of tasks performed by ontology engineers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bontcheva, K., Roberts, I., Derczynski, L., Rout, D.: The GATE Crowdsourcing Plugin: Crowdsourcing Annotated Corpora Made Easy. In: Proc. of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL). ACL (2014)
Brooke, J.: SUS: a quick and dirty usability scale. Taylor & Francis, London (1996)
Demartini, G., Difallah, D.E., Cudré-Mauroux, P.: ZenCrowd: Leveraging Probabilistic Reasoning and Crowdsourcing Techniques for Large-scale Entity Linking. In: Proceedings of the 21st International Conference on World Wide Web, pp. 469–478. ACM (2012)
Eckert, K., Niepert, M., Niemann, C., Buckner, C., Allen, C., Stuckenschmidt, H.: Crowdsourcing the Assembly of Concept Hierarchies. In: Proc. of the 10th Annual Joint Conference on Digital Libraries, JCDL 2010, pp. 139–148. ACM (2010)
Howe, J.: Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business (2009), http://crowdsourcing.typepad.com/
Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing User Studies with Mechanical Turk. In: Proc. of the 26th Conference on Human Factors in Computing Systems, pp. 453–456 (2008)
Landis, J., Koch, G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)
Laws, F., Scheible, C., Schütze, H.: Active Learning with Amazon Mechanical Turk. In: Proc. of the Conf. on Empirical Methods in NLP, pp. 1546–1556 (2011)
Markotschi, T., Völker, J.: Guess What?! Human Intelligence for Mining Linked Data. In: Proc. of the Workshop on Knowledge Injection into and Extraction from Linked Data at the International Conference on Knowledge Engineering and Knowledge Management, EKAW-2010 (2010)
Noy, N.F., Mortensen, J., Musen, M.A., Alexander, P.R.: Mechanical Turk As an Ontology Engineer?: Using Microtasks As a Component of an Ontology-engineering Workflow. In: Proceedings of the 5th Annual ACM Web Science Conference, WebSci 2013, pp. 262–271. ACM (2013)
Poesio, M., Kruschwitz, U., Chamberlain, J., Robaldo, L., Ducceschi, L.: Phrase Detectives: Utilizing Collective Intelligence for Internet-Scale Language Resource Creation. Transactions on Interactive Intelligent Systems 3(1), 1–44 (2013)
Sabou, M., Bontcheva, K., Scharl, A.: Crowdsourcing Research Opportunities: Lessons from Natural Language Processing. In: Proc. of the 12th International Conference on Knowledge Management and Knowledge Technologies (iKNOW). Special Track on Research 2.0 (2012)
Sabou, M., Bontcheva, K., Scharl, A., Föls, M.: Games with a Purpose or Mechanised Labour?: A Comparative Study. In: Proc. of the 13th International Conference on Knowledge Management and Knowledge Technologies, i-Know 2013, pp. 1–8. ACM (2013)
Sabou, M., Scharl, A., Föls, M.: Crowdsourced Knowledge Acquisition: Towards Hybrid-genre Workflows. International Journal of Semantic Web and Information Systems 9(3), 14–41 (2013)
Sarasua, C., Simperl, E., Noy, N.F.: CrowdMap: Crowdsourcing Ontology Alignment with Microtasks. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012, Part I. LNCS, vol. 7649, pp. 525–541. Springer, Heidelberg (2012)
Scharl, A., Sabou, M., Föls, M.: Climate Quiz: a Web Application for Eliciting and Validating Knowledge from Social Networks. In: Proceedings of the 18th Brazilian Symposium on Multimedia and the Web, WebMedia 2012, pp. 189–192. ACM (2012)
Siorpaes, K., Hepp, M.: Games with a Purpose for the Semantic Web. IEEE Intelligent Systems 23(3), 50–60 (2008)
Thaler, S., Simperl, E., Siorpaes, K.: SpotTheLink: Playful Alignment of Ontologies. In: Proceedings of the 2011 ACM Symposium on Applied Computing, pp. 1711–1712. ACM (2011)
Thaler, S., Simperl, E., Wölger, S.: An Experiment in Comparing Human-Computation Techniques. IEEE Internet Computing 16(5), 52–58 (2012)
von Ahn, L., Dabbish, L.: Designing games with a purpose. ACM Commun. 51(8), 58–67 (2008)
Waitelonis, J., Ludwig, N., Knuth, M., Sack, H.: WhoKnows? Evaluating Linked Data Heuristics with a Quiz that Cleans Up DBpedia. Interact. Techn. Smart Edu. 8(4), 236–248 (2011)
Wohlgenannt, G., Weichselbraun, A., Scharl, A., Sabou, M.: Dynamic Integration of Multiple Evidence Sources for Ontology Learning. Journal of Information and Data Management 3(3), 243–254 (2012)
Wolf, L., Knuth, M., Osterhoff, J., Sack, H.: RISQ! Renowned Individuals Semantic Quiz - a Jeopardy like Quiz Game for Ranking Facts. In: Proc. of the 7th International Conference on Semantic Systems, I-Semantics 2011, pp. 71–78. ACM (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Hanika, F., Wohlgenannt, G., Sabou, M. (2014). The uComp Protégé Plugin: Crowdsourcing Enabled Ontology Engineering. In: Janowicz, K., Schlobach, S., Lambrix, P., Hyvönen, E. (eds) Knowledge Engineering and Knowledge Management. EKAW 2014. Lecture Notes in Computer Science(), vol 8876. Springer, Cham. https://doi.org/10.1007/978-3-319-13704-9_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-13704-9_14
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-13703-2
Online ISBN: 978-3-319-13704-9
eBook Packages: Computer ScienceComputer Science (R0)