World Wide Web

, Volume 13, Issue 1–2, pp 33–59 | Cite as

Human Intelligence in the Process of Semantic Content Creation



Despite significant progress over the last years the large-scale adoption of semantic technologies is still to come. One of the reasons for this state of affairs is assumed to be the lack of useful semantic content, a prerequisite for almost every IT system or application using semantics. Through its very nature, this content can not be created fully automatically, but requires, to a certain degree, human contribution. The interest of Internet users in semantics, and in particular in creating semantic content, is, however, low. This is understandable if we think of several characteristics exposed by many of the most prominent semantic technologies, and the applications thereof. One of these characteristics is the high barrier of entry imposed. Interacting with semantic technologies today requires specific skills and expertise on subjects which are not part of the mainstream IT knowledge portfolio. A second characteristic are the incentives that are largely missing in the design of most semantic applications. The benefits of using machine-understandable content are in most applications fully decoupled from the effort of creating and maintaining this content. In other words, users do not have a motivation to contribute to the process. Initiatives in the areas of the Social Semantic Web acknowledged this problem, and identified mechanisms to motivate users to dedicate more of their time and resources to participate in the semantic content creation process. Still, even if incentives are theoretically in place, available human labor is limited and must only be used for those tasks that are heavily dependent on human intervention, and cannot be reliably automated. In this article, we concentrate on this step in between. As a first contribution, we analyze the process of semantic content creation in order to identify those tasks that are inherently human-driven. When building semantic applications involving these specific tasks, one has to install incentive schemes that are likely to encourage users to perform exactly these tasks that crucially rely on manual input. As a second contribution of the article, we propose incentives or incentive-driven tools that can be used to increase user interest in semantic content creation tasks. We hope that our findings will be adopted as recommendations for establishing a fundamentally new form of design of semantic applications by the semantic technologies community.


semantic technologies semantic content semantic content creation ontologies annotation alignment human intelligence human factor 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akkiraju, R., Farrell, J., Miller, J., Nagarajan, M., Schmidt, M.T., Sheth, A., Verma, K.: Web service semantics—wsdl-s. Technical Report, W3C Member Submission (2005)Google Scholar
  2. 2.
    Asirelli, P., Little, S., Martinelli, M., Salvetti, O.: Mulimedia metadata management: a proposal for an infrastructure. In: Proceedings of SWAP 2006 (2006)Google Scholar
  3. 3.
    Aussenac-Gilles, N., Despres, S., Szulman, S.: Ontology learning and population: bridging the gap between text and knowledge. Chapter: The TERMINAE method and platform for ontology engineering from texts, pp. 199–223. IOS, Amsterdam (2008)Google Scholar
  4. 4.
    Barras, C., Geoffrois, E., Wu, Z., Liberman, M.,: Transcriber: development and use of a tool for assisting speech corpora production. Speech Communication Special Issue on Speech Annotation and Corpus Tools 33(1–2) (2000)Google Scholar
  5. 5.
    Bernaras, A., Laresgoiti, I., Corera J.: Building and reusing ontologies for electrical network applications. In: European Conference on Articifial Intelligence (ECAI’96) (1996)Google Scholar
  6. 6.
    Bloehdorn, S., Petridis, K., Saathoff, C., Simou, N., Tzouvaras, V., Avrithis, Y., Handschuh, S., Kompatsiaris, Y., Staab, S., Strintzis, M.G.: Semantic annotation of images and videos for multimedia analysis. In: European Semantic Web Conference (ESWC). Springer LNCS, New York (2005)Google Scholar
  7. 7.
    Bontcheva, K., Kiryakov, A., Cunningham, H., Popov, B., Dimitrov, M.: Semantic web enabled, open source language technology. In: EACL Workshop on Language Technology and the Semantic Web: NLP and XML, Budapest, Hungary (2003)Google Scholar
  8. 8.
    Brank, J., Grobelnik, M., Dunja M.: A survey of ontology evaluation techniques. In: Conference on Data Mining and Data Warehouses (SiKDD 2005) (2005)Google Scholar
  9. 9.
    Braun, S., Schmidt, A., Walter, A., Nagypal, G., Zacharias, V.: Ontology maturing: a collaborative web 2.0 approach to ontology engineering. In: Workshop on Social and Collaborative Construction of Structured Knowledge at the 16th International World Wide Web Conference (WWW2007), Banff, Canada, 8 May 2007Google Scholar
  10. 10.
    Buerger, T., Ammendola, C.: A user centered annotation methodology for multimedia content. In: Poster Session at ESWC 2008. CEUR, vol. 367 (2008)Google Scholar
  11. 11.
    Campanini, S.E., Castagna, P., Tazzoli, R.: Platypus wiki: a semantic wiki wiki web. In: International Semantic Web Conference (ISWC), pp. 1–6, 10 December 2004Google Scholar
  12. 12.
    Cantador, I., Fernández, M., Castells, P.: Improving ontology recommendation and reuse in WebCORE by collaborative assessments. In: Proceedings of the 1st International Workshop on Social and Collaborative Construction of Structured Knowledge CKC 2007 at the at the 16th International World Wide Web Conference WWW 2007 (2007)Google Scholar
  13. 13.
    Chakarvarthy, A., Ciravegna, F., Lanfranchi, V.: Cross-media document annotation and enrichment. In: 1st Semantic Authoring and Annotation Workshop (SAAW2006) (2006)Google Scholar
  14. 14.
    Cimiano, P.: Ontology Learning and Population from Text: Algorithms, Evaluation and Applications. Springer, New York (2006)Google Scholar
  15. 15.
    Cimiano, P., Handschuh, S., Staab, S.: Towards the self-annotating web. In: Thirteenth International Conference on World Wide Web (2004)Google Scholar
  16. 16.
    Ciravegna, F., Wilks, Y.: Designing Adaptive Information Extraction for the Semantic web in Amilcare, Volume Annotation for the Semantic Web. IOS, Amsterdam (2003)Google Scholar
  17. 17.
    Ciravegna, F., Dingli, A., Petrelli, D., Wilks, Y.: User-system cooperation in document annotation based on information. In: 13th International Conference on Knowledge Engineering and KM (EKAW02) (2002)Google Scholar
  18. 18.
    De Moor, A., De Leenheer, P., Meersmann, R.: Dogma-mess: A meaning evolution support system for interorganizational ontology engineering. In: 14th International Conference on Conceptual Structures (ICCS 2006). Springer LNCS, New York (2006)Google Scholar
  19. 19.
    Dellschaft, K., Staab, S.: Ontology learning and population: briding the gap between text and knowledge. Chapter: Strategies for the Evaluation of Ontology Learning, pp. 253–272. IOS, Amsterdam (2008)Google Scholar
  20. 20.
    Dill, S., Gibson, N., Gruhl, D., Guha, R.V., Jhingran, A., Kanungo, T., Rajagopalan, S., Tomkins, A., Tomlin, J.A., Zien, J.Y.: Semtag and seeker: bootstrapping the semantic web via automated semantic annotation. In: Twelfth International World Wide Web Conference (2003)Google Scholar
  21. 21.
    Dimitrov, M., Simov, A., Momtchev, V., Konstantinov, M.: WSMO studio — a semantic web services modelling environment for WSMO (system description). In: European Semantic Web Conference (ESWC 2007), Innsbruck, Austria (2007)Google Scholar
  22. 22.
    Dingli, A., Ciravegna, F., Wilks, Y.: Automatic semantic annotation using unsupervised information extraction and integration. In: K-CAP 2003 Workshop on Knowledge Markup and Semantic Annotation (2003)Google Scholar
  23. 23.
    Domingue, J., Dzbor, M., Motta, E.: Magpie: Supporting browsing and navigation on the semantic web. In: ACM Conference on Intelligent User Interfaces (IUI) (2004)Google Scholar
  24. 24.
    Ehrig, M., Sure, Y.: Ontology mapping—an integrated approach. Technical Report, AIFB Karlsruhe (2004)Google Scholar
  25. 25.
    Euzenat, J., Shvaiko, P.: Ontology Matching. Springer, New York (2007)MATHGoogle Scholar
  26. 26.
    Euzenat, J., Mocan, A., Scharffe, F.: Ontology alignments. In: Semantic Web and Beyond, vol. 6. Springer, New York (2007)Google Scholar
  27. 27.
    Falconer, S.M., Storey, M.-A.: A cognitive support framework for ontology mapping. In: Asian Semantic Web Conference (ASWC 2007) (2007)Google Scholar
  28. 28.
    Farrell, J., Lausen, H.: Semantic annotations for wsdl and xml schema. Technical Report, W3C Recommendation (2007)Google Scholar
  29. 29.
    Fellbaum, C.: Wordnet: An Electronic Lexical Database. MIT, Cambridge (1998)MATHGoogle Scholar
  30. 30.
    Fensel, D., Lausen, H., Polleres, A., De, J., Bruijn, Stollberg, M., Roman, D., Domingue, J.: Enabling Semantic Web Services: The Web Service Modeling Ontology. Springer, New York (2006)Google Scholar
  31. 31.
    Fernandez-Lopez, M., Gomez-Perez, A., Juristo, N.: Methontology: from ontological art towards ontological engineering. In: AAAI97 Spring Symposium Series on Ontological Engineering, Stanford, USA (1997)Google Scholar
  32. 32.
    Gangemi, A., Catenacci, C., Ciaramita, M., Lehmann, J.: Modeling ontology evaluation and validation. In: Proceedings of the European Semantic Web Conference (ESWC 2006), pp. 140–154. Springer, New York (2006)Google Scholar
  33. 33.
    Gómez-Pérez, A.: Evaluation of ontologies. Int. J. Intell. Syst. 16(3), 391–409 (2001)MATHCrossRefGoogle Scholar
  34. 34.
    Gomez-Perez, A.: A survey of ontology learning methods and techniques. Technical Report, OntoWeb Deliverable 1.5 (2003)Google Scholar
  35. 35.
    Gomez-Perez, A., Fernandez-Lopez, M., Corcho, O.: Ontological engineering. In: Advanced Information and Knowledge Processing. Springer, New York (2004)Google Scholar
  36. 36.
    Gruber, T.R.: Toward principles for the design of ontologies used for knowledge sharing. Int. J. Human-Comput. Stud. 43, 907–928 (1995)CrossRefGoogle Scholar
  37. 37.
    Gruninger, M., Fox, M.S.: Methodology for the design and evaluation of ontologies. In: Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal, Canada 1995Google Scholar
  38. 38.
    Guarino, N., Welty, C.A.: Evaluating ontological decisions with ontoclean. Commun. ACM 45(2), 61–65 (2002)CrossRefGoogle Scholar
  39. 39.
    Handschuh, S., Staab, S., Ciravegna, F.: S-cream—semi-automatic creation of metadata. In: SAAKM 2002—Semantic Authoring, Annotation & Knowledge Markup (2002)Google Scholar
  40. 40.
    Hemetsberger, A.: When consumers produce on the internet: the relationship between cognitive-affective, socially-based, and behavioral involvement of prosumers. J. Soc. Psychol. (2003)Google Scholar
  41. 41.
    Holsapple, C.W., Joshi, K.D.: A collaborative approach to ontology design. Commun. ACM 45(2), 42–47 (2002)CrossRefGoogle Scholar
  42. 42.
    Jarrar, M., Meersman, R.: Formal ontology engineering in the dogma approach. In: On the Move to Meaningful Internet Systems 2002: CoopIS, DOA, and ODBASE (2002)Google Scholar
  43. 43.
    Kerrigan, M., Mocan, A., Simperl, E., Fensel, D.: Modeling semantic web services with the web service modeling toolkit. Technical Report, Semantic Technology Institute (STI) (2008)Google Scholar
  44. 44.
    Kerrigan, M., Mocan, A., Tanler, M., Fensel, D.: The web service modeling toolkit — an integrated development environment for semantic web services (system description). In: European Semantic Web Conference (ESWC 2007), Innsbruck, Austria (2007)Google Scholar
  45. 45.
    Kiryakov, A., Ognyanov, D., Manov, D.: Owlim—a pragmatic semantic repository for owl. In: Proceedings of International Workshop on Scalable Semantic Web Knowledge Base Systems (SSWS 2005), WISE 2005, New York City, USA, 20 November 2005Google Scholar
  46. 46.
    Kogut, P., Holmes, W.: Applying information extraction to generate daml annotations from web pages. In: First International Conference on Knowledge Capture 2001Google Scholar
  47. 47.
    Kotis, K., Vouros, G.A.: Human-centered ontology engineering: the hcome methodology. Knowl. Inf. Syst. 10(1), 109–131 (2005)CrossRefGoogle Scholar
  48. 48.
    Kuznetsov, S.: Motivations of contributors to wikipedia. ACM SIGCAS Comput. Soc. 36(2) (2006)Google Scholar
  49. 49.
    Lenat, D.B., Guha, R.V.: Building Large Knowledge-based Systems: Representation and Inference in the Cyc Project. Addison-Wesley, Reading (1990)Google Scholar
  50. 50.
    Lozano-Tello, A., Gomez-PerezL, A.: Ontometric: a method to choose the appropriate ontology. J. Database Manage. 15(2) (2004)Google Scholar
  51. 51.
    Lux, M., Becker, J., Krottmaier, H.: Caliph & Emir: semantic annotation and retrieval in personal digital photo libraries. In: CAISE Forum at the 15th Conference on Advanced Information Systems Engineering (2003)Google Scholar
  52. 52.
    Maedche, A., Staab, S.: Ontology learning for the semantic web. IEEE Intell. Syst. 16(2), 72–79 (2001)CrossRefGoogle Scholar
  53. 53.
    Marlow, C., Naaman, M., Boyd, D., Davis, M.: Position paper, tagging, taxonomy, flickr, article, toread. In: In Proceedings of the World Wide Web Conference (WWW2006), Edinburgh, Scotland, 22–26 May 2006. ACM (2006)Google Scholar
  54. 54.
    Martin, D., Burstein, M., Denker, G., Hobbs, J., Kagal, L., Lassila, O., McDermott, D., McIlraith, S., Paolucci, M., Parsia, B., Payne, T., Sabou, M., Sirin, E., Solanki, M., Srinivasan, N., Sycara, K.: Owl-s 1.0 release (2004)Google Scholar
  55. 55.
    Mika, P.: Ontologies are us: a unified model of social networks and semantics. Journal of Web Semantics (Elsevier) 5(1), 5–15 (2007)CrossRefMathSciNetGoogle Scholar
  56. 56.
    Nicola, A.D., Navigli, R., Missikoff, M.: Building an eprocurement ontology with upon methodology. In: Proceedings of the 15th e-Challenges Conferences, Lublijana, Slovenia (2005)Google Scholar
  57. 57.
    Noy, D.L., McGuinness, N.: Ontology development 101: a guide to creating your first ontology. Technical Report, Stanford University, Stanford (2000)Google Scholar
  58. 58.
    Noy, N., Musen, M.: Anchor-prompt: using non-logical context for semantic matching. In: IJCAI Workshop on Ontologies and Information Sharing, pp. 63–70. Seattle (WA US) (2001)Google Scholar
  59. 59.
    Noy, N.F., Musen, M.: The prompt suite: interactive tools for ontology merging and mapping. Int. J. Human-Comput. Stud. 59(6):983–1024 (2003)CrossRefGoogle Scholar
  60. 60.
    Patil, A., Oundhakar, S., Sheth, A., Verma, K.: Meteor-s web service annotation framework. In: World Wide Web Conference (WWW 2004) (2004)Google Scholar
  61. 61.
    Petridis, K., Anastasopoulos, D., Saathoff, C., Timmermann, N., Kompatasiaris, I., Staab, S.: M-ontomat-annotizer: image annotation. Linking ontologies and multimedia low-level features. In: Engineered Applications of Semantic Web Session (SWEA) at the 10th International Conference on Knowledge-Based & Intelligent Information & Engineering Systems (KES 2006) (2006)Google Scholar
  62. 62.
    Pinto, H.S., Martins, J.P.: A methodology for ontology integration. In: International Conference on Knowledge Capture (K-CAP), pp. 131–138. ACM, New York (2001)CrossRefGoogle Scholar
  63. 63.
    Popov, B., Kiryakov, A., Kirilov, A., Manov, D., Ognyanoff, D., Goranov, M.: Kim—semantic annotation platform. In: Symposium on Applied Computing, 20–23 October 2005. LNAI, vol. 2870, pp. 834–849. Springer (2005)Google Scholar
  64. 64.
    Reeve, L., Han, H.: Survey of Semantic Annotation Platforms, pp. 1634–1638. ACM, New York (2005)Google Scholar
  65. 65.
    Schaffert, S.: Ikewiki: a semantic wiki for collaborative knowledge management. In: 1st International Workshop on Semantic Technologies in Collaborative Applications STICA 06 (2006)Google Scholar
  66. 66.
    Simperl, E., Tempich, C., Vrandecic, D.: Ontology learning and population: bridging the gap between text and knowledge. Chapter: A Methodology for Ontology Learning, pp. 225–249. IOS, Amsterdam (2008)Google Scholar
  67. 67.
    Siorpaes, K., Hepp, M.: Games with a purpose for the semantic web. IEEE Intell. Syst. 23(3), 50–60 (2008)CrossRefGoogle Scholar
  68. 68.
    Siorpaes, K., Hepp, M.: OntoGame: weaving the semantic web by online games. In: European Semantic Web Conference (ESWC 2008), Teneriffe, Spain. Springer LNCS (2008)Google Scholar
  69. 69.
    Siorpaes, K., Hepp, M., Klotz, A., Hackl, M.: Myontology: tapping the wisdom of crowds for building ontologies. Technical report, STI Innsbruck Technical Report (2008)Google Scholar
  70. 70.
    Souzis, A.: Building a semantic wiki. IEEE Intell. Syst. 20(5), 87–91 (2005)CrossRefGoogle Scholar
  71. 71.
    Specia, L., Motta, E.: Integrating folksonomies with the semantic web. In: ESWC 2007. Springer, New York (2007)Google Scholar
  72. 72.
    Spyns, P., Tang, Y., Meersman, R.: A model theory inspired collaborative ontology engineering methodology. Technical Report, VUB StarLab (2007)Google Scholar
  73. 73.
    Staab, S., Schnurr, H.P., Studer, R., Sure, Y.: Knowledge processes and ontologies. IEEE Intell. Syst. 16(1), 26–34 (2001)CrossRefGoogle Scholar
  74. 74.
    Sure, Y.: Methodology, tools and case studies for ontology based knowledge management. PhD thesis, University of Karlsruhe (2003)Google Scholar
  75. 75.
    Uren, V., Cimiano, P., Iria, J., Handschuh, S., Vargas-Vera, M., Motta, E., Ciravegna, F.: Semantic annotation for knowledge management: requirements and a survey of the state of the art. Web Semantics: Science, Services and Agents on the World Wide Web 4(1), 14–28 (2006)CrossRefGoogle Scholar
  76. 76.
    Uschold, M., King, M.: Towards a methodology for building ontologies. In: Workshop on Basic Ontological Issues in Knowledge Sharing, Montreal, Canada (1995)Google Scholar
  77. 77.
    Van Damme, C., Hepp, M., Siorpaes, K.: Folksontology: An integrated approach for turning folksonomies into ontologies. In: Proceedings of the Workshop Bridging the Gap between Semantic Web and Web 2.0 at the ESWC 2007, Innsbruck, Austria, 7 June 2007, p. 15. LNCS Springer (2007)Google Scholar
  78. 78.
    Vargas-Vera, M., Motta, E., Domingue, J., Lanzoni, M., Stutt, A., Ciravegna, F.: Mnm: ontology driven semi-automatic and automatic support for semantic markup. In: 13th International Conference on Knowledge Engineering and Management (EKAW2002) (2002)Google Scholar
  79. 79.
    Völkel, M., Krötzsch, M., Vrandecic, D., Haller, H.: Semantic Wikipedia. 23–26 May 2006Google Scholar
  80. 80.
    Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: CHI 2004. ACM, New York (2004)Google Scholar
  81. 81.
    von Ahn, L., Dabbish, L.,: Designing games with a purpose. Commun. ACM 51(8), 58–67 (2008)Google Scholar
  82. 82.
    Vrandecic, D., Pinto, S., Tempich, C., Sure, Y.: The diligent knowledge process. J. Knowl. Manag. 9(5), 85–96 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Semantic Technology Institute (STI) InnsbruckUniversity InnsbruckInnsbruckAustria

Personalised recommendations