Semantic Enrichment by Non-experts: Usability of Manual Annotation Tools

  • Annika Hinze
  • Ralf Heese
  • Markus Luczak-Rösch
  • Adrian Paschke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7649)


Most of the semantic content available has been generated automatically by using annotation services for existing content. Automatic annotation is not of sufficient quality to enable focused search and retrieval: either too many or too few terms are semantically annotated. User-defined semantic enrichment allows for a more targeted approach. We developed a tool for semantic annotation of digital documents and conducted an end-user study to evaluate its acceptance by and usability for non-expert users. This paper presents the results of this user study and discusses the lessons learned about both the semantic enrichment process and our methodology of exposing non-experts to semantic enrichment.


User Study Annotation Tool Semantic Annotation Semantic Technology Semantic Enrichment 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Luczak-Rösch, M., Heese, R.: Linked data authoring for non-experts. In: Linked Data on the Web Workshop, World Wide Web Conference (April 2009)Google Scholar
  2. 2.
    Thomson Reuters Inc.: Open Calais website,
  3. 3.
    Zemanta Ltd.: Zemanta (2012),
  4. 4.
    Yesilada, Y., Bechhofer, S., Horan, B.: Cohse: Dynamic linking of web resources. Technical Report TR-2007-167, Sun Microsystems (August 2007)Google Scholar
  5. 5.
    Kalyanpur, A., Hendler, J., Parsia, B., Golbeck, J.: SMORE - semantic markup, ontology, and RDF editor. Technical Report ADA447989, Maryland University (2006)Google Scholar
  6. 6.
    McDowell, L., Etzioni, O., Gribble, S.D., Halevy, A., Levy, H., Pentney, W., Verma, D., Vlasseva, S.: Mangrove: Enticing Ordinary People onto the Semantic Web via Instant Gratification. In: Fensel, D., Sycara, K., Mylopoulos, J. (eds.) ISWC 2003. LNCS, vol. 2870, pp. 754–770. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  7. 7.
    Handschuh, S., Staab, S.: Cream: Creating metadata for the semantic web. Computer Networks 42(5), 579–598 (2003)zbMATHCrossRefGoogle Scholar
  8. 8.
    Booktate: Booktate website (2012),
  9. 9.
    Textensor Limited: A.nnotate (2012),
  10. 10.
    Olive Tree Bible Software, Inc.: Bible+ (2012), available online at
  11. 11.
    Khalili, A., Auer, S.: The RDFa content editor - from WYSIWYG to WYSIWYM (2011),
  12. 12.
    Morbidoni, C.: SWickyNotes Starting Guide. Net7 and Universita Politecnica delle Marche (April 2012),
  13. 13.
    Dello, K., Simperl, E.P.B., Tolksdorf, R.: Creating and using semantic web information with makna. In: First Workshop on Semantic Wikis - From Wiki to Semantics (2006)Google Scholar
  14. 14.
    Auer, S., Dietzold, S., Riechert, T.: OntoWiki – A Tool for Social, Semantic Collaboration. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 736–749. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  15. 15.
    Schaffert, S.: Ikewiki: A semantic wiki for collaborative knowledge management. In: Workshop on Semantic Technologies in Collaborative Applications, Manchester, UK (June 2006)Google Scholar
  16. 16.
    Hinze, A., Voisard, A., Buchanan, G.: Tip: Personalizing information delivery in a tourist information system. J. of IT & Tourism 11(3), 247–264 (2009)Google Scholar
  17. 17.
    Kruk, S.R., Woroniecki, T., Gzella, A., Dabrowski, M.: JeromeDL - a semantic digital library. In: Semantic Web Challenge (2007)Google Scholar
  18. 18.
    faviki: faviki - tags that make sense,
  19. 19.
    Passant, A.: MOAT-project,
  20. 20.
    Teufel, S.: Argumentative Zoning: Information Extraction from Scientific Text. PhD thesis, University of Edinburgh (1999)Google Scholar
  21. 21.
    Handschuh, S.: Creating Ontology-based Metadata by Annotation for the Semantic Web. Ph.D. thesis (dr. rer. pol.), University of Karlsruhe, TH (2005)Google Scholar
  22. 22.
    Bayerl, P.S., Lüngen, H., Gut, U., Paul, K.I.: Methodology for reliable schema development and evaluation of manual annotations. In: Workshop on Knowledge Markup and Semantic Annotation, pp. 17–23 (2003)Google Scholar
  23. 23.
    Erdmann, M., Maedche, A., Schnurr, H., Staab, S.: From manual to semi-automatic semantic annotation: About ontology-based text annotation tools. Group 6(i), 79–91 (2000)Google Scholar
  24. 24.
    Maynard, D., Dasiopoulou, S., et al.: D1. Benchmarking of annotation tools. Technical report, Knowledge Web Project (2007)Google Scholar
  25. 25.
    Castro, R.: Benchmarking Semantic Web technology. Studies on the Semantic Web. IOS Press (2010)Google Scholar
  26. 26.
    Degler, D., Henninger, S., Battle, L.: Semantic Web HCI: discussing research implications. In: Extended Abstracts on Human Factors in Computing Systems, pp. 1909–1912. ACM (2007)Google Scholar
  27. 27.
    Schraefel, M., Karger, D.: The pathetic fallacy of rdf. In: International Workshop on the Semantic Web and User Interaction (SWUI 2006) (2006)Google Scholar
  28. 28.
    Uren, V., Cimiano, P., et al.: Semantic annotation for knowledge management: Requirements and a survey of the state of the art. Web Semant. 4(1), 14–28 (2006)CrossRefGoogle Scholar
  29. 29.
    Karger, D.: Unference: Ui (not ai) as key to the semantic web. Panel on Interaction Design Grand Challenges and the Semantic Web at Semantic Web User Interaction Workshop (2006),
  30. 30.
    Jameson, A.: Usability and the Semantic Web. In: Sure, Y., Domingue, J. (eds.) ESWC 2006. LNCS, vol. 4011, pp. 3–3. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  31. 31.
    Duineveld, A.J., Stoter, R., et al.: Wondertools? a comparative study of ontological engineering tools. Journal of Human-Computer Studies 52(6), 1111–1133 (2000)CrossRefGoogle Scholar
  32. 32.
    Flouris, G., Manakanatas, D., Kondylakis, H., Plexousakis, D., Antoniou, G.: Ontology change: Classification and survey. Knowl. Eng. Rev. 23(2), 117–152 (2008)CrossRefGoogle Scholar
  33. 33.
    Maedche, A., Staab, S.: Ontology learning for the semantic web. IEEE Intelligent Systems 16(2), 72–79 (2001)CrossRefGoogle Scholar
  34. 34.
    Jones, W., Karger, D., Bergman, O., Franklin, M., Pratt, W., Bates, M.: Towards a Unification & Integration of PIM support. Technical report, University of Washington (2005)Google Scholar
  35. 35.
    Furst, F., Kuntz, P., Trichet, F.: Conceptual and Lexical Prototypicality Gradients Dedicated to Ontology Personalisation. In: Meersman, R., Tari, Z. (eds.) OTM 2008, Part II. LNCS, vol. 5332, pp. 1423–1439. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  36. 36.
    Hogben, G., Wilikens, M., Vakalis, I.: On the Ontology of Digital Identification. In: Meersman, R. (ed.) OTM Workshops 2003. LNCS, vol. 2889, pp. 579–593. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  37. 37.
    Handschuh, S., Staab, S.: Authoring and annotation of web pages in cream. In: WWW, pp. 462–473 (2002)Google Scholar
  38. 38.
    Heese, R., Luczak-Rösch, M., Paschke, A., Oldakowski, R., Streibel, O.: One click annotation. In: Workshop on Scripting and Development for the Semantic Web (May 2010)Google Scholar
  39. 39.
    Degler, D.: Design 10:5:2 for semantic applications. In: Semantic Technology Conference (2011),
  40. 40.
    Snyder, C.: Paper prototyping: The fast and easy way to design and refine user interfaces. Morgan Kaufmann Pub. (2003)Google Scholar
  41. 41.
  42. 42.
    Schlegel, A., Heese, R., Hinze, A.: Visualisation of semantic enrichment. In: Interaction and Visualisation in the Data Web, Workshop at Informatik 2012 (2012)Google Scholar
  43. 43.
    Hinze, A., Heese, R., Schlegel, A., Luczak-Rösch, M.: User-Defined Semantic Enrichment of Full-Text Documents: Experiences and Lessons Learned. In: Zaphiris, P., Buchanan, G., Rasmussen, E., Loizides, F. (eds.) TPDL 2012. LNCS, vol. 7489, pp. 209–214. Springer, Heidelberg (2012)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Annika Hinze
    • 1
  • Ralf Heese
    • 2
  • Markus Luczak-Rösch
    • 2
  • Adrian Paschke
    • 2
  1. 1.University of WaikatoNew Zealand
  2. 2.Freie Universität BerlinGermany

Personalised recommendations