A Controlled Crowdsourcing Approach for Practical Ontology Extensions and Metadata Annotations

  • Yolanda GilEmail author
  • Daniel Garijo
  • Varun Ratnakar
  • Deborah Khider
  • Julien Emile-Geay
  • Nicholas McKay
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10588)


Traditional approaches to ontology development have a large lapse between the time when a user using the ontology has found a need to extend it and the time when it does get extended. For scientists, this delay can be weeks or months and can be a significant barrier for adoption. We present a new approach to ontology development and data annotation enabling users to add new metadata properties on the fly as they describe their datasets, creating terms that can be immediately adopted by others and eventually become standardized. This approach combines a traditional, consensus-based approach to ontology development, and a crowdsourced approach where expert users (the crowd) can dynamically add terms as needed to support their work. We have implemented this approach as a socio-technical system that includes: (1) a crowdsourcing platform to support metadata annotation and addition of new terms, (2) a range of social editorial processes to make standardization decisions for those new terms, and (3) a framework for ontology revision and updates to the metadata created with the previous version of the ontology. We present a prototype implementation for the Paleoclimate community, the Linked Earth Framework, currently containing 700 datasets and engaging over 50 active contributors. Users exploit the platform to do science while extending the metadata vocabulary, thereby producing useful and practical metadata.


Metadata Crowdsourcing Semantic wiki Collaborative ontology engineering Semantic science Incremental vocabulary development 



We gratefully acknowledge funding from the US National Science Foundation under EarthCube grant ICER-1541029 and under grant IIS-1344272. We would like to thank the paleoclimate scientists who are participating in this community effort. We also thank Chris Duffy, Paul Hanson, Jie Ji, Tejal Patted, and Neha Suvarna for their contributions to the project.


  1. 1.
    Ahn, S., Khider, D., Lisiecki, L., Lawrence, C.E.: A probabilistic Pliocene-Pleistocene stack of benthic δ18O using a profile hidden Markov model. Dyn. Stat. Clim. Syst. (2017). doi: 10.1093/climsys/dzx002 Google Scholar
  2. 2.
    Auer, S., Dietzold, S., Riechert, T.: OntoWiki – a tool for social, semantic collaboration. In: Proceedings of International Semantic Web Conference (2006)Google Scholar
  3. 3.
    Bry, F., Schaffert, S., Vrandečić, D., Weiand, K.: Semantic Wikis: approaches, applications, and perspectives. In: Eiter, T., Krennwallner, T. (eds.) Reasoning Web 2012. LNCS, vol. 7487, pp. 329–369. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33158-9_9 CrossRefGoogle Scholar
  4. 4.
    Emile-Geay, J., McKay, N.P.: Paleoclimate data standards. Pages Mag. 24, 1 (2016). doi: 10.22498/pages.24.1.47 CrossRefGoogle Scholar
  5. 5.
    Evans, M.N., Tolwinski-Ward, S.E., Thompson, D.M., Anchukaitis, K.J.: Applications of proxy system modeling in high resolution paleoclimatology. Quatern. Sci. Rev. 76, 16–28 (2013). doi: 10.1016/j.quascirev.2013.05.024 CrossRefGoogle Scholar
  6. 6.
    Font, F., Serrà, J., Serra, X.: Analysis of the impact of a tag recommendation system in a real-world Folksonomy. ACM Trans. Intell. Syst. Technol. 7(1), 6 (2016)Google Scholar
  7. 7.
    Garijo, D.: WIDOCO: a wizard for documenting ontologies. In: Proceedings of 16th International Semantic Web Conference (ISWC) (2017)Google Scholar
  8. 8.
    Ghidini, C., Kump, B., Lindstaedt, S., Mabhub, N., Pammer, V., Rospocher, M., Serafini, L.: MoKi: the enterprise modelling Wiki. In: Proceedings of 6th Annual European Semantic Web Conference (ESWC) (2009)Google Scholar
  9. 9.
    Gil, Y., Michel, F., Ratnakar, V., Hauder, M.: Organic data science: a task-centered interface to on-line collaboration in science. In: Proceedings of ACM International Conference on Intelligent User Interfaces (IUI) (2015)Google Scholar
  10. 10.
    Gil, Y., Ratnakar, V.: Knowledge capture in the wild: a perspective from semantic Wiki communities. In: Proceedings of 7th ACM International Conference on Knowledge Capture (K-CAP) (2013)Google Scholar
  11. 11.
    Jiang, G., Solbrig, H.R.: Lex Wiki framework and use cases. In: First Meeting of Semantic Media Wiki Users, 22–23 November 2008, Boston, MA, USA (2008)Google Scholar
  12. 12.
    Jiang, G., Solbrig, H.R., Iberson-Hurst, D., Kush, R.D., Chute, C.G.: A collaborative framework for representation and harmonization of clinical study data elements using semantic MediaWiki. Summit Transl. Bioinform. 2010, 11–15 (2010)Google Scholar
  13. 13.
    Krötzsch, M., Vrandečić, D.: Semantic MediaWiki. In: Fensel, D. (ed.) Foundations for the Web of Information and Services, pp. 311–326. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-19797-0_16 CrossRefGoogle Scholar
  14. 14.
    McKay, N.P., Emile-Geay, J.: Technical note: the linked Paleo data framework – a common tongue for paleoclimatology. Clim. Past 12(4), 1093–1100 (2016). doi: 10.5194/cp-12-1093-2016 CrossRefGoogle Scholar
  15. 15.
    MediaWiki: The Free Wiki Engine (2017).
  16. 16.
    Nielsen, M.: Reinventing Discovery. Princeton University Press, Princeton (2011)CrossRefGoogle Scholar
  17. 17.
    PAGES 2k Consortium: Continental-scale temperature variability during the past two millennia. Nat. Geosci. 6(5), 339–346 (2013). doi: 10.1038/ngeo1797
  18. 18.
    PAGES 2k Consortium: A global multiproxy database for temperature reconstructions of the Common Era. Sci. Data 4, 170,088 EP (2017). doi: 10.1038/sdata.2017.88
  19. 19.
    Pangaea Interoperability and Services (2017).
  20. 20.
    Pinto, H.S., Tempich, C., Staab, S., Sure, Y.: Distributed engineering of ontologies (DILIGENT). In: Staab, S., Stuckenschmidt, H. (eds.) Semantic Web and Peer-to-Peer. Springer, Heidelberg (2005). doi: 10.1007/3-540-28347-1_16 Google Scholar
  21. 21.
    Ratnakar, V.: The Linked Earth Wiki [Dataset]. Zenodo.
  22. 22.
    Schandl, T., Blumauer, A.: PoolParty: SKOS thesaurus management utilizing linked data. In: Proceedings of 7th International Conference on the Semantic Web: Research and Applications (ESWC) (2010)Google Scholar
  23. 23.
    Suárez-Figueroa, M.-C., Gómez-Pérez, A., Motta, E., Gangemi, A.: Ontology Engineering in a Networked World. Springer, Berlin (2012). doi: 10.1007/978-3-642-24794-1 CrossRefGoogle Scholar
  24. 24.
    Tudorache, T., Noy, N.F., Tu, S., Musen, M.A.: Supporting collaborative ontology development in Protégé. In: Proceedings of International Semantic Web Conference (ISWC) (2008)Google Scholar
  25. 25.
    Tudorache, T., Nyulas, C., Noy, N.F., Musen, M.A.: WebProtégé: a collaborative ontology editor and knowledge acquisition tool for the web. Semant. Web 4(1), 89–99 (2013)Google Scholar
  26. 26.
    Wikipedia: The Free Encyclopedia (2017).
  27. 27.
    Zacharias, V., Braun, S.: SOBOLEO - social bookmarking and lightweight ontology engineering. In: 16th International World Wide Web Conference on Workshop on Social and Collaborative Construction of Structured Knowledge (2007)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yolanda Gil
    • 1
    Email author
  • Daniel Garijo
    • 1
  • Varun Ratnakar
    • 1
  • Deborah Khider
    • 2
  • Julien Emile-Geay
    • 2
  • Nicholas McKay
    • 3
  1. 1.Information Sciences InstituteUniversity of Southern CaliforniaLos AngelesUSA
  2. 2.Department of Earth SciencesUniversity of Southern CaliforniaLos AngelesUSA
  3. 3.School of Earth Sciences and Environmental SustainabilityNorth Arizona UniversityFlagstaffUSA

Personalised recommendations