Computing Authoring Tests from Competency Questions: Experimental Validation

  • Matt DennisEmail author
  • Kees van Deemter
  • Daniele Dell’Aglio
  • Jeff Z. Pan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10587)


This paper explores whether Authoring Tests derived from Competency Questions accurately represent the expectations of ontology authors. In earlier work we proposed that an ontology authoring interface can be improved by allowing the interface to test whether a given Competency Question (CQ) is able to be answered by the ontology at a given stage of its construction, an approach known as CQ-driven Ontology Authoring (CQOA). The experiments presented in the present paper suggest that CQOA’s understanding of CQs matches users’ understanding quite well, especially for inexperienced ontology authors.


Ontology authoring Competency questions 



This research has been funded by the EPSRC project: WhatIf: Answering “What if” questions for Ontology Authoring. EPSRC reference EP/J014176/1.


  1. 1.
    Ashburner, M., Ball, C.A., Blake, J.A., Botstein, D., Butler, H., Cherry, J.M., Davis, A.P., Dolinski, K., Dwight, S.S., Eppig, J.T., et al.: Gene ontology: tool for the unification of biology. Nat. Genet. 25(1), 25–29 (2000)CrossRefGoogle Scholar
  2. 2.
    Baader, F.: The Description Logic Handbook: Theory, Implementation and Applications. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  3. 3.
    Beaver, D.: Presupposition. In: van Benthem, J., ter Meulen, A. (eds.) Handbook of Logic and Language, pp. 939–1009. North Holland, Amsterdam (1997)CrossRefGoogle Scholar
  4. 4.
    Bezerra, C., Freitas, F., Santana, F.: Evaluating ontologies with competency questions. In: IEEE/WIC/ACM International Joint Conference on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 3, pp. 284–285. IEEE (2013)Google Scholar
  5. 5.
    Breheny, R., Katsos, N., Williams, J.: Are generalised scalar implicatures generated by default? An on-line investigation into the role of context in generating pragmatic inferences. Cognition 100, 434–463 (2006)CrossRefGoogle Scholar
  6. 6.
    Denaux, R., Dimitrova, V., Cohn, A.G., Dolbear, C., Hart, G.: Rabbit to OWL: ontology authoring with a CNL-based tool. In: Fuchs, N.E. (ed.) CNL 2009. LNCS (LNAI), vol. 5972, pp. 246–264. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-14418-9_15 CrossRefGoogle Scholar
  7. 7.
    Denaux, R., Thakker, D., Dimitrova, V., Cohn, A.G.: Interactive semantic feedback for intuitive ontology authoring. In: FOIS, pp. 160–173 (2012)Google Scholar
  8. 8.
    Dzbor, M., Motta, E., Buil, C., Gomez, J.M., Görlitz, O., Lewen, H.: Developing ontologies in OWL: an observational study. In: OWLED. CEUR Workshop Proceedings, vol. 216. (2006)Google Scholar
  9. 9.
    Dzbor, M., Motta, E., Gomez, J.M., Buil, C., Dellschaft, K., Görlitz, O., Lewen, H.: D4.1.1 analysis of user needs, behaviours & requirements wrt user interfaces for ontology engineering. Technical report, August 2006Google Scholar
  10. 10.
    Enderton, H.B.: A Mathematical Introduction to Logic. Academic Press, San Diego (2001)zbMATHGoogle Scholar
  11. 11.
    Fernandes, P.C.B., Guizzardi, R.S., Guizzardi, G.: Using goal modeling to capture competency questions in ontology-based systems. JIDM 2(3), 527 (2011)Google Scholar
  12. 12.
    Grueninger, M., Fox, M.: Methodology for the design and evaluation of ontologies. In: IJCAI Workshop on Basic Ontology Issues in Knowledge Sharing (1995)Google Scholar
  13. 13.
    Hofer, P., Neururer, S., Helga Hauffe, T.I., Zeilner, A., Göbel, G.: Semi-automated evaluation of biomedical ontologies for the biobanking domain based on competency questions. Stud. Health Tech. Inform. 212, 65–72 (2015)Google Scholar
  14. 14.
    Horridge, M., Drummond, N., Goodwin, J., Rector, A.L., Stevens, R., Wang, H.: The manchester OWL syntax. In: OWLed, vol. 216 (2006)Google Scholar
  15. 15.
    Huang, Y.T., Snedeker, J.: On-line interpretation of scalar quantifiers: insight into the semantic-pragmatics interface. Cogn. Psychol. 58, 376–415 (2009)CrossRefGoogle Scholar
  16. 16.
    Keet, C.M., Ławrynowicz, A.: Test-driven development of ontologies. In: Sack, H., Blomqvist, E., d’Aquin, M., Ghidini, C., Ponzetto, S.P., Lange, C. (eds.) ESWC 2016. LNCS, vol. 9678, pp. 642–657. Springer, Cham (2016). doi: 10.1007/978-3-319-34129-3_39 CrossRefGoogle Scholar
  17. 17.
    Knublauch, H., Fergerson, R.W., Noy, N.F., Musen, M.A.: The Protégé OWL plugin: an open development environment for semantic web applications. In: McIlraith, S.A., Plexousakis, D., Harmelen, F. (eds.) ISWC 2004. LNCS, vol. 3298, pp. 229–243. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-30475-3_17 CrossRefGoogle Scholar
  18. 18.
    Levinson, S.C.: Pragmatics. Cambridge University Press, Cambridge (1983)Google Scholar
  19. 19.
    Liebig, T., Noppens, O.: Ontotrack: a semantic approach for ontology authoring. Web Semant. Sci. Serv. Agents World Wide Web 3(2), 116–131 (2005)CrossRefGoogle Scholar
  20. 20.
    Lord, P.: The semantic web takes wing: programming ontologies with Tawny-OWL. In: OWLED 2013 (2013).
  21. 21.
    Malheiros, Y., Freitas, F.: A method to develop description logic ontologies iteratively based on competency questions: an implementation. In: ONTOBRAS, pp. 142–153 (2013)Google Scholar
  22. 22.
    Pan, J.Z., Ren, Y., Zhao, Y.: Tractable approximate deduction for OWL. Artif. Intell. 235, 95–155 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Power, R.: OWL simplified english: a finite-state language for ontology editing. In: Kuhn, T., Fuchs, N.E. (eds.) CNL 2012. LNCS, vol. 7427, pp. 44–60. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-32612-7_4 CrossRefGoogle Scholar
  24. 24.
    Rector, A., Drummond, N., Horridge, M., Rogers, J., Knublauch, H., Stevens, R., Wang, H., Wroe, C.: OWL pizzas: practical experience of teaching OWL-DL: common errors & common patterns. In: Motta, E., Shadbolt, N.R., Stutt, A., Gibbins, N. (eds.) EKAW 2004. LNCS, vol. 3257, pp. 63–81. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-30202-5_5 CrossRefGoogle Scholar
  25. 25.
    Ren, Y., Parvizi, A., Mellish, C., Pan, J.Z., Deemter, K., Stevens, R.: Towards competency question-driven ontology authoring. In: Presutti, V., d’Amato, C., Gandon, F., d’Aquin, M., Staab, S., Tordai, A. (eds.) ESWC 2014. LNCS, vol. 8465, pp. 752–767. Springer, Cham (2014). doi: 10.1007/978-3-319-07443-6_50 CrossRefGoogle Scholar
  26. 26.
    Sedivy, J.C.: Implicature during real time conversation: a view from language processing research. Philos. Compass 2(3), 275–496 (2007)CrossRefGoogle Scholar
  27. 27.
    Suárez-Figueroa, M.C., Gómez-Pérez, A.: Ontology requirements specification. In: Suárez-Figueroa, M., Gómez-Pérez, A., Motta, E., Gangemi, A. (eds.) Ontology Engineering in a Networked World, pp. 93–106. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-24794-1_5 CrossRefGoogle Scholar
  28. 28.
    Taylor, W.L.: Cloze procedure: a new tool for measuring readability. Journal. Q. 30, 415–433 (1953)Google Scholar
  29. 29.
    Uschold, M., Gruninger, M., et al.: Ontologies: principles, methods and applications. Knowl. Eng. Rev. 11(2), 93–136 (1996)CrossRefGoogle Scholar
  30. 30.
    Whetzel, P.L., Parkinson, H.E., Causton, H.C., Fan, L., Fostel, J., Fragoso, G., Game, L., Heiskanen, M., Morrison, N., Rocca-Serra, P., Sansone, S., Taylor, C.F., White, J., Stoeckert, C.J.: The MGED ontology: a resource for semantics-based description of microarray experiments. Bioinformatics 22(7), 866–873 (2006)CrossRefGoogle Scholar
  31. 31.
    Zemmouchi-Ghomari, L., Ghomari, A.R.: Translating natural language competency questions into SPARQL queries: a case study. In: The First International Conference on Building and Exploring Web Based Environments, WEB 2013, pp. 81–86 (2013)Google Scholar
  32. 32.
    Zuber, R., Zuber, R.: Non-declarative Sentences. John Benjamins Publishing, Amsterdam (1983)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Matt Dennis
    • 1
    • 2
    Email author
  • Kees van Deemter
    • 1
  • Daniele Dell’Aglio
    • 1
    • 3
  • Jeff Z. Pan
    • 1
  1. 1.University of AberdeenAberdeenUK
  2. 2.University of PortsmouthPortsmouthUK
  3. 3.University of ZurichZurichSwitzerland

Personalised recommendations