Skip to main content

On the Usefulness of Different Expert Question Types for Fault Localization in Ontologies

Part of the Lecture Notes in Computer Science book series (LNAI,volume 11606)


When ontologies reach a certain size and complexity, faults such as inconsistencies or wrong entailments are hardly avoidable. Locating the faulty axioms that cause these faults is a hard and time-consuming task. Addressing this issue, several techniques for semi-automatic fault localization in ontologies have been proposed. Often, these approaches involve a human expert who provides answers to system-generated questions about the intended (correct) ontology in order to reduce the possible fault locations. To suggest as few and as informative questions as possible, existing methods draw on various algorithmic optimizations as well as heuristics. However, these computations are often based on certain assumptions about the interacting user and the metric to be optimized.

In this work, we critically discuss these optimization criteria and suppositions about the user. As a result, we suggest an alternative, arguably more realistic metric to measure the expert’s effort and show that existing approaches do not achieve optimal efficiency in terms of this metric. Moreover, we detect that significant differences regarding user interaction costs arise if the assumptions made by existing works do not hold. As a remedy, we suggest a new notion of expert question that does not rely on any assumptions about the user’s way of answering. Experiments on faulty real-world ontologies testify that the new querying method minimizes the necessary expert consultations in the majority of cases and reduces the computation time for the best next question by at least 80 % in all scenarios.


  • Ontology debugging
  • Interactive debugging
  • Fault localization
  • Sequential diagnosis
  • Expert questions
  • Ontology quality assurance
  • Ontology repair
  • Test-driven debugging

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-22999-3_32
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   99.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-22999-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   129.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.


  1. 1.

    All information about OntoDebug can be found at

  2. 2.

    An ontology \(\mathcal {O}\) is coherent iff there do not exist any unsatisfiable classes in \(\mathcal {O}\). A class C is unsatisfiable in \(\mathcal {O}\) iff \(\mathcal {O}\models C \sqsubseteq \bot \). See also [13, Def. 1 and 2].

  3. 3.

    Throughout the presented examples, we use Description Logic notation. For details, see [1].

  4. 4.

    Note, the finally remaining diagnosis does not necessarily contain all faulty axioms in the ontology, as, e.g., some existing faults in the ontology might not yet have surfaced in terms of problems such as wrong entailments or unsatisfiable classes. However, the (faultiness of the) axioms in the final diagnosis do(es) explain all observed problems in the ontology.

  5. 5.

    Note, a positive answer (y) implicitly provides axiom-level information, i.e., the positive classification of all query-axioms. Thus, the discussed experts differ only in their negation behavior.

  6. 6.

    To stress the difference between singleton queries (Definition 2) and queries in terms of Definition 1, we will henceforth often refer to the latter as normal queries.

  7. 7.

    Such (singleton) queries consisting of only axioms explicitly included in the ontology are called explicit (singleton) queries [17].

  8. 8.

    The logical expressivity refers to the power of the logical language used in the ontology in terms of how much can be expressed using this language. In general, the higher the expressivity, the higher the cost of reasoning (and thus the cost of computing queries) with the respective logic tends to be. See [1] for more details on the logical expressivity of ontologies.

  9. 9.

    This is owed to the fact that the efficient generation of optimal singleton queries including “implicit” axioms, i.e., where \(Q \not \subseteq \mathcal {O}\) holds, is still an open research topic (cf. Sect. 4).

  10. 10.

    Note, the presented figures do not expose all results. However, the observations were greatly consistent over all studied ontologies. See the extended version [19] of this paper for all plots.


  1. Baader, F., Calvanese, D., McGuinness, D., Nardi, D., Patel-Schneider, P. (eds.): The Description Logic Handbook, 1st edn. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  2. Beck, K.: Test-Driven Development: By Example. Addison-Wesley Professional, Boston (2003)

    Google Scholar 

  3. Ceusters, W., Smith, B., Goldberg, L.: A terminological and ontological analysis of the NCI thesaurus. Methods Inf. Med. 44(4), 498 (2005)

    CrossRef  Google Scholar 

  4. Felfernig, A., Friedrich, G., Jannach, D., Stumptner, M.: Consistency-based diagnosis of configuration knowledge bases. Artif. Intell. 152(2), 213–234 (2004)

    MathSciNet  CrossRef  Google Scholar 

  5. Golbeck, J., Fragoso, G., Hartel, F., Hendler, J., Oberthaler, J., Parsia, B.: The national cancer institute’s thesaurus and ontology. JWS 1(1) (2003).

  6. Grau, B.C., Horrocks, I., Motik, B., Parsia, B., Patel-Schneider, P., Sattler, U.: OWL 2: the next step for OWL. JWS 6(4), 309–322 (2008)

    CrossRef  Google Scholar 

  7. Horridge, M., Bail, S., Parsia, B., Sattler, U.: The cognitive complexity of OWL justifications. In: Aroyo, L., et al. (eds.) ISWC 2011. LNCS, vol. 7031, pp. 241–256. Springer, Heidelberg (2011).

    CrossRef  Google Scholar 

  8. Jannach, D., Schmitz, T., Shchekotykhin, K.: Parallel model-based diagnosis on multi-core computers. JAIR 55, 835–887 (2016)

    MathSciNet  CrossRef  Google Scholar 

  9. Kalyanpur, A.: Debugging and repair of OWL ontologies. Ph.D. thesis, University of Maryland (2006)

    Google Scholar 

  10. de Kleer, J., Williams, B.C.: Diagnosing multiple faults. Artif. Intell. 32(1), 97–130 (1987)

    CrossRef  Google Scholar 

  11. Meilicke, C.: Alignment incoherence in ontology matching. Ph.D. thesis, University of Mannheim (2011)

    Google Scholar 

  12. Nikitina, N., Rudolph, S., Glimm, B.: Interactive ontology revision. JWS 12(0), 118–130 (2012)

    CrossRef  Google Scholar 

  13. Qi, G., Hunter, A.: Measuring incoherence in description logic-based ontologies. In: Aberer, K., et al. (eds.) ASWC/ISWC -2007. LNCS, vol. 4825, pp. 381–394. Springer, Heidelberg (2007).

    CrossRef  Google Scholar 

  14. Rector, A.L., Brandt, S., Schneider, T.: Getting the foot out of the pelvis: modeling problems affecting use of SNOMED CT hierarchies in practical applications. JAMIA 18(4), 432–440 (2011)

    Google Scholar 

  15. Reiter, R.: A theory of diagnosis from first principles. Artif. Intell. 32(1), 57–95 (1987)

    MathSciNet  CrossRef  Google Scholar 

  16. Rodler, P.: Interactive Debugging of Knowledge Bases. Ph.D. thesis, University of Klagenfurt (2015)

    Google Scholar 

  17. Rodler, P.: Towards better response times and higher-quality queries in interactive KB debugging. Technical report, University of Klagenfurt (2016).

  18. Rodler, P.: On active learning strategies for sequential diagnosis. In: DX, pp. 264–283 (2018)

    Google Scholar 

  19. Rodler, P., Eichholzer, M.: A new expert questioning approach to more efficient fault localization in ontologies. Technical report, University of Klagenfurt (2019).

  20. Rodler, P., Herold, M.: StaticHS: a variant of Reiter’s hitting set tree for efficient sequential diagnosis. In: SoCS, pp. 72–80 (2018)

    Google Scholar 

  21. Rodler, P., Jannach, D., Schekotihin, K., Fleiss, P.: Are query-based ontology debuggers really helping knowledge engineers? Knowledge-Based Systems (2019).

  22. Rodler, P., Schekotihin, K.: Reducing model-based diagnosis to knowledge base debugging. In: DX, pp. 284–296 (2018)

    Google Scholar 

  23. Rodler, P., Schmid, W.: On the impact and proper use of heuristics in test-driven ontology debugging. In: Benzmüller, C., Ricca, F., Parent, X., Roman, D. (eds.) RuleML+RR 2018. LNCS, vol. 11092, pp. 164–184. Springer, Cham (2018).

    CrossRef  Google Scholar 

  24. Rodler, P., Schmid, W., Schekotihin, K.: A generally applicable, highly scalable measurement computation and optimization approach to sequential model-based diagnosis. CoRR abs/1711.05508 (2017)

  25. Rodler, P., Schmid, W., Schekotihin, K.: Inexpensive cost-optimized measurement proposal for sequential model-based diagnosis. In: DX, pp. 200–218 (2018)

    Google Scholar 

  26. Rodler, P., Shchekotykhin, K., Fleiss, P., Friedrich, G.: RIO: minimizing user interaction in ontology debugging. In: Faber, W., Lembo, D. (eds.) RR 2013. LNCS, vol. 7994, pp. 153–167. Springer, Heidelberg (2013).

    CrossRef  Google Scholar 

  27. Schekotihin, K., Rodler, P., Schmid, W.: OntoDebug: interactive ontology debugging plug-in for Protégé. In: Ferrarotti, F., Woltran, S. (eds.) FoIKS 2018. LNCS, vol. 10833, pp. 340–359. Springer, Cham (2018).

    CrossRef  Google Scholar 

  28. Schekotihin, K., Rodler, P., Schmid, W., Horridge, M., Tudorache, T.: A Protégé plug-in for test-driven ontology development. In: ICBO (2018)

    Google Scholar 

  29. Schulz, S., Schober, D., Tudose, I., Stenzhorn, H.: The pitfalls of thesaurus ontologization-the case of the NCI thesaurus. In: AMIA Annual Symposium (2010)

    Google Scholar 

  30. Shchekotykhin, K., Friedrich, G., Fleiss, P., Rodler, P.: Interactive ontology debugging: two query strategies for efficient fault localization. JWS 12–13, 88–103 (2012)

    CrossRef  Google Scholar 

  31. Shchekotykhin, K., Jannach, D., Schmitz, T.: Mergexplain: fast computation of multiple conflicts for diagnosis. In: IJCAI, pp. 3221–3228 (2015)

    Google Scholar 

  32. Shchekotykhin, K.M., Friedrich, G., Rodler, P., Fleiss, P.: Sequential diagnosis of high cardinality faults in knowledge-bases by direct diagnosis generation. In: ECAI (2014)

    Google Scholar 

  33. Siddiqi, S.A., Huang, J.: Sequential diagnosis by abstraction. JAIR 41, 329–365 (2011)

    MathSciNet  CrossRef  Google Scholar 

  34. Smith, B., Ashburner, M., Rosse, C., et al.: The OBO foundry: coordinated evolution of ontologies to support biomedical data integration. Nature Biotechnol. 25(11), 1251 (2007)

    CrossRef  Google Scholar 

  35. Tudorache, T., Noy, N.F., Tu, S., Musen, M.A.: Supporting collaborative ontology development in Protégé. In: Sheth, A., et al. (eds.) ISWC 2008. LNCS, vol. 5318, pp. 17–32. Springer, Heidelberg (2008).

    CrossRef  Google Scholar 

Download references


This work was in part supported by the Carinthian Science Fund (KWF), contract KWF-3520/26767/38701. Moreover, we thank Wolfgang Schmid for his technical support during the implementation of our experiments.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Patrick Rodler .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Rodler, P., Eichholzer, M. (2019). On the Usefulness of Different Expert Question Types for Fault Localization in Ontologies. In: Wotawa, F., Friedrich, G., Pill, I., Koitz-Hristov, R., Ali, M. (eds) Advances and Trends in Artificial Intelligence. From Theory to Practice. IEA/AIE 2019. Lecture Notes in Computer Science(), vol 11606. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-22998-6

  • Online ISBN: 978-3-030-22999-3

  • eBook Packages: Computer ScienceComputer Science (R0)