Ranking with Ties of OWL Ontology Reasoners Based on Learned Performances

  • Nourhène Alaya
  • Sadok Ben Yahia
  • Myriam Lamolle
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 631)


Over the last decade, several ontology reasoners have been proposed to overcome the computational complexity of inference tasks on expressive ontology languages such as OWL 2 DL. Nevertheless, it is well-accepted that there is no outstanding reasoner that can outperform in all input ontologies. Thus, deciding the most suitable reasoner for an ontology based application is still a time and effort consuming task. In this paper, we suggest to develop a new system to provide user support when looking for guidance over ontology reasoners. At first, we will be looking at automatically predict a single reasoner empirical performances, in particular its robustness and efficiency, over any given ontology. Later, we aim at ranking a set of candidate reasoners in a most preferred order by taking into account information regarding their predicted performances. We conducted extensive experiments covering over 2500 well selected real-world ontologies and six state-of-the-art of the most performing reasoners. Our primary prediction and ranking results are encouraging and witnessing the potential benefits of our approach.


Ontology Reasoner Robustness Efficiency Supervised machine learning Prediction Ranking 


  1. 1.
    Abbott, D.: Applied Predictive Analytics: Principles and Techniques for the Professional Data Analyst, 1st edn. Wiley, Hoboken (2014)Google Scholar
  2. 2.
    Alaya, N., Ben Yahia, S., Lamolle, M.: Predicting the empirical robustness of the ontology reasoners based on machine learning techniques. In: Proceedings of the 7th International Conference on Knowledge Engineering and Ontology Development KEOD 2015, Lisbon, Portugal, pp. 61–73 (2015)Google Scholar
  3. 3.
    Alaya, N., Lamolle, M., Ben Yahia, S.: Towards unveiling the ontology key features altering reasoner performances. Technical report, IUT of Montreuil, France (2015).
  4. 4.
    Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D., Patel-Schneider, P.F.: The Description Logic Handbook: Theory Implementation and Applications, 2nd edn. Cambridge University Press, New York (2010)zbMATHGoogle Scholar
  5. 5.
    Baeza-Yates, R.A., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley Longman Publishing Co., Inc., Boston (1999)Google Scholar
  6. 6.
    Caruso, J.C., Cliff, N.: Empirical size, coverage, and power of confidence intervals for Spearman’s rho. J. Educ. Psychol. Meas. 57, 637–654 (1997)CrossRefGoogle Scholar
  7. 7.
    Chandrashekar, G., Sahin, F.: A survey on feature selection methods. Comput. Electr. Eng. 40(1), 16–28 (2014)CrossRefGoogle Scholar
  8. 8.
    Dumontier, M., Glimm, B., Gonçalves, R.S., Horridge, M., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Stamou, G.B., Stoilos, G. (eds.): Informal Proceedings of the 4th International Workshop on OWL Reasoner Evaluation (ORE-2015), Athens, Greece, vol. 1387. (2015)Google Scholar
  9. 9.
    Emond, E.J., Mason, D.: A new rank correlation coefficient with application to consensus ranking problem. J. Multicriteria Decis. Anal. 11, 17–28 (2002)CrossRefzbMATHGoogle Scholar
  10. 10.
    Fagin, R., Kumar, R., Mahdian, M., Sivakumar, D., Vee, E.: Comparing partial rankings. SIAM J. Discret. Math. 20, 628–648 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Garcia, S., Luengo, J., Saez, J., Lopez, V., Herrera, F.: A survey of discretization techniques: taxonomy and empirical analysis in supervised learning. IEEE Trans. Knowl. Data Eng. 25, 734–750 (2013)CrossRefGoogle Scholar
  12. 12.
    Gardiner, T., Tsarkov, D., Horrocks, I.: Framework for an automated comparison of description logic reasoners. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 654–667. Springer, Heidelberg (2006). doi: 10.1007/11926078_47 CrossRefGoogle Scholar
  13. 13.
    Gomes, C.P., Selman, B.: Algorithm portfolios. Artif. Intell. 126, 43–62 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Gonçalves, R.S., Matentzoglu, N., Parsia, B., Sattler, U.: The empirical robustness of description logic classification. In: Informal Proceedings of the 26th International Workshop on Description Logics, Ulm, Germany, pp. 197–208 (2013)Google Scholar
  15. 15.
    Gonçalves, R.S., Bail, S., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Glimm, B., Kazakov, Y.: Owl reasoner evaluation (ORE) workshop 2013 results: short report. In: ORE, pp. 1–18 (2013)Google Scholar
  16. 16.
    Gonçalves, R.S., Parsia, B., Sattler, U.: Performance heterogeneity and approximate reasoning in description logic ontologies. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012. LNCS, vol. 7649, pp. 82–98. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-35176-1_6 CrossRefGoogle Scholar
  17. 17.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P.: The weka data mining software: an update. SIGKDD Explor. Newsl. 11, 10–18 (2009)CrossRefGoogle Scholar
  18. 18.
    Hu, B., Dong, W.: A study on cost behaviors of binary classification measures in class-imbalanced problems. CoRR abs/1403.7100 (2014)Google Scholar
  19. 19.
    Kang, Y.-B., Krishnaswamy, S., Li, Y.-F.: R\(_2\)O\(_2\): an efficient ranking-based reasoner for OWL ontologies. In: Arenas, M., et al. (eds.) ISWC 2015. LNCS, vol. 9366, pp. 322–338. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-25007-6_19 CrossRefGoogle Scholar
  20. 20.
    Kang, Y.-B., Li, Y.-F., Krishnaswamy, S.: Predicting reasoning performance using ontology metrics. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012. LNCS, vol. 7649, pp. 198–214. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-35176-1_13 CrossRefGoogle Scholar
  21. 21.
    Kang, Y.B., Li, Y.F., Krishnaswamy, S.: How long will it take? Accurate prediction of ontology reasoning performance. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence, pp. 80–86 (2014)Google Scholar
  22. 22.
    Lee, M., Matentzoglu, N., Sattler, U., Parsia, B.: Verifying reasoner correctness - a justication based method. In: Informal Proceedings of the 4th International Workshop on OWL Reasoner Evaluation (ORE-2015), pp. 46–52 (2015)Google Scholar
  23. 23.
    LePendu, P., Noy, N.F., Jonquet, C., Alexander, P.R., Shah, N.H., Musen, M.A.: Optimize first, buy later: analyzing metrics to ramp-up very large knowledge bases. In: Patel-Schneider, P.F., Pan, Y., Hitzler, P., Mika, P., Zhang, L., Pan, J.Z., Horrocks, I., Glimm, B. (eds.) ISWC 2010. LNCS, vol. 6496, pp. 486–501. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-17746-0_31 CrossRefGoogle Scholar
  24. 24.
    Matentzoglu, N., Leo, J., Hudhra, V., Sattler, U., Parsia, B.: A survey of current, stand-alone OWL reasoners. In: Proceedings of the 4th International Workshop on OWL Reasoner Evaluation, pp. 68–79 (2015)Google Scholar
  25. 25.
    Mikolà\({\tilde{\rm s}}\)ek, V.: Dependability and robustness: state of the art and challenges. In: Proceedings of the First International Workshop on Software Technologies for Future Dependable Distributed Systems (STFSSD), pp. 25–31 (2009)Google Scholar
  26. 26.
    W3C OWL Working Group: OWL 2 Web Ontology Language: Document Overview. W3C Recommendation, October 2009.
  27. 27.
    Pandit, V., Kenkre, S., Khan, A.: On discovering bucket orders from preference data. In: Proceedings of the Eleventh SIAM International Conference on Data Mining, Arizona, USA, pp. 872–883 (2011)Google Scholar
  28. 28.
    Prudêncio, R.B.C., de Souto, M.C.P., Ludermir, T.B.: Selecting machine learning algorithms using the ranking meta-learning approach. In: Jankowski, N., Duch, W., Gra̧bczewski, K. (eds.) Meta-Learning in Computational Intelligence. SCI, vol. 358, pp. 225–243. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  29. 29.
    Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)CrossRefGoogle Scholar
  30. 30.
    Rosen, K.H.: Discrete Mathematics and Its Applications. McGraw Hill Higher Education, New York (1991)Google Scholar
  31. 31.
    Sazonau, V., Sattler, U., Brown, G.: Predicting performance of OWL reasoners: locally or globally? In: Proceedings of the Fourteenth International Conference on Principles of Knowledge Representation and Reasoning (2014)Google Scholar
  32. 32.
    Weithöner, T., Liebig, T., Luther, M., Böhm, S., Henke, F., Noppens, O.: Real-world reasoning with OWL. In: Franconi, E., Kifer, M., May, W. (eds.) ESWC 2007. LNCS, vol. 4519, pp. 296–310. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-72667-8_22 CrossRefGoogle Scholar
  33. 33.
    Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Satzilla: portfolio-based algorithm selection for SAT. J. Artif. Int. Res. 32, 565–606 (2008)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Nourhène Alaya
    • 1
    • 2
  • Sadok Ben Yahia
    • 1
  • Myriam Lamolle
    • 2
  1. 1.LIPAH-LR 11ES14, Faculty of Sciences of TunisUniversity of Tunis El ManarTunisTunisia
  2. 2.LIASD EA4383, IUT of MontreuilUniversity of Paris 8Saint-DenisFrance

Personalised recommendations