Skip to main content
SpringerLink
Log in

Empirically evaluating modeling language ontologies: the Peira framework

  • Regular Paper
  • Published:
Software and Systems Modeling Aims and scope Submit manuscript

Abstract

Conceptual modeling plays a central role in planning, designing, developing and maintaining software-intensive systems. One of the goals of conceptual modeling is to enable clear communication among stakeholders involved in said activities. To achieve effective communication, conceptual models must be understood by different people in the same way. To support such shared understanding, conceptual modeling languages are defined, which introduce rules and constraints on how individual models can be built and how they are to be understood. A key component of a modeling language is an ontology, i.e., a set of concepts that modelers must use to describe world phenomena. Once the concepts are chosen, a visual and/or textual vocabulary is adopted for representing the concepts. However, the choices both of the concepts and of the vocabulary used to represent them may affect the quality of the language under consideration: some choices may promote shared understanding better than other choices. To allow evaluation and comparison of alternative choices, we present Peira, a framework for empirically measuring the domain and comprehensibility appropriateness of conceptual modeling language ontologies. Given a language ontology to be evaluated, the framework is based on observing how prospective language users classify domain content under the concepts put forth by said ontology. A set of metrics is then used to analyze the observations and identify and characterize possible issues that the choice of concepts or the way they are represented may have. The metrics are abstract in that they can be operationalized into concrete implementations tailored to specific data collection instruments or study objectives. We evaluate the framework by applying it to compare an existing language against an artificial one that is manufactured to exhibit specific issues. We then test if the metrics indeed detect these issues. We find that the framework does offer the expected indications, but that it also requires good understanding of the metrics prior to committing to interpretations of the observations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. (/‘pi.ra/), the Greek word for experience, trial, experiment.

References

  1. Fettke, P.: How conceptual modeling is used. Commun. Assoc. Inf. Syst. 25, 43 (2009). https://doi.org/10.17705/1CAIS.02543

    Article  Google Scholar 

  2. Davies, I., Green, P., Rosemann, M., Indulska, M., Gallo, S.: How do practitioners use conceptual modeling in practice? Data Knowl. Eng. 58(3), 358–380 (2006). https://doi.org/10.1016/j.datak.2005.07.007

    Article  Google Scholar 

  3. Olivé, A.: Conceptual Modeling of Information Systems. Springer, Berlin, Heidelberg (2007)

    Google Scholar 

  4. Borgida, A.T., Chaudhri, V.K., Giorgini, P., Yu, E.S. (eds.): Conceptual Modeling: Foundations and Applications. Springer, Berlin, Heidelberg (2009)

    Google Scholar 

  5. Mylopoulos, J., Borgida, A., Jarke, M., Koubarakis, M.: Telos: representing knowledge about information systems. ACM Trans. Inf. Syst. 8(4), 325–362 (1990). https://doi.org/10.1145/102675.102676

    Article  Google Scholar 

  6. Karagiannis, D., Khun, H.: Metamodelling platforms. In: Proceedings of the Third International Conference on E-commerce and Web Technology (EC-Web 2002), pp. 182–197 (2002)

  7. Guarino, N., Oberle, D., Staab, S.: What is an ontology? In: Staab, S., Studer, R. (eds.) Handbook on Ontologies, pp. 1–17. Springer, Berlin (2009). https://doi.org/10.1007/978-3-540-92673-3_0

  8. Object Management Group: OMG Unified Modeling Language (OMG UML)—Version 2.5.1. (2017). Object Management Group. https://www.omg.org/spec/UML/2.5.1/PDF

  9. Dalpiaz, F., Franch, X., Horkoff, J.: iStar 2.0 language guide. The computing research repository (CoRR) abs/1605.0 (2016) arXiv:1605.07767

  10. The Open Group: ArchiMate® 3.1 specification. Technical report (2019)

  11. What’s New in ArchiMate 2.0? https://blog.opengroup.org/2012/01/31/whats-new-in-archimate-2-0/ (2012)

  12. What’s new in the ArchiMate 3.0 modeling language? https://blog.opengroup.org/2016/06/14/whats-new-in-archimate-3-0/ (2016)

  13. ArchiMate ® 3.1 Specification: The new version of the standard. https://blog.opengroup.org/2012/01/31/whats-new-in-archimate-2-0/ (2019)

  14. Liaskos, S., Mylopoulos, J., Khan, S.M.: Empirically evaluating the semantic qualities of language vocabularies. In: Ghose, A.K., Horkoff, J., Souza, V.E.S., Parsons, J., Evermann, J. (eds.) Proceedings of the 40th International Conference on Conceptual Modeling (ER 2021). Lecture Notes in Computer Science, vol. 13011, pp. 330–344. Springer, Berlin, Heidelberg (2021). https://doi.org/10.1007/978-3-030-89022-3_26

  15. Dickover, M.E., McGowan, C.L., Ross, D.T.: Software design using: SADT. In: Proceedings of the 1977 Annual Conference of the ACM. ACM ’77, pp. 125–133. Association for Computing Machinery, New York, NY, USA (1977). https://doi.org/10.1145/800179.810192

  16. Krogstie, J.: Model-Based Development and Evolution of Information Systems. Springer, Berlin, Heidelberg (2012)

    Book  Google Scholar 

  17. Krippendorff, K.: Content Analysis: An Introduction to It Methodology. SAGE, Thousand Oaks, London, New Delhi (2004)

    Google Scholar 

  18. Gwet, K.L.: Handbook of Inter-rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. Advanced Analytics, LLC, Gaithersburg (2014)

    Google Scholar 

  19. Wand, Y., Weber, R.: On the ontological expressiveness of information systems analysis and design grammars. Inf. Syst. J. 3(4), 217–237 (1993)

    Article  Google Scholar 

  20. Stoet, G.: PsyToolkit: a software package for programming psychological experiments using Linux. Behav. Res. Methods 42(4), 1096–1104 (2010). https://doi.org/10.3758/BRM.42.4.1096

    Article  Google Scholar 

  21. Stoet, G.: PsyToolkit: a novel web-based method for running online questionnaires and reaction-time experiments. Teach. Psychol. 44(1), 24–31 (2017). https://doi.org/10.1177/0098628316677643

    Article  Google Scholar 

  22. Prolific. https://www.prolific.co/ (2022)

  23. Peer, E., Rothschild, D., Gordon, A., Evernden, Z., Damer, E.: Data quality of platforms and panels for online behavioral research. Behav. Res. Methods 54(4), 1643–1662 (2022). https://doi.org/10.3758/s13428-021-01694-3

    Article  Google Scholar 

  24. Liaskos, S., Zarbaf, S.: Replication data for: empirically evaluating modeling language ontologies: the Peira Framework. https://doi.org/10.5683/SP3/O1E4PL .

  25. Dikta, G., Scheer, M.: Bootstrap Methods With Applications in R, 1st edn. Springer, Berlin, Heidelberg (2021). https://doi.org/10.1007/978-3-030-73480-0

    Book  Google Scholar 

  26. Rosnow, R.L., Rosenthal, R.: Beginning Behavioral Research: A Conceptual Primer, 6th edn. Pearson Prentice Hall, NJ (2008)

    Google Scholar 

  27. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977). https://doi.org/10.2307/2529310

    Article  Google Scholar 

  28. Alothman, N., Zhian, M., Liaskos, S.: User perception of numeric contribution semantics for goal models: an exploratory experiment. In: Proceedings of the 36th International Conference on Conceptual Modeling (ER 2017), Xi’an, China, pp. 451–465 (2017). http://www.yorku.ca/liaskos/Docs/ER17.pdf

  29. Liaskos, S., Ronse, A., Zhian, M.: Assessing the intuitiveness of qualitative contribution relationships in goal models: an exploratory experiment. In: Proceedings of the 11th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’17), Toronto, Canada, pp. 466–471 (2017). http://www.yorku.ca/liaskos/Docs/ESEM17.pdf

  30. Liaskos, S., Dundjerovic, T., Gabriel, G.: Comparing alternative goal model visualizations for decision making: an exploratory experiment. In: Proceedings of the 33rd Annual ACM Symposium on Applied Computing (SAC’18), Pau, France, pp. 1272–1281 (2018). http://www.yorku.ca/liaskos/Papers/SAC2018/Visualizations/SAC2018.pdf

  31. Henderson-Sellers, B., Gonzalez-Perez, C.: Granularity in conceptual modelling: application to metamodels. In: Proceedings of the 29th International Conference on Conceptual Modeling (ER 2010), Vancouver, BC, Canada, pp. 219–232 (2010)

  32. Nelson, H.J., Poels, G., Genero, M., Piattini, M.: A conceptual modeling quality framework. Softw. Qual. J. 20, 201–228 (2012)

    Article  Google Scholar 

  33. Wand, Y., Weber, R.: Toward a theory of the deep structure of information systems. In: Proceedings of the Conference on Information Systems (ICIS 1990), pp. 61–71 (1990)

  34. Lindland, O.I., Sindre, G., Solvberg, A.: Understanding quality in conceptual modeling. IEEE Softw. 11(2), 42–49 (1994)

    Article  Google Scholar 

  35. Moody, D.L.: The “physics" of notations: toward a scientific basis for constructing visual notations in software engineering. IEEE Trans. Softw. Eng. 35(6), 756–779 (2009). https://doi.org/10.1109/TSE.2009.67

    Article  Google Scholar 

  36. Moody, D.L., Heymans, P., Matulevičius, R.: Visual syntax does matter: improving the cognitive effectiveness of the i\(^{*}\) visual notation. Requir. Eng. 15(2), 141–175 (2010)

    Article  Google Scholar 

  37. Bork, D., Roelens, B.: A technique for evaluating and improving the semantic transparency of modeling language notations. Softw. Syst. Model. 20(4), 939–963 (2021). https://doi.org/10.1007/s10270-021-00895-w

    Article  Google Scholar 

  38. Bork, D., Karagiannis, D., Pittl, B.: How are metamodels specified in practice? Empirical insights and recommendations. In: Proceedigns of the 24th Americas Conference on Information Systems (AMCIS’18) (2018)

  39. Houy, C., Fettke, P., Loos, P.: Understanding understandability of conceptual models—What are we actually talking about? In: Proceedings of the 31st International Conference on Conceptual Modeling (ER 2012), vol. LNCS 7532, pp. 64–77 (2012)

  40. Caire, P., Genon, N., Heymans, P., Moody, D.L.: Visual notation design 2.0: towards user comprehensible requirements engineering notations. In: Proceedings of the 21st IEEE International Requirements Engineering Conference (RE’13), Rio de Janeiro, Brasil, pp. 115–124 (2013)

  41. Estrada, H., Rebollar, A.M., Pastor, O., Mylopoulos, J.: An empirical evaluation of the i* framework in a model-based software generation environment. In: Proceedings of the 18th International Conference on Advanced Information Systems Engineering (CAiSE’06), pp. 513–527. Springer, Luxembourg, Luxembourg (2006)

  42. Hadar, I., Reinhartz-Berger, I., Kuflik, T., Perini, A., Ricca, F., Susi, A.: Comparing the comprehensibility of requirements models expressed in Use Case and Tropos: results from a family of experiments. Inf. Softw. Technol. 55(10), 1823–1843 (2013)

    Article  Google Scholar 

  43. Horkoff, J., Yu, E.: Finding solutions in goal models: an interactive backward reasoning approach. In: Proceedings of the 29th International Conference on Conceptual Modeling (ER’10). ER’10, Vancouver, Canada, pp. 59–75 (2010)

  44. Santos, M., Gralha, C., Goulão, M., Araújo, J.: Increasing the semantic transparency of the KAOS goal model concrete syntax. In: Proceedings of the 37th International Conference on Conceptual Modeling (ER’18), Xi’an, China, pp. 424–439 (2018)

  45. Liaskos, S., Tambosi, W.: Factors affecting comprehension of contribution links in goal models: an experiment. In: Proceedings of the 38th International Conference on Conceptual Modeling (ER’19), Salvador, Brazil, pp. 525–539 (2019)

  46. Cimiano, P., Mädche, A., Staab, S., Völker, J.: Ontology learning. In: Staab, S., Studer, R. (eds.) Handbook on Ontologies, pp. 245–267. Springer, Berlin, Heidelberg (2009). https://doi.org/10.1007/978-3-540-92673-3_11

    Chapter  Google Scholar 

  47. Wong, W., Liu, W., Bennamoun, M.: Ontology learning from text: a look back and into the future. ACM Comput. Surv. 44(4), 1–36 (2012). https://doi.org/10.1145/2333112.2333115

    Article  Google Scholar 

  48. Obrst, L., Ceusters, W., Mani, I., Ray, S., Smith, B.: The evaluation of ontologies. In: Baker, C.J.O., Cheung, K.-H. (eds.) Semantic Web: Revolutionizing Knowledge Discovery in the Life Sciences, pp. 139–158. Springer, Boston (2007). https://doi.org/10.1007/978-0-387-48438-9_8

    Chapter  Google Scholar 

  49. Medelyan, O., Witten, I.H.: Thesaurus-based index term extraction for agricultural documents. In: Proceedings of 2005 EFITA/WCCA Joint Congress on IT in Agriculture, pp. 1122–1129. EFITA/WICCA, Conference held at Vila Real, Portugal (2005). https://hdl.handle.net/10289/8101

  50. Guizzardi, G.: Ontological foundations for structural conceptual models. PhD thesis, University of Twente (2005)

  51. Krantz, D.H., Luce, D.R., Suppes, P., Tversky, A.: Foundations of Measurement Volume I: Additive and Polynomial Representations. Academic Press, Cambridge (1971)

  52. Narens, L.: Introduction to the Theories of Measurement and Meaningfulness and the Use of Symmetry in Science. Lawrence Erlbaum Associates, Inc., Mahwah (2007)

    Book  Google Scholar 

  53. Hand, D.J.: Statistics and the theory of measurement. J. R. Stat. Soc. Ser. A (Stat. Soc.) 159(3), 445 (1996). https://doi.org/10.2307/2983326

Download references

Author information

Authors and Affiliations

  1. School of Information Technology, York University, 4700 Keele St., Toronto, ON, M3J 1P3, Canada

    Sotirios Liaskos

  2. Department of Electrical Engineering and Computer Science, York University, 4700 Keele St., Toronto, ON, M3J 1P3, Canada

    Saba Zarbaf

  3. Department of Computer Science, University of Toronto, 40 St.George St., Toronto, ON, M5S 2E4, Canada

    John Mylopoulos

  4. Department of Computer Science, University of Regina, 3737 Wascana Parkway, Regina, SK, S4S 0A2, Canada

    Shakil M. Khan

Authors
  1. Sotirios Liaskos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saba Zarbaf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John Mylopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shakil M. Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sotirios Liaskos.

Additional information

Communicated by Timothy Lethbridge.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liaskos, S., Zarbaf, S., Mylopoulos, J. et al. Empirically evaluating modeling language ontologies: the Peira framework. Softw Syst Model (2024). https://doi.org/10.1007/s10270-023-01147-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10270-023-01147-9

Keywords

Search

Navigation