Example-driven modeling: on effects of using examples on structural model comprehension, what makes them useful, and how to create them

  • Dina Zayan
  • Atrisha Sarkar
  • Michał Antkiewicz
  • Rita Suzana Pitangueira Maciel
  • Krzysztof Czarnecki
Regular Paper


We present a controlled experiment for the empirical evaluation of example-driven modeling (EDM), an approach that systematically uses examples for model comprehension and domain knowledge transfer. We conducted the experiment with 26 graduate (Masters and Ph.D. level) and undergraduate (Bachelor level) students from electrical and computer engineering, computer science, and software engineering programs at the University of Waterloo. The experiment involves a domain model, with UML class diagrams representing the domain abstractions and UML object diagrams representing examples of using these abstractions. The goal is to provide empirical evidence of the effects of suitable examples on model comprehension, compared to having model abstractions only, by having the participants perform model comprehension tasks. Our results show that EDM is superior to having model abstractions only, with an improvement of 39% for diagram completeness, 33% for questions completeness, 71% for efficiency, and a reduction in the number of mistakes by 80%. We provide qualitative results showing that participants receiving model abstractions augmented with examples experienced lower perceived difficulty in performing the comprehension tasks, higher perceived confidence in their tasks’ solutions, and asked 90% fewer clarifying domain questions. We also present participants’ feedback regarding the usefulness of the provided examples, their number and types, as well as the use of partial examples. We present a taxonomy of the different types of examples, explain their significance, and propose guidelines for manual and automatic creation of useful examples.


Software engineering Structural modeling Empirical study Example-driven modeling 



This work was partly sponsored by CAPES Science Abroad scholarship - number 1505/14-9 and NECSIS-Automotive Partnership Canada.


  1. 1.
    Agresti, A.: A survey of exact inference for contingency tables. Stat. Sci. 7, 131–153 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Andrews, A., France, R., Ghosh, S., Craig, G.: Test adequacy criteria for UML design models. Softw. Test. Verif. Reliab. 13(2), 95–127 (2003)CrossRefGoogle Scholar
  3. 3.
    Antkiewicz, M., Bąk, K., Zayan, D., Czarnecki, K., Wąsowski, A., Diskin, Z.: Example-driven modeling using Clafer. In: First International Workshop on Model-driven Engineering By Example (2013)Google Scholar
  4. 4.
    Bąk, K., Czarnecki, K., Wąsowski, A.: Feature and meta-models in Clafer: mixed, specialized and coupled. In: International Conference on Software Language Engineering, pp. 291–301 (2010)Google Scholar
  5. 5.
    Bąk, K., Diskin, Z., Antkiewicz, M., Czarnecki, K., Wąsowski, A.: Partial instances via subclassing. In: 6th International Conference on Software Language Engineering (2013)Google Scholar
  6. 6.
    Bąk, K., Zayan, D., Czarnecki, K., Antkiewicz, M., Diskin, Z., Wąsowski, A., Rayside, D.: Example-driven modeling. Model = Abstractions + Examples. In: New Ideas and Emerging Results (NIER) track of ICSE’13 (2013)Google Scholar
  7. 7.
    Balogh, A., Varró, D.: Advanced model transformation language constructs in the VIATRA2 framework. In: Proceedings of the 2006 ACM Symposium on Applied Computing, pp. 1280–1287. ACM (2006)Google Scholar
  8. 8.
    Batot, E.: Generating examples for knowledge abstraction in MDE: a multi-objective framework. In: SRC@ MoDELS, pp. 1–6 (2015)Google Scholar
  9. 9.
    Bauer, J., Baader, F.: Model exploration to support understanding of ontologies. Ph.D. thesis, Master’s thesis, Technische Universität Dresden (2009)Google Scholar
  10. 10.
    Brosch, P., Langer, P., Seidl, M., Wieland, K., Wimmer, M., Kappel, G., Retschitzegger, W., Schwinger, W.: An example is worth a thousand words: composite operation modeling by-example. In: Model Driven Engineering Languages and Systems, Volume 5795 of Lecture Notes in Computer Science, pp. 271–285 (2009)Google Scholar
  11. 11.
    Cerpa, N., Verner, J.M.: Why did your project fail? Commun. ACM 52(12), 130–134 (2009)CrossRefGoogle Scholar
  12. 12.
    Chi, M.T., Feltovich, P.J., Glaser, R.: Categorization and representation of physics problems by experts and novices. Cogn. Sci. 5(2), 121–152 (1981)CrossRefGoogle Scholar
  13. 13.
    Cho, H., Gray, J., Syriani, E.: Creating visual domain-specific modeling languages from end-user demonstration. In: Proceedings of the 4th International Workshop on Modeling in Software Engineering, MiSE ’12, pp. 22–28 (2012)Google Scholar
  14. 14.
    Cornelissen, B., Zaidman, A., Van Deursen, A., Moonen, L., Koschke, R.: A systematic survey of program comprehension through dynamic analysis. IEEE Trans. Softw. Eng. 35(5), 684–702 (2009)CrossRefGoogle Scholar
  15. 15.
    Demuth, B.: The dresden OCL toolkit and its role in information systems development. In: Proceedings of the 13th International Conference on Information Systems Development (ISDŠ2004), vol. 7 (2004)Google Scholar
  16. 16.
    Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock, J., Larsen, P. (eds.) FME ’93: Industrial-Strength Formal Methods, volume 670 of Lecture Notes in Computer Science, pp. 268–284. Springer, Berlin (1993)Google Scholar
  17. 17.
    Dobing, B., Parsons, J.: How UML is used. Commun. ACM 49(5), 109–113 (2006)CrossRefGoogle Scholar
  18. 18.
    Dobing, B., Parsons, J.: Dimensions of UML diagram use: a survey of practitioners. J. Database Manag. 19(1), 1–18 (2008)CrossRefGoogle Scholar
  19. 19.
    Ehrig, K., Küster, J., Täntzer, G.: Generating instance models from meta models. Softw. Syst. Model. 8(4), 479–500 (2009)CrossRefGoogle Scholar
  20. 20.
    Fisler, K., Krishnamurthi, S., Meyerovich, L.A., Tschantz, M.C.: Verification and change-impact analysis of access-control policies. In: Proceedings of the 27th International Conference on Software Engineering, pp. 196–205. ACM (2005)Google Scholar
  21. 21.
    Fleurey, F., Baudry, B., Muller, P.-A., Le Traon, Y.: Qualifying input test data for model transformations. Softw. Syst. Model. 8(2), 185–203 (2009)CrossRefGoogle Scholar
  22. 22.
    Gick, M.L., Holyoak, K.J.: Schema induction and analogical transfer. Cogn. Psychol. 15(1), 1–38 (1983)CrossRefGoogle Scholar
  23. 23.
    Gick, M.L., Paterson, K.: Do contrasting examples facilitate schema acquisition and analogical transfer? Can. J. Psychol. 46(4), 539–550 (1992)CrossRefGoogle Scholar
  24. 24.
    Gogolla, M., Büttner, F., Richters, M.: USE: a UML-based specification environment for validating UML and OCL. Sci. Comput. Progr. 69(1), 27–34 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Goldstone, R.L., Son, J.Y.: The transfer of scientific principles using concrete and idealized simulations. J. Learn. Sci. 14(1), 69 (2005)CrossRefGoogle Scholar
  26. 26.
    Guan, P., Reiss, L., Hirshberg, D.A., Weiss, A., Black, M.J.: Drape: dressing any person. ACM Trans. Graph. 31(4), 35 (2012)CrossRefGoogle Scholar
  27. 27.
    Halbert, D.C.: Programming by example. Ph.D. thesis, University of California, Berkeley (1984)Google Scholar
  28. 28.
    Harel, D., Marron, A., Weiss, G.: Behavioral programming. Commun. ACM 55(7), 90–100 (2012)CrossRefGoogle Scholar
  29. 29.
    Hutchinson, J., Whittle, J., Rouncefield, M., Kristoffersen, S.: Empirical assessment of MDE in industry. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pp. 471–480, New York. ACM (2011)Google Scholar
  30. 30.
    Jackson, D.: Software Abstractions: Logic, Language, and Analysis. MIT Press, London (2012)Google Scholar
  31. 31.
    Jackson, E.K., Kang, E., Dahlweid, M., Seifert, D., Santen, T.: Components, platforms and possibilities: towards generic automation for MDA. In: Proceedings of the Tenth ACM International Conference on Embedded Software, pp. 39–48. ACM (2010)Google Scholar
  32. 32.
    Janzen, D., Saiedian, H.: Test-driven development concepts, taxonomy, and future direction. Computer 38(9), 43–50 (2005)CrossRefGoogle Scholar
  33. 33.
    Juristo, N., Moreno, A.M.: Basics of Software Engineering Experimentation, 1st edn. Springer, Berlin (2010)zbMATHGoogle Scholar
  34. 34.
    Kalyuga, S.: Knowledge elaboration: a cognitive load perspective. Cognitive load in interactive knowledge construction. Learn. Instr. 19(5), 402–410 (2009)CrossRefGoogle Scholar
  35. 35.
    Kappel, G., Langer, P., Retschitzegger, W., Schwinger, W., Wimmer, M.: Conceptual modelling and its theoretical foundations. In: Chapter Model Transformation by-Example: A Survey of the First Wave, pp. 197–215 (2012)Google Scholar
  36. 36.
    Karasneh, B., Jolak, R., Chaudron, M.R.V.: Using examples for teaching software design: an experiment using a repository of UML class diagrams. In: 2015 Asia–Pacific Software Engineering Conference (APSEC), pp. 261–268 (2015)Google Scholar
  37. 37.
    Kessentini, M., Sahraoui, H., Boukadoum, M., Omar, O.B.: Search-based model transformation by example. Softw. Syst. Model. 11(2), 209–226 (2012)CrossRefGoogle Scholar
  38. 38.
    Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)CrossRefGoogle Scholar
  39. 39.
    Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)CrossRefGoogle Scholar
  40. 40.
    Lauesen, S., Vinter, O.: Preventing requirement defects: an experiment in process improvement. Requir. Eng. 6(1), 37–50 (2001)zbMATHCrossRefGoogle Scholar
  41. 41.
    Likert, R.: A technique for the measurement of attitudes. Arch. Psychol. 140, 1–55 (1932)Google Scholar
  42. 42.
    López-Fernández, J.J., Cuadrado, J.S., Guerra, E., de Lara, J.: Example-driven meta-model development. Softw. Syst. Model. 14, 1–25 (2013)Google Scholar
  43. 43.
    López-Fernández, J.J., Guerra, E., de Lara, J.: Combining unit and specification-based testing for meta-model validation and verification. Inf. Syst. 62, 104–135 (2016)CrossRefGoogle Scholar
  44. 44.
    Magicdraw. Accessed 21 Oct 2017
  45. 45.
    Maoz, S., Ringert, J.O., Rumpe, B.: Modal object diagrams. In: European Conference on Object-Oriented Programming, pp. 281–305. Springer (2011)Google Scholar
  46. 46.
    Mendel, L.: Modeling by example. Ph.D. thesis, Massachusetts Institute of Technology (2007)Google Scholar
  47. 47.
    Montaghami, V., Rayside, D.: Extending alloy with partial instances. In: Proceedings of Abstract State Machines, Alloy, B, VDM, and Z: Third International Conference, ABZ 2012, June 18–21, pp. 122–135 (2012)Google Scholar
  48. 48.
    Moody, D.: The “physics” of notations: toward a scientific basis for constructing visual notations in software engineering. IEEE Trans. Softw. Eng. 35(6), 756–779 (2009)CrossRefGoogle Scholar
  49. 49.
    Nebeling, M., Geel, M., Norrie, M.C.: Engineering information management tools by example. In: Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces, pp. 313–320. ACM (2014)Google Scholar
  50. 50.
    Nelson, T., Saghafi, S., Dougherty, D.J., Fisler, K., Krishnamurthi, S.: Aluminum: principled scenario exploration through minimality. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 232–241. IEEE Press (2013)Google Scholar
  51. 51.
    North, D.: Introducing BDD. Better Softw. 12, 1 (2006)Google Scholar
  52. 52.
    Nugroho, A.: Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf. Softw. Technol. 51(12), 1670–1685 (2009)CrossRefGoogle Scholar
  53. 53.
    Offutt, J., Abdurazik, A.: Generating tests from UML specifications. In: France, R., Rumpe, B. (eds.) UML’99—The Unified Modeling Language, Volume 1723 of Lecture Notes in Computer Science, pp. 416–429 (1999)Google Scholar
  54. 54.
    Olkin, I.: Contributions to Probability and Statistics: Essays in Honor of Harold Hotelling. Stanford University Press, Stanford (1960)zbMATHGoogle Scholar
  55. 55.
    Park, S.S.: Communicating domain knowledge through example-driven story testing. Master’s thesis, University of Calgary, Alberta (2011)Google Scholar
  56. 56.
    Petre, M.: UML in practice. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 722–731 (2013)Google Scholar
  57. 57.
    Purchase, H.C., Colpoys, L., McGill, M., Carrington, D., Britton, C.: UML class diagram syntax: an empirical study of comprehension. In: Proceedings of the 2001 Asia–Pacific Symposium on Information Visualisation—Volume 9, APVis’01, pp. 113–120. Australian Computer Society, Inc. (2001)Google Scholar
  58. 58.
    Ricca, F., Di Penta, M., Torchiano, M., Tonella, P., Ceccato, M.: The role of experience and ability in comprehension tasks supported by UML stereotypes. In: 29th International Conference on Software Engineering, pp. 375–384 (2007)Google Scholar
  59. 59.
    Sadilek, D.A., Weißleder, S.: Testing metamodels. In: Model Driven Architecture–Foundations and Applications, pp. 294–309. Springer, Berlin (2008)Google Scholar
  60. 60.
    Salay, R., Chechik, M., Horkoff, J., Sandro, A.D.: Managing requirements uncertainty with partial models. Requir. Eng. 18(2), 107–128 (2013)CrossRefGoogle Scholar
  61. 61.
    Scanniello, G., Ricca, F., Torchiano, M.: On the effectiveness of the UML object diagrams: a replicated experiment. In: 15th Annual Conference on Evaluation Assessment in Software Engineering (EASE 2011), pp. 76–85 (2011)Google Scholar
  62. 62.
    Sun, Y., White, J., Gray, J.: Model transformation by demonstration. In: Schurr, A., Selic, B. (eds.) Model Driven Engineering Languages and Systems, pp. 712–726. Springer, Berlin (2009)Google Scholar
  63. 63.
    Supplemental materials for the controlled experiment presented in Zayan et al.: Effects of Using Examples on Structural Model Comprehension (2013). Accessed 21 Oct 2017
  64. 64.
    The R project for statistical computing. Accessed 21 Oct 2017
  65. 65.
    Torchiano, M.: Empirical assessment of UML static object diagrams. In: Proceedings of 12th IEEE International Workshop on Program Comprehension, pp. 226–230 (2004)Google Scholar
  66. 66.
    Torlak, E., Jackson, D.: Kodkod: a relational model finder. In: Tools and Algorithms for the Construction and Analysis of Systems, pp. 632–647. Springer, Berlin (2007)Google Scholar
  67. 67.
    van Gog, T., Kester, L., Paas, F.: Effects of worked examples, example-problem, and problem-example pairs on novices’ learning. Contemp. Educ. Psychol. 36(3), 212–218 (2011)CrossRefGoogle Scholar
  68. 68.
    van Solingen, R., Basili, V., Caldiera, G., Rombach, H.D.: Goal Question Metric (GQM) Approach. Wiley, New York (2002)CrossRefGoogle Scholar
  69. 69.
    Warmer, J., Kleppe, A.: The Object Constraint Language: Getting Your Models Ready for MDA, 2nd edn. Addison-Wesley Longman Publishing Co. Inc., Boston (2003)Google Scholar
  70. 70.
    Welch’s t test. Accessed 21 Oct 2017
  71. 71.
    Whittle, J.: What do 449 MDE practitioners think about MDE? In: EESSMod (2011)Google Scholar
  72. 72.
    Wimmer, M., Strommer, M., Kargl, H., Kramler, G.: Towards model transformation generation by-example. In: 40th Annual Hawaii International Conference on System Sciences, 2007. HICSS 2007, pp. 285b–285b. IEEE (2007)Google Scholar
  73. 73.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, Norwell (2000)zbMATHCrossRefGoogle Scholar
  74. 74.
    Yusuf, S., Kagdi, H., Maletic, J.I.: Assessing the comprehension of UML class diagrams via eye tracking. In: 15th International Conference on Program Comprehension (ICPC’07), pp. 113–122. IEEE Computer Society (2007)Google Scholar
  75. 75.
    Zayan, D., Antkiewicz, M., Czarnecki, K.: Effects of using examples on structural model comprehension: a controlled experiment. In: Proceedings of the 36th International Conference on Software Engineering, pp. 955–966. ACM (2014)Google Scholar
  76. 76.
    Zloof, M.: QBE/OBE: a language for office and business automation. Computer 14(5), 13–22 (1981)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Dina Zayan
    • 1
  • Atrisha Sarkar
    • 1
  • Michał Antkiewicz
    • 1
  • Rita Suzana Pitangueira Maciel
    • 2
  • Krzysztof Czarnecki
    • 1
  1. 1.University of WaterlooWaterlooCanada
  2. 2.Federal University of BahiaSalvadorBrazil

Personalised recommendations