Empirical study on the effectiveness and efficiency of model-driven architecture techniques


Previous studies have reported conflicting opinions on the feasibility of model-driven architecture (MDA). Studies have investigated the mechanics of MDA, but few have examined its effectiveness and efficiency from a developer’s perception. This study conducted empirical research in which a system was implemented by subjects using MDA; afterward, evaluated its perceived efficiency and effectiveness. In the model construction phase, Unified Modeling Language and Object Constraint Language were perceived as effective and efficient. In the model transformation phase, the query/view/transformation standard was perceived as marginally efficient rather than effective, and the round-trip engineering phase was not perceived as effective or efficient. These findings are explained using 12 themes identified in subjects’ opinions. This study may help scholars understand the importance of efficiency and effectiveness on MDA techniques and facilitate the development of more acceptable and practical MDA.

This is a preview of subscription content, log in to check access.

Fig. 1


  1. 1.

    Abrahao, S., Bourdeleau, F., Cheng, B., Kokaly, S., Paige, R., Stöerrle, H., Whittle, J.: User experience for model-driven engineering: challenges and future directions. In: Proceedings of the MODELS, pp. 229–236. IEEE, Austin, TX (2017)

  2. 2.

    Abran, A., Khelifi, A., Suryn, W., Seffah, A.: Usability meanings and interpretations in ISO standards. Softw. Qual. J. 11(4), 325–338 (2003)

    Article  Google Scholar 

  3. 3.

    Agarwal, R., Sinha, A.P.: Object-oriented modeling with UML: a study of developers’ perceptions. Commun. ACM 46(9), 248–256 (2003)

    Article  Google Scholar 

  4. 4.

    Albuquerque, D., Cafeo, B., Garcia, A., Barbosa, S., Abrahao, S., Ribeiro, A.: Quantifying usability of domain-specific languages: an empirical study on software maintenance. J Syst. Softw. 101(C), 245–259 (2015)

    Article  Google Scholar 

  5. 5.

    Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E., Moret-Bonillo, V.: Usability: a critical analysis and a taxonomy. Int. J. Hum.Comput. Int. 26(1), 53–74 (2010)

    Article  Google Scholar 

  6. 6.

    Al-Qutaish, R.E.: An investigation of the weaknesses of the ISO 9126 international standard. In: Proceedings of the ICCEE, pp. 275–279. IEEE, Dubai, United Arab Emirates (2009)

  7. 7.

    Armonas, A., Nemuraite, L.: Using Attributes and merging algorithms for transforming OCL expressions to code. Inf. Technol. Control 38(4), 283–293 (2009)

    Google Scholar 

  8. 8.

    Asadi, M., Ramsin, R.: MDA-based methodologies: an analytical survey. In: Schieferdecker I., Hartman A. (eds.) Model Driven Architecture—Foundations and Applications. ECMDA-FA 2008. Lecture Notes in Computer Science, vol. 5095, pp. 419–431. Springer, Berlin (2008)

  9. 9.

    Bajwa, I.S., Lee, M.G.: Transformation rules for translating business rules to OCL constraints. In: France R.B., Kuester J.M., Bordbar B., Paige R.F. (eds.) Modelling Foundations and Applications. ECMFA 2011. Lecture Notes in Computer Science, vol. 6698, pp. 132–143. Springer, Berlin (2011)

  10. 10.

    Barišić, A., Amaral, V., Goulão, M., Barroca, B.: Quality in use of domain-specific languages: a case study. In: Proceedings of the 3rd ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools, pp. 65–72. ACM, Portland, OR (2011)

  11. 11.

    Bendraou, R., Jézéquel, J.-M., Gervais, M.-P., Blanc, X.: A comparison of six UML-based languages for software process modeling. IEEE Trans. Softw. Eng. 36(5), 662–675 (2010)

    Article  Google Scholar 

  12. 12.

    Bennett, J., Cooper, K., Dai, L.: Aspect-oriented model-driven skeleton code generation: a graph-based transformation approach. Sci. Comput. Program. 75(8), 689–725 (2010)

    MATH  Article  Google Scholar 

  13. 13.

    Berardinelli, L., Bernardo, M., Cortellessa, V., Di Marco, A.: Multidimensional context modeling applied to non-functional analysis of software. Softw. Syst. Model (2017). https://doi.org/10.1007/s10270-017-0645-2

    Article  Google Scholar 

  14. 14.

    Berg, K.E., Latin, R.W.: Essentials of Research Methods in Health, Physical Education, Exercise Science, and Recreation. Lippincott Williams & Wilkins, Philadelphia (2008)

    Google Scholar 

  15. 15.

    Bevan, N.: Measuring usability as quality of use. Softw. Qual. J. 4(2), 115–130 (1995)

    Article  Google Scholar 

  16. 16.

    Bevan, N., Carter, J., Earthy, J., Geis, T., Harker, S.: New ISO standards for usability, usability reports and usability measures. In: Kurosu, M. (ed.) Human-Computer Interaction. Theory, Design, Development and Practice. HCI 2016. Lecture Notes in Computer Science, vol. 9731, pp. 268–278. Springer, Cham (2016)

    Google Scholar 

  17. 17.

    Bhiri, M.T., Mourad, K., Graiet, M., Aniorté, P.: UML/OCL and refinement. In: Proceedings of the International Conference on Engineering of Computer Based Systems, pp. 149–158. IEEE, Las Vegas (2011)

  18. 18.

    Boehm, B.W., Brown, J.R., Kaspar, H.: Characteristics of Software Quality. North-Holland Publishing Co., Amsterdam (1978)

    Google Scholar 

  19. 19.

    Braga, C., Menezes, R., Comicio, T., Santos, C., Landim, E.: Transformation contracts in practice. IET Softw. 6(1), 16–32 (2012)

    Article  Google Scholar 

  20. 20.

    Cabot, J., Teniente, E.: Transformation techniques for OCL constraints. J. Sci. Comput. Program. 68(3), 152–168 (2007)

    MathSciNet  MATH  Google Scholar 

  21. 21.

    Chen, X., Zhou, D., Wu, L.: Theory of automatic transformation from Computation Independent Model to Platform Independent Model for MDA. In: Proceedings of the ICETA, pp. 235–244. Tsingtao, China (2014)

  22. 22.

    Compeau, D., Higgins, C.A., Huff, S.: Social cognitive theory and individual reactions to computing technology: a longitudinal study. MIS Quart. 23(2), 145–158 (1999)

    Article  Google Scholar 

  23. 23.

    Curtis, B., Krasner, H., Iscoe, N.: A field study of the software design process for large systems. Commun. ACM 31(11), 1268–1287 (1988)

    Article  Google Scholar 

  24. 24.

    Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart. 13(3), 319–340 (1989)

    Article  Google Scholar 

  25. 25.

    De Castro, V., Marcos, E., Vara, J.M.: Applying CIM-to-PIM model transformations for the service-oriented development of information systems. Inf. Softw. Technol. 53(1), 87–105 (2011)

    Article  Google Scholar 

  26. 26.

    De Sousa Saraiva, J., Da Silva, A.R.: Chapter 5: Evaluation of MDE tools from a metamodeling perspective. In: Siau, K., Erickson, J. (eds.) Principal Advancements in Database Management Technologies, pp. 105–131. IGI Global, Hershey (2010). https://doi.org/10.4018/978-1-60566-904-5.ch005

  27. 27.

    Diskin, Z., Wider, A., Gholizadeh, H., Czarnecki, K.: Towards a rational taxonomy for increasingly symmetric model synchronization. In: Di Ruscio D., Varró D. (eds.) Theory and Practice of Model Transformations. ICMT 2014. Lecture Notes in Computer Science, vol. 8568, pp. 57–73. Springer, Cham (2014)

  28. 28.

    Domínguez-Mayo, F.J., Escalona, M.J., Mejías, M., Ross, M., Staples, G.: A quality management based on the quality model life cycle. Comput. Stand. Int. 34(4), 396–412 (2012)

    Article  Google Scholar 

  29. 29.

    Fabra, J., De Castro, V., Álvarez, P., Marcos, E.: Automatic execution of business process models: exploiting the benefits of model-driven engineering approaches. J. Syst. Softw. 85(3), 607–625 (2012)

    Article  Google Scholar 

  30. 30.

    Falessi, D., Juristo, N., Wohlin, C., Turhan, B., Münch, J., Jedlitschka, A., Oivo, M.: Empirical software engineering experts on the use of students and professionals in experiments. Empir. Softw. Eng. 23(1), 452–489 (2018)

    Article  Google Scholar 

  31. 31.

    Fouad, A., Phalp, K., Kanyaru, J.M., Jeary, S.: Embedding requirements within Model-Driven Architecture. Softw. Qual. J. 19(2), 411–430 (2011)

    Article  Google Scholar 

  32. 32.

    France, R.B., Rumpe, B.: Model-driven development of complex software: a research roadmap. In: Proceedings of the International Conference on Software Engineering, pp. 37–54. IEEE, Minneapolis (2007)

  33. 33.

    Gammaitoni, L., Kelsen, P.: F-Alloy: a relational model transformation language based on Alloy. Softw. Syst. Model. 1–35 (2017)

  34. 34.

    Gopher, D., Braune, R.: On the psychophysics of workload: Why bother with subjective measures? Hum. Factors 26(5), 519–532 (1984)

    Article  Google Scholar 

  35. 35.

    Guo, Y., Guan, W., Sierszecki, K., Angelov, C.: Building a software development environment for embedded systems. In: Proceedings of the International Conference on Green Computing and Communications and International Conference on Cyber, Physical and Social Computing, pp. 916–921. IEEE/ACM, Washington, DC (2010)

  36. 36.

    Hailpern, B., Tarr, P.: Model-driven development: the good, the bad, and the ugly. IBM Syst. J. 45(3), 451–461 (2006)

    Article  Google Scholar 

  37. 37.

    Hawkins, J.L., Ivkovic, I., Kontogiannis, K., McKegney, R., Ng, W., Tong, T.: U.S. Patent No. 8,418,125. Washington, DC: U.S. Patent and Trademark Office (2013)

  38. 38.

    Höfer, A., Tichy, W.F.: Status of empirical research in software engineering. In Basili V.R., Rombach D., Schneider K., Kitchenham B., Pfahl D., Selby R.W. (eds.) Empirical Software Engineering Issues. Critical Assessment and Future Directions. Lecture Notes in Computer Science, vol. 4336, pp. 10–19. Springer, Berlin (2007)

  39. 39.

    ISO 25010: System and software quality models. https://www.iso.org/standard/35733.html (2011)

  40. 40.

    ISO 9126-4, Quality in use metrics (2004)https://www.iso.org/standard/39752.html. Accessed 07 Oct 2018

  41. 41.

    ISO 9241-11: Usability: Definitions and concepts (2018). https://www.iso.org/standard/63500.html. Accessed 13 Feb 2018

  42. 42.

    Kuhlmann, M., Sohr, K., Gogolla, M.: Employing UML and OCL for designing and analysing role-based access control. Math. Struct. Comput. Sci. 23(04), 796–833 (2013)

    MathSciNet  Article  Google Scholar 

  43. 43.

    Lehman, M.M.: Process models, process programs, programming support. In: Proceedings of the 9th International Conference on Software Engineering, pp. 14–16. IEEE, Los Alamitos (1987)

  44. 44.

    Lewis, J.R.: Usability: lessons learned … and yet to be learned. Int. J. Hum. Comput. Int. 30(9), 663–684 (2014)

    Article  Google Scholar 

  45. 45.

    Masood, Z., Xuequn, S., Yousaf, J.: Usability evaluation framework for software engineering methodologies. Lect. Notes Softw. Eng. 2(3), 225–232 (2014)

    Article  Google Scholar 

  46. 46.

    Mattsson, A., Fitzgerald, B., Lundell, B., Lings, B.: An approach for modeling architectural design rules in UML and its application to embedded software. ACM Trans. Softw. Eng. Methods 21(2), 10 (2012)

    Article  Google Scholar 

  47. 47.

    McCall, J.A., Richards, P.K., Walters, G.F.: Factors in software quality. Vol. 1. Concepts and definitions of software quality. Technical Report ADA049014. General Electric Co., Sunnyvale, California (1977)

  48. 48.

    McMeekin, D., von Konsky, B., Robey, M., Cooper, D.: The significance of participant experience when evaluating software inspection techniques. In: Proceedings of the ASWEC, pp. 200–209. IEEE, Gold Coast (2009)

  49. 49.

    Mepham, W.: Discrete event calculus using Semantic Web technologies. Ph.D. Thesis, University of Glamorgan, Trefforest, Wales (2012)

  50. 50.

    Meservy, T.O., Fenstermacher, K.D.: Transforming software development: an MDA road map. IEEE Comput. 38(9), 52–58 (2005)

    Article  Google Scholar 

  51. 51.

    Mohagheghi, P., Aagedal, J.: Evaluating quality in model-driven engineering. In: Proceedings of the International Workshop on Modeling in Software, IEEE, Minneapolis (2007)

  52. 52.

    Mohagheghi, P., Dehlen, V.: Developing a quality framework for model-driven engineering. In: Proceedings of the International Conference on Model Driven Engineering Languages and Systems, pp. 275–286. Springer, Berlin (2007)

  53. 53.

    Mohagheghi, P., Dehlen, V., Neple, T.: Definitions and approaches to model quality in model-based software development—a review of literature. Inf. Softw. Technol. 51(12), 1646–1669 (2009)

    Article  Google Scholar 

  54. 54.

    Moody, D.: The “physics” of notations: toward a scientific basis for constructing visual notations in software engineering. IEEE Trans. Softw. Eng. 35(6), 756–779 (2009)

    Article  Google Scholar 

  55. 55.

    Moore, G.C., Benbasat, I.: Development of an instrument to measure the perceptions of adopting an information technology innovation. Inf. Syst. Res. 2(3), 192–222 (1991)

    Article  Google Scholar 

  56. 56.

    Mussbacher, G., Amyot, D., Breu, R., Bruel, J.M., Cheng, B.H., Collet, P., et al.: The relevance of model-driven engineering thirty years from now. In: Dingel J., Schulte W., Ramos I., Abrahão S., Insfran E. (eds.) Model-Driven Engineering Languages and Systems. MODELS 2014. Lecture Notes in Computer Science, vol. 8767, pp. 183–200. Springer, Cham (2014)

  57. 57.

    Nielsen, J., Mack, R.L.: Usability Inspection Methods. Wiley, New York (1994)

    Google Scholar 

  58. 58.

    Object Management Group. MDA Guide rev. 2.0 (2014). http://www.omg.org/cgi-bin/doc?ormsc/14-06-01.pdf. Accessed 13 Feb 2018

  59. 59.

    O’Donnell, R.D., Eggemeier, F.T.: Workload assessment methodology. In: Boff, K.R., Kaufman, L., Thomas, J.P. (eds.) Handbook of Perception and Human Performance, pp. 1–49. Wiley, New York (1986)

    Google Scholar 

  60. 60.

    Paas, F., Van Merriënboer, J.J.G., Adam, J.J.: Measurement of cognitive-load in instructional research. Percept. Mot. Skills 79(1), 419–430 (1994)

    Article  Google Scholar 

  61. 61.

    Panach, J.I., Condori-Fernandez, N., Vos, T., Aquino, N., Valverde, F.: Early usability measurement in model-driven development: definition and empirical evaluation. Int. J. Softw. Eng. Knowl. 21(03), 339–365 (2011)

    Article  Google Scholar 

  62. 62.

    Perez-Medina, J.L., Dupuy-Chessa, S., Front, A.: A survey of model driven engineering tools for user interface design. In: Winckler M., Johnson H., Palanque P. (eds.) Task Models and Diagrams for User Interface Design. TAMODIA 2007. Lecture Notes in Computer Science, vol. 4849. pp. 84–97. Springer, Berlin (2007)

  63. 63.

    Petre, M.: UML in practice. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 722–731. IEEE, Piscataway (2013)

  64. 64.

    Porter, A., Votta, L.: Comparing detection methods for software requirements inspections: a replication using professional subjects. Empir. Softw. Eng. 3(4), 355–379 (1998)

    Article  Google Scholar 

  65. 65.

    Ricca, F., Di Penta, M., Torchiano, M., Tonella, P., Ceccato, M.: How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans. Softw. Eng. 36(1), 96–118 (2010)

    Article  Google Scholar 

  66. 66.

    Runeson, P.: Using students as experiment subjects—an analysis on graduate and freshmen student data. In: Proceedings of the 7th International Conference on Empirical Assessment in Software Engineering, pp. 95–102. Keele, Staffordshire (2003)

  67. 67.

    Salman, I., Misirli, A.T., Juristo, N.: Are students representatives of professionals in software engineering experiments?. In: Proceedings of the ICSE, pp. 666–676. IEEE, Florence (2015)

  68. 68.

    Seddon, P.B.: A respecification and extension of the DeLone and McLean model of IS success. Inf. Syst. Res. 8(3), 240–253 (1997)

    Article  Google Scholar 

  69. 69.

    Seffah, A., Donyaee, M., Kline, R.B., Padda, H.K.: Usability measurement and metrics: a consolidated model. Softw. Qual. J. 14(2), 159–178 (2006)

    Article  Google Scholar 

  70. 70.

    Shin, S.S.: A study on the difficulties of learning phase transition in object-oriented analysis and design from the viewpoint of semantic distance. IEEE Trans. Educ. 58(2), 117–123 (2015)

    Article  Google Scholar 

  71. 71.

    Shin, S.S.: Evaluation of Model Driven Architecture-based instruction for understanding phase transitions in object-oriented analysis and design. ACM Trans. Comput. Educ. 16(4), 17 (2016)

    Article  Google Scholar 

  72. 72.

    Sjoberg, D.I., Anda, B., Arisholm, E., Dyba, T., Jorgensen, M., Karahasanovic, A., Koren, E.F., Vokác, M.: Conducting realistic experiments in software engineering. In: Proceedings of the International Symposium on Empirical Software Engineering, pp. 17–26. IEEE, Nara (2002)

  73. 73.

    Sjøberg, D.I., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N.K., Rekdal, A.C.: A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9), 733–753 (2005)

    Article  Google Scholar 

  74. 74.

    Sorokin, L., Montero, F., Märtin, C.: Flex RIA development and usability evaluation. In: Weske M., Hacid MS., Godart C. (eds.) Web Information Systems Engineering—WISE 2007 Workshops. WISE 2007. Lecture Notes in Computer Science, vol. 4832, pp. 447–452. Springer, Berlin (2007)

  75. 75.

    Stevens, P.: Bidirectional model transformations in QVT: semantic issues and open questions. Softw. Syst. Model. 9(1), 7–20 (2010)

    MathSciNet  Article  Google Scholar 

  76. 76.

    Stoter, J., Visser, T., van Oosterom, P., Quak, W., Bakker, N.: A semantic-rich multi-scale information model for topography. Int. J. Geogr. Inf. Sci. 25(5), 739–763 (2011)

    Article  Google Scholar 

  77. 77.

    Thompson, R.L., Higgins, C.A., Howell, J.M.: Personal computing: toward a conceptual model of utilization. MIS Quart. 15(1), 124–143 (1991)

    Article  Google Scholar 

  78. 78.

    Uhl, A.: Model-driven development in the enterprise. IEEE Softw. 25(1), 46–49 (2008)

    Article  Google Scholar 

  79. 79.

    Vara, J.M., Marcos, E.: A framework for model-driven development of information systems: technical decisions and lessons learned. J. Syst. Softw. 85(10), 2368–2384 (2012)

    Article  Google Scholar 

  80. 80.

    Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Quart. 27(3), 425–478 (2003)

    Article  Google Scholar 

  81. 81.

    Wang, S., Ma, J., He, Q., Wan, J.: Formal behavior modeling and effective automatic refinement. Inf. Sci. 180(20), 3894–3913 (2010)

    Article  Google Scholar 

  82. 82.

    Whittle, J., Hutchinson, J., Rouncefield, M., Burden, H., Heldal, R.: A taxonomy of tool-related issues affecting the adoption of model-driven engineering. Softw. Syst. Model. 16(2), 313–331 (2017)

    Article  Google Scholar 

  83. 83.

    Whittle, J., Hutchinson, J., Rouncefield, M.: The state of practice in model-driven engineering. IEEE Softw. 31(3), 79–85 (2014)

    Article  Google Scholar 

  84. 84.

    Yeh, R.K.J., Teng, J.T.C.: Extended conceptualisation of perceived usefulness: empirical test in the context of information system use continuance. Behav. Inf. Technol. 31(5), 525–540 (2012)

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Shin-Shing Shin.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by Dr. Timothy Lethbridge.


Appendix A: Question items used in estimating effectiveness and efficiency of MDA techniques

Anchors for all constructs are 1–7 (1 = Strongly Disagree; 7 = Strongly Agree) Items are adapted from Davis [24] and Yeh and Teng [84].


Appendix B: Narrative of the purchase book use case of the e-bookstore system

Use case name Purchase book
Primary business actor Customer
Other participating actors Warehouse
Description This use case describes the event of a customer browsing and submitting an order for books. On completion, the customer will be sent an order confirmation
Precondition There are books in this online bookshop
Trigger This use case begins when a customer accesses the book list
Typical course of event Actor action System response
Step 1: A customer accesses the book list on the web page and clicks an item he/she wants
Step 3: The customer sets the quantity ordered and presses “Add Item” button
Step 5: The customer presses the “Check out” button
Step 7: The customer enters his ID and password
Step 12: The customer selects one of the options
1. Go back to catalog
2. Remove books from the shopping cart or reset quantity
3. Check out
Step 2: The system responds by providing the book information
Step 4: The system verifies the product availability
Step 6: The system requests the customer to provide his ID and password
Step 8: The system verifies the customer’s account
Step 9: The system determines the price to be charged to the customer
Step 10: The system determines the total cost of the order
Step 11: The system displays the shopping cart information, including item, price, quantity, total, and so on. There are three options for the customer in the next step
Step 13: The system responds
1. Display book list
2. Remove books from the shopping cart or reset quantity
3. Transfer the shopping cart into an order and send the order confirmation to the customer
Alternate courses Alt-Step 1: The customer wants to leave the system, even though the shopping cart has books Alt-Step 2: The system deletes the shopping cart and terminates the use case
Conclusion This use case concludes when the customer receives a confirmation of the order
Postcondition The order has been recorded
Implementation Constraints and Specifications: Web screen to be provided for customers

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Shin, S. Empirical study on the effectiveness and efficiency of model-driven architecture techniques. Softw Syst Model 18, 3083–3096 (2019). https://doi.org/10.1007/s10270-018-00711-y

Download citation


  • Model-driven architecture
  • Effectiveness
  • Efficiency
  • Model transformation
  • Unified Modeling Language
  • Query/view/transformation