Software & Systems Modeling

, Volume 14, Issue 4, pp 1349–1365 | Cite as

Evaluating the effort of composing design models: a controlled experiment

  • Kleinner FariasEmail author
  • Alessandro Garcia
  • Jon Whittle
  • Christina von Flach Garcia Chavez
  • Carlos Lucena
Special Section Paper


Model composition plays a key role in many tasks in model-centric software development, e.g., evolving UML diagrams to add new features or reconciling models developed in parallel by different software development teams. However, based on our experience in previous empirical studies, one of the main impairments for the widespread adoption of composition techniques is the lack of empirical knowledge about their effects on developers’ effort. This problem applies to both existing categories of model composition techniques, i.e., specification-based (e.g., Epsilon) and heuristic-based techniques (e.g., IBM RSA). This paper, therefore, reports on a controlled experiment that investigates the effort of (1) applying both categories of model composition techniques and (2) detecting and resolving inconsistencies in the output composed models. We evaluate the techniques in 144 evolution scenarios, where 2,304 compositions of elements of UML class diagrams were produced. The main results suggest that (1) the employed heuristic-based techniques require less effort to produce the intended model than the chosen specification-based technique, (2) there is no significant difference in the correctness of the output composed models generated by these techniques, and (3) the use of manual heuristics for model composition outperforms their automated counterparts.


Model composition effort Empirical studies Effort measurement 


  1. 1.
    France, R., Rumpe, B.: Model-driven development of complex software: a research roadmap. In: Future of Software Engineering at ICSE’07, pp. 37–54 (2007)Google Scholar
  2. 2.
    Mens, T.: A state-of-the-art survey on software merging. IEEE Trans. Softw. Eng. 28(5), 449–462 (2002)CrossRefGoogle Scholar
  3. 3.
    Clarke, S.: Composition of object-oriented software design models. Ph.D. thesis, Dublin City University (2001)Google Scholar
  4. 4.
  5. 5.
    IBM Rational Software Architect. (2013)
  6. 6.
    Farias, K.: Empirical evaluation of effort on composing design models. Ph.D. thesis, PUC-Rio, Rio de Janeiro, Brazil (2012)Google Scholar
  7. 7.
    Farias, K.: Empirical evaluation of effort on composing design models. In: 32nd ACM/IEEE International Conference on Software Engineering, Doctoral Symposium, vol. 2, pp. 405–408, Cape Town, South Africa (2010)Google Scholar
  8. 8.
    Clarke, S., Baniassad, E.: Aspect-Oriented Analysis and Design: The Theme Approach. Addison-Wesley, Upper Saddle River (2005)Google Scholar
  9. 9.
    Farias, K., Garcia, A., Whittle, J., Chavez, C., Lucena, C.: Evaluating the effort of composing design models: a controlled experiment, In: Proceedings of the 15th International Conference on Model-Driven Engineering Languages and Systems (MODELS’12), Innsbruck, Austria, vol. 7590, pp. 676–691 (2012)Google Scholar
  10. 10.
    Farias, K., Garcia, A., Whittle, J.: Assessing the impact of aspects on model composition effort. In: 9th International Conference on Aspect-Oriented Software Development (AOSD’12), Saint Malo, France, pp. 73–84 (2010)Google Scholar
  11. 11.
    Clarke, S., Walker, R.: Composition patterns: an approach to designing reusable aspects. In: 23rd International Conference on Software Engineering (ICSE’01), Toronto, ON, pp. 5–14 (2001)Google Scholar
  12. 12.
    Ricca, F., Penta, M., Torchiano, M., Tonella, P., Ceccato, M.: How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans. Softw. Eng. 96(1), 96–118 (2010)CrossRefGoogle Scholar
  13. 13.
    Wohlin, C., et al.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, Norwell (2000)CrossRefGoogle Scholar
  14. 14.
    Cook, T., Campbell, D., Day, A.: Quasi-experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin, Boston (1979)Google Scholar
  15. 15.
    Kitchenham, B., Al-Kilidar, H., Babar, M., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., Zhu, L.: Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Eng. 13(1), 97–112 (2008)CrossRefGoogle Scholar
  16. 16.
    Kitchenham, B.: Empirical paradigm: the role of experiments. In: Empirical Software Engineering Issues, pp. 25–32 (2006)Google Scholar
  17. 17.
    Sjøberg, D., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Koren, E., Vokác, M.: Conducting realistic experiments in software engineering. In: 1st International Symposium on Empirical Software Engineering, pp. 17–26 (2002)Google Scholar
  18. 18.
    Basili, V., Caldiera, G., Rombach, H.: The Goal Question Metric Paradigm. Encyclopedia of Software Engineering. Wiley, New York (1994)Google Scholar
  19. 19.
    Devore, J., et al.: Applied Statistics for Engineers and Scientists. Duxbury, North Scituate (1999)Google Scholar
  20. 20.
    Campbell, D., Russo, M.: Social Experimentation. SAGE Classics, Beverly Hills (1998)Google Scholar
  21. 21.
    Research method knowledge base: improving conclusion validity. (2011)
  22. 22.
    Jørgensen, M.: Practical guidelines for expert-judgment-based software effort estimation. IEEE Software, pp. 57–63 (2005)Google Scholar
  23. 23.
    Brewer, M.: Research design and issues of validity. In: Reis, H., Judd, C. (eds.) Handbook of Research Methods in Social and Personality Psychology. Cambridge University Press, Cambridge (2000)Google Scholar
  24. 24.
    Shadish, W., Cook, T., Campbell, D.: Experimental and Quasi-experimental Designs for Generalized Causal Inference. Houghton Mifflin, Boston (2002)Google Scholar
  25. 25.
    Mitchell, M., Jolley, J.: Research Design Explained, 4th edn. Harcourt, New York (2001)Google Scholar
  26. 26.
    Whittle, J., Jayaraman, P.: Synthesizing hierarchical state machines from expressive scenario descriptions. ACM Trans. Softw. Eng. Methodol. 19(3), 1–45 (2010)Google Scholar
  27. 27.
    Thaker, S., Batory, D., Kitchin, D., Cook, W.: Safe composition of product lines. In: 6th Generative Programming: Concepts and Experiences, Salzburg, Austria, pp. 95–104 (2007)Google Scholar
  28. 28.
    Klein, J., Hélouët, L., Jézéquel, J., Semantic-based weaving of scenarios. In: 5th Aspect-Oriented Software Development (AOSD’06), Bonn, Germany, pp. 27–38 (2006)Google Scholar
  29. 29.
    Asklund, U.: Identifying inconsistencies during structural merge. In: Proceedings of Nordic Workshop Programming Environment Research, pp. 86–96 (1994)Google Scholar
  30. 30.
    Farias, K., Garcia, A., Lucena, C.: Effects of stability on model composition effort: an exploratory study. J. Softw. Syst. Model. (SoSym) 12(1), 1–22 (2013)Google Scholar
  31. 31.
    Norris, N., Letkeman, K.: Governing and managing enterprise models: part 1. Introduction and concepts. IBM Developer Works, (2011)
  32. 32.
    Perry, D., Siya, P., Votta, L.: Parallel changes in large scale software development: an observational case study. In: International Conference on Software Engineering (ICSE’98), pp. 251–260 (1998)Google Scholar
  33. 33.
    Lange, C., Chaudron, M.: Effects of defects in UML models: an experimental investigation. In: International Conference on Software Engineering (ICSE’06), China, pp. 401–410 (2006)Google Scholar
  34. 34.
    Petre, M.: UML in practice. In: 35th International Conference on Software Engineering (ICSE 2013), San Francisco, CA, pp. 18–26 (2013)Google Scholar
  35. 35.
    Briand, L., Labiche, Y., Di Penta, M., BondocL, H.: An experimental investigation of formality in UML-based development. IEEE Trans. Softw. Eng. 31(10), 833–849 (2005)CrossRefGoogle Scholar
  36. 36.
    Unified Modeling Language: Infrastructure, Object Management Group (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Kleinner Farias
    • 1
    Email author
  • Alessandro Garcia
    • 2
  • Jon Whittle
    • 3
  • Christina von Flach Garcia Chavez
    • 4
  • Carlos Lucena
    • 2
  1. 1.Interdisciplinary Postgraduate Program in Applied Computing (PIPCA)University of Vale do Rio dos Sinos (Unisinos)São LeopoldoBrazil
  2. 2.OPUS Research Group, LES, Informatics DepartmentPontifical Catholic University of Rio de JaneiroRio de JaneiroBrazil
  3. 3.School of Computing and CommunicationsLancaster UniversityCity of LancasterUK
  4. 4.Department of Computer ScienceFederal University of BahiaSalvadorBrazil

Personalised recommendations