Empirical Evaluation of UML Modeling Tools–A Controlled Experiment

  • Safdar Aqeel Safdar
  • Muhammad Zohaib Iqbal
  • Muhammad Uzair Khan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9153)


Model driven software engineering (MDSE) has shown to provide mark improvement in productivity and quality of software products. UML is a standard modeling language that is widely used in the industry to support MDSE. To provide tool support for MDSE, a large number of UML modeling tools are available, ranging from open-source tools to commercial tools with high price tag. A common decision faced while applying UML in practice is the selection of an appropriate tool for modeling. In this paper we conduct a study to compare three of the well-known modeling tools: IBM Rational Software Architect (RSA), MagicDraw, and Papyrus. In this study we conducted an experiment with undergraduate and graduate students. The goal is to compare the productivity of the software engineers while modeling with the tools. We measure the productivity in terms of modeling effort required to correctly complete a task, learnability, time and number of clicks required, and memory load required for the software engineer to complete a task. Our results show that MagicDraw performed significantly better in terms of learnability, memory load, and completeness of tasks. In terms of time and number of clicks, IBM RSA was significantly better while modeling class diagrams and state machines when compared to Papyrus. However no single tool outperformed others in all the modeling tasks with respect to time and number of clicks.


Model driven software engineering UML
 Modeling tools Controlled experiment Empirical software engineering 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Iqbal, M.Z., Ali, S., Yue, T., Briand, L.: Applying UML/MARTE on industrial projects: challenges, experiences, and guidelines. Software & Systems Modeling, 1–19 (2014)Google Scholar
  2. 2.
    Khaled, L.: A comparison between UML tools. In: Second International Conference on Environmental and Computer Science, pp. 111–114. IEEE (2009)Google Scholar
  3. 3.
    Eichelberger, H., Eldogan, Y., Schmid, K.: A Comprehensive Survey of UML Compliance in Current Modelling Tools. Software Engineering 143, 39–50 (2009)Google Scholar
  4. 4.
    Heena, R.: A comparative study of UML tools. In: Proceedings of 11th International Conference on Advances in Computing and Artificial Intelligence, pp. 1–4 (2011)Google Scholar
  5. 5.
    Bellon, S., Koschke, R., Antoniol, G., Krinke, J., Merlo, E.: Comparison and evaluation of clone detection tools. IEEE Transactions on Software Engineering 33, 577–591 (2007)CrossRefGoogle Scholar
  6. 6.
    Ali, S., Yue, T., Briand, L.: Assessing quality and effort of applying aspect state machines for robustness testing: a controlled experiment. In: IEEE 6th International Conference on Software Testing, Verification and Validation, pp. 212–221. IEEE (2013)Google Scholar
  7. 7.
    Dias Neto, A.C., Subramanyan, R., Vieira, M., Travassos, G.H.: A survey on model-based testing approaches: a systematic review. In: Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies: Held in Conjunction with ASE Conference, pp. 31–36. ACM (2007)Google Scholar
  8. 8.
    Sengoz, Y.S., Jawawi, A., Deris, S.B.: A Comparison UML Tools for Real-time Software Modeling (2009)Google Scholar
  9. 9.
    Rani, T., Garg, S.: Comparison of different UML tool: Tool approach. International Journal Of Engineering And Computer Science 2, 1900–1908 (2013)Google Scholar
  10. 10.
    Bobkowska, A.E., Reszke, K.: Usability of UML modeling tools. In: Proceedings of the Conference on Software Engineering: Evolution and Emerging Technologies, pp. 75–86 (2005)Google Scholar
  11. 11.
    Smith, H.H.: On tool selection for illustrating the use of UML in system development. Journal of Computing Sciences in Colleges 19, 53–63 (2004)Google Scholar
  12. 12.
    Ali, S., Iqbal, M.Z., Arcuri, A., Briand, L.: A search-based OCL constraint solver for model-based test data generation. In: 11th International Conference on Quality Software (QSIC), pp. 41–50. IEEE (2011)Google Scholar
  13. 13.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer (2012)Google Scholar
  14. 14.
    Usman, M., Iqbal, M.Z., Khan, M.U.: A model-driven approach to generate mobile applications for multiple platforms. In: Asia-Pacific Software Engineering Conference (2014)Google Scholar
  15. 15.
    Jilani, A.A., Iqbal, M.Z., Khan, M.U.: A search based test data generation approach for model transformations. In: Di Ruscio, D., Varró, D. (eds.) ICMT 2014. LNCS, vol. 8568, pp. 17–24. Springer, Heidelberg (2014)Google Scholar
  16. 16.
    Khan, M.U., Iqbal, M.Z., Ali, S.: A Heuristic-based approach to refactor crosscutting behaviors in UML state machines. In: International Conference on Software Maintenance and Evolution. IEEE (2014)Google Scholar
  17. 17.
    Eriksson, H.-E., Penker, M., Lyons, B., Fado, D.: UML 2 toolkit. John Wiley & Sons (2003)Google Scholar
  18. 18.
    Binder, R.V.: Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Professional (2000)Google Scholar
  19. 19.
    Safdar, S.A., Iqbal, M.Z., Khan, M.U.: Empirical Evaluation of Productivity of Software Engineer in UML Modeling Tools- A Controlled Experiment. Technical report# 3515. Software Quality Engineering & Testing (QUEST) Lab (2015)Google Scholar
  20. 20.
    Sheskin, D.J.: Handbook of parametric and nonparametric statistical procedures. Chapman & Hall/CRC (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Safdar Aqeel Safdar
    • 1
  • Muhammad Zohaib Iqbal
    • 1
    • 2
  • Muhammad Uzair Khan
    • 1
  1. 1.Software Quality Engineering and Testing Lab (QUEST)National University of Computer and Emerging ScienceIslamabadPakistan
  2. 2.Interdisciplinary Centre for Security, Reliability and TrustLuxembourgLuxembourg

Personalised recommendations