Empirical Software Engineering

, Volume 9, Issue 3, pp 149–195 | Cite as

A Controlled Experiment Comparing the Maintainability of Programs Designed with and without Design Patterns—A Replication in a Real Programming Environment

  • Marek Vokáč
  • Walter Tichy
  • Dag I. K. SjØberg
  • Erik Arisholm
  • Magne Aldrin


Software “design patterns” seek to package proven solutions to design problems in a form that makes it possible to find, adapt and reuse them. To support the industrial use of design patterns, this research investigates when, and how, using patterns is beneficial, and whether some patterns are more difficult to use than others. This paper describes a replication of an earlier controlled experiment on design patterns in maintenance, with major extensions. Experimental realism was increased by using a real programming environment instead of pen and paper, and paid professionals from multiple major consultancy companies as subjects. Measurements of elapsed time and correctness were analyzed using regression models and an estimation method that took into account the correlations present in the raw data. Together with on-line logging of the subjects’ work, this made possible a better qualitative understanding of the results. The results indicate quite strongly that some patterns are much easier to understand and use than others. In particular, the Visitor pattern caused much confusion. Conversely, the patterns Observer and, to a certain extent, Decorator were grasped and used intuitively, even by subjects with little or no knowledge of patterns. The implication is that design patterns are not universally good or bad, but must be used in a way that matches the problem and the people. When approaching a program with documented design patterns, even basic training can improve both the speed and quality of maintenance activities.

Controlled experiment design patterns real programming environment qualitative results 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Alexander, C. 1978. A Pattern Language: Towns, Buildings, Construction. New York, USA: Oxford University Press, Inc.Google Scholar
  2. Alexander, C. 1987. The Timeless Way of Building. New York, USA: Oxford University Press Inc.Google Scholar
  3. Arisholm, E. 2001. Empirical assessment of changeability in object-oriented software. Ph.D. thesis, University of Oslo.Google Scholar
  4. Arisholm, E., Sjøberg, D. I. K., Carelius, G. J., and Lindsjørn, Y. 2002a. SESE an experiment support environment for evaluating software engineering technologies. In NW-PER2002 (Tenth Nordic Workshop on Programming and Software Development Tools and Techniques). Copenhagen, Denmark, pp. 81-98.Google Scholar
  5. Arisholm, E., Sjøberg, D. I. K., Carelius, G. J., and Lindsjørn, Y. 2002b. A web-based support environment for software engineering experiments. Nordic Journal of Computing 9(4): 231–247.Google Scholar
  6. Arisholm, E., Sjøberg, D. I. K., and Jørgensen, M. 2001. Assessing the changeability of two objectoriented design alternatives-a controlled experiment. Empirical Software Engineering 6, 231–277.Google Scholar
  7. Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., and Stal, M. 1996. Pattern-Oriented Software Architecture. Chichester: Wiley.Google Scholar
  8. Diggle, P., Liang, K., and Zeger, S. 1994. The Analysis of Longitudinal Data. Oxford: Oxford University Press.Google Scholar
  9. Efron, B., and Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability. London: Chapman &; Hall.Google Scholar
  10. Gamma, E., Helm, R., Johnson, R., and Vlissides, J. 1995. Design Patterns: Elements of Reusable Object-Oriented Software. Reading, MA: Addison-Wesley.Google Scholar
  11. Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El-Emam, K., and Rosenberg, J. 2002. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8): 721–734.Google Scholar
  12. Liang, K., and Zeger, S. 1986. Longitudinal Data Analysis using Generalized Linear Models. Biometrika 73, 13–22.Google Scholar
  13. Lindsay, R., and Ehrenberg, A. 1993. The design of replicated studies. The American Statistician 47(3): 217–228.Google Scholar
  14. McCullagh, P., and Nelder, J. 1989. Generalized Linear Models. New York: Chapman and Hall.Google Scholar
  15. Prechelt, L. 2000. An empirical study of working speed differences between software engineers for various kinds of task. Submitted to IEEE Transactions on Software Engineering, to be revised.Google Scholar
  16. Prechelt, L., Unger, B., Tichy, W. F., Brössler, P., and Votta, L. G. 2001. A controlled experiment in maintenance comparing design patterns to simpler solutions. IEEE Transactions on Software Engineering 27(12): 1134–1144.Google Scholar
  17. Sjøberg, D., Anda, B., Arisholm, E., Dybå, T., Jørgensen, M., Karahasanovic, A., Koren, E., and Vokác, M. 2002. Conducting realistic experiments in software engineering. In ISESE2002 (First International Symposium on Empirical Software Engineering). Nara, Japan, pp. 17-26, IEEE Computer Society.Google Scholar
  18. Smith, D., Robertson, W., and Diggle, P. 1996. Object-oriented software for the analysis of longitudinal data in S. Technical Report Technical Report MA96/192, Department of Mathematics and Statistics, University of Lancaster.Google Scholar
  19. Yourdon, E. 1999. Death March. Indianapolis, Indiana: Prentice Hall PTR.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Marek Vokáč
    • 1
  • Walter Tichy
    • 2
  • Dag I. K. SjØberg
    • 1
  • Erik Arisholm
    • 1
  • Magne Aldrin
    • 3
  1. 1.Simula Research LaboratoryLysakerNorway
  2. 2.Universität KarlsruheKarlsruheGermany
  3. 3.Norwegian Computing CenterOsloNorway

Personalised recommendations