Empirical Software Engineering

, Volume 18, Issue 4, pp 699–745 | Cite as

Do background colors improve program comprehension in the #ifdef hell?

  • Janet Feigenspan
  • Christian Kästner
  • Sven Apel
  • Jörg Liebig
  • Michael Schulze
  • Raimund Dachselt
  • Maria Papendieck
  • Thomas Leich
  • Gunter Saake
Article

Abstract

Software-product-line engineering aims at the development of variable and reusable software systems. In practice, software product lines are often implemented with preprocessors. Preprocessor directives are easy to use, and many mature tools are available for practitioners. However, preprocessor directives have been heavily criticized in academia and even referred to as “#ifdef hell”, because they introduce threats to program comprehension and correctness. There are many voices that suggest to use other implementation techniques instead, but these voices ignore the fact that a transition from preprocessors to other languages and tools is tedious, erroneous, and expensive in practice. Instead, we and others propose to increase the readability of preprocessor directives by using background colors to highlight source code annotated with ifdef directives. In three controlled experiments with over 70 subjects in total, we evaluate whether and how background colors improve program comprehension in preprocessor-based implementations. Our results demonstrate that background colors have the potential to improve program comprehension, independently of size and programming language of the underlying product. Additionally, we found that subjects generally favor background colors. We integrate these and other findings in a tool called FeatureCommander, which facilitates program comprehension in practice and which can serve as a basis for further research.

Keywords

Empirical software engineering Software visualization Program comprehension Software product lines Preprocessors FeatureCommander 

References

  1. Adams B et al (2008) Aspect mining in the presence of the C preprocessor. In: Proc.AOSD workshop on linking aspect technology and evolution. ACM Press, pp 1–6Google Scholar
  2. Adams B et al (2009) Can we refactor conditional compilation into aspects? In: Proc. Int’l conf. aspect-oriented software development (AOSD). ACM Press, pp 243–254Google Scholar
  3. Anderson T, Finn J (1996) The new statistical analysis of data. SpringerGoogle Scholar
  4. Apel S, Kästner C (2009) An overview of feature-oriented software development. J Obj Techn 8(4):1–36Google Scholar
  5. Apel S et al (2008) Aspectual feature modules. IEEE Trans Softw Eng 34(2):162–180CrossRefGoogle Scholar
  6. Atkins D et al (2002) Using version control data to evaluate the impact of software tools: a case study of the version editor. IEEE Trans Softw Eng 28(7):625–637CrossRefGoogle Scholar
  7. Aversano L et al (2002) Handling preprocessor-conditioned declarations. In: Proc. IEEE int’l workshop on source code analysis and manipulation. IEEE CS, pp 83–92Google Scholar
  8. Basili VR (1992) Software modeling and measurement: the goal/question/metric paradigm. Tech. Rep. CS-TR-2956 (UMIACS-TR-92-96)Google Scholar
  9. Baxter ID, Mehlich M (2001) Preprocessor conditional removal by simple partial evaluation. In: Proc. working conf. reverse engineering (WCRE). IEEE CS, pp 281–290Google Scholar
  10. Boehm B (1981) Software engineering economics. Prentice HallGoogle Scholar
  11. Boysen J (1977) Factors affecting computer program comprehension. PhD thesis, Iowa State UniversityGoogle Scholar
  12. Brooks R (1978) Using a behavioral theory of program comprehension in software engineering. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 196–201Google Scholar
  13. Chevalier F et al (2010) Using text animated transitions to support navigation in document histories. In: Proc. conf. human factors in computing systems (CHI). ACM Press, pp 683–692Google Scholar
  14. Chu-Carroll M et al (2003) Visual separation of concerns through multidimensional program storage. In: Proc. int’l conf. aspect-oriented software development (AOSD). ACM Press, pp 188–197Google Scholar
  15. Clements P, Northrop L (2001) Software product lines: practice and patterns. Addison-Wesley, Reading, MAGoogle Scholar
  16. Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494–509MathSciNetCrossRefGoogle Scholar
  17. Cohen J (1969) Statistical power analysis for the behavioral sciences. Academic PressGoogle Scholar
  18. Coppit D et al (2007) Spotlight: a prototype tool for software plans. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 754–757Google Scholar
  19. Couto MV et al (2011) Extracting software product lines: a case study using conditional compilation. In: Proc. Europ. conf. software maintenance and reengineering (CSMR), pp 191–200Google Scholar
  20. Czarnecki K, Antkiewicz M (2005) Mapping features to models: a template approach based on superimposed variants. In: Proc. int’l conf. generative programming and component engineering (GPCE). Springer, pp 422–437Google Scholar
  21. Daly J et al (1995) The effect of inheritance on the maintainability of object-oriented software: an empirical study. In: Proc. int’l conf. software maintenance (ICSM). IEEE CS, pp 20–29Google Scholar
  22. Diehl S (2007) Software visualization: visualizing the structure, behaviour, and evolution of software. SpringerGoogle Scholar
  23. Dunsmore A, Roper M (2000) A comparative evaluation of program comprehension measures. Tech. Rep. EFoCS 35-2000, Department of Computer Science, University of StrathclydeGoogle Scholar
  24. Eick S et al (1992) SeeSoft—a tool for visualizing line oriented software statistics. IEEE Trans Softw Eng 18(11):957–968CrossRefGoogle Scholar
  25. Ernst M et al (2002) An empirical analysis of C preprocessor use. IEEE Trans Softw Eng 28(12):1146–1170MathSciNetCrossRefGoogle Scholar
  26. Favre J-M (1995) The CPP paradox. In: Proc. European workshop on software maintenanceGoogle Scholar
  27. Favre J-M (1997) Understanding-in-the-large. In: Proc. int’l workshop on program comprehension (IWPC). IEEE CS, p 29Google Scholar
  28. Feigenspan J (2009) Empirical comparison of FOSD approaches regarding program comprehension—a feasibility study. Master’s thesis, University of MagdeburgGoogle Scholar
  29. Feigenspan J et al (2009) How to compare program comprehension in FOSD empirically - an experience report. In: Proc. int’l workshop on feature-oriented software development. ACM Press, pp 55–62Google Scholar
  30. Feigenspan J et al (2010) Visual support for understanding product lines. In: Proc. int’l conf. program comprehension (ICPC). IEEE CS, Demo Paper, pp 34–35Google Scholar
  31. Feigenspan J et al (2011a) FeatureCommander: colorful #ifdef world. In: Software product line conference (SPLC), paper 3. ACM Press, pp 1–2Google Scholar
  32. Feigenspan J et al (2011b) Using background colors to support program comprehension in software product lines. In: Proc. int’l conf. evaluation and assessment in software engineering (EASE). Institution of Engineering and Technology, pp 66–75Google Scholar
  33. Figueiredo E et al (2008) Evolving software product lines with aspects: an empirical study on design stability. In: Proc. int’l conf. software engineering (ICSE). ACM Press, pp 261–270Google Scholar
  34. Fisher D, Tan K (1989) Visual displays: the highlighting paradox. Human Factors 31(1):17–30Google Scholar
  35. Garrido A, Johnson RE (2005) Analyzing multiple configurations of a C program. In: Proc. int’l conf. software maintenance (ICSM). IEEE CS, pp 379–388Google Scholar
  36. Giventer L (2008) Statistical analysis for public administration, 2nd edn. Jones and Bartlett PublishingGoogle Scholar
  37. Goldstein B (2002) Sensation and perception, 5th edn. Cengage Learning ServicesGoogle Scholar
  38. Hanenberg S (2010) An experiment about static and dynamic type sytems. In: Proc. int’l conf. object-oriented programming, systems, languages and applications (OOPSLA). ACM Press, pp 22–35Google Scholar
  39. Harrison W, Ossher H (1993) Subject-oriented programming: a critique of pure objects. In: Proc. int’l conf. object-oriented programming, systems, languages and applications (OOPSLA). IEEE CS, pp 411–428Google Scholar
  40. Heidenreich F et al (2008) FeatureMapper: mapping features to models. In: Comp. int’l conf. software engineering (ICSE). ACM Press, Demo Paper, pp 943–944Google Scholar
  41. Hofer W et al (2010) Toolchain-independent variant management with the Leviathan filesystem. In: Proc. int’l workshop on feature-oriented software development (FOSD). ACM Press, pp 18–24Google Scholar
  42. Hu Y et al (2000) C/C++ conditional compilation analysis using symbolic execution. In: Proc. int’l conf. software maintenance (ICSM). IEEE CS, pp 196–206Google Scholar
  43. Kästner C (2010) Virtual separation of concerns: preprocessors 2.0. PhD thesis, University of MagdeburgGoogle Scholar
  44. Kästner C et al (2008) Granularity in software product lines. In: Proc. int’l conf. software engineering (ICSE). ACM Press, pp 311–320Google Scholar
  45. Kästner C et al (2009a) A model of refactoring physically and virtually separated features. In: Proc. int’l conf. generative programming and component engineering (GPCE). ACM Press, pp 157–166Google Scholar
  46. Kästner C et al (2009b) FeatureIDE: tool framework for feature-oriented software development. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, Demo Paper, pp 611–614Google Scholar
  47. Kiczales G et al (1997) Aspect-oriented programming. In: Proc. Europ. conf. object-oriented programming (ECOOP). Springer, pp 220–242Google Scholar
  48. Kitchenham B et al (2008) Evaluating guidelines for reporting empirical software engineering studies. Empir Software Eng 13(1):97–121CrossRefGoogle Scholar
  49. Koenemann J, Robertson S (1991) Expert problem solving strategies for program comprehension. In: Proc. conf. human factors in computing systems (CHI). ACM Press, pp 125–130Google Scholar
  50. Krone M, Snelting G (1994) On the inference of configuration structures from source code. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 49–57Google Scholar
  51. Kullbach B, Riediger V (2001) Folding: an approach to enable program understanding of preprocessed languages. In: Proc. working conf. reverse engineering (WCRE). IEEE CS, pp 3–12Google Scholar
  52. Levkowitz H, Herman GT (1992) Color scales for image data. IEEE Comput Graph Appl 12(1):72–80CrossRefGoogle Scholar
  53. Liebig J et al (2010) An analysis of the variability in forty preprocessor-based software product lines. In: Proc. int’l conf. software engineering (ICSE). ACM Press, pp 105–114Google Scholar
  54. Liebig J et al (2011) Analyzing the discipline of preprocessor annotations in 30 million lines of C code. In: Proc. int’l conf. aspect-oriented software development (AOSD). ACM Press, pp 191–202Google Scholar
  55. Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22(140):1–55Google Scholar
  56. Livadas P, Small D (1994) Understanding code containing preprocessor constructs. In: Proc. int’l workshop program comprehension (IWPC). IEEE CS, pp 89–97Google Scholar
  57. Lohmann D et al (2006) A quantitative analysis of aspects in the eCos kernel. In: Proc. Europ. conf. computer systems (EuroSys). ACM Press, pp 191–204Google Scholar
  58. McCloskey B, Brewer E (2005) ASTEC: a new approach to refactoring C. In: Proc. Europ. software engineering conf./foundations of software engineering (ESEC/FSE). ACM Press, pp 21–30Google Scholar
  59. Miller G (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev 63(2):81–97CrossRefGoogle Scholar
  60. Mook D (1996) Motivation: the organization of action, 2nd edn. W.W. Norton & CoGoogle Scholar
  61. Muthig D, Patzke T (2003) Generic implementation of product line components. In: Int’l conf. NetObjectDays. Springer, pp 313–329Google Scholar
  62. Oberg B, Notkin D (1992) Error reporting with graduated color. IEEE Softw 9(6):33–38CrossRefGoogle Scholar
  63. Otero M, Dolado J (2004) Evaluation of the comprehension of the dynamic modeling in UML. J Inf Softw Technol 46(1):35–53CrossRefGoogle Scholar
  64. Overbey JL, Johnson RE (2009) Software language engineering. In: Generating rewritable abstract syntax trees, pp 114–133.Google Scholar
  65. Pearse T, Oman P (1997) Experiences developing and maintaining software in a multi-platform environment. In: Proc. int’l conf. software maintenance (ICSM). IEEE CS, pp 270–277Google Scholar
  66. Pennington N (1987) Stimulus structures and mental representations in expert comprehension of computer programs. Cogn Psychol 19(3):295–341CrossRefGoogle Scholar
  67. Pohl K et al (2005) Software product line engineering: foundations, principles, and techniques. SpringerGoogle Scholar
  68. Prechelt L et al (2002) Two controlled experiments assessing the usefulness of design pattern documentation in program maintenance. IEEE Trans Softw Eng 28(6):595–606CrossRefGoogle Scholar
  69. Prehofer C (1997) Feature-oriented programming: a fresh look at objects. In: Europ. conf. on objectoriented programming (ECOOP). Springer, pp 419–443Google Scholar
  70. Rambally G (1986) The influence of color on program readability and comprehensibility. In: Proc. technical symposium on computer science education (SIGCSE). ACM Press, pp 173–181Google Scholar
  71. Ribeiro M et al (2010) Emergent feature modularization. In: Proceedings of the ACM international conference companion on object oriented programming systems languages and applications companion (SPLASH). ACM Press, pp 11–18Google Scholar
  72. Rice J (1991) Display color coding: 10 rules of thumb. IEEE Softw 8(1):86–88Google Scholar
  73. Riggs R et al (2003) Programming wireless devices with the java 2 platform, micro edition. Sun Microsystems, IncGoogle Scholar
  74. Shaft T, Vessey I (1995) The relevance of application domain knowledge: the case of computer program comprehension. Inf Syst Res 6(3):286–299CrossRefGoogle Scholar
  75. Shapiro S, Wilk M (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611MathSciNetMATHCrossRefGoogle Scholar
  76. Shneiderman B, Mayer R (1979) Syntactic/semantic interactions in programmer behavior: a model and experimental results. Int J Parallel Prog 8(3):219–238MATHGoogle Scholar
  77. Singh N et al (2006) CViMe: viewing conditionally compiled C/C++ sources through java. In: Companion to the 21st ACM SIGPLAN symposium on object-oriented programming systems, languages, and applications. ACM Press, pp 730–731Google Scholar
  78. Singh N et al (2007) C-CLR: a tool for navigating highly configurable system software. In: Proc. workshop aspects, components, and patterns for infrastr. software. ACM PressGoogle Scholar
  79. Smaragdakis Y, Batory D (1998) Implementing layered designs with mixin layers. In: Proc. Europ. conf. object-oriented programming (ECOOP). Springer, pp 550–570Google Scholar
  80. Soloway E, Ehrlich K (1984) Empirical studies of programming knowledge. IEEE Trans Softw Eng 10(5):595–609CrossRefGoogle Scholar
  81. Someren M et al (1994) The think aloud method: a practical guide to modelling cognitive processes. Academic PressGoogle Scholar
  82. Spencer H, Collyer G (1992) #ifdef considered harmful or portability experience with C news. In: Proc. USENIX conf. USENIX Association, pp 185–198Google Scholar
  83. Spinellis D (2003) Global analysis and transformations in preprocessed languages. IEEE Trans Softw Eng 29(11):1019–1030CrossRefGoogle Scholar
  84. Standish T (1984) An essay on software reuse. IEEE Trans Softw Eng SE–10(5):494–497CrossRefGoogle Scholar
  85. Stengel M et al (2011) View infinity: a zoomable interface for feature-oriented software development. In: Proc. int’l conf. software engineering (ICSE). ACM Press, pp 1031–1033Google Scholar
  86. Tamborello F, Byrne M (2007) Adaptive but non-optimal visual search with highlighted displays. Cogn Syst Res 8(3):182–191CrossRefGoogle Scholar
  87. Tartler R et al (2011) Feature consistency in compile-time configurable system software. In: Proc. Europ. conf. computer systems conference (EuroSys). ACM Press, pp 47–60Google Scholar
  88. Tiarks R (2011) What programmers really do: an observational study’. In: Proc. workshop software reengineering (WSR), pp 36–37Google Scholar
  89. Tichy WF (1998) Should computer scientists experiment more? Computer 31(5):32–40MathSciNetCrossRefGoogle Scholar
  90. Vidacs L et al (2004) Columbus schema for C/C++ preprocessing. In: Proc. Europ. conf. software maintenance and reengineering (CSMR). IEEE CS, pp 75–84Google Scholar
  91. von Mayrhauser A, Vans A (1993) From program comprehension to tool requirements for an industrial environment. In: Proc. int’l workshop program comprehension (IWPC). IEEE CS, pp 78–86Google Scholar
  92. von Mayrhauser A et al (1997) Program understanding behaviour during enhancement of large-scale software. J Softw Maint: Res Pract 9(5):299–327CrossRefGoogle Scholar
  93. von Mayrhauser A, Vans M (1995) Program comprehension during software maintenance and evolution. Computer 28(8):44–55CrossRefGoogle Scholar
  94. Ware C (2000) Information visualization: perception for design. Morgan KaufmannGoogle Scholar
  95. Wijffelaars M et al (2008) Generating color palettes using intuitive parameters. Comput Graph Forum 27(3):743–750CrossRefGoogle Scholar
  96. Yang W (1994) How to merge program texts. J Syst Softw 27(2):129–135CrossRefGoogle Scholar
  97. Yellott J (1971) Correction for fast guessing and the speed accuracy trade-off in choice reaction time. J Math Psych 8(2):159–199MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Janet Feigenspan
    • 1
  • Christian Kästner
    • 2
  • Sven Apel
    • 3
  • Jörg Liebig
    • 3
  • Michael Schulze
    • 4
  • Raimund Dachselt
    • 1
  • Maria Papendieck
    • 1
  • Thomas Leich
    • 5
  • Gunter Saake
    • 1
  1. 1.School of Computer ScienceUniversity of MagdeburgMagdeburgGermany
  2. 2.Philipps University MarburgMarburgGermany
  3. 3.University of PassauPassauGermany
  4. 4.pure-systemsMagdeburgGermany
  5. 5.Metop Research InstituteMagdeburgGermany

Personalised recommendations