Skip to main content

The ‘as code’ activities: development anti-patterns for infrastructure as code

Abstract

Context:

The ‘as code’ suffix in infrastructure as code (IaC) refers to applying software engineering activities, such as version control, to maintain IaC scripts. Without the application of these activities, defects that can have serious consequences may be introduced in IaC scripts. A systematic investigation of the development anti-patterns for IaC scripts can guide practitioners in identifying activities to avoid defects in IaC scripts. Development anti-patterns are recurring development activities that relate with defective IaC scripts.

Goal:

The goal of this paper is to help practitioners improve the quality of infrastructure as code (IaC) scripts by identifying development activities that relate with defective IaC scripts.

Methodology:

We identify development anti-patterns by adopting a mixed-methods approach, where we apply quantitative analysis with 2,138 open source IaC scripts and conduct a survey with 51 practitioners.

Findings:

We observe five development activities to be related with defective IaC scripts from our quantitative analysis. We identify five development anti-patterns namely, ‘boss is not around’, ‘many cooks spoil’, ‘minors are spoiler’, ‘silos’, and ‘unfocused contribution’.

Conclusion:

Our identified development anti-patterns suggest the importance of ‘as code’ activities in IaC because these activities are related to quality of IaC scripts.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    https://aws.amazon.com/

  2. 2.

    https://github.com/Mirantis/puppet-manifests

  3. 3.

    https://figshare.com/s/c88ece116f803c09beb6

  4. 4.

    https://research.ncsu.edu/sparcs/compliance/irb/using-the-eirb-system/

References

  1. Adams B, McIntosh S (2016) Modern release engineering in a nutshell – why researchers should care. In: 2016 IEEE 23Rd international conference on software analysis, evolution, and reengineering (SANER). https://doi.org/10.1109/SANER.2016.108, vol 5, pp 78–90

  2. Alali A, Kagdi H, Maletic JI (2008) What’s a typical commit? a characterization of open source software repositories. In: 2008 16th IEEE international conference on program comprehension, pp 182–191. https://doi.org/10.1109/ICPC.2008.24

  3. Anderson S, Allen P, Peckham S, Goodwin N (2008) Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Res Policy Syst 6(1):7

    Google Scholar 

  4. Ansible (2019) Nasa: Increasing cloud efficiency with ansible and ansible tower. Tech. Rep., Ansible, https://www.ansible.com/hubfs/pdf/Ansible-Case-Study-NASA.pdf?hsLang=en-us

  5. Arisholm E, Briand LC, Johannessen EB (2010) A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. J Syst Software 83(1):2–17. https://doi.org/10.1016/j.jss.2009.06.055. sI: Top Scholars

    Article  Google Scholar 

  6. Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8(1):19–32. https://doi.org/10.1080/1364557032000119616

    Google Scholar 

  7. Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code!: examining the effects of ownership on software quality. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th european conference on foundations of software engineering, ACM, New York, NY, USA, ESEC/FSE ’11, pp 4–14. https://doi.org/10.1145/2025113.2025119

  8. Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324

    MATH  Google Scholar 

  9. Breiman L et al (1984) Classification and Regression Trees, 1st. Chapman & Hall, New York. http://www.crcpress.com/catalog/C4841.htm

    MATH  Google Scholar 

  10. Bright J (2017) Slalom’s approach to breaking down silos between devops and security teams. https://blog.chef.io/2017/08/16/slaloms-approach-to-breaking-down-silos-between-devops-and-security/. [Online; Accessed 18-Feb-2019]

  11. Brooks FP Jr (1995) The Mythical Man-month (Anniversary Ed.) Addison-Wesley Longman Publishing Co., Inc, Boston

    Google Scholar 

  12. Brown WH, Malveau RC, McCormick HWS, Mowbray TJ (1998) Antipatterns: Refactoring Software, Architectures, and Projects in Crisis, 1st. John Wiley & Sons, Inc., New York

    Google Scholar 

  13. Businge J, Kawuma S, Bainomugisha E, Khomh F, Nabaasa E (2017) Code authorship and fault-proneness of open-source android applications: an empirical study. In: Proceedings of the 13th international conference on predictive models and data analytics in software engineering, ACM, New York, NY, USA, PROMISE, pp 33–42. https://doi.org/10.1145/3127005.3127009

  14. C SN, Menzies T (2019) Assessing developer beliefs: a reply to “perceptions, expectations, and challenges in defect prediction”. arXiv:1904.05794

  15. Chen B, Jiang ZMJ (2017) Characterizing and detecting anti-patterns in the logging code. In: Proceedings of the 39th international conference on software engineering, IEEE Press, ICSE ’17, pp 71–81. https://doi.org/10.1109/ICSE.2017.15

  16. Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494–509

    Google Scholar 

  17. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46. https://doi.org/10.1177/001316446002000104

    Google Scholar 

  18. Commons W (2017) Incident documentation/20170118-Labs. https://wikitech.wikimedia.org/wiki/Incident_documentation/20170118-Labs [Online; accessed 27-Jan-2019]

  19. Cramer D, Howitt DL (2004) The Sage dictionary of statistics: a practical resource for students in the social sciences. Sage

  20. Devanbu P, Zimmermann T, Bird C (2016) Belief and evidence in empirical software engineering. In: Proceedings of the 38th international conference on software engineering, ACM, New York, NY, USA, ICSE ’16, pp 108–119. https://doi.org/10.1145/2884781.2884812

  21. Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. Springer London, London, pp 285–311

    Google Scholar 

  22. Elberzhager F, Kremer S, Münch J, Assmann D (2012) Guiding testing activities by predicting defect-prone parts using product and inspection metrics. In: 2012 38th Euromicro conference on software engineering and advanced applications, pp 406–413. https://doi.org/10.1109/SEAA.2012.30

  23. Freedman D (2005) Statistical models : theory and practice. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  24. Fu W, Menzies T, Shen X (2016) Tuning for software analytics: is it really necessary?. Inf Softw Technol 76:135–146. http://www.sciencedirect.com/science/article/pii/S0950584916300738

    Google Scholar 

  25. Garousi V, Küçük B (2018) Smells in software test code: a survey of knowledge in industry and academia. J Syst Software 138:52–81. http://www.sciencedirect.com/science/article/pii/S0164121217303060

    Google Scholar 

  26. Ghotra B, McIntosh S, Hassan AE (2015) Revisiting the impact of classification techniques on the performance of defect prediction models. In: Proceedings of the 37th international conference on software engineering - volume 1. IEEE Press, Piscataway, pp 789–800. http://dl.acm.org/citation.cfm?id=2818754.2818850

  27. Hall T, Beecham S, Bowes D, Gray D, Counsell S (2012) A systematic literature review on fault prediction performance in software engineering. IEEE T Software Eng 38(6):1276–1304. https://doi.org/10.1109/TSE.2011.103

    Google Scholar 

  28. Hassan AE (2009) Predicting faults using the complexity of code changes. In: Proceedings of the 31st international conference on software engineering, IEEE computer society, Washington, DC, USA, ICSE ’09, pp 78–88. https://doi.org/10.1109/ICSE.2009.5070510

  29. Hersher R (2017) Incident documentation/20170118-Labs. https://www.npr.org/sections/thetwo-way/2017/03/03/518322734/amazon-and-the-150-million-typo, [Online; accessed 27-Jan-2019]

  30. Hove SE, Anda B (2005) Experiences from conducting semi-structured interviews in empirical software engineering research. In: 11th IEEE International software metrics symposium (METRICS’05), pp 10 pp.–23 https://doi.org/10.1109/METRICS.2005.24

  31. Huberty CJ, Olejnik S (2006) Applied MANOVA and discriminant analysis, vol 498. John Wiley & Sons, New York

    MATH  Google Scholar 

  32. Hudak P (1998) Modular domain specific languages and tools. In: Proceedings. Fifth international conference on software reuse (Cat. No.98TB100203), pp 134–142. https://doi.org/10.1109/ICSR.1998.685738

  33. Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st. Addison-Wesley Professional, Boston

    Google Scholar 

  34. IEEE (2010) Ieee standard classification for software anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993) pp 1–23. https://doi.org/10.1109/IEEESTD.2010.5399061

  35. Jiang Y, Adams B (2015) Co-evolution of infrastructure and source code: an empirical study. In: Proceedings of the 12th working conference on mining software repositories, ieee press, Piscataway, NJ, USA, MSR ’15, pp 45–55. http://dl.acm.org/citation.cfm?id=2820518.2820527

  36. Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. Springer London, London, pp 63–92. https://doi.org/10.1007/978-1-84800-044-5_3

    Google Scholar 

  37. Labs P (2018) Puppet documentation. https://docs.puppet.com/, [Online; accessed 08-Aug-2018]

  38. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. http://www.jstor.org/stable/2529310

    MATH  Google Scholar 

  39. Leone M (2016) The economic benefits of puppet enterprise. Tech. rep., ESG. https://puppet.com/resources/analyst-report/the-economic-benefits-puppet-enterprise

  40. MacLeod L, Greiler M, Storey M, Bird C, Czerwonka J (2018) Code reviewing in the trenches: challenges and best practices. IEEE Softw 35 (4):34–42. https://doi.org/10.1109/MS.2017.265100500

    Google Scholar 

  41. Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Statist 18(1):50–60. http://www.jstor.org/stable/2236101

    MathSciNet  MATH  Google Scholar 

  42. Martin RC (2011) The clean coder: a code of conduct for professional programmers. Pearson Education

  43. McCune JT, Jeffrey (2011) Pro Puppet, 1st edn. Apress. https://doi.org/10.1007/978-1-4302-3058-8. https://www.springer.com/gp/book/9781430230571

  44. Meneely A, Williams L (2009) Secure open source collaboration: an empirical study of linus’ law. In: Proceedings of the 16th ACM conference on computer and communications security, ACM, New York, NY, USA, CCS ’09, pp 453–462. https://doi.org/10.1145/1653662.1653717

  45. Meneely A, Smith B, Williams L (2013) Validating software metrics: a spectrum of philosophies. ACM Trans Softw Eng Methodol 21(4):24:1–24:28. https://doi.org/10.1145/2377656.2377661

    Google Scholar 

  46. Menzies T, Greenwald J, Frank A (2007) Data mining static code attributes to learn defect predictors. IEEE T Software Eng 33(1):2–13. https://doi.org/10.1109/TSE.2007.256941

    Google Scholar 

  47. Morris K (2016) Infrastructure as code: managing servers in the cloud. “ O’Reilly Media, Inc.”

  48. Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating github for engineered software projects. Empirical Software Engineering pp 1–35. https://doi.org/10.1007/s10664-017-9512-6

  49. Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E (2018) Systematic review or scoping review? guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 18 (1):143

    Google Scholar 

  50. Oktaba P (2015) Keep your commits small. https://dzone.com/articles/keep-your-commits-small, [Online; accessed 08-Feb-2019]

  51. Ostrand TJ, Weyuker EJ, Bell RM (2004) Where the bugs are. In: Proceedings of the 2004 ACM SIGSOFT international symposium on software testing and analysis, ACM, New York, NY, USA, ISSTA ’04, pp 86–96. https://doi.org/10.1145/1007512.1007524

  52. Pinzger M, Nagappan N, Murphy B (2008) Can developer-module networks predict failures?. In: Proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, SIGSOFT ’08/FSE-16, pp 2–12. https://doi.org/10.1145/1453101.1453105

  53. Rahman A, Williams L (2018) Characterizing defective configuration scripts used for continuous deployment. In: 2018 IEEE 11th International conference on software testing, verification and validation (ICST), pp 34–45. https://doi.org/10.1109/ICST.2018.00014

  54. Rahman A, Williams L (2019) Source code properties of defective infrastructure as code scripts. Information and Software Technology. https://doi.org/10.1016/j.infsof.2019.04.013, http://www.sciencedirect.com/science/article/pii/S0950584919300965

  55. Rahman A, Partho A, Morrison P, Williams L (2018) What questions do programmers ask about configuration as code?. In: Proceedings of the 4th international workshop on rapid continuous software engineering, ACM, New York, NY, USA, RCoSE ’18, pp 16–22. https://doi.org/10.1145/3194760.3194769

  56. Rahman A, Parnin C, Williams L (2019) The seven sins: Security smells in infrastructure as code scripts. In: Proceedings of the 41st international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’19, pp 164–175. https://doi.org/10.1109/ICSE.2019.00033

  57. Rahman A, Farhana E, Parnin C, Williams L (2020) Gang of eight: a defect taxonomy for infrastructure as code scripts. In: Proceedings of the 42nd international conference on software engineering, ICSE ’20, to appear

  58. Rahman F, Devanbu P (2013a) How, and why, process metrics are better. In: Proceedings of the 2013 international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’13, pp 432–441. http://dl.acm.org/citation.cfm?id=2486788.2486846

  59. Rahman F, Devanbu P (2013b) How, and why, process metrics are better. In: Proceedings of the 2013 international conference on software engineering, IEEE press, Piscataway, NJ, USA, ICSE ’13, pp 432–441. http://dl.acm.org/citation.cfm?id=2486788.2486846

  60. Rigby PC, German DM, Storey MA (2008) Open source software peer review practices: a case study of the apache server. In: Proceedings of the 30th international conference on software engineering, ACM, New York, NY, USA, ICSE ’08, pp 541–550. https://doi.org/10.1145/1368088.1368162

  61. Romano J, Kromrey J, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys?. In: Annual meeting of the florida association of institutional research, pp 1–3

  62. Saldana J (2015) The coding manual for qualitative researchers. Sage

  63. Shambaugh R, Weiss A, Guha A (2016) Rehearsal: a configuration verification tool for puppet. SIGPLAN Not 51(6):416–430. https://doi.org/10.1145/2980983.2908083

    Google Scholar 

  64. Sharma T, Fragkoulis M, Spinellis D (2016) Does your configuration code smell?. In: Proceedings of the 13th international conference on mining software repositories, ACM, New York, NY, USA, MSR ’16, pp 189–200. https://doi.org/10.1145/2901739.2901761

  65. Shihab E, Jiang ZM, Adams B, Hassan AE, Bowerman R (2011) Prioritizing the creation of unit tests in legacy software systems. Software Pract Exper 41(10):1027–1048. https://doi.org/10.1002/spe.1053

    Google Scholar 

  66. Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 2013 6th International workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92. https://doi.org/10.1109/CHASE.2013.6614738

  67. Sullivan GM, Feinn R (2012) Using effect size-or why the p value is not enough. J Grad Med Educ 4(3):279–282. https://doi.org/10.4300/JGME-D-12-00156.1

    Google Scholar 

  68. Tan PN, Steinbach M, Kumar V (2005) Introduction to data mining, 1st. Addison-Wesley Longman Publishing Co., Inc., Boston

    Google Scholar 

  69. Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2016) Automated parameter optimization of classification techniques for defect prediction models. In: Proceedings of the 38th international conference on software engineering, ACM, New York, NY, USA, ICSE ’16, pp 321–332 https://doi.org/10.1145/2884781.2884857

  70. Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18. https://doi.org/10.1109/TSE.2016.2584050

    Google Scholar 

  71. Tosun A, Bener A, Turhan B, Menzies T (2010) Practical considerations in deploying statistical methods for defect prediction: a case study within the turkish telecommunications industry. Inf Softw Technol 52(11):1242–1257. https://doi.org/10.1016/j.infsof.2010.06.006

    Google Scholar 

  72. Tufano M, Bavota G, Poshyvanyk D, Di Penta M, Oliveto R, De Lucia A (2017) An empirical study on developer-related factors characterizing fix-inducing commits. J Softw Evol Proc 29(1):e1797. https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.1797

    Google Scholar 

  73. Turhan B, Kocak G, Bener A (2009) Data mining source code for locating software bugs: a case study in telecommunication industry. Expert Syst Appl 36(6):9986–9990. https://doi.org/10.1016/j.eswa.2008.12.028. http://www.sciencedirect.com/science/article/pii/S0957417408009275

    Google Scholar 

  74. Turnbull J (2007) Pulling strings with puppet: automated system administration done right. Apress

  75. van der Bent E, Hage J, Visser J, Gousios G (2018) How good is your puppet? an empirically defined and validated quality model for puppet. In: 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER), pp 164–174. https://doi.org/10.1109/SANER.2018.8330206

  76. Van Wyk E, Krishnan L, Bodin D, Schwerdfeger A (2007) Attribute grammar-based language extensions for java. In: Proceedings of the 21st european conference on object-oriented programming, springer-verlag, Berlin, Heidelberg, ECOOP’07, pp 575–599. http://dl.acm.org/citation.cfm?id=2394758.2394796

  77. Voelter M (2013) DSL engineering: designing implementing and using domain-specific languages. CreateSpace Independent Publishing Platform, USA

    Google Scholar 

  78. Weinberg GM (1992) Quality software management (vol. 1): systems thinking. Dorset House Publishing Co., Inc., New York

    MATH  Google Scholar 

  79. Weiss A, Guha A, Brun Y (2017) Tortoise: Interactive system configuration repair. In: Proceedings of the 32Nd IEEE/ACM international conference on automated software engineering, IEEE press, Piscataway, NJ, USA, ASE 2017, pp 625–636. http://dl.acm.org/citation.cfm?id=3155562.3155641

Download references

Acknowledgements

The NSA Science of Security Lablet (award H98230-17-D-0080) at the North Carolina State University supported this research study. We thank the Realsearch research group members for their useful feedback. We also thank the practitioners who answered our questions.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Akond Rahman.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Daniel Méndez

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rahman, A., Farhana, E. & Williams, L. The ‘as code’ activities: development anti-patterns for infrastructure as code. Empir Software Eng 25, 3430–3467 (2020). https://doi.org/10.1007/s10664-020-09841-8

Download citation

Keywords

  • Anti-pattern
  • Bugs
  • Configuration script
  • Continuous deployment
  • Defect
  • Devops
  • Infrastructure as code
  • Practice
  • Puppet
  • Quality