Abstract
Context:
The ‘as code’ suffix in infrastructure as code (IaC) refers to applying software engineering activities, such as version control, to maintain IaC scripts. Without the application of these activities, defects that can have serious consequences may be introduced in IaC scripts. A systematic investigation of the development anti-patterns for IaC scripts can guide practitioners in identifying activities to avoid defects in IaC scripts. Development anti-patterns are recurring development activities that relate with defective IaC scripts.
Goal:
The goal of this paper is to help practitioners improve the quality of infrastructure as code (IaC) scripts by identifying development activities that relate with defective IaC scripts.
Methodology:
We identify development anti-patterns by adopting a mixed-methods approach, where we apply quantitative analysis with 2,138 open source IaC scripts and conduct a survey with 51 practitioners.
Findings:
We observe five development activities to be related with defective IaC scripts from our quantitative analysis. We identify five development anti-patterns namely, ‘boss is not around’, ‘many cooks spoil’, ‘minors are spoiler’, ‘silos’, and ‘unfocused contribution’.
Conclusion:
Our identified development anti-patterns suggest the importance of ‘as code’ activities in IaC because these activities are related to quality of IaC scripts.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adams B, McIntosh S (2016) Modern release engineering in a nutshell – why researchers should care. In: 2016 IEEE 23Rd international conference on software analysis, evolution, and reengineering (SANER). https://doi.org/10.1109/SANER.2016.108, vol 5, pp 78–90
Alali A, Kagdi H, Maletic JI (2008) What’s a typical commit? a characterization of open source software repositories. In: 2008 16th IEEE international conference on program comprehension, pp 182–191. https://doi.org/10.1109/ICPC.2008.24
Anderson S, Allen P, Peckham S, Goodwin N (2008) Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Res Policy Syst 6(1):7
Ansible (2019) Nasa: Increasing cloud efficiency with ansible and ansible tower. Tech. Rep., Ansible, https://www.ansible.com/hubfs/pdf/Ansible-Case-Study-NASA.pdf?hsLang=en-us
Arisholm E, Briand LC, Johannessen EB (2010) A systematic and comprehensive investigation of methods to build and evaluate fault prediction models. J Syst Software 83(1):2–17. https://doi.org/10.1016/j.jss.2009.06.055. sI: Top Scholars
Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8(1):19–32. https://doi.org/10.1080/1364557032000119616
Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code!: examining the effects of ownership on software quality. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th european conference on foundations of software engineering, ACM, New York, NY, USA, ESEC/FSE ’11, pp 4–14. https://doi.org/10.1145/2025113.2025119
Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324
Breiman L et al (1984) Classification and Regression Trees, 1st. Chapman & Hall, New York. http://www.crcpress.com/catalog/C4841.htm
Bright J (2017) Slalom’s approach to breaking down silos between devops and security teams. https://blog.chef.io/2017/08/16/slaloms-approach-to-breaking-down-silos-between-devops-and-security/. [Online; Accessed 18-Feb-2019]
Brooks FP Jr (1995) The Mythical Man-month (Anniversary Ed.) Addison-Wesley Longman Publishing Co., Inc, Boston
Brown WH, Malveau RC, McCormick HWS, Mowbray TJ (1998) Antipatterns: Refactoring Software, Architectures, and Projects in Crisis, 1st. John Wiley & Sons, Inc., New York
Businge J, Kawuma S, Bainomugisha E, Khomh F, Nabaasa E (2017) Code authorship and fault-proneness of open-source android applications: an empirical study. In: Proceedings of the 13th international conference on predictive models and data analytics in software engineering, ACM, New York, NY, USA, PROMISE, pp 33–42. https://doi.org/10.1145/3127005.3127009
C SN, Menzies T (2019) Assessing developer beliefs: a reply to “perceptions, expectations, and challenges in defect prediction”. arXiv:1904.05794
Chen B, Jiang ZMJ (2017) Characterizing and detecting anti-patterns in the logging code. In: Proceedings of the 39th international conference on software engineering, IEEE Press, ICSE ’17, pp 71–81. https://doi.org/10.1109/ICSE.2017.15
Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494–509
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46. https://doi.org/10.1177/001316446002000104
Commons W (2017) Incident documentation/20170118-Labs. https://wikitech.wikimedia.org/wiki/Incident_documentation/20170118-Labs [Online; accessed 27-Jan-2019]
Cramer D, Howitt DL (2004) The Sage dictionary of statistics: a practical resource for students in the social sciences. Sage
Devanbu P, Zimmermann T, Bird C (2016) Belief and evidence in empirical software engineering. In: Proceedings of the 38th international conference on software engineering, ACM, New York, NY, USA, ICSE ’16, pp 108–119. https://doi.org/10.1145/2884781.2884812
Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. Springer London, London, pp 285–311
Elberzhager F, Kremer S, Münch J, Assmann D (2012) Guiding testing activities by predicting defect-prone parts using product and inspection metrics. In: 2012 38th Euromicro conference on software engineering and advanced applications, pp 406–413. https://doi.org/10.1109/SEAA.2012.30
Freedman D (2005) Statistical models : theory and practice. Cambridge University Press, Cambridge
Fu W, Menzies T, Shen X (2016) Tuning for software analytics: is it really necessary?. Inf Softw Technol 76:135–146. http://www.sciencedirect.com/science/article/pii/S0950584916300738
Garousi V, Küçük B (2018) Smells in software test code: a survey of knowledge in industry and academia. J Syst Software 138:52–81. http://www.sciencedirect.com/science/article/pii/S0164121217303060
Ghotra B, McIntosh S, Hassan AE (2015) Revisiting the impact of classification techniques on the performance of defect prediction models. In: Proceedings of the 37th international conference on software engineering - volume 1. IEEE Press, Piscataway, pp 789–800. http://dl.acm.org/citation.cfm?id=2818754.2818850
Hall T, Beecham S, Bowes D, Gray D, Counsell S (2012) A systematic literature review on fault prediction performance in software engineering. IEEE T Software Eng 38(6):1276–1304. https://doi.org/10.1109/TSE.2011.103
Hassan AE (2009) Predicting faults using the complexity of code changes. In: Proceedings of the 31st international conference on software engineering, IEEE computer society, Washington, DC, USA, ICSE ’09, pp 78–88. https://doi.org/10.1109/ICSE.2009.5070510
Hersher R (2017) Incident documentation/20170118-Labs. https://www.npr.org/sections/thetwo-way/2017/03/03/518322734/amazon-and-the-150-million-typo, [Online; accessed 27-Jan-2019]
Hove SE, Anda B (2005) Experiences from conducting semi-structured interviews in empirical software engineering research. In: 11th IEEE International software metrics symposium (METRICS’05), pp 10 pp.–23 https://doi.org/10.1109/METRICS.2005.24
Huberty CJ, Olejnik S (2006) Applied MANOVA and discriminant analysis, vol 498. John Wiley & Sons, New York
Hudak P (1998) Modular domain specific languages and tools. In: Proceedings. Fifth international conference on software reuse (Cat. No.98TB100203), pp 134–142. https://doi.org/10.1109/ICSR.1998.685738
Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation, 1st. Addison-Wesley Professional, Boston
IEEE (2010) Ieee standard classification for software anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993) pp 1–23. https://doi.org/10.1109/IEEESTD.2010.5399061
Jiang Y, Adams B (2015) Co-evolution of infrastructure and source code: an empirical study. In: Proceedings of the 12th working conference on mining software repositories, ieee press, Piscataway, NJ, USA, MSR ’15, pp 45–55. http://dl.acm.org/citation.cfm?id=2820518.2820527
Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. Springer London, London, pp 63–92. https://doi.org/10.1007/978-1-84800-044-5_3
Labs P (2018) Puppet documentation. https://docs.puppet.com/, [Online; accessed 08-Aug-2018]
Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. http://www.jstor.org/stable/2529310
Leone M (2016) The economic benefits of puppet enterprise. Tech. rep., ESG. https://puppet.com/resources/analyst-report/the-economic-benefits-puppet-enterprise
MacLeod L, Greiler M, Storey M, Bird C, Czerwonka J (2018) Code reviewing in the trenches: challenges and best practices. IEEE Softw 35 (4):34–42. https://doi.org/10.1109/MS.2017.265100500
Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Statist 18(1):50–60. http://www.jstor.org/stable/2236101
Martin RC (2011) The clean coder: a code of conduct for professional programmers. Pearson Education
McCune JT, Jeffrey (2011) Pro Puppet, 1st edn. Apress. https://doi.org/10.1007/978-1-4302-3058-8. https://www.springer.com/gp/book/9781430230571
Meneely A, Williams L (2009) Secure open source collaboration: an empirical study of linus’ law. In: Proceedings of the 16th ACM conference on computer and communications security, ACM, New York, NY, USA, CCS ’09, pp 453–462. https://doi.org/10.1145/1653662.1653717
Meneely A, Smith B, Williams L (2013) Validating software metrics: a spectrum of philosophies. ACM Trans Softw Eng Methodol 21(4):24:1–24:28. https://doi.org/10.1145/2377656.2377661
Menzies T, Greenwald J, Frank A (2007) Data mining static code attributes to learn defect predictors. IEEE T Software Eng 33(1):2–13. https://doi.org/10.1109/TSE.2007.256941
Morris K (2016) Infrastructure as code: managing servers in the cloud. “ O’Reilly Media, Inc.”
Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating github for engineered software projects. Empirical Software Engineering pp 1–35. https://doi.org/10.1007/s10664-017-9512-6
Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E (2018) Systematic review or scoping review? guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 18 (1):143
Oktaba P (2015) Keep your commits small. https://dzone.com/articles/keep-your-commits-small, [Online; accessed 08-Feb-2019]
Ostrand TJ, Weyuker EJ, Bell RM (2004) Where the bugs are. In: Proceedings of the 2004 ACM SIGSOFT international symposium on software testing and analysis, ACM, New York, NY, USA, ISSTA ’04, pp 86–96. https://doi.org/10.1145/1007512.1007524
Pinzger M, Nagappan N, Murphy B (2008) Can developer-module networks predict failures?. In: Proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, SIGSOFT ’08/FSE-16, pp 2–12. https://doi.org/10.1145/1453101.1453105
Rahman A, Williams L (2018) Characterizing defective configuration scripts used for continuous deployment. In: 2018 IEEE 11th International conference on software testing, verification and validation (ICST), pp 34–45. https://doi.org/10.1109/ICST.2018.00014
Rahman A, Williams L (2019) Source code properties of defective infrastructure as code scripts. Information and Software Technology. https://doi.org/10.1016/j.infsof.2019.04.013, http://www.sciencedirect.com/science/article/pii/S0950584919300965
Rahman A, Partho A, Morrison P, Williams L (2018) What questions do programmers ask about configuration as code?. In: Proceedings of the 4th international workshop on rapid continuous software engineering, ACM, New York, NY, USA, RCoSE ’18, pp 16–22. https://doi.org/10.1145/3194760.3194769
Rahman A, Parnin C, Williams L (2019) The seven sins: Security smells in infrastructure as code scripts. In: Proceedings of the 41st international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’19, pp 164–175. https://doi.org/10.1109/ICSE.2019.00033
Rahman A, Farhana E, Parnin C, Williams L (2020) Gang of eight: a defect taxonomy for infrastructure as code scripts. In: Proceedings of the 42nd international conference on software engineering, ICSE ’20, to appear
Rahman F, Devanbu P (2013a) How, and why, process metrics are better. In: Proceedings of the 2013 international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’13, pp 432–441. http://dl.acm.org/citation.cfm?id=2486788.2486846
Rahman F, Devanbu P (2013b) How, and why, process metrics are better. In: Proceedings of the 2013 international conference on software engineering, IEEE press, Piscataway, NJ, USA, ICSE ’13, pp 432–441. http://dl.acm.org/citation.cfm?id=2486788.2486846
Rigby PC, German DM, Storey MA (2008) Open source software peer review practices: a case study of the apache server. In: Proceedings of the 30th international conference on software engineering, ACM, New York, NY, USA, ICSE ’08, pp 541–550. https://doi.org/10.1145/1368088.1368162
Romano J, Kromrey J, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys?. In: Annual meeting of the florida association of institutional research, pp 1–3
Saldana J (2015) The coding manual for qualitative researchers. Sage
Shambaugh R, Weiss A, Guha A (2016) Rehearsal: a configuration verification tool for puppet. SIGPLAN Not 51(6):416–430. https://doi.org/10.1145/2980983.2908083
Sharma T, Fragkoulis M, Spinellis D (2016) Does your configuration code smell?. In: Proceedings of the 13th international conference on mining software repositories, ACM, New York, NY, USA, MSR ’16, pp 189–200. https://doi.org/10.1145/2901739.2901761
Shihab E, Jiang ZM, Adams B, Hassan AE, Bowerman R (2011) Prioritizing the creation of unit tests in legacy software systems. Software Pract Exper 41(10):1027–1048. https://doi.org/10.1002/spe.1053
Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 2013 6th International workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92. https://doi.org/10.1109/CHASE.2013.6614738
Sullivan GM, Feinn R (2012) Using effect size-or why the p value is not enough. J Grad Med Educ 4(3):279–282. https://doi.org/10.4300/JGME-D-12-00156.1
Tan PN, Steinbach M, Kumar V (2005) Introduction to data mining, 1st. Addison-Wesley Longman Publishing Co., Inc., Boston
Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2016) Automated parameter optimization of classification techniques for defect prediction models. In: Proceedings of the 38th international conference on software engineering, ACM, New York, NY, USA, ICSE ’16, pp 321–332 https://doi.org/10.1145/2884781.2884857
Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2017) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18. https://doi.org/10.1109/TSE.2016.2584050
Tosun A, Bener A, Turhan B, Menzies T (2010) Practical considerations in deploying statistical methods for defect prediction: a case study within the turkish telecommunications industry. Inf Softw Technol 52(11):1242–1257. https://doi.org/10.1016/j.infsof.2010.06.006
Tufano M, Bavota G, Poshyvanyk D, Di Penta M, Oliveto R, De Lucia A (2017) An empirical study on developer-related factors characterizing fix-inducing commits. J Softw Evol Proc 29(1):e1797. https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.1797
Turhan B, Kocak G, Bener A (2009) Data mining source code for locating software bugs: a case study in telecommunication industry. Expert Syst Appl 36(6):9986–9990. https://doi.org/10.1016/j.eswa.2008.12.028. http://www.sciencedirect.com/science/article/pii/S0957417408009275
Turnbull J (2007) Pulling strings with puppet: automated system administration done right. Apress
van der Bent E, Hage J, Visser J, Gousios G (2018) How good is your puppet? an empirically defined and validated quality model for puppet. In: 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER), pp 164–174. https://doi.org/10.1109/SANER.2018.8330206
Van Wyk E, Krishnan L, Bodin D, Schwerdfeger A (2007) Attribute grammar-based language extensions for java. In: Proceedings of the 21st european conference on object-oriented programming, springer-verlag, Berlin, Heidelberg, ECOOP’07, pp 575–599. http://dl.acm.org/citation.cfm?id=2394758.2394796
Voelter M (2013) DSL engineering: designing implementing and using domain-specific languages. CreateSpace Independent Publishing Platform, USA
Weinberg GM (1992) Quality software management (vol. 1): systems thinking. Dorset House Publishing Co., Inc., New York
Weiss A, Guha A, Brun Y (2017) Tortoise: Interactive system configuration repair. In: Proceedings of the 32Nd IEEE/ACM international conference on automated software engineering, IEEE press, Piscataway, NJ, USA, ASE 2017, pp 625–636. http://dl.acm.org/citation.cfm?id=3155562.3155641
Acknowledgements
The NSA Science of Security Lablet (award H98230-17-D-0080) at the North Carolina State University supported this research study. We thank the Realsearch research group members for their useful feedback. We also thank the practitioners who answered our questions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Daniel Méndez
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rahman, A., Farhana, E. & Williams, L. The ‘as code’ activities: development anti-patterns for infrastructure as code. Empir Software Eng 25, 3430–3467 (2020). https://doi.org/10.1007/s10664-020-09841-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-020-09841-8