Skip to main content
SpringerLink
Log in

Fixing vulnerabilities potentially hinders maintainability

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Security is a requirement of utmost importance to produce high-quality software. However, there is still a considerable amount of vulnerabilities being discovered and fixed almost weekly. We hypothesize that developers affect the maintainability of their codebases when patching vulnerabilities. This paper evaluates the impact of patches to improve security on the maintainability of open-source software. Maintainability is measured based on the Better Code Hub’s model of 10 guidelines on a dataset, including 1300 security-related commits. Results show evidence of a trade-off between security and maintainability for 41.90% of the cases, i.e., developers may hinder software maintainability. Our analysis shows that 38.29% of patches increased software complexity and 37.87% of patches increased the percentage of LOCs per unit. The implications of our study are that changes to codebases while patching vulnerabilities need to be performed with extra care; tools for patch risk assessment should be integrate into the CI/CD pipeline; computer science curricula needs to be updated; and, more secure programming languages are necessary.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Listing 1
Listing 2
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Zero Day Initiative website available at https://www.zerodayinitiative.com/advisories/published/(Accessed on September 20, 2021)

  2. SIG’s website: https://www.sig.eu/ (Accessed on September 20, 2021)

  3. BCH’s website: https://bettercodehub.com/ (Accessed on September 20, 2021)

  4. OpenSSL is a toolkit that contains open-source implementations of the SSL and TLS cryptographic protocols. Repository available at https://github.com/openssl/openssl (Accessed on September 20, 2021)

  5. CVE-2016-6304 details available at http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6304(Accessed on September 20, 2021)

  6. CVE-2016-6304 fix available at https://github.com/openssl/openssl/commit/e408c09bbf7c3057bda4b8d20bec1b3a7771c15b(Accessed on September 20, 2021)

  7. CVE-2014-1608 details available at https://github.com/mantisbt/mantisbt/commit/00b4c17088fa56594d85fe46b6c6057bb3421102(Accessed on September 20, 2021)

  8. CWE-89 details available at https://cwe.mitre.org/data/definitions/89.html (Accessed on September 20, 2021)

  9. Research Concepts list available at https://cwe.mitre.org/data/definitions/1000.html

  10. Information available here: https://www.softwareimprovementgroup.com/methodologies/iso-iec-25010-2011-standard/

  11. Check the answer to How can I adjust the threshold for passing/not passing a guideline? at https://bettercodehub.com/docs/faq (Accessed on September 20, 2021)

  12. Research Concepts is a tree-view provided by the Common Weakness Enumeration (CWE) website that intends to facilitate research into weaknesses. It is organized according to abstractions of behaviors instead of how they can be detected, their usual location in code, and when they are introduced in the development life cycle. The list is available here: https://cwe.mitre.org/data/definitions/1000.html

  13. CVE-2016-0799 patch details available at https://github.com/openssl/openssl/commit/9cb177301fdab492e4cfef376b28339afe3ef663 (Accessed on September 20, 2021)

References

  • Acar Y, Stransky C, Wermke D, Weir C, Mazurek ML, Fahl S (2017) Developers need support, too: A survey of security advice for software developers. In: 2017 IEEE cybersecurity development (SecDev), pp 22–26. https://doi.org/10.1109/SecDev.2017.17

  • Alves TL, Correia JP, Visser J (2011) Benchmark-based aggregation of metrics to ratings. In: 2011 Joint conference of the 21st international workshop on software measurement and the 6th international conference on software process and product measurement, pp 20–29. https://doi.org/10.1109/IWSM-MENSURA.2011.15

  • Alves TL, Ypma C, Visser J (2010) Deriving metric thresholds from benchmark data. In: 2010 IEEE international conference on software maintenance, pp 1–10. https://doi.org/10.1109/ICSM.2010.5609747

  • Baggen R, Correia JP, Schill K, Visser J (2012) Standardized code quality benchmarking for improving software maintainability. Softw Qual J 20 (2):287–307. https://doi.org/10.1007/s11219-011-9144-9

    Article  Google Scholar 

  • Berger ED, Hollenbeck C, Maj P, Vitek O, Vitek J (2019) On the impact of programming languages on code quality. arXiv:1901.10220

  • Bijlsma D, Ferreira MA, Luijten B, Visser J (2012) Faster issue resolution with higher technical quality of software. Softw Qual J. 20(2):265–285. https://doi.org/10.1007/s11219-011-9140-0

    Article  Google Scholar 

  • Chowdhury I, Zulkernine M (2010) Can complexity, coupling, and cohesion metrics be used as early indicators of vulnerabilities?. In: Proceedings of the 2010 ACM symposium on applied computing, SAC ’10. pp 1963–1969, Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1774088.1774504

  • Common Criteria Working Group (2009) Common methodology for information technology security evaluation. Tech. rep., Technical report, Common Criteria Interpretation Management Board

  • Cruz L, Abreu R, Grundy J, Li L, Xia X (2019) Do energy-oriented changes hinder maintainability?. In: 2019 IEEE International conference on software maintenance and evolution (ICSME), pp 29–40

  • di Biase M, Rastogi A, Bruntink M, van Deursen A (2019) The delta maintainability model: Measuring maintainability of fine-grained code changes. In: 2019 IEEE/ACM international conference on technical debt (TechDebt), pp 113–122

  • Elkhail AA, Cerny T (2019) On relating code smells to security vulnerabilities. In: 2019 IEEE 5th intl conference on big data security on cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing, (HPSC) and IEEE intl conference on intelligent data and security (IDS), pp 7–12

  • Foundation TO (2017) Owasp top 10 - 2017, The ten most critical web application security risks. Tech. rep., The OWASP Foundation. Release Candidate

  • Foundation TO (2017) Owasp top 10 - 2017, The ten most critical web application security risks. Tech. rep., The OWASP Foundation. Release Candidate

  • Hegedűs P, Kádár I, Ferenc R, Gyimóthy T (2018) Empirical evaluation of software maintainability based on a manually validated refactoring dataset. Inf Softw Technol 95:313–327. https://doi.org/10.1016/j.infsof.2017.11.012

    Article  Google Scholar 

  • Hegedűs P, Bán D, Ferenc R, Gyimóthy T (2012) Myth or reality? analyzing the effect of design patterns on software maintainability. In: Computer applications for software engineering, disaster recovery, and business continuity. Springer, Berlin, pp 138–145

  • Heitlager I, Kuipers T, Visser J (2007) A practical model for measuring maintainability. In: 6th International conference on the quality of information and communications technology (QUATIC 2007), pp 30–39. https://doi.org/10.1109/QUATIC.2007.8

  • International Organization for Standardization (2011) International standard ISO/IEC 25010 systems and software engineering - systems and software quality requirements and evaluation (SQuaRE) - system and software quality models

  • Islam MR, Zibran MF (2016) A comparative study on vulnerabilities in categories of clones and non-cloned code. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), vol 3, pp 8–14

  • Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 654–665

  • Kataoka Y, Imai T, Andou H, Fukaya T (2002) A quantitative evaluation of maintainability enhancement by refactoring. In: International conference on software maintenance, 2002. Proceedings., pp 576–585. https://doi.org/10.1109/ICSM.2002.1167822

  • Khomh F, Gueheneuce Y (2008) Do design patterns impact software quality positively?. In: 2008 12th European conference on software maintenance and reengineering, pp 274–278. https://doi.org/10.1109/CSMR.2008.4493325

  • Kurilova D, Potanin A, Aldrich J (2014) Wyvern: Impacting software security via programming language design. In: Proceedings of the 5th workshop on evaluation and usability of programming languages and tools, pp 57–58

  • Li F, Paxson V (2017) A large-scale empirical study of security patches. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, CCS ’17, pp 2201–2215, Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3133956.3134072

  • Malavolta I, Verdecchia R, Filipovic B, Bruntink M, Lago P (2018) How maintainability issues of android apps evolve. In: 2018 IEEE international conference on software maintenance and evolution (ICSME), pp 334–344. https://doi.org/10.1109/ICSME.2018.00042

  • Maruyama K, Tokoda K (2008) Security-aware refactoring alerting its impact on code vulnerabilities. In: 2008 15th Asia-pacific software engineering conference, pp 445–452. https://doi.org/10.1109/APSEC.2008.57

  • McCabe TJ (1976) A complexity measure. IEEE Trans Softw Eng SE-2(4):308–320. https://doi.org/10.1109/TSE.1976.233837

    Article  MathSciNet  Google Scholar 

  • McGraw G (2004) Software security. IEEE Secur Priv 2(2):80–83

    Article  Google Scholar 

  • McGraw KO, Wong SP (1992) A common language effect size statistic psychological bulletin. https://doi.org/10.1037/0033-2909.111.2.361

  • Nistor L, Kurilova D, Balzer S, Chung B, Potanin A, Aldrich J (2013) Wyvern: A simple, typed, and pure object-oriented language. In: Proceedings of the 5th Workshop on MechAnisms for SPEcialization, Generalization and InHerItance, MASPEGHI ’13, pp 9–16, Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2489828.2489830

  • Olivari M (2018) Maintainable production: A model of developer productivity based on source code contributions. Master’s thesis University of Amsterdam

  • Palomba F, Bavota G, Penta MD, Fasano F, Oliveto R, Lucia AD (2018) On the diffuseness and the impact on maintainability of code smells: A large scale empirical investigation. Empirical Softw Engg 23(3):1188–1221. https://doi.org/10.1007/s10664-017-9535-z

    Article  Google Scholar 

  • Ponta SE, Plate H, Sabetta A, Bezzi M, Dangremont C (2019) A manually-curated dataset of fixes to vulnerabilities of open-source software. In: Proceedings of the 16th international conference on mining software repositories, MSR ’19. IEEE Press, p 383–387. https://doi.org/10.1109/MSR.2019.00064

  • Pothamsetty V (2005) Where security education is lacking. In: Proceedings of the 2Nd annual conference on information security curriculum development, InfoSecCD ’05, pp 54–58, ACM, New York, NY, USA. https://doi.org/10.1145/1107622.1107635

  • Pratt JW (1959) Remarks on zeros and ties in the wilcoxon signed rank procedures. J Am Stat Assoc 54(287):655–667

    Article  MathSciNet  Google Scholar 

  • Ray B, Posnett D, Devanbu P, Filkov V (2017) A large-scale study of programming languages and code quality in github. Commun ACM 60 (10):91–100. https://doi.org/10.1145/3126905

    Article  Google Scholar 

  • Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in Github. In: Proceedings of the 22Nd ACM SIGSOFT international symposium on foundations of software engineering, FSE 2014, 155–165, ACM, New York, NY, USA. https://doi.org/10.1145/2635868.2635922

  • Reis S, Abreu R (2017) A database of existing vulnerabilities to enable controlled testing studies. Int J Secur Softw Eng (IJSSE) 8(3). https://doi.org/10.4018/IJSSE.2017070101

  • Reis S, Abreu R (2017) Secbench: A database of real security vulnerabilities. In: Proceedings of the international workshop on secure software engineering in devops and agile development (SecSE 2017)

  • Schneier B (2006) Beyond fear: Thinking sensibly about security in an uncertain world. Berlin, Springer Science & Business Media

    Google Scholar 

  • Shin Y, Meneely A, Williams L, Osborne JA (2010) Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Trans Softw Eng 37(6):772–787

    Article  Google Scholar 

  • Slaughter SA, Harter DE, Krishnan MS (1998) Evaluating the cost of software quality. Commun ACM 41(8):67–73

    Article  Google Scholar 

  • Telang R, Wattal S (2007) An empirical analysis of the impact of software vulnerability announcements on firm stock price. IEEE Trans Softw Eng 33(8):544–557

    Article  Google Scholar 

  • The OWASP Foundation (2009) OWASP application security verification standard 2009 - web application standard. Tech rep

  • Visser J (2016) Building maintainable software, java edition: Ten guidelines for future-proof code. O’Reilly Media, Inc

  • Visser J (2020) Sig/tUvit evaluation criteria trusted product maintainability: Guidance for producers. Available: https://bit.ly/3hnY0Am

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics Bulletin 1(6):80–83

    Article  Google Scholar 

  • Xu H, Heijmans J, Visser J (2013) A practical model for rating software security. In: 2013 IEEE seventh international conference on software security and reliability companion, pp 231–232. https://doi.org/10.1109/SERE-C.2013.11

  • Zazworka N, Shaw MA, Shull F, Seaman C (2011) Investigating the impact of design debt on software quality. In: Proceedings of the 2nd workshop on managing technical debt, MTD ’11, pp 17–23, Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1985362.1985366

  • Zibran MF, Saha RK, Asaduzzaman M, Roy CK (2011) Analyzing and forecasting near-miss clones in evolving software: An empirical study. In: 2011 16th IEEE international conference on engineering of complex computer systems, pp 295–304

Download references

Acknowledgements

We thank SIG’s Better Code Hub team for all the support as well as help in validating our methodology and results; and, Pedro Adão for the invaluable feedback in the early stages of the project.

This work is financed by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia with reference UIDB/50021/2020, a PhD scholarship (ref. SFRH/BD/143319/2019), the SecurityAware Project (ref. CMU/TIC/0064/2019)—also funded by the Carnegie Mellon Program—, and the FaultLocker Project (ref. PTDC/CCI-COM/29300/2017).

Author information

Authors and Affiliations

  1. INESC-ID and IST, University of Lisbon, Lisbon, Portugal

    Sofia Reis

  2. INESC-ID and FEUP, University of Porto, Porto, Portugal

    Rui Abreu

  3. Delft University of Technology, Delft, The Netherlands

    Luis Cruz

Authors
  1. Sofia Reis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Abreu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sofia Reis.

Additional information

Communicated by: Burak Turhan

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reis, S., Abreu, R. & Cruz, L. Fixing vulnerabilities potentially hinders maintainability. Empir Software Eng 26, 127 (2021). https://doi.org/10.1007/s10664-021-10019-z

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10019-z

Keywords

Search

Navigation