Skip to main content
SpringerLink
Log in

Assessment of Security Defense of Native Programs Against Software Faults

  • Chapter
  • First Online:
System Dependability and Analytics

Part of the book series: Springer Series in Reliability Engineering ((RELIABILITY))

  • 419 Accesses

Abstract

This chapter explores the possibility of building a unified assessment methodology for software reliability and security. The fault injection methodology originally designed for reliability assessment is extended to quantify and characterize the security defense aspect of native applications. Native application refers to system software written in C/C++ programming language. Specifically, software fault injection is used to measure the portion of injected software faults caught by the built-in error detection mechanisms of a target program (e.g., the detection coverage of assertions). To automatically activate as many injected faults as possible, a gray box fuzzing technique is used. Using dynamic analyzers during fuzzing further helps us catch the critical error propagation paths of injected (but undetected) faults, and identify code fragments as targets for security hardening. Because conducting software fault injection experiments for fuzzing is an expensive process, a novel, locality-based fault selection algorithm is presented. The presented algorithm increases the fuzzing failure ratios by 3–19 times, accelerating the speed of experiment. The case studies use all the above experimental techniques in order to compare the effectiveness of fuzzing and testing, and consequently assess the security defense of native benchmark programs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    UIUC DEPEND research group founded by Professor Ravishankar K. Iyer (Fellow of AAAS, ACM, and IEEE) has been one of the leading academic research groups in this field. Many SWIFI tools reviewed in this section were built by the DEPEND research group.

  2. 2.

    Ph.D. alumni of UIUC DEPEND group and IEEE Fellow for the contributions on software reliability.

  3. 3.

    In other words, mutant types.

  4. 4.

    libFuzzer, https://llvm.org/docs/LibFuzzer.html.

  5. 5.

    The used structural tests were contained in the benchmark programs. For the fuzzing, ones available at https://github.com/google/fuzzer-test-suite were used that were developed by a fuzzing team at Google.

  6. 6.

    American Fuzzy Lop (AFL), http://lcamtuf.coredump.cx/afl/.

  7. 7.

    The seeds and dictionaries available at https://github.com/mirrorer/afl/ and https://github.com/google/fuzzer-test-suite were used. Otherwise, seeds were generated by running fuzzing for a sufficiently long period of time (e.g., for sqliıe).

  8. 8.

    This work is rooted in the fault injection methodology and demonstrates a new application area of fault injection in software security evaluation. Since the author joined Google, there have been other works done to improve the dependability and security of mobile cloud computing applications. Interested readers are referred to [59] for big data service monitoring, [60] for big data software release, [5] for cloud virtualization platform security, and [61] Android platform ecosystem security.

References

  1. Chen S, Xu J, Kalbarczyk Z, Iyer R, Whisnant K (2004) Modeling and evaluating the security threats of transient errors in firewall software. Perform Eval 56(1):53–72

    Article  Google Scholar 

  2. Nakka N, Kalbarczyk Z, Iyer R, Xu J (2004) An architectural framework for providing reliability and security support. In: Proceedings of the IEEE/IFIP international conference on dependable systems and networks (DSN), pp 585–594

    Google Scholar 

  3. Pham C, Estrada Z, Cao P, Kalbarczyk Z, Iyer RK (2014) Reliability and security monitoring of virtual machines using hardware architectural invariants. In: Proceedings of the IEEE/IFIP international conference on dependable systems and networks (DSN), pp 13–24

    Google Scholar 

  4. Sanders WH (2014) Quantitative security metrics: Unattainable holy grail or a vital breakthrough within our reach? IEEE Secur Priv 12(2):67–69

    Article  Google Scholar 

  5. Yim KS (2016) The rowhammer attack injection methodology. In: 2016 IEEE 35th symposium on reliable distributed systems (SRDS), pp 1–10

    Google Scholar 

  6. Iyer R, Nakka N, Gu W, Kalbarczyk Z (2010) Fault injection. In: Encyclopedia of software engineering, pp 287–299

    Google Scholar 

  7. Pendleton M, Garcia-Lebron R, Cho J-H, Xu S (2017) A survey on systems security metrics. ACM Comput Surv 49(4):1–35

    Article  Google Scholar 

  8. Arlat J, Aguera M, Amat L, Crouzet Y, Fabre JC, Laprie JC, Martins E, Powell D (1990) Fault injection for dependability validation: a methodology and some applications. IEEE Trans Softw Eng 16(2):166–182

    Article  Google Scholar 

  9. Madeira H, Silva JG (1994) Experimental evaluation of the fail-silent behavior in computers without error masking. In: Proceedings of IEEE 24th international symposium on fault-tolerant computing, pp 350–359

    Google Scholar 

  10. Madeira H, Rela M, Moreira F, Silva JG (1994) Rifle: a general purpose pin-level fault injector. In: Echtle K, Hammer D, Powell D (eds) Dependable computing—EDCC-1. Springer, Berlin, pp 197–216d

    Google Scholar 

  11. Karlsson J, Liden P, Dahlgren P, Johansson R, Gunneflo U (1994) Using heavy-ion radiation to validate fault-handling mechanisms. IEEE Micro 14(1):8–23

    Article  Google Scholar 

  12. Ando H, Kan R, Tosaka Y, Takahisa K, Hatanaka K (2008) Validation of hardware error recovery mechanisms for the sparc64 v microprocessor. In: 2008 IEEE international conference on dependable systems and networks with FTCS and DCC (DSN), pp 62–69

    Google Scholar 

  13. Govindavajhala S, Appel AW (2003) Using memory errors to attack a virtual machine. In: 2003 symposium on security and privacy, pp 154–165

    Google Scholar 

  14. Miremadi G, Harlsson J, Gunneflo U, Torin J (1992) Two software techniques for on-line error detection. In: [1992] digest of papers. FTCS-22: the twenty-second international symposium on fault-tolerant computing, pp 328–335

    Google Scholar 

  15. Choi GS, Iyer RK (1992) Focus: an experimental environment for fault sensitivity analysis. IEEE Trans Comput 41(12):1515–1526

    Article  Google Scholar 

  16. Jenn E, Arlat J, Rimen M, Ohlsson J, Karlsson J (1994) Fault injection into vhdl models: the mefisto tool. In: Proceedings of IEEE 24th international symposium on fault-tolerant computing, pp 66–75

    Google Scholar 

  17. Segall Z, Vrsalovic D, Siewiorek D, Yaskin D, Kownacki J, Barton J, Dancey R, Robinson A, Lin T (1988) Fiat-fault injection based automated testing environment. In: [1988] The eighteenth international symposium on fault-tolerant computing. Digest of papers, pp 102–107

    Google Scholar 

  18. Yim KS, Pham C, Saleheen M, Kalbarczyk Z, Iyer R (2011) Hauberk: Lightweight silent data corruption error detector for gpgpu. In: Proceedings of the IEEE international parallel distributed processing symposium (IPDPS), pp 287–300

    Google Scholar 

  19. Stott DT, Floering B, Burke D, Kalbarczpk Z, Iyer RK (2000) Nftape: a framework for assessing dependability in distributed systems with lightweight fault injectors. In: Proceedings IEEE international computer performance and dependability symposium. IPDS 2000, pp 91–100

    Google Scholar 

  20. Carreira J, Madeira H, Silva JG (1998) Xception: a technique for the experimental evaluation of dependability in modern computers. IEEE Trans Softw Eng 24(2):125–136

    Article  Google Scholar 

  21. Arlat J, Fabre JC, Rodriguez M (2002) Dependability of cots microkernel-based systems. IEEE Trans Comput 51(2):138–163

    Article  Google Scholar 

  22. Kanawati GA, Kanawati NA, Abraham JA (1995) Ferrari: a flexible software-based fault and error injection system. IEEE Trans Comput 44(2):248–260

    Article  Google Scholar 

  23. Han S, Shin KG, Rosenberg HA (1995) Doctor: an integrated software fault injection environment for distributed real-time systems. In: Proceedings of 1995 IEEE international computer performance and dependability symposium, pp 204–213

    Google Scholar 

  24. Pattabiraman K, Nakka NM, Kalbarczyk ZT, Iyer RK (2013) Symplfied: symbolic program-level fault injection and error detection framework. IEEE Trans Comput 62(11):2292–2307

    Article  MathSciNet  Google Scholar 

  25. Jacques-Silva G, Kalbarczyk Z, Gedik B, Andrade H, Wu K-L, Iyer RK (2011) Modeling stream processing applications for dependability evaluation. In: 2011 IEEE/IFIP 41st international conference on dependable systems networks (DSN), pp 430–441

    Google Scholar 

  26. Kao WI, Iyer RK, Tang D (1993) Fine: a fault injection and monitoring environment for tracing the Unix system behavior under faults. IEEE Trans Softw Eng 19(11):1105–1118

    Article  Google Scholar 

  27. Kao W-L, Iyer RK (1994) Define: a distributed fault injection and monitoring environment. In: Proceedings of IEEE workshop on fault-tolerant parallel and distributed systems, pp 252–259

    Google Scholar 

  28. Tsai TK, Iyer RK, Jewitt D (1996) An approach towards benchmarking of fault-tolerant commercial systems. In: Proceedings of annual symposium on fault tolerant computing, pp 314–323

    Google Scholar 

  29. Chillarege R, Bhandari IS, Chaar JK, Halliday MJ, Moebus DS, Ray BK, Wong MY (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans Softw Eng 18(11):943–956

    Article  Google Scholar 

  30. Sullivan M, Chillarege R (1991) Software defects and their impact on system availability-a study of field failures in operating systems. In: [1991] digest of papers. Fault-tolerant computing: the twenty-first international symposium, pp 2–9

    Google Scholar 

  31. Thakur A, Iyer RK, Young L, Lee I (1995) Analysis of failures in the tandem nonstop-ux operating system. In: Proceedings of the sixth international symposium on software reliability engineering, pp 40–50

    Google Scholar 

  32. Herder JN, Bos H, Gras B, Homburg P, Tanenbaum AS (2009) Fault isolation for device drivers. IEEE/IFIP international conference on dependable systems networks 2009:33–42

    Article  Google Scholar 

  33. Herder JN, Bos H, Gras B, Homburg P, Tanenbaum AS (2007) Failure resilience for device drivers. In: Proceedings of the IEEE/IFIP international conference on dependable systems and networks (DSN), pp 41–50

    Google Scholar 

  34. Moraes R, Barbosa R, Durães J, Mendes N, Martins E, Madeira H (2006) Injection of faults at component interfaces and inside the component code: are they equivalent? In: Proceedings of the European dependable computing conference, pp 53–64

    Google Scholar 

  35. Johansson A, Suri N, Murphy B (2007) On the selection of error model(s) for os robustness evaluation. In: 37th annual IEEE/IFIP international conference on dependable systems and networks (DSN’07), pp 502–511

    Google Scholar 

  36. Winter S, Sarbu C, Suri N, Murphy B (2011) The impact of fault models on software robustness evaluations. In: 2011 33rd international conference on software engineering (ICSE), pp 51–60

    Google Scholar 

  37. Durães J, Madeira H (2002) Emulation of software faults by educated mutations at machine-code level. In: Proceedings of 13th international symposium on software reliability engineering, pp 329–340

    Google Scholar 

  38. Durães JA, Madeira HS (2006) Emulation of software faults: a field data study and a practical approach. IEEE Trans Softw Eng 32(11):849–867

    Article  Google Scholar 

  39. Durães J, Madeira H (2003) Definition of software fault emulation operators: a field data study. In: Proceedings of 2003 international conference on dependable systems and networks, pp 105–114

    Google Scholar 

  40. Natella R, Cotroneo D, Duraes JA, Madeira HS (2013) On fault representativeness of software fault injection. IEEE Trans Softw Eng 39(1):80–96

    Article  Google Scholar 

  41. Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng 37(5):649–678

    Article  Google Scholar 

  42. Hudak JJ, Suh BH, Siewiorek DP, Segall Z (1993) Evaluation and comparison of fault-tolerant software techniques. IEEE Trans Reliab 42(2):190–204

    Article  Google Scholar 

  43. Bondavalli A, Chiaradonna S, Cotroneo D, Romano L (2004) Effective fault treatment for improving the dependability of cots and legacy-based applications. IEEE Trans Dependable Secure Comput 1(4):223–237

    Article  Google Scholar 

  44. Ng WT, Chen PM (2001) The design and verification of the rio file cache. IEEE Trans Comput 50(4):322–337

    Article  Google Scholar 

  45. Chandra S, Chen PM (1998) How fail-stop are faulty programs? In: Digest of papers. Twenty-eighth annual international symposium on fault-tolerant computing (Cat. No.98CB36224), pp 240–249

    Google Scholar 

  46. Swift MM, Bershad BN, Levy HM (2003) Improving the reliability of commodity operating systems. In: Proceedings of the nineteenth ACM symposium on operating systems principles, ser. SOSP ’03. ACM, New York, pp 207–222

    Google Scholar 

  47. Swift MM, Annamalai M, Bershad BN, Levy HM (2006) Recovering device drivers. ACM Trans Comput Syst 24(4):333–360

    Article  Google Scholar 

  48. Neves N, Antunes J, Correia M, Verissimo P, Neves R (2006) Using attack injection to discover new vulnerabilities. In: International conference on dependable systems and networks (DSN’06), pp 457–466

    Google Scholar 

  49. Antunes J, Neves N, Correia M, Verissimo P, Neves R (2010) Vulnerability discovery with attack injection. IEEE Trans Softw Eng 36(3):357–370

    Article  Google Scholar 

  50. Fonseca J, Vieira M, Madeira H (2014) Evaluation of web security mechanisms using vulnerability attack injection. IEEE Trans Dependable Secure Comput 11(5):440–453

    Article  Google Scholar 

  51. Kanoun K, Spainhower L (2008) Dependability benchmarking for computer systems. Wiley, IEEE Computer Society Pr

    Book  Google Scholar 

  52. Tsai TK, Hsueh M-C, Zhao H, Kalbarczyk Z, Iyer RK (1999) Stress-based and path-based fault injection. IEEE Trans Comput 48(11):1183–1201

    Article  Google Scholar 

  53. Serebryany K, Bruening D, Potapenko A, Vyukov D (2012) Addresssanitizer: a fast address sanity checker. In: 2012 USENIX annual technical conference (USENIX ATC 12). USENIX Association, Boston, pp 309–318. [Online]. Available: https://www.usenix.org/conference/atc12/technical-sessions/presentation/serebryany

  54. Kim S, Zimmermann T, Jr EJW, Zeller A (2007) Predicting faults from cached history. In: Proceedings of the international conference on software engineering (ICSE), pp 489–498

    Google Scholar 

  55. Yim KS, Kalbarczyk ZT, Iyer RK (2009) Quantitative analysis of long-latency failures in system software. In: 2009 15th IEEE Pacific rim international symposium on dependable computing, pp 23–30

    Google Scholar 

  56. Razavi K, Gras B, Bosman E, Preneel B, Giuffrida C, Bos H (2016) Flip feng shui: Hammering a needle in the software stack. In: 25th USENIX security symposium (USENIX Security 16). USENIX Association, Austin, pp 1–18

    Google Scholar 

  57. Cadar C, Ganesh V, Pawlowski PM, Dill DL, Engler DR (2008) Exe: automatically generating inputs of death. ACM Trans Inf Syst Secur 12(2):10:1–10:38

    Google Scholar 

  58. Cadar C, Dunbar D, Engler D (2008) Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th USENIX conference on operating systems design and implementation, ser. OSDI’08. USENIX Association, Berkeley, pp 209–224

    Google Scholar 

  59. Yim KS (2016) Evaluation metrics of service-level reliability monitoring rules of a big data service. In: 2016 IEEE 27th international symposium on software reliability engineering (ISSRE), pp 376–387

    Google Scholar 

  60. Yim KS (2014) Norming to performing: Failure analysis and deployment automation of big data software developed by highly iterative models. In: 2014 IEEE 25th international symposium on software reliability engineering, pp 144–155

    Google Scholar 

  61. Yim KS, Malchev I, Hsieh A, Burke D (2019) Treble: fast software updates by creating an equilibrium in an active software ecosystem of globally distributed stakeholders. ACM Trans Embed Comput Syst 18(5s). [Online]. Available: https://doi.org/10.1145/3358237

Download references

Author information

Authors and Affiliations

  1. Google, Alphabet Inc., attn yim, 1600 Amphitheatre Parkway, Mountain View, CA, 94043, USA

    Keun Soo Yim

Authors
  1. Keun Soo Yim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Keun Soo Yim .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Long Wang

  2. University of British Columbia, Vancouver, BC, Canada

    Karthik Pattabiraman

  3. Nokia Bell Labs, São Paulo, Brazil

    Catello Di Martino

  4. Mayo Clinic, Rochester, MN, USA

    Arjun Athreya

  5. Purdue University, West Lafayette, IN, USA

    Saurabh Bagchi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Yim, K.S. (2023). Assessment of Security Defense of Native Programs Against Software Faults. In: Wang, L., Pattabiraman, K., Di Martino, C., Athreya, A., Bagchi, S. (eds) System Dependability and Analytics. Springer Series in Reliability Engineering. Springer, Cham. https://doi.org/10.1007/978-3-031-02063-6_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-02063-6_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-02062-9

  • Online ISBN: 978-3-031-02063-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation