Abstract
Effective and scalable quality assurance techniques are essential for realizing formal model-based development techniques for high-assurance systems. In this paper, we present the GUMBOX property-based testing framework for the SAE standard Architecture and Analysis Definition Language (AADL) integrated with HAMR AADL code generation tool chain. In GUMBOX, automated testing infrastructure for AADL component application code is automatically generated from AADL models and formal specifications written in the GUMBO contract language. This testing framework complements our previous work on using code-level symbolic execution to verify that component source code conforms to model-level GUMBO contracts, and it allows developers to switch between using testing and formal verification with specifications derived from a common contract language. We describe how the GUMBOX framework is incorporated in continuous integration infrastructure with parallel and distributed execution of tests in industrial workflows.
This work is supported in part by the U.S. Army Combat Capabilities Development Command, Aviation and Missile Center under Contract No.W911W6-20-C-2020 and the U.S. Defense Advanced Research Projects Agency (DARPA).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
As with any testing/code-level formal method, I/O and interactions with physical devices or stateful services may need to be supported by manually crafted stubs.
References
Barbosa, H., et al.: cvc5: A versatile and industrial-strength SMT solver. In: TACAS 2022. LNCS, vol. 13243, pp. 415–442. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99524-9_24
Barrett, C., et al.: CVC4. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 171–177. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_14
Belt, J., Hatcliff, J., Robby, Chalin, P., Hardin, D., Deng, X.: Bakar kiasan: flexible contract checking for critical systems using symbolic execution. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 58–72. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20398-5_6
Belt, J., et al.: Model-driven development for the seL4 microkernel using the HAMR framework. J. Syst. Archit. (2022)
Borde, E., Rahmoun, S., Cadoret, F., Pautet, L., Singhoff, F., Dissaux, P.: Architecture models refinement for fine grain timing analysis of embedded systems. In: 2014 25nd IEEE International Symposium on Rapid System Prototyping, pp. 44–50 (2014)
Burdy, L., et al.: An overview of JML tools and applications. Int. J. Softw. Tools Technol. Transf. 7(3), 212–232 (2005)
C, S.A.R.: Architecture analysis and design language (AADL) (2017)
Cadoret, F., Borde, E., Gardoll, S., Pautet, L.: Design patterns for rule-based refinement of safety critical embedded systems models. In: 2012 IEEE 17th International Conference on Engineering of Complex Computer Systems, pp. 67–76. IEEE (2012)
Cheon, Y., Leavens, G.T.: A simple and practical approach to unit testing: the JML and JUNIT way. In: ECOOP 2002 – Object-Oriented Programming, pp. 231–255 (2002)
Claessen, K., Hughes, J.: Quickcheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming, pp. 268–279 (2000)
Cofer, D.D., et al.: Cyberassured systems engineering at scale. IEEE Secur. Priv. 20(3), 52–64 (2022)
Cofer, D., Gacek, A., Miller, S., Whalen, M.W., LaValley, B., Sha, L.: Compositional verification of architectural models. In: Goodloe, A.E., Person, S. (eds.) NFM 2012. LNCS, vol. 7226, pp. 126–140. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28891-3_13
Conchon, S., Coquereau, A., Iguernlala, M., Mebsout, A.: Alt-ergo 2.2. In: SMT Workshop: International Workshop on Satisfiability Modulo Theories (2018)
Deng, X., Robby, Hatcliff, J.: Kiasan: a verification and test-case generation framework for java based on symbolic execution. In: Leveraging Applications of Formal Methods, Second International Symposium, ISoLA 2006, Paphos, Cyprus, 15–19 November 2006, pp. 137 (2006)
Deng, X., Robby, Hatcliff, J.: Kiasan/kunit: automatic test case generation and analysis feedback for open object-oriented systems. In: Testing: Academic and Industrial Conference Practice and Research Techniques (TAICPART 2007) (2007)
Dong, Y.W., Wang, G., Zhao, H.B.: A model-based testing for AADL model of embedded software. In: 2009 Ninth International Conference on Quality Software, pp. 185–190 (2009)
Fähndrich, M.: Static verification for code contracts. In: Static Analysis, pp. 2–5 (2010)
Feiler, P.H., Gluch, D.P.: Model-Based Engineering with AADL: An Introduction to the SAE Architecture Analysis and Design Language. Addison-Wesley, Boston (2013)
Fisher, K., Launchbury, J., Richards, R.: The HACMS program: using formal methods to eliminate exploitable bugs. Philos. Trans. Roy. Soc. A Math. Phys. Eng. Sci. 375(2104) (2017)
Godefroid, P.: Test generation using symbolic execution. In: Foundations of Software Technology and Theoretical Computer Science (2012)
Godefroid, P., Klarlund, N., Sen, K.: Dart: directed automated random testing. In: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 213–223 (2005)
GUMBOX property-based testing case studies (2023). https://github.com/santoslab/gumbox-case-studies
HAMR project website (2022). https://hamr.sireum.org
HARDENS: high assurance rigorous digital engineering for nuclear safety (artifacts repository). https://github.com/GaloisInc/HARDENS
Hatcliff, J., Belt, J., Robby, Carpenter, T.: HAMR: an AADL multi-platform code generation toolset. In: Margaria, T., Steffen, B. (eds.) ISoLA 2021. LNCS, vol. 13036, pp. 274–295. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89159-6_18
Hatcliff, J., Hugues, J., Stewart, D., Wrage, L.: Formalization of the AADL run-time services. In: Margaria, T., Steffen, B. (eds.) ISoLA 2022. LNCS, vol. 13702, pp. 105–134. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19756-7_7
Hatcliff, J., Leavens, G.T., Leino, K.R.M., Müller, P., Parkinson, M.: Behavioral interface specification languages. ACM Comput. Surv. 44(3) (2012)
Hatcliff, J., Stewart, D., Belt, J., Robby, Schwerdfeger, A.: An AADL contract language supporting integrated model- and code-level verification. In: Proceedings of the 2022 ACM Workshop on High Integrity Language Technology. HILT ’22 (2022)
Hoang, D., Moy, Y., Wallenburg, A., Chapman, R.: SPARK 2014 and GNATprove. Int. J. Softw. Tools Technol. Transf. 17(6) (2015)
Hugues, J., Wrage, L., Hatcliff, J., Stewart, D.: Mechanization of a large DSML: an experiment with AADL and coq. In: 20th ACM-IEEE International Conference on Formal Methods and Models for System Design, MEMOCODE 2022, Shanghai, China, 13–14 October 2022, pp. 1–9. IEEE (2022)
Johnsen, A., Pettersson, P., Lundqvist, K.: An architecture-based verification technique for AADL specifications. In: Crnkovic, I., Gruhn, V., Book, M. (eds.) ECSA 2011. LNCS, vol. 6903, pp. 105–113. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23798-0_11
Kanig, J.: Leading-edge ADA verification technologies: combining testing and verification with gnattest and gnatprove - the hi-lite project. In: Proceedings of the 2012 ACM Conference on High Integrity Language Technology, HILT ’12, 2–6 December 2012, Boston, Massachusetts, USA, pp. 5–6. ACM (2012)
Kline, S., Hudak, J., O’Neill, A.: Automated test and re-test for AADL (SBIR project between Innovative Defense Technologies and the Software Engineering Institute) (2022). https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=651952
Larson, B.R., Chalin, P., Hatcliff, J.: BLESS: formal specification and verification of behaviors for embedded systems with software. In: Brat, G., Rungta, N., Venet, A. (eds.) NFM 2013. LNCS, vol. 7871, pp. 276–290. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38088-4_19
Lasnier, G., Zalila, B., Pautet, L., Hugues, J.: Ocarina: an environment for AADL models analysis and automatic code generation for high integrity applications. In: Kordon, F., Kermarrec, Y. (eds.) Ada-Europe 2009. LNCS, vol. 5570, pp. 237–250. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-01924-1_17
Lempia, D., Miller, S.: DOT/FAA/AR-08/32. Requirements Engineering Management Handbook, Federal Aviation Administration (2009)
Leroy, X., Blazy, S., Kästner, D., Schommer, B., Pister, M., Ferdinand, C.: CompCert-a formally verified optimizing compiler. In: ERTS 2016: Embedded Real Time Software and Systems, 8th European Congress (2016)
Sireum logika (2022). https://logika.sireum.org
Meyer, B.: Eiffel as a framework for verification. In: Meyer, B., Woodcock, J. (eds.) VSTTE 2005. LNCS, vol. 4171, pp. 301–307. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-69149-5_32
de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24
Moy, Y., Ledinot, E., Delseny, H., Wiels, V., Monate, B.: Testing or formal verification: DO-178C alternatives and industrial experience. IEEE Softw. 30(3), 50–57 (2013)
Nilsson, R.: ScalaCheck: The Definitive Guide. Artima Press, Walnut Creek (2014)
Robby, Hatcliff, J.: Slang: the Sireum programming language. In: Margaria, T., Steffen, B. (eds.) Leveraging Applications of Formal Methods, Verification and Validation - ISoLA 2021. Lecture Notes in Computer Science, vol. 13036, pp. 253–273. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89159-6_17
sel4 microkernel (2015). sel4.systems/
Stewart, D., Liu, J.J., Cofer, D., Heimdahl, M., Whalen, M.W., Peterson, M.: AADL-based safety analysis using formal methods applied to aircraft digital systems. Reliab. Eng. Syst. Saf. 213, 107649 (2021)
Ward, D.T., Helton, S.B.: Estimating return on investment for SAVI (a model-based virtual integration process. SAE International J. Aerosp. (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hatcliff, J., Belt, J., Robby, Legg, J., Stewart, D., Carpenter, T. (2023). Automated Property-Based Testing from AADL Component Contracts. In: Cimatti, A., Titolo, L. (eds) Formal Methods for Industrial Critical Systems. FMICS 2023. Lecture Notes in Computer Science, vol 14290. Springer, Cham. https://doi.org/10.1007/978-3-031-43681-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-43681-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43680-2
Online ISBN: 978-3-031-43681-9
eBook Packages: Computer ScienceComputer Science (R0)