Improving an Industrial Test Generation Tool Using SMT Solver
 1.9k Downloads
Abstract
We present an SMT solving based test generation approach for MATLAB Simulink designs, implemented in the HiLiTE tool developed by Honeywell for verification of avionic systems. The test requirements for a Simulink model are represented by a set of behavioral equivalence classes for each block in the model, in terms of its input(s) and output. A unique feature of our approach is that the equivalence class definitions, as well as the upstream subgraph of a block under test, are translated as constraints into SMT expressions. An SMT solver is called at the backend of HiLiTE to find a satisfiable solution that is further augmented into an endtoend test case at the model level.
Keywords
Equivalence Class Test Case Generation Product Block Switch Block Automate Test Case Generation1 Introduction
As the industry practices engage modelbased design increasingly, modelbased verification and testing [1] techniques emerge to keep up with the trends. In avionics area, comprehensive testing methods and tools are required to assure that safetycritical systems like flight controls are certified to the guidelines established by standard processes such as the DO178C [2].
At Honeywell, researchers have developed the Honeywell Integrated Lifecycle Tools & Environment (HiLiTE) suite of tools for the automated verification of avionics applications developed using MATLAB Simulink/Stateflow. HiLiTE performs automatic test generation [3] on Simulink models based upon the lowlevel requirements (LLRs) expressed by the model elements. The tests are then applied to the executable object code generated from the model to verify that the code complies with the LLRs in the design model. HiLiTE has been qualified as a DO178C verification tool and deployed in several avionics product certifications to deliver significant cost savings in the verification effort.
This paper presents an SMT solving technique to extend the earlier heuristicsbased test case generation approaches implemented in HiLiTE, providing improved performance on models with complex constraints or nonlinear arithmetic computations. SMT solving is the decision procedure of determining whether a formula in firstorderlogic is satisfiable and finding a concrete solution if it is. SMT Solvers, such as Z3 [4], Yices [5], etc., have rapidly matured over the last 5 years and have been used in various areas including automated test case generation [6, 7]. In our SMT solving based approach, each LLR equivalence class for a block type is represented by a set of constraints, applied on blocklevel input(s) and expected output. Meanwhile, test space is also constrained by the subgraph environment that the block under test (BUT) is embedded in. The collection of constraints can be formulated as an SMT problem and expressed in a standard format by HiLiTE in an automatic fashion. SMT solver is then called to generate the satisfiable solution once for all ports in the related subgraph. The solution is merged back to the entire graph for a complete modellevel inputtooutput test case. With the integration of heuristics and SMT solving techniques, HiLiTE has been successfully used to generate requirementbased test cases for a great range of largescale complex constrained avionics models.
Section 2 describes the HiLiTE normal test case generation approach and the need for improvements. Section 3 describes the formalized language of equivalence classes of block’s behaviors and SMT solving based test case generation approach. Finally, the conclusion and future work are discussed in Sect. 4.
2 HiLiTE Test Generation Approach
When the switch block is the BUT, the equivalence class requires different values at its data inputs (FalseIn, TrueIn) to verify unique impact of an input on the block’s output. One test case template assigns 44 to FalseIn and 46 to TrueIn, but this leads to a conflict at the model input AdjustPct after backward computation through the two lookup tables since their data points are in the same range. HiLiTE then further tries several alternative templates based on heuristics, yet all result in search failure. The root cause is that HiLiTE templates heuristics in the equivalence class domain prematurely pick block’s local input values, while this problem involves taking into account constraints imposed by the lookup table blocks driven by the same input AdjustPct.
3 Applying SMT Solving in Test Case Generation
Test generation difficulties such as those noted above can be addressed by an approach that solves computational constraints of the upstream subgraph of BUT in conjunction with the constraints on BUT inputs imposed by the behavior equivalence class. SMT can be thought of the constraint satisfaction problem expressed in Boolean formulas, linear/nonlinear arithmetic in integer/real domain, bitvectors and so on. In HiLiTE, we added SMT solving based approach that embodies formulating test case generation constraints from both equivalence classes, constraints related to upstream source ports and the subgraph computations upstream of the BUT into an SMT problem. Therefore, constraints can be solved together to find a satisfying solution which excludes any conflicts.
3.1 Formal Specification of Equivalence Classes of Block Behaviors
Equivalence class definitions for switch block.
3.2 SMT Logic Formula for the Blocks’ Computation

Sum: \(\bigwedge _{j=0}^{m1}(\texttt {Out}_j=\Sigma _{i=1}^{n}{} \texttt {In\_i}_j)\).

Comparator: \(\bigwedge _{j=0}^{m1}(\texttt {Out}_j=\texttt {In\_1}_j \sim \texttt {In\_2}_j)\), \(\sim \in \{=,\ne ,>,<,\ge ,\le \}\).

Switch: \(\bigwedge _{j=0}^{m1}(((!\texttt {In\_3}_j)\wedge (\texttt {Out}_j=\texttt {In\_1}_j))\vee (\texttt {In\_3}_j\wedge (\texttt {Out}_j=\texttt {In\_2}_j)))\).

1D Lookup Table: \(\bigwedge _{j=0}^{m1}((\bigvee _{i=1}^n ((\texttt {In}_j\in Range_i)\wedge (\texttt {Out}_j=f_i(\texttt {In}_j))))\), where \(f_i\) is a linear function of \(\texttt {In}_j\) given the value of \(\texttt {In}_j\) in \(Range_i\).

UnitDelay: \((\texttt {Out}_0={initial\_constant})\wedge \bigwedge _{j=1}^{m1}(\texttt {Out}_j=\texttt {In}_{j1})\).
Note: support for timedependent blocks (e.g., UnitDelay) also allows us to explore feedback loops in the model for bounded number of steps.
3.3 Formulated SMT Problem
3.4 Tool Architecture
3.5 Nonlinear Applications
4 Conclusion and Future Work
We extended the HiLiTE test generation capability with an SMT solving based approach for solving certain complex constrained problems. The improved tool combines HiLiTE normal search method and SMT solving, and has been successfully applied on many largescale industrial models. HiLiTE is also being extended to derive invariant bounds on the number of time steps (e.g., for a timer) that will help bound the array size. We are also applying SMT solving to support such invariant generation, in which each conditionguarded path that captures a certain pattern of model behavior can be validated by SMT solving.
References
 1.Bhatt, D., Madl, G., Oglesby, D.: System Architecture Driven Software Design Analysis Methodology and Toolset. In: SAE International (2012)Google Scholar
 2.RTCA DO178C, Software Considerations in Airborne Systems and Equipment Certification, RTCA Inc. (2011)Google Scholar
 3.Bhatt, D., Madl, G., Oglesby, D., Schloegel, K.: Towards scalable verification of commercial avionics software. In: Proceedings of the AIAA Infotech @ Aerospace Conference, April 2010Google Scholar
 4.Z3Prover. https://github.com/Z3Prover/z3/wiki/
 5.The Yices SMT Solver. http://yices.csl.sri.com/
 6.Beyer, D., Chlipala, A.J., Henzinger, T.A., Jhala, R., Majumdar, R.: Generating Tests from Counterexamples. In: ICSE (2004)Google Scholar
 7.Peleska, J., Vorobev, E., Lapschies, F.: Automated test case generation with SMTsolving and abstract interpretation. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 298–312. Springer, Heidelberg (2011)CrossRefGoogle Scholar
Copyright information
Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, a link is provided to the Creative Commons license and any changes made are indicated.
The images or other third party material in this chapter are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.