Skip to main content
SpringerLink
Log in

Automating requirements analysis and test case generation

  • RE 2018
  • Published:
Requirements Engineering Aims and scope Submit manuscript

Abstract

Writing clear and unambiguous requirements that are conflict-free and complete is no easy task. Incorrect requirements lead to errors being introduced early in the design process. The longer the gap between error introduction and error discovery, the higher the cost associated with the error. To address the growing cost of system development, we introduce a tool called Analysis of Semantic Specifications and Efficient generation of Requirements-based Tests (ASSERT™) for capturing requirements, backed by a formal requirements analysis engine. ASSERT also automatically generates a complete set of requirements-based test cases. The requirements are captured in a structured natural language that is both human- and machine-readable. Formal analysis of these requirements with an automated theorem prover identifies errors as soon as requirements are written. It also addresses the historical problem that analysis engines are hard to use and understand for someone without formal methods expertise and analysis results are often difficult for the end-user to understand and make actionable. ASSERT’s major contribution is to bring powerful requirements capture and analysis capability to the domain of the end-user. We provide explainable and automated formal analysis, something we found important for a tool’s adoptability in industry. Automating test case generation in ASSERT also provides clear and measurable productivity gains in system development.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Siu K, Moitra A, Durling M, Crapo A, Li M, Yu H, Herencia-zapana H, Castillo-Effen M, Sen S, McMillan C, Manolios P (2017) Flight critical software and systems development using ASSERT™. In: IEEE/AIAA 36th digital avionics systems conference (DASC), St. Petersburg, FL, USA

  2. Crapo A, Moitra A, McMillan C, Russell D (2017) Requirements capture and analysis in ASSERT™. In: IEEE 25th international requirements engineering conference (RE), Lisbon, Portugal

  3. Moitra A, Siu K, Crapo AW, Chamarti H, Durling M, Li M, Yu H, Manolios P, Meiners M (2018) Towards development of complete and conflict-free requirements. In: IEEE 26th international requirements engineering conference (RE), Banff, Canada

  4. McMillan C, Crapo A, Durling M, Li M, Moitra A, Manolios P, Stephens M, Russell D (2019) Increasing development assurance for system and software development with validation and verification using ASSERT™. In: SAE technical paper no. 2019-01-1370

  5. RTCA (2011) DO-178C software considerations in airborne systems and equipment certification 12/13/2011

  6. Owre S, Rushby J, Shankar N (1992) PVS: a prototype verification system. In: International conference on automated deduction, Springer, Berlin

  7. PVS: property verification system. http://www.csl.sri.com/projects/pvs. Accessed 3 May 2019

  8. Heimdahl MPK, Czerny BJ (1996) Using PVS to analyze hierarchical state-based requirements for completeness and consistency. In: IEEE high-assurance systems engineering workshop (Cat. No. 96TB100076)

  9. Owre S, Rushby J, Shankar N (1995) Analyzing tabular and state-transition requirements specifications in PVS. In: Proceedings of TACAS

  10. Rayadurgam S, Joshi A, Heimdahl M (2003) Using PVS to prove properties of systems modelled in a synchronous dataflow language. In: International conference on formal engineering methods

  11. Leveson N, Heimdahl M, Hildreth H, Reese J (1994) Requirements specification for process-control systems. In: IEEE transaction on software engineering

  12. Leveson N, Heimdahl M, Reese J (1999) Designing specification languages for process control systems: lessons learned and steps to the future. In: Nierstrasz O, Lemoine M (eds) Software engineering—ESEC/FSE’99. Lecture notes in computer science, vol 1687, Springer, Berlin

  13. Easterbrook S, Lutz R, Covington R, Kelly J, Ampo Y, Hamilton D (1998) Experiences using lightweight formal methods for requirements modeling. IEEE Trans Softw Eng 24(1):4–14

    Article  Google Scholar 

  14. Holtzmann GJ, Lieberman WS (1991) Design and validation of computer protocols. Prentice Hall, Englewood Cliffs

    Google Scholar 

  15. Berry D (1999) Formal methods: the very idea, some thought about why they work when they work. Electron Notes Theor Comput Sci 25:10–22

    Article  Google Scholar 

  16. Nuseibeh B, Easterbrook S (2000) Requirements engineering: a roadmap. In: Proceedings of the conference on the future of software engineering, ACM

  17. Mavin A, Wilkinson P, Harwood A, Novak M (2009) Easy approach to requirements syntax (EARS). In: 17th IEEE international requirements engineering conference, Atlanta, USA

  18. Mavin A, Wilkinson P, Gregory S, Uusitalo E (2016) Listens learned (8 lessons learned applying EARS). In: IEEE 24th international requirements engineering conference, Beijing, China

  19. Badger J, Throop D, Claunch C (2014) VARED verification and analysis of requirements and early designs. In: IEEE 22nd requirements engineering conference, Karlskrona, Sweden

  20. Gross K, Fifarek A, Hoffman J (2016) Incremental formal methods based design approach demonstrated on a coupled tank control system. In: IEEE 17th international symposium on high assurance systems engineering

  21. GitHub SpeAR. https://github.com/lgwagner/SpeAR. Accessed 3 May 2019

  22. Feiler PH, Gluch DP, Hudak JJ (2006) The architecture analysis and design language (AADL): an introduction (no. CMU/SEI-2006-TN-011). Carnegie-Mellon Univ Pittsburgh PA Software Engineering Inst

  23. Brat G, Bushnell D, Davies M, Giannakopoulou D, Howar F, Kahsai T (2015) Verifying the safety of a flight-critical system. In: International symposium on formal methods, Springer, Cham

  24. W3C: Semantic Web Standards. Web Ontology Language (OWL). https://www.w3.org/OWL/. Accessed 3 May 2019

  25. Crapo A, Moitra A (2019) Using OWL ontologies as a domain-specific language for capturing requirements for formal analysis and test case generation. In: International conference on semantic computing, Newport Beach

  26. Whalen M, Gacek A, Cofer D, Murugesan A, Heimdahl M, Rayadurgam S (2013) Your “what” is my “how”: iteration and hierarchy in system design. IEEE software, vol 30, no 2

  27. Some Famous Unit Conversion Errors. https://spacemath.gsfc.nasa.gov/weekly/6Page53.pdf. Accessed 3 May 2019

  28. Nuseibeh B (1997) Ariane 5: who dunnit? IEEE Softw 14(3):15–16

    Article  Google Scholar 

  29. Litt J, Simon D, Garg S, Guo TH, Mercer C, Millar R, Behbahani A, Bajwa A, Jensen DT (2004) A survey of intelligent control and health management technologies for aircraft propulsion systems. JACIC 1(12):543–563

    Article  Google Scholar 

  30. Semantic Application Design Language (SADL). http://sadl.sourceforge.net/index.html. Accessed 3 May 2019

  31. Crapo A, Moitra A (2013) Toward a unified English-like representation of semantic models, data, and graph patterns for subject matter experts. Int J Semant Comput 7(3):215–236

    Article  Google Scholar 

  32. Laprie JC, Arlat J, Beounes C, Kanoun K (1990) Definition and analysis of hardware- and software-fault-tolerant architectures. Computer 23(7):39–51

    Article  Google Scholar 

  33. IBM Rational DOORS. https://www.ibm.com/us-en/marketplace/rational-doors. Accessed 3 May 2019

  34. Manolios P (2017) Scalable methods for analyzing formalized requirements and localizing errors. Patent 9,639,450, 2 May 2017

  35. Chamarthi HR, Dillinger PC, Manolios P, Vroon D (2011) The ACL2 Sedan theorem proving system. In: TACAS, Springer

  36. Manolios P, Vroon D (2006) Termination analysis with calling context graphs. In: Computer aided verification (CAV). Lecture notes in computer science, Springer, vol 4144, pp 401–414

  37. Manolios P, Vroon D (2010) Interactive termination proofs using termination cores. In: Interactive theorem proving, lecture notes in computer science, Springer, vol 6172

  38. Chamarthi HR, Dillinger PC, Kaufmann M, Manolios P (2011) Integrating testing and interactive theorem proving. In: ACL2 2011, EPTCS 70, pp 4–19

  39. Chamarthi HR, Manolios P (2011) Automated specification analysis using an interactive theorem prover. In: FMCAD

  40. Chamarthi HR, Dillinger PC, Manolios P (2014) Data definitions in the ACL2 Sedan. In: ACL2

  41. ACL2 Tutorial. http://www.cs.utexas.edu/users/moore/acl2/v7-4/combined-manual/. Accessed 3 May 2019

  42. Moore J, Kaufmann M, Manolios P (2000) Computer-aided reasoning: an approach. Kluwer, Dordrecht

    Google Scholar 

  43. Li M (2014) Integrated automated test case generation for safety-critical software. Patent U.S. Patent Application 20160170864A1, filed December 2014

  44. Li M, Durling M, Siu K, Oliveira I, Yu H, De Conto A (2018) System and method for safety-critical software automated requirements-based test case generation. Patent 9,940,222, 10 Apr 2018

  45. De Conto A, Li M, Manolios P, Oliveira I (2016) System and method for equivalence class analysis-based automated requirements-based test case generation. Patent U.S. Patent Application 20170228309A1, filed February 2016

  46. Durrieu G, Laurent O, Seguin C, Wiels V (2004) Formal proof and test case generation for critical embedded systems using SCADE. In: Jacquart R (ed) Building the information society, Springer, Boston, MA, vol 156, pp 499–504

  47. Wiels V, Delmas R, Doose D, Garoche PL, Cazin J, Durrieu G (2012) Formal verification of critical aerospace software. In: Aerospace Lab

  48. Bochot T, Virelizier P, Waeselynck H, Wiels V (2009) Model checking flight control systems: the Airbus experience. In: International conference on software engineering

  49. ANSYS SCADE Suite. https://www.ansys.com/products/embedded-software/ansys-scade-suite. Accessed 3 May 2019

  50. Z3. https://github.com/Z3Prover/z3. Accessed 3 May 2019

  51. dReal. dreal.github.io/. Accessed 3 May 2019

  52. Federal Aviation Administration (2001) Rationale for accepting masking MC/DC in certification projects. Certification Authorities Software Team, Position Paper, CAST-6

Download references

Acknowledgements

The authors gratefully acknowledge Scott Stacey, Tony Lanzillotti, Jeff Gilton, Shreecharan Kanchanavally, Gary Quackenbush, Christin Rauche, and Michael Idelchik for their unwavering support throughout the course of this research.

Author information

Authors and Affiliations

  1. GE Global Research, Niskayuna, NY, USA

    Abha Moitra, Kit Siu, Andrew W. Crapo, Michael Durling & Meng Li

  2. Northeastern University, Boston, MA, USA

    Panagiotis Manolios

  3. GE Aviation Systems, Grand Rapids, MI, USA

    Michael Meiners & Craig McMillan

Authors
  1. Abha Moitra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kit Siu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew W. Crapo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Durling
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Meng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Panagiotis Manolios
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Michael Meiners
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Craig McMillan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abha Moitra.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: Domain model and requirements

Appendix: Domain model and requirements

The semantic domain model is as follows.

figure ac
figure ad

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moitra, A., Siu, K., Crapo, A.W. et al. Automating requirements analysis and test case generation. Requirements Eng 24, 341–364 (2019). https://doi.org/10.1007/s00766-019-00316-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00766-019-00316-x

Keywords

Search

Navigation