Skip to main content
SpringerLink
Log in

Generating model-based test cases from natural language requirements for space application software

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Natural Language (NL) deliverables suffer from ambiguity, poor understandability, incompleteness, and inconsistency. Howewer, NL is straightforward and stakeholders are familiar with it to produce their software requirements documents. This paper presents a methodology, SOLIMVA, which aims at model-based test case generation considering NL requirements deliverables. The methodology is supported by a tool that makes it possible to automatically translate NL requirements into Statechart models. Once the Statecharts are derived, another tool, GTSC, is used to generate the test cases. SOLIMVA uses combinatorial designs to identify scenarios for system and acceptance testing, and it requires that a test designer defines the application domain by means of a dictionary. Within the dictionary there is a Semantic Translation Model in which, among other features, a word sense disambiguation method helps in the translation process. Using as a case study a space application software product, we compared SOLIMVA with a previous manual approach developed by an expert under two aspects: test objectives coverage and characteristics of the Executable Test Cases. In the first aspect, the SOLIMVA methodology not only covered the test objectives associated to the expert’s scenarios but also proposed a better strategy with test objectives clearly separated according to the directives of combinatorial designs. The Executable Test Cases derived in accordance with the SOLIMVA methodology not only possessed similar characteristics with the expert’s Executable Test Cases but also predicted behaviors that did not exist in the expert’s strategy. The key benefits from applying the SOLIMVA methodology/tool within a Verification and Validation process are the ease of use and, at the same time, the support of a formal method consequently leading to a potential acceptance of the methodology in complex software projects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25

Similar content being viewed by others

Notes

  1. At present, SOLIMVA uses an open source tool, TConfig (University of Ottawa 2008), which has implemented the IPO procedure. The IPO procedure was originally conceived for 2-way (pairwise) testing. The TConfig tool can generate from 2-way to 6-way Mixed-Level Covering Arrays. However, it is not clear if the tool has implemented the In-Parameter-Order-General (IPOG) (Lei et al. 2007) algorithm which deals with general t-way testing or else if another approach was adopted.

  2. Lexical category or part of speech is a linguistic category of words (or more precisely lexical items), which is generally defined by the syntactic or morphological behavior of the lexical item in question. Common linguistic categories include noun and verb, among others.

  3. The Stanford POS Tagger adopts the Penn Treebank POS tagset (Marcus et al. 1993).

  4. In linguistics, one definition of lemma is: the canonical form, dictionary form, or citation form of a set of words (headword). For instance, in English, “run”, “runs”, and “ran” are forms of the same lexeme, with “run” as the lemma.

  5. A path is a finite sequence of adjacent transitions.

  6. POS tagging is also known as word category disambiguation.

  7. Traditionally, an FSM is seen as a computational model and not as a language (Hopcroft and Ullman, 1979).

  8. In hidden inconsistencies, the inconsistency occurs due to the consequences of some requirements rather than the requirements themselves.

  9. Actually, CARL does not really apply model checking according to its most common definition (Clarke and Lerda 2007; Baier and Katoen 2008).

References

  • Abrial, J. R. (2006). Formal methods in industry: Achievements, problems, future. In Proceedings of the 28th international conference on software engineering (ICSE) (pp. 761–768). New York, NY: ACM.

  • Ambriola V., & Gervasi V. (1997). Processing natural language requirements. In Proceedings of the 12th international conference on automated software engineering (ASE) (pp. 36–45). Washington, DC: IEEE Computer Society.

  • Ambriola, V., & Gervasi, V. (2006). On the systematic analysis of natural language requirements with CIRCE. Automated Software Engineering, 13(1), 107–167.

    Article  Google Scholar 

  • Ambrósio, A. M., Mattiello-Francisco, F., Santiago, V. A., Silva, W. P., & Martins, E. (2007). Designing fault injection experiments using state-based model to test a space software. In Dependable computing (LNCS, Vol. 4746, pp. 170–178). Berlin/Heidelberg, Germany: Springer.

  • Antoniol, G., Briand, L. C., Di Penta, M., & Labiche, Y. (2002). A case study using the round-trip strategy for state-based class testing. In Proceedings of the 13th IEEE international symposium on software reliability engineering (ISSRE) (pp. 269–279). Washington, DC: IEEE Computer Society.

  • Baier, C., & Katoen, J. P. (2008). Principles of model checking. Cambridge, MA: The MIT Press.

    MATH  Google Scholar 

  • Balcer, M., Hasling, W., & Ostrand, T. (1989). Automatic generation of test scripts from formal test specifications. ACM SIGSOFT Software Engeneering Notes, 14(8), 210–218.

    Article  Google Scholar 

  • Basili, V. R., Green, S., Laitenberger, O., Lanubile, F., Shull, F., Sørumgård, S., et al. (1996). The empirical investigation of perspective-based reading. Empirical Software Engineering Journal, 1(2), 133–164.

    Article  Google Scholar 

  • Bertolino, A., & Gnesi, S. (2003). Use case-based testing of product lines. ACM SIGSOFT Software Engineering Notes, 28(5), 355–358.

    Article  Google Scholar 

  • Binder, R. V. (1999). Testing object-oriented systems. USA: Addison-Wesley.

    Google Scholar 

  • Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., & Mylopoulos, J. (2004). Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems, 8(3), 203–236.

    Article  Google Scholar 

  • Briand, L. C., & Labiche, Y. (2002). A UML-based approach to system testing. Journal of Software and Systems Modeling, 1(1), 10–42.

    Article  Google Scholar 

  • Briand, L. C., Labiche. Y., & Wang, Y. (2004). Using simulation to empirically investigate test coverage criteria based on Statechart. In Proceedings of the 26th international conference on software engineering (ICSE) (pp. 86–95). Washington, DC: IEEE Computer Society.

  • Chow, T. S. (1978). Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering, SE-4(3), 178–187.

    Article  Google Scholar 

  • Clarke, E. M., & Lerda, F. (2007). Model checking: Software and beyond. Journal of Universal Computer Science, 13(5), 639–649.

    MathSciNet  Google Scholar 

  • Cristiá, M., & Monetti, P. (2009). Implementing and applying the Stocks–Carrington framework for model-based testing. In K. Breitman, & A. Cavalcanti (Eds.), Formal methods and software engineering (LNCS, Vol. 5885, pp. 167–185). Berlin/Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Cristiá, M., Santiago, V., & Vijaykumar, N. L. (2010). On comparing and complementing two MBT approaches. In Proceedings of the 11th Latin–American test workshop (LATW) (pp. 1–6). Washington, DC: IEEE Computer Society.

  • ECSS. (2008). ECSS-S-ST-00C: ECSS system—description, implementation and general requirements. Noordwijk, The Netherlands: European Cooperation for Space Standardization (ECSS).

    Google Scholar 

  • ECSS. (2009). ECSS-E-ST-10-06C: ECSS space engineering—technical requirements specification. Noordwijk, The Netherlands: European Cooperation for Space Standardization (ECSS).

    Google Scholar 

  • El-Far, I. K., & Whittaker, J. A. (2001). Model-based software testing. In J. J. Marciniak (Ed.), Encyclopedia of software engineering. USA: Wiley.

    Google Scholar 

  • Fantechi, A., Gnesi, S., Lami, G., & Maccari, A. (2003). Applications of linguistic techniques for use case analysis. Requirements Engineering, 8(3), 161–170.

    Article  Google Scholar 

  • Fantechi, A., & Spinicci, E. (2005). A content analysis technique for inconsistency detection in software requirements documents. In Proceedings of the VIII workshop on requirements engineering (WER) (pp. 245–256).

  • Fraser, M. D., Kumar, K., & Vaishnavi, V. K. (1991). Informal and formal requirements specification languages: Bridging the gap. IEEE Transactions on Software Engineering, 17(5), 454–466.

    Article  Google Scholar 

  • Frohlich, P., & Link, J. (2000). Automated test case generation from dynamic models. In ECOOP 2000: Object-oriented programming (LNCS, Vol. 1850, pp. 472–491). Berlin/Heidelberg, Germany: Springer.

  • Fuchs, N. E., Schwertel, U., & Schwitter, R. (1999). Attempto Controlled English—not just another logic specification language. In Logic-based program synthesis and transformation (LNCS, Vol. 1559, pp. 1–20). Berlin/Heidelberg, Germany: Springer.

  • Fuchs, N. E., Schwertel, U., & Torge, S. (2000). A natural language front-end to model generation. Journal of Language and Computation, 1(2), 199–214.

    Google Scholar 

  • Gervasi, V., & Zowghi, D. (2005). Reasoning about inconsistencies in natural language requirements. ACM Transactions on Software Engineering and Methodology, 14(3), 277–330.

    Article  Google Scholar 

  • Harel, D. (1987). Statecharts: A visual formalism for complex systems. Science of Computer Programming, 8, 231–274.

    Article  MATH  MathSciNet  Google Scholar 

  • Harel, D., Pnueli, A., Schmidt, J. P., & Sherman, R. (1987). On the formal semantics of Statecharts (extended abstract). In Proceedings of the 2nd IEEE symposium on logic in computer science (LICS) (pp. 54–64). Washington, DC: IEEE Computer Society.

  • Hierons, R. M. (1997). Testing from a Z specification. The Journal of Software Testing, Verification and Reliability, 7(1), 19–33.

    Article  Google Scholar 

  • Hierons, R. M., Bogdanov, K., Bowen, J. P., Cleaveland, R., Derrick, J., Dick, J., et al. (2009). Using formal specifications to support testing. ACM Computing Surveys, 41(2), 1–76.

    Article  Google Scholar 

  • Hoare, C. A. R. (1985). Communicating sequential processes. Englewood Cliffs, NJ: Prentice-Hall.

    MATH  Google Scholar 

  • Hopcroft, J. E., & Ullman, J. D. (1979). Introduction to automata theory, languages, and computation. Reading, MA: Addison Wesley.

    MATH  Google Scholar 

  • Howden, W. E. (1976). Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering, SE-2(3), 208–215.

    Article  MathSciNet  Google Scholar 

  • IEEE. (1990). IEEE Std 610.12-1990: IEEE standard glossary of software engineering terminology. New York, NY: The Institute of Electrical and Electronics Engineers (IEEE).

    Google Scholar 

  • Jiang, J. J., & Conrath, D. W. (1997). Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th international conference research on computational linguistics (ROCLING) (pp. 19–33).

  • Kim, H. Y., & Sheldon, F. T. (2004). Testing software requirements with Z and Statecharts applied to an embedded control system. Software Quality Journal, 12(3), 231–264.

    Article  Google Scholar 

  • Leacock, C., & Chodorow, M. (1998). Combining local context and WordNet similarity for word sense identification. In C. Fellbaum (Ed.), WordNet: An electronic lexical database, chap 11 (pp. 265–283). Cambridge, MA: The MIT Press.

    Google Scholar 

  • Lee, D., & Yannakakis, M. (1996). Principles and methods of testing finite state machines: A survey. Proceedings of the IEEE, 84(8), 1090–1123.

  • Lei, Y., Kacker, R., Kuhn, D. R., Okun, V., & Lawrence, J. (2007). IPOG: A general strategy for t-way software testing. In Proceedings of the 14th annual IEEE international conference and workshops on the engineering of computer-based systems (ECBS) (pp. 549–556). Washington, DC: IEEE Computer Society.

  • Lei, Y., & Tai, K. C. (1998). In-parameter-order: A test generation strategy for pairwise testing. In Proceedings of the 3rd IEEE international symposium on high-assurance systems engineering (HASE) (pp. 254–261). Washington, DC: IEEE Computer Society.

  • Lesk, M. (1986). Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the 5th international conference on systems documentation (SIGDOC) (pp. 24–26). New York, NY: ACM.

  • Liang, J., & Palmer, J. D. (1994). A pattern matching and clustering based approach for supporting requirements transformation. In Proceedings of the 1st IEEE international conference on requirements engineering (ICRE) (pp. 180–183). Washington, DC: IEEE Computer Society.

  • Lu, C. W., Chang, C. H., Chu, W. C., Cheng, Y. W., & Chang, H.C. (2008). A requirement tool to support model-based requirement engineering. In Proceedings of the 32nd annual IEEE international computer software and applications conference (COMPSAC) (pp. 712–717). Washington, DC: IEEE Computer Society.

  • Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330.

    Google Scholar 

  • Masiero, P. C., Maldonado, J. C., & Boaventura, I. G. (1994). A reachability tree for statecharts and analysis of some properties. Information and Software Technology, 36(10), 615–624.

    Article  Google Scholar 

  • Mathur, A. P. (2008). Foundations of software testing. Delhi, India: Dorling Kindersley (India) Pearson Education in South Asia.

    Google Scholar 

  • Mich, L. (1996). NL-OOPS: From natural language to object oriented requirements using the natural language processing system LOLITA. Natural Language Engineering, 2(2), 161–187.

    Article  Google Scholar 

  • Mich, L., Franch, M., & Inverardi, P. (2004). Market research for requirements analysis using linguistic tools. Requirements Engineering Journal, 9(1), 40–56.

    Article  Google Scholar 

  • Miller, G. A. (1998). Nouns in WordNet. In C. Fellbaum (Ed.), WordNet: An electronic lexical database, chap 1 (pp. 23–46). Cambridge, MA: The MIT Press.

    Google Scholar 

  • Miller, G. A., Leacock, C., Tengi, R., & Bunker, R. T. (1993). A semantic concordance. In Proceedings of the workshop on human language technology (HLT) (pp. 303–308). Morristown, NJ: Association for Computational Linguistics.

  • Morgan, R., Garigliano, R., Callaghan, P., Poria, S., Smith, M., Urbanowicz, A., et al. (1995). University of Durham: Description of the LOLITA system as used in MUC-6. In Proceedings of the 6th message understanding conference (MUC-6) (pp. 71–85).

  • Navigli, R. (2009). Word sense disambiguation: A survey. ACM Computing Surveys, 41(2), 1–69.

    Article  Google Scholar 

  • Offutt, J., & Abdurazik, A. (1999). Generating tests from UML specifications. In UML’99: The unified modeling language (LNCS, Vol. 1723, pp 416–429). Berlin/Heidelberg, Germany: Springer.

  • OMG. (2007). OMG unified modeling language (OMG UML), Superstructure, V2.1.2. Needham, MA: The Object Management Group (OMG).

    Google Scholar 

  • Ostrand, T. J., & Balcer, M. J. (1988). The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6), 676–686.

    Article  Google Scholar 

  • Paradkar, A. (2003). Towards model-based generation of self-priming and self-checking conformance tests for interactive systems. In Proceedings of the 18th ACM symposium on applied computing (SAC) (pp. 1110–1117). New York, NY: ACM.

  • Pedersen, T., Patwardhan, S., & Michelizzi, J. (2004). WordNet::Similarity: Measuring the relatedness of concepts. In Proceedings of the 5th annual conference of the North American chapter of the association for computational linguistics (NAACL) (pp. 38–41). Morristown, NJ: Association for Computational Linguistics.

  • Petrenko, A., & Yevtushenko, N. (2005). Testing from partial deterministic FSM specifications. IEEE Transactions on Computers, 54(9), 1154–1165.

    Article  Google Scholar 

  • Pimont, S., & Rault, J. C. (1976). A software reliability assessment based on a structural and behavioral analysis of programs. In Proceedings of the 2nd international conference on software engineering (ICSE) (pp. 486–491). New York, NY: ACM.

  • Pressman, R. S. (2001). Software Engineering: A practitioner’s approach (5th ed.). New York, NY: McGraw-Hill.

    Google Scholar 

  • Santiago, V., Amaral, A. S. M., Vijaykumar, N. L., Mattiello-Francisco, M. F., Martins, E., & Lopes, O. C. (2006). A practical approach for automated test case generation using statecharts. In Proceedings of the 30th annual international computer software and applications conference (COMPSAC)—international workshop on testing and quality assurance for component-based systems (TQACBS) (pp. 183–188). Los Alamitos, CA: IEEE Computer Society.

  • Santiago, V., Mattiello-Francisco, F., Costa, R., Silva, W. P., & Ambrosio, A. M. (2007). QSEE project: An experience in outsourcing software development for space applications. In Proceedings of the 19th international conference on software engineering and knowledge engineering (SEKE) (pp. 51–56). Skokie, IL: Knowledge Systems Institute Graduate School.

  • Santiago, V., Silva, W. P., & Vijaykumar, N. L. (2008a). Shortening test case execution time for embedded software. In Proceedings of the 2nd IEEE international conference on secure system integration and reliability improvement (SSIRI) (pp.81–88). Washington, DC: IEEE Computer Society.

  • Santiago, V., Vijaykumar, N. L., Guimaraes, D., Amaral, A. S., & Ferreira, E. (2008b). An environment for automated test case generation from statechart-based and finite state machine-based behavioral models. In Proceedings of the 1st international conference on software testing, verification and validation (ICST)—workshop on advances in model based testing (A-MOST) (pp. 63–72). Washington, DC: IEEE Computer Society.

  • Santiago Júnior, V. A., Cristiá, M., & Vijaykumar, N. L. (2010). Model-based test case generation using Statecharts and Z: A comparison and a combined approach. INPE, São José dos Campos, http://www.urlib.net/sid.inpe.br/mtc-m19@80/2010/02.26.14.05, (INPE-16677-RPQ/850)

  • Sarma, M., & Mall, R. (2009). Automatic generation of test specifications for coverage of system state transitions. Information and Software Technology, 51(2), 418–432.

    Article  Google Scholar 

  • Sidhu, D. P., & Leung, T. K. (1989). Formal methods for protocol testing: A detailed study. IEEE Transactions on Software Engineering, 15(4), 413–426.

    Article  Google Scholar 

  • Sinha, R., & Mihalcea, R. (2007). Unsupervised graph-based word sense disambiguation using measures of word semantic similarity. In Proceedings of the international conference on semantic computing (ICSC) (pp. 363–369). Washington, DC: IEEE Computer Society.

  • Sleator, D. D., & Temperley, D. (1993). Parsing English with a link grammar. In Proceedings of the 3rd international workshop on parsing technologies (pp. 277–292).

  • Sneed, H. M. (2007). Testing against natural language requirements. In Proceedings of the 7th international conference on quality software (QSIC) (pp. 380–387). Washington, DC: IEEE Computer Society.

  • Souza, É. F. (2010). Geração de casos de teste para sistemas da área espacial usando critérios de teste para máquinas de estados finitos. Master Dissertation (133 p.). Instituto Nacional de Pesquisas Espaciais, São José dos Campos, SP.

  • Souza, S. R. S. (2000). Validação de especificações de sistemas reativos: Definição e análise de critérios de teste. PhD Thesis (264 p.). Universidade de São Paulo, São Carlos, SP.

  • Spivey, J. M. (1989). The Z notation: a reference manual. Upper Saddle River, NJ: Prentice-Hall

    MATH  Google Scholar 

  • Toutanova, K., Klein, D., Manning, C. D., & Singer, Y. (2003). Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the conference of the North American chapter of the association for computational linguistics on human language technology (pp. 173–180).

  • University of Ottawa. (2008). Alan Williams’ page. Available from: http://www.site.uottawa.ca/~awilliam/. Accessed in July 22, 2010.

  • University of Sussex. (2010). David Hope’s page. Available from: http://www.cogs.susx.ac.uk/users/drh21/. Accessed in July 22, 2010.

  • Vijaykumar, N. L., Carvalho, S. V., Francês, C. R. L., Abdurahiman, V., & Amaral, A. S. M. (2006). Performance evaluation from statecharts representation of complex systems: Markov approach. In Anais do XXVI Congresso da Sociedade Brasileira de Computação (CSBC)—Workshop em Desempenho de Sistemas Computacionais e de Comunicação (pp. 183–202). Porto Alegre, RS: Sociedade Brasileira de Computação.

Download references

Acknowledgments

We would like to thank and dedicate this research to Prof. José Demísio Simões da Silva (in memoriam) for his relevant contributions to the better development of our research.

Author information

Authors and Affiliations

  1. Instituto Nacional de Pesquisas Espaciais (INPE), Av. dos Astronautas, 1758, 12227-010, São José dos Campos, SP, Brazil

    Valdivino Alexandre de Santiago Júnior & Nandamudi Lankalapalli Vijaykumar

Authors
  1. Valdivino Alexandre de Santiago Júnior
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nandamudi Lankalapalli Vijaykumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valdivino Alexandre de Santiago Júnior.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Santiago Júnior, V.A., Vijaykumar, N.L. Generating model-based test cases from natural language requirements for space application software. Software Qual J 20, 77–143 (2012). https://doi.org/10.1007/s11219-011-9155-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-011-9155-6

Keywords

Search

Navigation