Software & Systems Modeling

, Volume 14, Issue 1, pp 483–524 | Cite as

Environment modeling and simulation for automated testing of soft real-time embedded software

  • Muhammad Zohaib Iqbal
  • Andrea Arcuri
  • Lionel Briand
Regular Paper


Given the challenges of testing at the system level, only a fully automated approach can really scale up to industrial real-time embedded systems (RTES). Our goal is to provide a practical approach to the model-based testing of RTES by allowing system testers, who are often not familiar with the system’s design but are application domain experts, to model the system environment in such a way as to enable its black-box test automation. Environment models can support the automation of three tasks: the code generation of an environment simulator to enable testing on the development platform or without involving actual hardware, the selection of test cases, and the evaluation of their expected results (oracles). From a practical standpoint—and such considerations are crucial for industrial adoption—environment modeling should be based on modeling standards (1) that are at an adequate level of abstraction, (2) that software engineers are familiar with, and (3) that are well supported by commercial or open source tools. In this paper, we propose a precise environment modeling methodology fitting these requirements and discuss how these models can be used to generate environment simulators. The environment models are expressed using UML/MARTE and OCL, which are international standards for real-time systems and constraint modeling. The presented techniques are evaluated on a set of three artificial problems and on two industrial RTES.


Environment modeling Environment simulation Automated testing Model-based testing  Real-time embedded systems Search based software engineering 



The work presented in this paper was supported by Norwegian Research Council and was produced as part of the ITEA 2 VERDE project. Lionel Briand was partly supported by a PEARL grant from the National Research Fund, Luxembourg (FNR/P10/03 - Validation and Verification Laboratory). We are thankful to Christine Husa, Tor Sjøwall, John Roger Johansen, Erling Marhussen, Dag Kristensen, and Anders Emil Olsen, all from Tomra and Bjørn Nordmoen, Petter Eide, Tore Andre Nilsen, and Sofia Wegger from WesternGeco for their technical support.


  1. 1.
    Iqbal, M.Z., Arcuri, A., Briand, L.: Environment modeling with UML/MARTE to support Black-Box system testing for real-time embedded systems: methodology and industrial case studies. In: Model Driven Engineering Languages and Systems, pp. 286–300. Springer, Berlin (2010)Google Scholar
  2. 2.
    Artemis: (2011) Artemis Joint Undertaking—The public private partnership for R & D Embedded Systems (June 13, 2011).
  3. 3.
    Broekman, B.M., Notenboom, E.: Testing Embedded Software. Addison-Wesley, Boston (2003)Google Scholar
  4. 4.
    OMG: Unified Modeling Language Superstructure, Version 2.3 (2010).
  5. 5.
    OMG: Modeling and Analysis of Real-time and Embedded systems (MARTE), Version 1.0 (2009).
  6. 6.
    OMG: Object Constraint Language Specification, Version 2.2. Object Management Group Inc. (2010).
  7. 7.
    Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design patterns: elements of reusable object-oriented software (1995)Google Scholar
  8. 8.
  9. 9.
    Wainer, G.A.: Discrete-Event Modeling and Simulation: A Practitioner’s Approach: CRC, Boca Raton (2009)Google Scholar
  10. 10.
    Fritzson, P., Engelson, V.: Modelica–a unified object-oriented language for system modeling and simulation. In: ECOOP’98–Object-Oriented Programming, p. 67. Springer, Berlin (1998)Google Scholar
  11. 11.
    Schamai, W., Fritzson, P., Paredis, C., Pop, A.: Towards unified system modeling and simulation with ModelicaML: modeling of executable behavior using graphical notations. Presented at the 7th International Modelica Conference. Como, Italy (2009)Google Scholar
  12. 12.
    Mooney, J., Sarjoughian, H.: A framework for executable UML models. Presented at the Proceedings of the 2009 Spring Simulation Multiconference. San Diego, California (2009)Google Scholar
  13. 13.
    Kruse, P.M., Wegener, J., Wappler, S.: A highly configurable test system for evolutionary black-box testing of embedded systems. Presented at the Proceedings of the 11th Annual conference on Genetic and evolutionary computation. Montreal, Canada (2009)Google Scholar
  14. 14.
    Lindlar, F., Windisch, A., Wegener, J.: Integrating Model-Based Testing with Evolutionary Functional Testing. Presented at the Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops (2010)Google Scholar
  15. 15.
    Lindlar, F., Windisch, A.: A search-based approach to functional hardware-in-the-loop testing. Presented at the Proceedings of the 2nd International Symposium on Search Based, Software Engineering (2010)Google Scholar
  16. 16.
    Short, M., Pont, M.J.: Assessment of high-integrity embedded automotive control systems using hardware in the loop simulation. J. Syst. Softw. 81, 1163–1183 (2008)CrossRefGoogle Scholar
  17. 17.
    Francis, G., Burgos, R., Rodriguez, P., Wang, F., Boroyevich, D., Liu, R. Monti, A.: Virtual Prototyping of Universal Control Architecture Systems by means of Processor in the Loop Technology. Presented at the Twenty Second Annual IEEE Applied Power Electronics Conference, APEC 2007 (2007)Google Scholar
  18. 18.
  19. 19.
  20. 20.
  21. 21.
    Kishi, T., Noda, N.: Aspect-oriented context modeling for embedded systems. Aspect-Oriented Requirements Engineering and Architecture Design, Presented at the Workshop on Early Aspects (2004)Google Scholar
  22. 22.
    Karsai, G., Neema, S., Sharp, D.: Model-driven architecture for embedded software: a synopsis and an example. Sci. Comput. Progr. 73, 26–38 (2008)CrossRefMATHMathSciNetGoogle Scholar
  23. 23.
    Choi, K.S., Jung, S.C., Kim, H.J., Bae, D.H., Lee, D.H.: UML-based Modeling and Simulation Method for Mission-Critical Real-Time Embedded System Development. Presented at the IASTED International Conference Proceedings (2006)Google Scholar
  24. 24.
    Kreiner, C., Steger, C., Weiss, R.: Improvement of control software for automatic logistic systems using executable environment models. Presented at the EUROMICRO ’98: Proceedings of the 24th Conference on EUROMICRO, 1998Google Scholar
  25. 25.
    Burmeister, C.: Real-time environment modeling. In: IEEE (ed.) IEEE Workshop on Real-Time Applications, pp. 142–146. New York (1993)Google Scholar
  26. 26.
    Ubayashi, N., Seto, T., Kanagawa, H., Taniguchi, S., Yoshida, J., Sumi, T., Hirayama, M.: A context analysis method for constructing reliable embedded systems. In: Proceedings of the International Workshop on Models in Software Engineering, Leipzig, pp. 57–62 (2008)Google Scholar
  27. 27.
    Pettit IV, R.G., Street, J.A.: Lessons learned applying UML in the design of mission critical software. In: UML: satellite activities. Lecture Notes in Computer Science. Springer, Berlin, pp. 129–137 (2004)Google Scholar
  28. 28.
    Axelsson, J.: Unified modeling of real-time control systems and their physical environments using UML. Presented at the Eighth Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ECBS ’01) (2001)Google Scholar
  29. 29.
    Gomaa, H.: Designing Concurrent. Distributed and Real-Time Applications with UML. Addison-Wesley Educational Publishers Inc, Boston (2000)Google Scholar
  30. 30.
    Friedenthal, S., Moore, A., Steiner, R.: A Practical Guide to SysML: The Systems Modeling Language. Elsevier, Amsterdam (2008)Google Scholar
  31. 31.
    Auguston, M., Michael, J.B., Shing, M.: Environment behavior models for automation of testing and assessment of system safety. Inf. Softw. Technol. 48, 971–980 (2006)CrossRefGoogle Scholar
  32. 32.
    Heisel, M., Hatebur, D., Santen, T., Seifert, D.: Testing against requirements using UML environment models. In: Fachgruppentreffen Requirements Engineering und Test, Analyse & Verifikation, pp. 28–31 (2008)Google Scholar
  33. 33.
    Adjir, N., Saqui-Sannes, P., Rahmouni, K.M.: Testing real-time systems using TINA. In: Testing of software and communication systems. Lecture Notes in Computer Science. Springer, Berlin (2009)Google Scholar
  34. 34.
    Larsen, K.G., Mikucionis, M., Nielsen, B.: Online testing of real-time systems using Uppaal. In: Formal Approaches to Software Testing. Lecture Notes in Computer Science, Springer, Berlin (2005)Google Scholar
  35. 35.
    Hessel, A., Larsen, K., Mikucionis, M., Nielsen, B., Pettersson, P., Skou, A.: Testing real-time systems using uppaal. In: Formal Methods and Testing, pp. 77–117 (2008)Google Scholar
  36. 36.
    Larsen, K.G., Mikucionis, M., Nielsen, B., Skou, A.: Testing real-time embedded software using UPPAAL-TRON: an industrial case study. In: Proceedings of the 5th ACM International Conference on Embedded Software, pp. 299–306 (2005)Google Scholar
  37. 37.
    Krichen, M., Tripakis, S.: Conformance testing for real-time systems. Formal Methods Syst. Design 34, 238–304 (2009)CrossRefMATHGoogle Scholar
  38. 38.
    Du Bousquet, L., Ouabdesselam, F., Richier, J.L., Zuanon, N.: Lutess: a specification-driven testing environment for synchronous software. Presented at the ICSE ’99: Proceedings of the 21st International Conference on Software Engineering, Los Angeles (1999)Google Scholar
  39. 39.
    Peleska, J., Lapschies, F., Vorobev, E., Loeding, H., Smuda, P., Schmid, H., Zahlten, C.: A real-world benchmark model for testing concurrent real-time systems in the automotive domain. In: Testing Software and Systems. Springer, Berlin, pp. 146–161 (2011)Google Scholar
  40. 40.
    Peleska, J., Vorobev, E., Lapschies, F., Zahlten, C.: Automated Model-Based Testing with RT-Tester, University of Bremen (2011)Google Scholar
  41. 41.
    David, A., Larsen, K.G., Li, S., Nielsen, B.: Timed testing under partial observability. In: International Conference on Software Testing Verification and Validation, ICST’09, pp. 61–70 (2009)Google Scholar
  42. 42.
  43. 43.
    Iqbal, M.Z., Arcuri, A., Briand, L.: Empirical investigation of search algorithms for environment model-based testing of real-time embedded software. In: International Symposium on Software Testing and Analysis (ISSTA) (2012)Google Scholar
  44. 44.
    Arcuri, A., Iqbal, M., Briand, L.: Black-Box system testing of real-time embedded systems using random and search-based testing. In: Testing Software and Systems, pp. 95–110. Springer, Berlin (2010)Google Scholar
  45. 45.
    Iqbal, M.Z., Arcuri, A., Briand, L.: Combining search-based and adaptive random testing strategies for environment model-based testing of real-time embedded systems. In: Symposium on Search-based, Software Engineering (2012)Google Scholar
  46. 46.
    Pilitowski, R., Dereziñska, A.: Code generation and execution framework for UML 2.0 classes and state machines. In: Innovations and Advanced Techniques in Computer and Information Sciences and Engineering, pp. 421–427. Springer, Netherlands (2007)Google Scholar
  47. 47.
    Chauvel, F., Jézéquel, J.-M.: Code generation from UML models with semantic variation points. In: Model Driven Engineering Languages and Systems, pp. 54–68. Springer, Berlin (2005)Google Scholar
  48. 48.
    SmartState: SmartState–UML statemachine code generation tool (2011).
  49. 49.
    IBM: IBM Rational Rhapsody (2011).
  50. 50.
    Samek, M.: Practical UML statecharts in C/C++: event-driven programming for embedded systems: Newnes (2009)Google Scholar
  51. 51.
    Ferreira, L., Rubira, C.: The reflective state pattern. Presented at the Proceedings of the Pattern Languages of Program Design, Monticello, IL, USA (1998)Google Scholar
  52. 52.
    Chin, B., Millstein, T.: An extensible state machine pattern for interactive applications. In: ECOOP–Object-Oriented Programming, pp. 566–591. Springer, Berlin (2008)Google Scholar
  53. 53.
    Holt, N., Anda, B., Asskildt, K., Briand, L., Endresen, J., Frøystein, S.: Experiences with precise state modeling in an industrial safety critical system. Presented at the Critical Systems Development Using Modeling Lanuguages, CSDUML’06 (2006)Google Scholar
  54. 54.
    Palfinger, G.: State Action Mapper. Presented at the 4th Pattern Languages of Programming. PLoP), USA (1997)Google Scholar
  55. 55.
    Niaz, I.A., Tanaka, J.: An object-oriented approach to generate Java code from UML Statecharts. Int. J. Comput. Inf. Sci. 6, 83–98 (2005)Google Scholar
  56. 56.
    Quadri, I.R., Meftali, S., Dekeyser, J.L.: Designing dynamically reconfigurable SoCs: From UML MARTE models to automatic code generation. In: Conference on design and architectures for signal and image processing (DASIP), pp. 68–75 (2010)Google Scholar
  57. 57.
    Rodrigues, W., Guyomarc’h, F., Dekeyser, J.L.: An MDE approach for automatic code generation from UML/MARTE to OpenCL. Comput. Sci. Eng. (2012)Google Scholar
  58. 58.
    Piel É., Atitallah R.B., Marquet P., Meftali S., Niar S., Etien A., Dekeyser J.L., Boulet P., Europe I.L.N.: Gaspard2: from MARTE to SystemC simulation. In: Modeling and Analysis of Real-Time and Embedded Systems with the MARTE UML profile DATE, vol. 8, p. 65 (2008)Google Scholar
  59. 59.
    Peñil, P., Medina, J., Posadas, H., Villar, E.: Generating heterogeneous executable specifications in SystemC from UML/MARTE models. Innov. Syst. Softw. Eng. 6, 65–71 (2010)CrossRefGoogle Scholar
  60. 60.
    Vidal, J., De Lamotte, F., Gogniat, G., Soulard, P., Diguet, J. P.: A co-design approach for embedded system modeling and code generation with UML and MARTE. In: Design, Automation and Test in Europe Conference and Exhibition (DATE ’09), pp. 226–231 (2009)Google Scholar
  61. 61.
    Mraidha, C., Tanguy, Y., Jouvray, C., Terrier, F., Gérard, S.: An execution framework for MARTE-based models. In: 13th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS), pp. 222–227 (2008)Google Scholar
  62. 62.
    Yu, H., Talpin, J.P., Besnard, L., Gautier, T., Marchand, H., Le Guernic, P.: Polychronous controller synthesis from MARTE CCSL timing specifications. In: 9th IEEE/ACM International Conference on Formal Methods and Models for Codesign (MEMOCODE), pp. 21–30 (2011)Google Scholar
  63. 63.
    Selic, B.: A systematic approach to domain-specific language design using UML. In: 10th IEEE International Symposium on Object and Component-Oriented Real-Time, Distributed Computing, 2007. ISORC’07, pp. 2–9 (2007)Google Scholar
  64. 64.
    Larman, C.: Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and the Unified Process. Prentice Hall PTR, Upper Saddle River (2001)Google Scholar
  65. 65.
    OMG: Concrete Syntax for UML Action Language (Action Language for Foundational UML–ALF), Version 1.0–Beta 1 (2010).
  66. 66.
  67. 67.
    Musa, J.D.: The operational profile in software reliability engineering: an overview. Presented at the Third International Symposium on Software, Reliability Engineering (1992)Google Scholar
  68. 68.
    Oldevik, J.: MOFScript user guide. Version 0.6 (MOFScript v 1.1. 11) (2006)Google Scholar
  69. 69.
    Bruegge, B., Dutoit, A.: Object-Oriented Software Engineering Using UML, Patterns, and Java. Prentice Hall, Upper Saddle River (2009)Google Scholar
  70. 70.
    Sun Java Real-Time System. Accessed on 09/02/2012
  71. 71.
    SUSE Linux Enterprise Real Time Extension. Accessed on 09/02/2012
  72. 72.
    McMinn, P.: Search based software test data generation: a survey. Softw. Test. Verif. Reliab. 14, 105–156 (2004)CrossRefGoogle Scholar
  73. 73.
    Ali, S., Iqbal, M.Z., Arcuri, A., Briand, L.: A Search-based OCL constraint solver for model-based test data generation. Presented at the 11th International Conference on Quality Software (2011)Google Scholar
  74. 74.
    Runeson, P., Rainer, A., Höst, M., Regnell, B.: Case Study Research in Software Engineering: Guidelines and Examples. Wiley, New York (2012)CrossRefGoogle Scholar
  75. 75.
    Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14(2), 131–164 (2009)CrossRefGoogle Scholar
  76. 76.
    Zheng, M., Alagar, V., Ormandjieva, O.: Automated generation of test suites from formal specifications of real-time reactive systems. J. Syst. Softw. 81, 286–304 (2008)CrossRefGoogle Scholar
  77. 77.
    Iqbal, M., Ali, S., Yue, T., Briand, L.: Experiences of applying UML/MARTE on three industrial projects. In: Model driven engineering languages and systems, pp. 642–658 (2012)Google Scholar
  78. 78.
    Fiorentini, C., Momigliano, A., Ornaghi, M., Poernomo, I.: A constructive approach to testing model transformations. In: Theory and Practice of Model Transformations, pp. 77–92 (2010)Google Scholar
  79. 79.
    Andrews, J., Briand, L., Labiche, Y., Namin, A.: Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32, 608–624 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Muhammad Zohaib Iqbal
    • 1
  • Andrea Arcuri
    • 2
  • Lionel Briand
    • 3
  1. 1.Department of Computer ScienceNational University of Computer and Emerging SciencesIslamabadPakistan
  2. 2.Simula Research LaboratoryFornebuNorway
  3. 3.SnT Centre for Security, Reliability, and TrustUniversity of LuxembourgWalferdangeLuxembourg

Personalised recommendations