An Industrial Application of Robustness Testing Using Aspect-Oriented Modeling, UML/MARTE, and Search Algorithms

  • Shaukat Ali
  • Lionel C. Briand
  • Andrea Arcuri
  • Suneth Walawege
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6981)


Systematic and rigorous robustness testing is very critical for embedded systems, as for example communication and control systems. Robustness testing aims at testing the behavior of a system in the presence of faulty situations in its operating environment (e.g., sensors and actuators). In such situations, the system should gracefully degrade its performance instead of abruptly stopping execution. To systematically perform robustness testing, one option is to resort to model-based robustness testing (MBRT), based for example on UML/MARTE models. However, to successfully apply MBRT in industrial contexts, new technology needs to be developed to scale to the complexity of real industrial systems. In this paper, we report on our experience of performing MBRT on video conferencing systems developed by Cisco Systems, Norway. We discuss how we developed and integrated various techniques and tools to achieve a fully automated MBRT that is able to detect previously uncaught software faults in those systems. We provide an overview of how we achieved scalable modeling of robustness behavior using aspect-oriented modeling, test case generation using search algorithms, and environment emulation for test case execution. Our experience and lessons learned identify challenges and open research questions for the industrial application of MBRT.


Model-based testing aspect-oriented modeling search algorithms MARTE UML robustness 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    IEEE Standard Glossary of Software Engineering Terminology. IEEE, IEEE Std 610.12-1990, p. 1 (1990) Google Scholar
  2. 2.
    IEEE Standard Dictionary of Measures of the Software Aspects of Dependability. IEEE Std 982.1-2005 (Revision of IEEE Std 982.1-1988), pp. 1–34 (2006) Google Scholar
  3. 3.
    Standard for Software Quality Characteristics. International Organization for Standardization, ISO-9126-3 (2003) Google Scholar
  4. 4.
    Software Assurance Standard. NASA Technical Standard, NASA-STD-8739.8 (2005) Google Scholar
  5. 5.
    Shafique, M., Labiche, Y.: A Systematic Review of Model Based Testing Tools. Carleton University, Department of Systems and Computer Engineering, Technical Report, SCE-10-04 (2010)Google Scholar
  6. 6.
    Ali, S., Hemmati, H., Holt, N.E., Arisholm, E., Briand, L.C.: Model Transformations as a Strategy to Automate Model-Based Testing - A Tool and Industrial Case Studies. Simula Research Laboratory, Technical Report (2010-01) (2010)Google Scholar
  7. 7.
    Ali, S., Briand, L.C., Hemmati, H.: Modeling Robustness Behavior Using Aspect-Oriented Modeling to Support Robustness Testing of Industrial Systems. Simula Research Laboratory, Technical Report (2010-03) (2010)Google Scholar
  8. 8.
    Modeling and Analysis of Real-time and Embedded systems, MARTE (2010),
  9. 9.
    Binder, R.V.: Testing object-oriented systems: models, patterns, and tools. Addison-Wesley Longman Publishing Co., Inc., Amsterdam (1999)Google Scholar
  10. 10.
    Yedduladoddi, R.: Aspect Oriented Software Development: An Approach to Composing UML Design Models. VDM Verlag Dr. Müller (2009)Google Scholar
  11. 11.
    Kermeta - Breathe Life into Your Metamodels (2010),
  12. 12.
    Ali, S., Iqbal, M.Z., Arcuri, A., Briand, L.C.: A Search-based OCL Constraint Solver for Model-based Test Data Generation. In: Proceedings of the 11th International Conference On Quality Software, QSIC 2011 (2011)Google Scholar
  13. 13.
    Egea, M.: EyeOCL Software (2010),
  14. 14.
  15. 15.
    Ali, S., Yue, T., Briand, L.C., Malik, Z.I.: Does Aspect-Oriented Modeling Help Improve the Readability of UML State Machines? (2010) Google Scholar
  16. 16.
    Query/View/Transformation QVT (2011),
  17. 17.
    Cabot, J., Claris, R., Riera, D.: Verification of UML/OCL Class Diagrams using Constraint Programming. In: Proceedings of the 2008 IEEE International Conference on Software Testing Verification and Validation Workshop. IEEE Computer Society, Los Alamitos (2008)Google Scholar
  18. 18.
    Krieger, M., Knapp, A.: Executing Underspecified OCL Operation Contracts with a SAT Solver. In: 8th International Workshop on OCL Concepts and Tools. ECEASST, vol. 15 (2008)Google Scholar
  19. 19.
    Hemmati, H., Briand, L., Arcuri, A., Ali, S.: An Enhanced Test Case Selection Approach for Model-Based Testing: An Industrial Case Study. In: 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York (2010)Google Scholar
  20. 20.
    Pintér, G., Majzik, I.: Modeling and Analysis of Exception Handling by Using UML Statecharts. Scientific Engineering of Distributed Java Applications, 58–67 (2005)Google Scholar
  21. 21.
    Lei, B., Liu, Z., Morisset, C., Li, X.: State Based Robustness Testing for Components. Electronic Notes of Theoratical Computer Science 260, 173–188Google Scholar
  22. 22.
    Nebut, C., Fleurey, F., Traon, Y.L., Jezequel, J.-M.: Automatic Test Generation: A Use Case Driven Approach. IEEE Transactions of Software Engineering 32, 140–155 (2006)CrossRefGoogle Scholar
  23. 23.
    Afzal, W., Torkar, R., Feldt, R.: A Systematic Review of Search-based Testing for Non-functional System Properties. Information and Software Technology 51, 957–976 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Shaukat Ali
    • 1
    • 2
  • Lionel C. Briand
    • 1
    • 2
  • Andrea Arcuri
    • 1
  • Suneth Walawege
    • 3
  1. 1.Simula Research LaboratoryNorway
  2. 2.The University of OsloNorway
  3. 3.Cisco Systems IncNorway

Personalised recommendations