Efficient and Flexible Test Automation in Production Systems Engineering

  • Dietmar WinklerEmail author
  • Kristof Meixner
  • Petr Novak


Context and background: In Production Systems Engineering (PSE), software and systems testing are success-critical along the production automation life cycle to identify defects early and efficiently. Although test automation concepts enable continuous integration and tests during engineering and maintenance, tool chains are often hardwired, less flexible, and inefficient. Thus, there is a need for more flexible tool chains to support verification and validation of control code variants. Objective: In this book chapter, we (a) describe a flexible Test Automation Framework (TAF) to enable continuous integration and tests and (b) provide an adapted maintenance process to enable efficient verification and validation of control code variants. Method: We build on best practices from Software Engineering and Software Testing to establish a flexible TAF based on Behavior-Driven Testing. We use the Abstract Syntax Tree (AST) as foundation for human-based verification and validation. We developed an initial prototype derived from industry partners and used an Industry 4.0 Testbed for evaluation. Results and conclusion: First results of the prototype implementation with selected testing tools showed the capability of the TAF concept for supporting flexible configurations of testing tool chains. The AST concept can support the human-based verification and validation of control code variants.


Production systems engineering Test automation Behavior-driven test Model quality assurance Verification and validation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The financial support by the Christian Doppler Research Association, the Austrian Federal Ministry for Digital and Economic Affairs, and the National Foundation for Research, Technology and Development is gratefully acknowledged. The research done by Petr Novak has been supported by the DAMiAS project funded by the Technology Agency of the Czech Republic.


  1. Aurum, A., Petersson, H., & Wohlin, C. (2002). State-of-the-art: Software inspections after 25 years. Journal of Software Testing, Verification and Reliability, 12(3), 133–154.CrossRefGoogle Scholar
  2. Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.Google Scholar
  3. Biffl, S., Lüder, A., & Winkler, D. (2016). Multi-disciplinary engineering for Industrie 4.0: Semantic challenges, needs, and capabilities. In S. Biffl & M. Sabou (Eds.), Semantic web for intelligent engineering applications. New York: Springer.Google Scholar
  4. Brambilla, M., Cabot, J., & Wimmer, M. (2012). Model-driven software engineering in practice. San Rafael, CA: Morgan & Claypool.CrossRefGoogle Scholar
  5. Broy, M. (2006). The ‘grand challenge’ in informatics: Engineering software-intensive systems. IEEE Computer, 39(10), 72–80.CrossRefGoogle Scholar
  6. Duvall, P. M., Matyas, S., & Glover, A. (2007). Continuous integration: Improving software quality and reducing risk. London: Pearson.Google Scholar
  7. Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3), 226–237.CrossRefGoogle Scholar
  8. Gilb, T., Graham, D., & Finzi, S. (1993). Software inspection. Boston, MA: Addison Wesley.Google Scholar
  9. Häser, F., Felderer, M., & Breu, R. (2016). Is business domain language support beneficial for creating test case specifications: A controlled experiment. Information and Software Technology, 79, 52–62.CrossRefGoogle Scholar
  10. ISO 29119-2. (2013). Software and systems engineering – Software testing – Part 2: Test processes, ISO/IEC/IEEE291192.Google Scholar
  11. ISO 29119-5. (2016). Software and systems engineering – Software testing – Part 5: Keyword-driven testing, ISO/IEC/IEEE 29119-5.Google Scholar
  12. Jones, J. (2003). Abstract syntax tree implementation idioms. In Proceedings of the 10th Conference on Pattern Languages of Programs (PLOP) (pp. 1–10). North Carolina: The Hillside Group.Google Scholar
  13. Laitenberger, O., & DeBaud, J. M. (2000). An encompassing life cycle centric survey of software inspection. Journal of Systems and Software, 50(1), 5–31.CrossRefGoogle Scholar
  14. Laukkanen, E., Itkonen, J., & Lassenius, C. (2017). Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology, 82, 55–79.CrossRefGoogle Scholar
  15. Lindstrom, L., & Jeffries, R. (2003). Extreme programming and agile software development methodologies. In IS management handbook (8th ed.). Abingdon: Taylor & Francis.Google Scholar
  16. Mårtensson, T., Hammarstrom, P., & Bosch, J. (2017). Continuous integration is not about build systems. In Proceedings of the 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE.Google Scholar
  17. Mårtensson, T., Ståhl, D., & Bosch, J. (2018). Enable more frequent integration of software in industry projects. Journal of Systems and Software, 142, 223–236.CrossRefGoogle Scholar
  18. Micallef, M., & Colombo, C. (2015). Lessons learnt from using DSLs for automated software testing. In Proceedings of 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.Google Scholar
  19. Moser, T., Mordinyi, R., & Winkler, D. (2012). Extending mechatronic objects for automation systems engineering in heterogeneous engineering environments. In Proceedings of the 17th International Conference on Emerging Technologies & Factory Automation (ETFA). IEEE.Google Scholar
  20. Musil, J., Musil, A., Weyns, D., & Biffl, S. (2015). An architecture framework for collective intelligence systems. In Proceedings of the 12th Working IEEE/IFIP Conference on Software Architecture. ACM.Google Scholar
  21. Roche, J. (2013). Adopting DevOps practices in quality assurance. Communications of the ACM, 56(11), 38–43.CrossRefGoogle Scholar
  22. Rosen, R., von Wichert, G., Lo, G., & Bettenhausen, K. D. (2015). About the importance of autonomy and digital twins for the future of manufacturing. IFAC-PapersOnLine, 48(3), 567–572.CrossRefGoogle Scholar
  23. Sabou, M., Winkler, D., Penzerstadler, P., & Biffl, S. (2018). Verifying conceptual domain models with human computation: A casestudy in software engineering. In Proceedings of the 6th AAAI conference on human computation and crowdsourcing (HCOMP). Zurich: AAAI.Google Scholar
  24. Schatten, A., Biffl, S., Demolsky, M., Gostischa-Fanta, E., Östereicher, T., & Winkler, D. (2010). Best Practice Software-Engineering: Eine praxiserprobte Zusammenstellung von komponentenorientierten Konzepten, Methoden und Werkzeugen. Springer.Google Scholar
  25. Schwaber, K., & Beedle, M. (2002). Agile software development with Scrum. London: Pearson.zbMATHGoogle Scholar
  26. Shahin, M., Babar, M. A., & Zhu, L. (2017). Continuous integration, delivery and deployment: A systematic review on approaches, tools, challenges and practices. IEEE Access, 5, 3909–3943.CrossRefGoogle Scholar
  27. Soeken, M., Wille, R., & Drechsler, R. (2012). Assisted behavior driven development using natural language processing. In Objects, models, components, patterns, tools (pp. 269–287). Berlin: Springer.CrossRefGoogle Scholar
  28. Solis, C., & Wang, X. (2011). A study of the characteristics of behaviour driven development. In Proceedings of the 37th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA). IEEE.Google Scholar
  29. Spillner, A., Linz, T., & Schaefer, H. (2014). Software testing foundations: A study guide for the certified tester exam. San Rafael, CA: Rocky Nook.Google Scholar
  30. Ståhl, D., & Bosch, J. (2014). Modeling continuous integration practice differences in industry software development. Journal of Systems and Software, 87, 48–59.CrossRefGoogle Scholar
  31. Sunindyo, W. D., Moser, T., Winkler, D., & Biffl, S. (2010). Foundations for event-based process analysis in heterogeneous software engineering environments. In Proceedings of the 36th EUROMICRO conference on software engineering and advanced applications (SEAA). IEEE.Google Scholar
  32. Van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Beck, K., & Beedle, M. (2001). Manifesto for agile software development. Alicante: Universidad de Alicante.Google Scholar
  33. Vogel-Heuser, B., Kegel, G., Bender, K., & Wucherer, K. (2009). Global information architecture for industrial automation. In Automatisierungstechnische Praxis (atp). Munich: Oldenbourg-Verlag.Google Scholar
  34. Vogel-Heuser, B., Fay, A., Schaefer, I., & Tichy, M. (2015). Evolution of software in automated production systems: Challenges and research directions. Journal of Systems and Software, 110, 54–84.CrossRefGoogle Scholar
  35. Whitehead, J. (2007). Collaboration in software engineering: A roadmap. In Future of software engineering (FOSE). IEEE.Google Scholar
  36. Winkler, D., Wimmer, M., Berardinelli, L., & Biffl, S. (2017a). Towards model quality assurance for multi-disciplinary engineering.Needs, challenges, and solution concepts in an AutomationMLcontext. In S. Biffl, A. Lüder, & D. Gerhard (Eds.), Multi-disciplinary engineering for cyber-physical production systems, chapter 16 (pp. 433–457). Cham: Springer.CrossRefGoogle Scholar
  37. Winkler, D., Sabou, M., Petrovic, S., Biffl, S., Kalinowski, M., & Carneiro, G. (2017b). Improving model inspection with crowdsourcing. In Proceedings of the 4th International Workshop on Crowdsourcing in Software Engineering. Buenos Aires: ACM/IEEE International Conference on Software Engineering (ICSE).Google Scholar
  38. Winkler, D., Sabou, M., & Biffl, S. (2017c). Improving quality assurance in multidisciplinary engineering environments with semantic technologies. In Quality control and assurance - an ancient Greek term re-mastered. London: Intec Publishing.Google Scholar
  39. Winkler, D., Meixner, K., & Biffl, S. (2018a). Towards flexible and automated testing in production systems engineering projects. In Proceedings of the 23rd International Conference on Emerging Technologies and Factory Automation (ETFA). IEEE.Google Scholar
  40. Winkler, D., Sabou, M., Petrovic, S., Biffl, S., & Kalinowski, M. (2018b). Investigating a distributed and scalable model review process. Centro Latinoamericano de Estudios en Informatica (CLEI), 21(1).Google Scholar
  41. Zhu, Y.-M. (2016). Software reading techniques: Twenty techniques for more effective software review and inspection. New York: Apress.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Christian Doppler Laboratory for Security and Quality Improvement in the Production System Lifecycle (CDL-SQI), Institute of Information Systems EngineeringTechnische Universität WienViennaAustria
  2. 2.Czech Technical UniversityPragueCzech Republic

Personalised recommendations