Skip to main content

Integration testing for robotic systems

Software Quality Journal volume 30, pages 3–35 (2022)Cite this article

Abstract

Conventional approaches for the validation of robotic systems include simulations and functional software testing. Although these approaches are interesting, they do not offer coverage information to guide the testing activity. In this sense, the introduction of coverage testing to the validation of mobile robotic systems seems to promise. This paper proposes a integration testing approach for robotic systems in a simulation environment. The approach is composed of test scenarios, constructed based on functional testing, and structural testing towards revealing defects and providing a better systematization of the testing activity in this application domain. The proposed approach focuses on systems developed in ROS, the robot operating system, in which communication can be established through a publish/subscribe interaction schema. The paper provides an example of the way the integration testing approach can be applied to robotic systems. An experimental study conducted with developers of mobile robotic systems evaluated the gains and proved the applicability of the approach to the industry. The results confirmed its advantages for the integration testing of mobile robotic systems. It can check functional behavior (test scenarios), or reach a high structural coverage (coverage criteria), and reveal common defects in mobile robotic systems.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Notes

  1. http://www.ros.org

  2. http://www.ros.org

  3. http://lrm.icmc.usp.br/web/index.php?n=Port.ProjCarina2Info

  4. https://www.openrobots.org/wiki/morse/

  5. http://wiki.ros.org/rviz

  6. https://github.com/ros

  7. https://bit.ly/2DkDAnw

  8. http://wiki.ros.org/gazebo

  9. http://www.robafis.fr/RobAFIS

References

  • Abdelgawad, M., McLeod, S., Andrews, A., & Xiao, J. (2017). Model-based testing of real-time adaptive motion planning (RAMP). In: IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots, SIMPAR, 162–169.

  • Ammann, P., & Offutt, J. (2008). Introduction to Software Testing. Cambridge University Press.

  • Baresi, L., & Pezzé, M. (2006). An introduction to software testing. Electronic Notes in Theoretical Computer Science, proceedings of the School of SegraVis Research Training Network on Foundations of Visual Modelling Techniques (FoVMT 2004), 148(1), 89–111.

  • Belli, F., Hollmann, A., & Padberg, S. (2009). Communication sequence graphs for mutation-oriented integration testing. In: Secure Software Integration and Reliability Improvement - SSIRI, Third IEEE International Conference on, 387–392.

  • Blondeau, V., Etien, A., Anquetil, N., Cresson, S., Croisy, P., & Ducasse, S. (2017). Test case selection in industry: an analysis of issues related to static approaches. Software Quality Journal, 25(4), 1203–1237.

    Article  Google Scholar 

  • Brito, MAS. (2017). Study and definition of integration software testing for mobile robotic systems. PhD thesis, ICMC/USP, Sao Carlos - Sao Paulo - Brasil, portuguese.

  • Brito, MAS., Santos, M., Souza, PSL., & Souza, SRS. (2015). Integration testing criteria for mobile robotic systems. In: International Conference on Software Engineering & Knowledge Engineering.

  • Brito, P., De Lemos, R., Rubira, C., & Martins, E. (2009). Architecting fault tolerance with exception handling: Verification and validation. Journal of Computer Science and Technology, 24(2), 212–237.

    Article  Google Scholar 

  • Carlson, J., Murphy, R. R., & Nelson, A. (2004). Follow-up analysis of mobile robot failures. International Conference on Robotics and Automation, ICRA, 5, 4987–4994.

    Google Scholar 

  • Chen, Q., Wang, L., Yang, Z., & Stoller, S. (2009). HAVE: Detecting atomicity violations via integrated dynamic and static analysis. Fundamental Approaches to Software Engineering, Springer, Berlin Heidelberg, 5503, 425–439.

    Article  Google Scholar 

  • Damm, W., Hungar, H., Josko, B., Peikenkamp, T., & Stierand, I. (2011). Using contract-based component specifications for virtual integration testing and architecture design. In: Design, Automation Test in Europe Conference Exhibition, 1023–1028.

  • Eugster, P., Felber, P. A., Guerraoui, R., & Kermarrec, A. (2003). The many faces of publish/subscribe. ACM Comput Surv, 35(2), 114–131.

    Article  Google Scholar 

  • Fernandes, LC., Souza, JR., Pessim, G., Shinzato, PY., Sales, DO., Grassi, Jr., V., ... Wolf, DF. (2013). CaRINA intelligent robotic car: Architectural design and implementations. Journal of Systems Architecture.

  • Frankl, F., & Weyuker, E. (1986). Data ow testing in the presence of unexecutable paths. In: Proceedings of the Workshop on Software Testing, 4–13.

  • Garlan, D., Khersonsky, S., & Kim, J. S. (2003). Model checking publish-subscribe systems. In T. Ball & S. K. Rajamani (Eds.), Model Checking Software Heidelberg: Springer (pp. 166–180). Berlin: Berlin Heidelberg.

    Chapter  Google Scholar 

  • Kaner, C. (2003). An introduction to scenario testing. Tech. rep.: University Florida Tech.

  • Kang, JS., & Park, HS. (2011). RPIST: Required and provided interface specification-based test case generation and execution methodology for robot software component. In: Int. Conference on Ubiquitous Robots and Ambient Intelligence, 647–651.

  • Kang, J. S., & Park, H. S. (2012). Web-based automated black-box testing framework for component based robot software. Proceedings of the ACM Conference on Ubiquitous Computing, UbiComp, 852–859. https://doi.org/10.1145/2370216.2370410, http://doi.acm.org/10.1145/2370216.2370410.

  • Kang, SS., Maeng, SW., Kim, SW., & Park, HS. (2010). SITAT: Simulation-based interface testing automation tool for robot software component. In: Int. Conference on Control Automation and Systems, 1781–1784.

  • Kim, D., Hong, J. E., & Chung, L. (2018). Investigating relationships between functional coupling and the energy efficiency of embedded software. Software Quality Journal, 26(2), 491–519.

    Article  Google Scholar 

  • Kramer, J., & Scheutz, M. (2007). Development environments for autonomous mobile robots: A survey. Auton Robots, 22(2), 101–132.

    Article  Google Scholar 

  • Lee, N., Kim, T., & Cha, S. (2002). Construction of global finite state machine for testing task interactions written in message sequence charts. International conference on Software engineering and knowledge engineering, 27, 369–376.

    Google Scholar 

  • Leung, HKN., & White, L. (1990). A study of integration testing and software regression at the integration level. In: Conference on Software Maintenance, 290–301.

  • Lim, JH., Song, SH., Kuc, TY., Park, HS., & Kim, HS. (2009). A hierarchical test model and automated test framework for RTC. In: Int. Conf. on Future Generation Information Technology, 198–207.

  • Lima, LP., & Cavalli, AR. (1997). A pragmatic approach to generating test sequences for embedded systems. In: International Federation for Information Processing, 288–307.

  • Lu, XL., Dong, YW., Sun, B., & Zhao, HB. (2011). Research of embedded software testing method based on AADL modes. In: 3rd International Conference on Communication Software and Networks, 89–92.

  • Lucas, C., Elbaum, S., & Rosenblum, DS. (2012). Detecting problematic message sequences and frequencies in distributed systems. In: Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, ACM, New York, NY, USA, OOPSLA ’12, 915–926.

  • Lussier, B., Chatila, R., Ingrand, F., Killijian, M., & Powell, D. (2004). On fault tolerance and robustness in autonomous systems. In: 3rdI ARPIEEE/RASEURON Joint Workshop on Technical Challenges for Dependable Robots in Human Environments.

  • Matias, LPN., Santos, TC., Wolf, DF., & Souza, JR. (2015). Path planning and autonomous navigation using amcl and AD*. In: 12th Latin American Robotics Symposium and 3rd Brazilian Symposium on Robotics (LARS-SBR), 320–324.

  • Mattiello-Francisco, F., Martins, E., Cavalli, A., & Yano, E. (2012). Inrob: An approach for testing interoperability and robustness of real-time embedded software. Journal of Systems and Software, 85(1), 3–15.

    Article  Google Scholar 

  • Melo, S., Souza, S. R. S., Sarmanho, F. S., & Souza, P. S. L. (2017). Contributions for the structural testing of multithreaded programs: coverage criteria, testing tool, and experimental evaluation. Software Quality Journal, 26(1), 921–959.

    Google Scholar 

  • Myers, G. J., Sandler, C., Badgett, T., & Thomas, T. M. (2004). The Art of Software Testing. Hoboken, New Jersey: John Wiley & Sons Inc.

    Google Scholar 

  • Neves, VO., Delamaro, ME., Masiero, P., Mendes, CCT., & Wolf, DF. (2013). Structural testing of autonomous vehicles. In: 25th International Conference on Software Engineering & Knowledge Engineering, 200–205.

  • Pakala, H., Khan, I., & Raju, K. (2010). Sensors integration in embedded systems. In: International Conference on Power, Control and Embedded Systems.

  • Pakala, H. G. M., Raju, K. V. S. V. N., & Khan, I. (2011). Integration testing of multiple embedded processing components. Advanced Computing, 133, 200–209.

    Article  Google Scholar 

  • Park, H., & Seok Kang, J. (2012). SITAF: Simulation-based interface testing automation framework for robot software component (pp. 453–470). InTech: Automation.

    Google Scholar 

  • Planthaber, S., Vogelgesang, J., & Nieben, E. (2011). CoHoN: A fault-tolerant publish/subscribe tree-based middleware for robots with heterogeneous communication hardware. In: International Conference on Robotics and Biomimetics. IEEE International Conference on Robotics and Biomimetics (IEEE-ROBIO), 232–235.

  • Prado, RR., Souza, PSL., Souza, SRS., Dourado, GGM., & Batista, RN. (2016). Valipar service: structural testing of concurrent programs a web service composition. In: 13th International Conference on Information Technology: New Generations (ITNG), 581–590.

  • Pressman, R. S. (2005). Software Engineering: Practitioner’s Approach (6th ed.). McGraw-Hill.

  • Rapps, S., & Weyuker, E. J. (1985). Selecting software test data using data flow information. IEEE Transactions on Software Engineering, SE, 11(4), 367–375.

    Article  Google Scholar 

  • Siegwart, R., & Nourbakhsh, I. R. (2004). Introduction to Autonomous Mobile Robots. Scituate, MA, USA: Bradford Company.

    Google Scholar 

  • Son, J., Kuc, T., Park, J., & Kim, H., (2011). Simulation based functional and per- formance evaluation of robot components and modules. In: International Conference on Information Science and Applications, 1–7.

  • Sotiropoulos, T., Waeselynck, H., Guiochet, J., & Ingrand, F. (2017). Can robot navigation bugs be found in simulation? an exploratory study. In: IEEE Inter- national Conference on Software Quality, Reliability and Security (QRS), 150–159.

  • Souza, P. S. L., Souza, S. R. S., & Zaluska, E. (2014). Structural testing for message-passing concurrent programs: an extended test model. Concurrency and Computation: Practice and Experience, 26(1), 21–50.

    Article  Google Scholar 

  • Souza, S. R. S., Vergilio, S. R., Souza, P. S. L., Simão, A. S., & Hausen, A. C. (2008). Structural testing criteria for message-passing parallel programs. Concurrency and Computation: Practice and Experience, 20(16), 1893–1916.

    Article  Google Scholar 

  • Söylemez, M., & Tarhan, A. (2018). Challenges of software process and product quality improvement: catalyzing defect root-cause investigation by process enactment data analysis. Software Quality Journal, 26(2), 779–807.

    Article  Google Scholar 

  • Steinbauer, G. (2013). A survey about faults of robots used in robocup. In X. Chen, P. Stone, L. E. Sucar, & T. van der Zant (Eds.), RoboCup 2012: Robot Soccer World Cup XVI Heidelberg: Springer (pp. 344–355). Berlin: Berlin Heidelberg.

    Chapter  Google Scholar 

  • Szyperski, C. (2002). Component Software: Beyond Object-Oriented Program- ming. Boston, MA: Addison-Wesley.

    Google Scholar 

  • Tzafestas, S. (2014). Introduction to Mobile Robot Control. Elsevier.

  • Weisshardt, F., Kett, J., Araujo, De Freitas Oliveira, T., Bubeck, A., & Verl, A., (2014). Enhancing software portability with a testing and evaluation platform. Proceedings for the Joint Conference of ISR 2014–45th International Sym- posium on Robotics and Robotik - 8th German Conference on Robotics Munich, Germany: ISR/ROBOTIK, 219–224.

Download references

Acknowledgements

The authors acknowledge Brazilian funding agency FAPESP, under processes 2013/03459-4 and 2019/06937-0 for the financial support provided for this research.

Author information

Authors and Affiliations

  1. Departamento de Ciência da Computação, Universidade Federal de Lavras, Lavras, Brazil

    Maria A. S. Brito

  2. Instituto de Ciências Matemáticas e de Computação, Universidade de Sao Paulo, Sao Carlos, Brazil

    Simone R. S. Souza & Paulo S. L. Souza

Authors
  1. Maria A. S. Brito
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simone R. S. Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paulo S. L. Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria A. S. Brito.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Brito, M.A.S., Souza, S.R.S. & Souza, P.S.L. Integration testing for robotic systems. Software Qual J 30, 3–35 (2022). https://doi.org/10.1007/s11219-020-09535-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-020-09535-w

Keywords

  • Software testing
  • Integration testing
  • Test scenarios
  • Testing criteria
  • Robotic system
  • Experimental study