Systematic and Realistic Testing in Simulation of Control Code for Robots in Collaborative Human-Robot Interactions

  • Dejanira Araiza-Illan
  • David Western
  • Anthony G. Pipe
  • Kerstin Eder
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9716)


Industries such as flexible manufacturing and home care will be transformed by the presence of robotic assistants. Assurance of safety and functional soundness for these robotic systems will require rigorous verification and validation. We propose testing in simulation using Coverage-Driven Verification (CDV) to guide the testing process in an automatic and systematic way. We use a two-tiered test generation approach, where abstract test sequences are computed first and then concretized (e.g., data and variables are instantiated), to reduce the complexity of the test generation problem. To demonstrate the effectiveness of our approach, we developed a testbench for robotic code, running in ROS-Gazebo, that implements an object handover as part of a human-robot interaction (HRI) task. Tests are generated to stimulate the robot’s code in a realistic manner, through stimulating the human, environment, sensors, and actuators in simulation. We compare the merits of unconstrained, constrained and model-based test generation in achieving thorough exploration of the code under test, and interesting combinations of human-robot interactions. Our results show that CDV combined with systematic test generation achieves a very high degree of automation in simulation-based verification of control code for robots in HRI.


Model Check Test Generation Coverage Model System Under Test Robotic Assistant 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is part of the EPSRC-funded project “Trustworthy Robotic Assistants” (refs. EP/K006320/1 and EP/K006223/1).


  1. 1.
    Alexander, R., Hawkins, H., Rae, D.: Situation coverage - a coverage criterion for testing autonomous robots. Technical report, Department of Computer Science, University of York (2015)Google Scholar
  2. 2.
    Araiza-Illan, D., Western, D., Pipe, A., Eder, K.: Coverage-driven verification — an approach to verify code for robots that directly interact with humans. In: Piterman, N., et al. (eds.) HVC 2015. LNCS, vol. 9434, pp. 69–84. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-26287-1_5 CrossRefGoogle Scholar
  3. 3.
    Bird, D., Munoz, C.: Automatic generation of random self-checking test cases. IBM Syst. J. 22(3), 229–245 (1983)CrossRefGoogle Scholar
  4. 4.
    Boren, J., Cousins, S.: The SMACH high-level executive. IEEE Robot. Autom. Mag. 17(4), 18–20 (2010)CrossRefGoogle Scholar
  5. 5.
    Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press, Cambridge (1999)Google Scholar
  6. 6.
    Eder, K., Harper, C., Leonards, U.: Towards the safety of human-in-the-loop robotics: challenges and opportunities for safety assurance of robotic co-workers. In: Proceedings of ROMAN, pp. 660–665 (2014)Google Scholar
  7. 7.
    Gaudel, M.-C.: Counting for random testing. In: Wolff, B., Zaïdi, F. (eds.) ICTSS 2011. LNCS, vol. 7019, pp. 1–8. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  8. 8.
    Grigore, E.C., Eder, K., Lenz, A., Skachek, S., Pipe, A.G., Melhuish, C.: Towards safe human-robot interaction. In: Groß, R., Alboul, L., Melhuish, C., Witkowski, M., Prescott, T.J., Penders, J. (eds.) TAROS 2011. LNCS, vol. 6856, pp. 323–335. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  9. 9.
    Haedicke, F., Le, H., Grosse, D., Drechsler, R.: CRAVE: an advanced constrained random verification environment for SystemC. In: Proceedings of SoC, pp. 1–7 (2012)Google Scholar
  10. 10.
    Hartmanns, A., Hermanns, H.: A modest approach to checking probabilistic timed automata. In: Proceedings of QEST, pp. 187–196 (2009)Google Scholar
  11. 11.
    Julius, A.A., Fainekos, G.E., Anand, M., Lee, I., Pappas, G.J.: Robust test generation and coverage for hybrid systems. In: Bemporad, A., Bicchi, A., Buttazzo, G. (eds.) HSCC 2007. LNCS, vol. 4416, pp. 329–342. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  12. 12.
    Kim, J., Esposito, J.M., Kumar, R.: Sampling-based algorithm for testing and validating robot controllers. Int. J. Robot. Res. 25(12), 1257–1272 (2006)CrossRefzbMATHGoogle Scholar
  13. 13.
    Lackner, H., Schlingloff, B.: Modeling for automated test generation a comparison. In: Proceedings of MBEES Workshop (2012)Google Scholar
  14. 14.
    Lakhotia, K., McMinn, P., Harman, M.: Automated test data generation for coverage: havent we solved this problem yet? In: Proceedings of TAIC (2009)Google Scholar
  15. 15.
    Lenz, A., Skachek, S., Hamann, K., Steinwender, J., Pipe, A., Melhuish, C.: The BERT2 infrastructure: an integrated system for the study of human-robot interaction. In: Proceedings of IEEE-RAS Humanoids, pp. 346–351 (2010)Google Scholar
  16. 16.
    Mossige, M., Gotlieb, A., Meling, H.: Testing robot controllers using constraint programming and continuous integration. Inf. Softw. Technol. 57, 169–185 (2014)CrossRefGoogle Scholar
  17. 17.
    Nielsen, B., Skou, A.: Automated test generation from timed automata. Int. J. Softw. Tools Technol. Transfer. 5, 59–77 (2003)CrossRefzbMATHGoogle Scholar
  18. 18.
    Nielsen, B.: Towards a method for combined model-based testing and analysis. In: Proceedings of MODELSWARD, pp. 609–618 (2014)Google Scholar
  19. 19.
    Petters, S., Thomas, D., Friedmann, M., von Stryk, O.: Multilevel testing of control software for teams of autonomous mobile robots. In: Carpin, S., Noda, I., Pagello, E., Reggiani, M., von Stryk, O. (eds.) SIMPAR 2008. LNCS (LNAI), vol. 5325, pp. 183–194. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Pinho, T., Moreira, A.P., Boaventura-Cunha, J.: Framework using ROS and SimTwo simulator for realistic test of mobile robot controllers. In: Proceedings of CONTROLO, pp. 751–759 (2014)Google Scholar
  21. 21.
    Piziali, A.: Functional verification coverage measurement and analysis. Kluwer Academic (2004)Google Scholar
  22. 22.
    Sankaranarayanan, S., Fainekos, G.E.: Falsification of temporal properties of hybrid systems using the cross-entropy method. In: Proceedings of HSCC, pp. 125–134 (2012)Google Scholar
  23. 23.
    Stocker, R., Dennis, L., Dixon, C., Fisher, M.: Verifying brahms human-robot teamwork models. In: del Cerro, L.F., Herzig, A., Mengin, J. (eds.) JELIA 2012. LNCS, vol. 7519, pp. 385–397. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  24. 24.
    Trojanek, P., Eder, K.: Verification and testing of mobile robot navigation algorithms: a case study in SPARK. In: Proceedings of IROS, pp. 1489–1494 (2014)Google Scholar
  25. 25.
    Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Testi., Verification Reliab. 22, 297–312 (2012)CrossRefGoogle Scholar
  26. 26.
    Webster, M., Dixon, C., Fisher, M., Salem, M., Saunders, J., Koay, K.L., Dautenhahn, K.: Formal verification of an autonomous personal robotic assistant. In: Proceedings of AAAI FVHMS, pp. 74–79 (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Dejanira Araiza-Illan
    • 1
  • David Western
    • 1
  • Anthony G. Pipe
    • 2
  • Kerstin Eder
    • 1
  1. 1.Department of Computer Science and Bristol Robotics LaboratoryUniversity of BristolBristolUK
  2. 2.Faculty of Engineering Technology and Bristol Robotics LaboratoryUniversity of the West of EnglandBristolUK

Personalised recommendations