Skip to main content

Model checking learning agent systems using Promela with embedded C code and abstraction

Abstract

As autonomous systems become more prevalent, methods for their verification will become more widely used. Model checking is a formal verification technique that can help ensure the safety of autonomous systems, but in most cases it cannot be applied by novices, or in its straight “off-the-shelf” form. In order to be more widely applicable it is crucial that more sophisticated techniques are used, and are presented in a way that is reproducible by engineers and verifiers alike. In this paper we demonstrate in detail two techniques that are used to increase the power of model checking using the model checker Spin. The first of these is the use of embedded C code within Promela specifications, in order to accurately reflect robot movement. The second is to use abstraction together with a simulation relation to allow us to verify multiple environments simultaneously. We apply these techniques to a fairly simple system in which a robot moves about a fixed circular environment and learns to avoid obstacles. The learning algorithm is inspired by the way that insects learn to avoid obstacles in response to pain signals received from their antennae. Crucially, we prove that our abstraction is sound for our example system—a step that is often omitted but is vital if formal verification is to be widely accepted as a useful and meaningful approach.

References

  1. ACH+95

    Alur R, Courcoubetis C, Halbwachs N, Henzinger T, Ho P-H, Nicollin X, Olivero A, Sifakis J, Yovine S (1995) The algorithmic analysis of hybrid systems. Theor Comput Sci 138:3–34

  2. AGLS01

    Alur R, Grosu R, Lee I, Sokolsky O (2001) Compositional refinement of hiererarchical hybrid systems. Vol. 2034 of Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Rome, pp 33–48.

  3. AICE15

    Antuna L, Ilan D, CAmpos S, Eder K (2015) symmetry reduction enables model checking of more complex emergent behaviours of swarm navigation algorithms. In: Towards autonomous robotic systems, vol. 9287 of Lecture Notes in Computing Science. Springer, New York, pp 26–37

  4. BBDB14

    Brambilla M, Brutschy A, Dorigo M, Birattari M: Property-driven design for robot swarms: A design method based on prescriptive modeling and model checking. ACM Trans Auton Adapt Syst 9(4), 17–11728 (2014)

    Article  Google Scholar 

  5. Bra84

    Braitenberg V: Vehicles: experiments in synthetic psychology. MIT Press, Colorado (1984)

    Google Scholar 

  6. Büc60

    Büchi J (1960) On a decision method in restricted second order arithmetic. In: Proceedings of the International Congress on Logic, Method, and Philosophy of Science. Stanford University Press, pp 1–12

  7. Cat94

    Cattel T (1994) Modeling and verification of a multiprocessor realtime OS kernel. In: Hogrefe D, Leue S (eds) Proceedings of the 7th WG6.1 International Conference on Formal Description Techniques (FORTE ‘94), vol. 6 of International Federation For Information Processing. Chapman and Hall, Berne, Switzerland, pp 55–70

  8. CGM+97

    Cimatti A, Giunchiglia F, Mingardi G, Romano D, Torielli F, Traverso P (1997) Model checking safety critical software with SPIN: an application to a railway interlocking system. In: Langerak R (ed) Proceedings of the 3rd SPIN Workshop (SPIN‘97). Twente University, The Netherlands, pp. 5–17

  9. CGP99

    Clarke E, Grumberg O, Peled D: Model checking. The MIT Press, Cambridge (1999)

    Google Scholar 

  10. CM08

    Calder M, Miller A: An automatic abstraction technique for verifying featured, parameterised systems. Theor Comput Sci 404(3), 235–255 (2008)

    MathSciNet  Article  MATH  Google Scholar 

  11. Dil96

    Dill D (1996) The Murϕ verification system. In: Alur R, Henzinger T (eds) Proceedings of the 8th International Conference on Computer Aided Verification (CAV ‘96), vol. 1102 of Lecture Notes in Computer Science. Springer, New Brunswick, pp 390–393

  12. DWFZ12

    Dixon C, Winfield A, Fisher M, Zeng C: Towards temporal verification of swarm robotic systems. Robot Automomous Syst 60, 1429–1441 (2012)

    Article  Google Scholar 

  13. FDW13

    Fisher M, Dennis L, Webster M: Verification of autonomous systems. Commun ACM 56(9), 84–93 (2013)

    Article  Google Scholar 

  14. FGKGP09

    Fainekos G, Girard A, Kress-Gazit H, Pappas G: Temporal logic motion planning for dynamic robots. Automatica 45(2), 343–352 (2009)

    MathSciNet  Article  MATH  Google Scholar 

  15. GJD13

    Guo M, Johansson K, Dimarogonas (2013) Revising motion planning under linear temporal logic specifications in partially known workspaces. In: IEEE international Conference Robotics and Automation

  16. GMS16

    Gallardo M, Merino P, Salmerón A (2016) River basin management with SPIN. In: Proceedings of the 23rd International SPIN symposium on Model Checking of Software (SPIN 2016) (to appear)

  17. HIM+16

    Hoffman R, Ireland M, Miller A, Norman G, Veres S (2016) Autonomous agent behaviour modelled in PRISM. In: Proceedings of the 23rd International SPIN symposium on Model Checking of Software (SPIN 2016) (to appear)

  18. HKNP06

    Hinton A, Kwiatkowska M, Norman G, Parker D: Prism: a tool for automatic verification of probabilistic systems. LNCS 3920, 441–444 (2006)

    Google Scholar 

  19. Hol03

    Holzmann GJ (2003) Trends in software verification. In: Araki K, Gnesi S, Mandrioli D (eds) Proceedings of the International Symposium on Formal Methods Europe (FME 2006), vol. 2805 of Lecture Notes in Computer Science. Springer, Pisa, Italy, pp 40–50

  20. Hol04

    Holzmann G (2004) The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Pearson Education

  21. hsc02

    (2002) Proceedings of the 5th International Workshop on Hybrid Systems: Computation and Control (HSCC ’02), vol. 2289 of Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Santa Barbara

  22. Hum13

    Humphrey L (2013) Model checking for verification in uav cooperative control applications. Vol. 444 of Lecture Notes in Computer Science. Springer, New York, pp 69–117.

  23. JTZW06

    Jeyaraman S, Tsourdos A, Zbikowski R, White B (2006) Kripke modelling approaches of a multiple robots system with minimalist communication: a formal approach of choice. Int J Syst Sci 37(6):339–349

  24. KB06

    Kloetzer M, Belta C (2006) Hierarchical abstractions for robotic swarms. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2006), pp 952–957

  25. KB07

    Kloetzer M, Belta C: Temporal logic planning and control robotic swarms by heirarchical abstractions. IEEE Trans Robot 213(2), 320–330 (2007)

    Article  Google Scholar 

  26. KDF12

    Konur S, Dixon C, Fisher M: Analysing robot swarm behaviour via probabilistic model checking. Robot Auton Syst 60(2), 199–213 (2012)

    Article  Google Scholar 

  27. Kir14

    Kirwan RF (2014) Applying model checking to agent-based learning systems. Ph.D. thesis, University of Glasgow

  28. KKT+10

    Kulvicius T, Kolodziejski C, Tamosiunaite T, Porr B, Worgotter F: Behavioral analysis of differential Hebbian learning in closed-loop systems. Biol Cybern 103(4), 255–271 (2010)

    Article  MATH  Google Scholar 

  29. KL02

    Kumar S, Li K (2002) Using model checking to debug device firmware. In: Proceedings of the 5th Symposium on Operating System Design and Implementation (OSDI 2002). USENIX, Boston

  30. KL15

    Kouvaros P, Lomuscio A (2015) A counter abstraction technique for the verification of robot swarms. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, pp 2081–2088

  31. KMP

    Kirwan R, Miller A, Porr B (2016). doi:10.5525/gla.researchdata.310

  32. KMPD13

    Kirwan R, Miller A, Porr B, Di Prodi P: Formal modelling of robot behaviour with learning. Neural Comput 25(11), 2976–3019 (2013)

    Article  Google Scholar 

  33. Kri63

    Kripke S: Semantical considerations on modal logics. Acta Philos Fenn 16, 83–94 (1963)

    MathSciNet  MATH  Google Scholar 

  34. McM93

    McMillan K (1993) Symbolic model checking. Boston

  35. Mer00

    Merz S (2000) Model checking: a tutorial overview. In: Cassez F Jard C, Rozoy B, Ryan M (eds) Modeling and verification of parallel processes, 4th Summer School, MOVEP, vol. 2067 of Lecture Notes in Computer Science. Springer, Nantes

  36. Mil71

    Milner R (1971) An algebraic definition of simulation between programs. In: Proceedings of the 2nd International Joint Conference on Artificial Intelligence, pp 481–489

  37. MOSS99

    Müller-Olm M, Schmidt D, Steffen B (1999) Model-checking: a tutorial introduction. In: Cortesi A, File G (eds) Proceedings of the 6th International Static Analysis Symposium (SAS’99), vol. 1694 of Lecture Notes in Computer Science (LNCS). Springer, Venice, pp 330–354

  38. PvW03

    Porr B, von Ferber C, Wörgötter F (2003) ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioural paradigm. Neural Comput 15(4):865–884

  39. PW06

    Porr B, Wörgötter F: Strongly improved stability and faster convergence of temporal sequence learning by utilising input correlations only. Neural Comput 18(6), 1380–1412 (2006)

    Article  MATH  Google Scholar 

  40. SB87

    Sutton R, Barto A (1987) A temporal-difference model of classical conditioning. In: Proceedings of the Ninth Annual Conference of the Cognitive Science Society. Seattle, Washington, pp 355–378

  41. SDDF12

    Stocker R, Dennis L, Dixon C, Fisher M (2012) Verifiying Brahms human-robot teamwork models. In: Proceedings of the 13th European conference on logics in Artificial Intelligence (JELIA-2012), pp 385–397

  42. Sie01

    Sierhuis M (2001) Modeling and simulating work practice. BRAHMS: A multi-agent modeling and simulation language for work system analysis and design. Ph.D. thesis, University of Amsterdam

  43. SLM+09

    Sharma O, Lewis J, Miller A, Dearle A, Balasubramaniam D, Morrison R, Sventek J (2009) Towards verifying correctness of wireless sensor network applications using insense and spin. In: Păsrăeanu C (ed) Proceedings of the 16th International SPIN Workshop (SPIN 2009), Lecture Notes in Computer Science. Springer, Grenoble, pp 223–240

  44. Sta02

    Stauner T (2002) Discrete-time refinement of hybrid automata. In: Sta02 [hsc02], pp 407–420

  45. TK02

    Tiwari A, Khanna G (2002) Series of abstractions for hybrid automata. In: TiwKha02 [hsc02], pp 465–478

  46. WBBK11

    Weissman M, Bedenk S, Buckl C, Knoll A (2011) Model checking industrial robot systems. In: Groce A, Musuvathi M (eds) Proceedings of the 18th International SPIN Workshop (SPIN 2009), vol. 6823 of Lecture Notes in Computer Science. Springer, Snowbird, pp 161–176

  47. WDF+15

    Webster M, Dixon C, Fisher M, Salem M, Saunders J, Koay J, Dautenhahn K, Saez-Pons J: Toward reliable autonomous robotic assistants through formal verification: a case study. IEEE Trans Human Mach Syst 99, 1–11 (2015)

    Google Scholar 

  48. YT01

    Yuen C, Tjioe W (2001) Modeling and verifying a price model for congestion control in computer networks using Promela/Spin. In: Dwyer MB (ed) Proceedings of the 8th International SPIN Workshop (SPIN 2001), vol. 2057 of Lecture Notes in Computer Science. Springer, Toronto, Canada, pp 272–287

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Alice Miller.

Additional information

Ryan Kirwan was supported by the Engineering and Physical Sciences Research Council [grant number EP/P504937/1].

Michael Butler

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kirwan, R., Miller, A. & Porr, B. Model checking learning agent systems using Promela with embedded C code and abstraction. Form Asp Comp 28, 1027–1056 (2016). https://doi.org/10.1007/s00165-016-0382-2

Download citation

Keywords

  • Model checking
  • abstraction
  • Agent
  • Simulation
  • Learning