## Abstract

As autonomous systems become more prevalent, methods for their verification will become more widely used. Model checking is a formal verification technique that can help ensure the safety of autonomous systems, but in most cases it cannot be applied by novices, or in its straight “off-the-shelf” form. In order to be more widely applicable it is crucial that more sophisticated techniques are used, and are presented in a way that is reproducible by engineers and verifiers alike. In this paper we demonstrate in detail two techniques that are used to increase the power of model checking using the model checker Spin. The first of these is the use of embedded C code within Promela specifications, in order to accurately reflect robot movement. The second is to use abstraction together with a simulation relation to allow us to verify multiple environments simultaneously. We apply these techniques to a fairly simple system in which a robot moves about a fixed circular environment and learns to avoid obstacles. The learning algorithm is inspired by the way that insects learn to avoid obstacles in response to pain signals received from their antennae. Crucially, we prove that our abstraction is sound for our example system—a step that is often omitted but is vital if formal verification is to be widely accepted as a useful and meaningful approach.

## References

Alur R, Courcoubetis C, Halbwachs N, Henzinger T, Ho P-H, Nicollin X, Olivero A, Sifakis J, Yovine S (1995) The algorithmic analysis of hybrid systems. Theor Comput Sci 138:3–34

Alur R, Grosu R, Lee I, Sokolsky O (2001) Compositional refinement of hiererarchical hybrid systems. Vol. 2034 of Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Rome, pp 33–48.

Antuna L, Ilan D, CAmpos S, Eder K (2015) symmetry reduction enables model checking of more complex emergent behaviours of swarm navigation algorithms. In: Towards autonomous robotic systems, vol. 9287 of Lecture Notes in Computing Science. Springer, New York, pp 26–37

Brambilla M, Brutschy A, Dorigo M, Birattari M: Property-driven design for robot swarms: A design method based on prescriptive modeling and model checking. ACM Trans Auton Adapt Syst

**9**(4), 17–11728 (2014)Braitenberg V: Vehicles: experiments in synthetic psychology. MIT Press, Colorado (1984)

Büchi J (1960) On a decision method in restricted second order arithmetic. In: Proceedings of the International Congress on Logic, Method, and Philosophy of Science. Stanford University Press, pp 1–12

Cattel T (1994) Modeling and verification of a multiprocessor realtime OS kernel. In: Hogrefe D, Leue S (eds) Proceedings of the 7th WG6.1 International Conference on Formal Description Techniques (FORTE ‘94), vol. 6 of International Federation For Information Processing. Chapman and Hall, Berne, Switzerland, pp 55–70

Cimatti A, Giunchiglia F, Mingardi G, Romano D, Torielli F, Traverso P (1997) Model checking safety critical software with SPIN: an application to a railway interlocking system. In: Langerak R (ed) Proceedings of the 3rd SPIN Workshop (SPIN‘97). Twente University, The Netherlands, pp. 5–17

Clarke E, Grumberg O, Peled D: Model checking. The MIT Press, Cambridge (1999)

Calder M, Miller A: An automatic abstraction technique for verifying featured, parameterised systems. Theor Comput Sci

**404**(3), 235–255 (2008)Dill D (1996) The Murϕ verification system. In: Alur R, Henzinger T (eds) Proceedings of the 8th International Conference on Computer Aided Verification (CAV ‘96), vol. 1102 of Lecture Notes in Computer Science. Springer, New Brunswick, pp 390–393

Dixon C, Winfield A, Fisher M, Zeng C: Towards temporal verification of swarm robotic systems. Robot Automomous Syst

**60**, 1429–1441 (2012)Fisher M, Dennis L, Webster M: Verification of autonomous systems. Commun ACM

**56**(9), 84–93 (2013)Fainekos G, Girard A, Kress-Gazit H, Pappas G: Temporal logic motion planning for dynamic robots. Automatica

**45**(2), 343–352 (2009)Guo M, Johansson K, Dimarogonas (2013) Revising motion planning under linear temporal logic specifications in partially known workspaces. In: IEEE international Conference Robotics and Automation

Gallardo M, Merino P, Salmerón A (2016) River basin management with SPIN. In: Proceedings of the 23rd International SPIN symposium on Model Checking of Software (SPIN 2016) (to appear)

Hoffman R, Ireland M, Miller A, Norman G, Veres S (2016) Autonomous agent behaviour modelled in PRISM. In: Proceedings of the 23rd International SPIN symposium on Model Checking of Software (SPIN 2016) (to appear)

Hinton A, Kwiatkowska M, Norman G, Parker D: Prism: a tool for automatic verification of probabilistic systems. LNCS

**3920**, 441–444 (2006)Holzmann GJ (2003) Trends in software verification. In: Araki K, Gnesi S, Mandrioli D (eds) Proceedings of the International Symposium on Formal Methods Europe (FME 2006), vol. 2805 of Lecture Notes in Computer Science. Springer, Pisa, Italy, pp 40–50

Holzmann G (2004) The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Pearson Education

(2002) Proceedings of the 5th International Workshop on Hybrid Systems: Computation and Control (HSCC ’02), vol. 2289 of Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Santa Barbara

Humphrey L (2013) Model checking for verification in uav cooperative control applications. Vol. 444 of Lecture Notes in Computer Science. Springer, New York, pp 69–117.

Jeyaraman S, Tsourdos A, Zbikowski R, White B (2006) Kripke modelling approaches of a multiple robots system with minimalist communication: a formal approach of choice. Int J Syst Sci 37(6):339–349

Kloetzer M, Belta C (2006) Hierarchical abstractions for robotic swarms. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2006), pp 952–957

Kloetzer M, Belta C: Temporal logic planning and control robotic swarms by heirarchical abstractions. IEEE Trans Robot

**213**(2), 320–330 (2007)Konur S, Dixon C, Fisher M: Analysing robot swarm behaviour via probabilistic model checking. Robot Auton Syst

**60**(2), 199–213 (2012)Kirwan RF (2014) Applying model checking to agent-based learning systems. Ph.D. thesis, University of Glasgow

Kulvicius T, Kolodziejski C, Tamosiunaite T, Porr B, Worgotter F: Behavioral analysis of differential Hebbian learning in closed-loop systems. Biol Cybern

**103**(4), 255–271 (2010)Kumar S, Li K (2002) Using model checking to debug device firmware. In: Proceedings of the 5th Symposium on Operating System Design and Implementation (OSDI 2002). USENIX, Boston

Kouvaros P, Lomuscio A (2015) A counter abstraction technique for the verification of robot swarms. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, pp 2081–2088

Kirwan R, Miller A, Porr B (2016). doi:10.5525/gla.researchdata.310

Kirwan R, Miller A, Porr B, Di Prodi P: Formal modelling of robot behaviour with learning. Neural Comput

**25**(11), 2976–3019 (2013)Kripke S: Semantical considerations on modal logics. Acta Philos Fenn

**16**, 83–94 (1963)McMillan K (1993) Symbolic model checking. Boston

Merz S (2000) Model checking: a tutorial overview. In: Cassez F Jard C, Rozoy B, Ryan M (eds) Modeling and verification of parallel processes, 4th Summer School, MOVEP, vol. 2067 of Lecture Notes in Computer Science. Springer, Nantes

Milner R (1971) An algebraic definition of simulation between programs. In: Proceedings of the 2nd International Joint Conference on Artificial Intelligence, pp 481–489

Müller-Olm M, Schmidt D, Steffen B (1999) Model-checking: a tutorial introduction. In: Cortesi A, File G (eds) Proceedings of the 6th International Static Analysis Symposium (SAS’99), vol. 1694 of Lecture Notes in Computer Science (LNCS). Springer, Venice, pp 330–354

Porr B, von Ferber C, Wörgötter F (2003) ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioural paradigm. Neural Comput 15(4):865–884

Porr B, Wörgötter F: Strongly improved stability and faster convergence of temporal sequence learning by utilising input correlations only. Neural Comput

**18**(6), 1380–1412 (2006)Sutton R, Barto A (1987) A temporal-difference model of classical conditioning. In: Proceedings of the Ninth Annual Conference of the Cognitive Science Society. Seattle, Washington, pp 355–378

Stocker R, Dennis L, Dixon C, Fisher M (2012) Verifiying Brahms human-robot teamwork models. In: Proceedings of the 13th European conference on logics in Artificial Intelligence (JELIA-2012), pp 385–397

Sierhuis M (2001) Modeling and simulating work practice. BRAHMS: A multi-agent modeling and simulation language for work system analysis and design. Ph.D. thesis, University of Amsterdam

Sharma O, Lewis J, Miller A, Dearle A, Balasubramaniam D, Morrison R, Sventek J (2009) Towards verifying correctness of wireless sensor network applications using insense and spin. In: Păsrăeanu C (ed) Proceedings of the 16th International SPIN Workshop (SPIN 2009), Lecture Notes in Computer Science. Springer, Grenoble, pp 223–240

Stauner T (2002) Discrete-time refinement of hybrid automata. In: Sta02 [hsc02], pp 407–420

Tiwari A, Khanna G (2002) Series of abstractions for hybrid automata. In: TiwKha02 [hsc02], pp 465–478

Weissman M, Bedenk S, Buckl C, Knoll A (2011) Model checking industrial robot systems. In: Groce A, Musuvathi M (eds) Proceedings of the 18th International SPIN Workshop (SPIN 2009), vol. 6823 of Lecture Notes in Computer Science. Springer, Snowbird, pp 161–176

Webster M, Dixon C, Fisher M, Salem M, Saunders J, Koay J, Dautenhahn K, Saez-Pons J: Toward reliable autonomous robotic assistants through formal verification: a case study. IEEE Trans Human Mach Syst

**99**, 1–11 (2015)Yuen C, Tjioe W (2001) Modeling and verifying a price model for congestion control in computer networks using Promela/Spin. In: Dwyer MB (ed) Proceedings of the 8th International SPIN Workshop (SPIN 2001), vol. 2057 of Lecture Notes in Computer Science. Springer, Toronto, Canada, pp 272–287

## Author information

### Authors and Affiliations

### Corresponding author

## Additional information

Michael Butler

Ryan Kirwan was supported by the Engineering and Physical Sciences Research Council [grant number EP/P504937/1].

## Rights and permissions

**Open Access** This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## About this article

### Cite this article

Kirwan, R., Miller, A. & Porr, B. Model checking learning agent systems using Promela with embedded C code and abstraction.
*Form Asp Comp* **28**, 1027–1056 (2016). https://doi.org/10.1007/s00165-016-0382-2

Received:

Revised:

Accepted:

Published:

Issue Date:

DOI: https://doi.org/10.1007/s00165-016-0382-2

### Keywords

- Model checking
- abstraction
- Agent
- Simulation
- Learning