Abstract
As autonomous systems become more prevalent, methods for their verification will become more widely used. Model checking is a formal verification technique that can help ensure the safety of autonomous systems, but in most cases it cannot be applied by novices, or in its straight “off-the-shelf” form. In order to be more widely applicable it is crucial that more sophisticated techniques are used, and are presented in a way that is reproducible by engineers and verifiers alike. In this paper we demonstrate in detail two techniques that are used to increase the power of model checking using the model checker Spin. The first of these is the use of embedded C code within Promela specifications, in order to accurately reflect robot movement. The second is to use abstraction together with a simulation relation to allow us to verify multiple environments simultaneously. We apply these techniques to a fairly simple system in which a robot moves about a fixed circular environment and learns to avoid obstacles. The learning algorithm is inspired by the way that insects learn to avoid obstacles in response to pain signals received from their antennae. Crucially, we prove that our abstraction is sound for our example system—a step that is often omitted but is vital if formal verification is to be widely accepted as a useful and meaningful approach.
Article PDF
Similar content being viewed by others
References
Alur R, Courcoubetis C, Halbwachs N, Henzinger T, Ho P-H, Nicollin X, Olivero A, Sifakis J, Yovine S (1995) The algorithmic analysis of hybrid systems. Theor Comput Sci 138:3–34
Alur R, Grosu R, Lee I, Sokolsky O (2001) Compositional refinement of hiererarchical hybrid systems. Vol. 2034 of Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Rome, pp 33–48.
Antuna L, Ilan D, CAmpos S, Eder K (2015) symmetry reduction enables model checking of more complex emergent behaviours of swarm navigation algorithms. In: Towards autonomous robotic systems, vol. 9287 of Lecture Notes in Computing Science. Springer, New York, pp 26–37
Brambilla M, Brutschy A, Dorigo M, Birattari M: Property-driven design for robot swarms: A design method based on prescriptive modeling and model checking. ACM Trans Auton Adapt Syst 9(4), 17–11728 (2014)
Braitenberg V: Vehicles: experiments in synthetic psychology. MIT Press, Colorado (1984)
Büchi J (1960) On a decision method in restricted second order arithmetic. In: Proceedings of the International Congress on Logic, Method, and Philosophy of Science. Stanford University Press, pp 1–12
Cattel T (1994) Modeling and verification of a multiprocessor realtime OS kernel. In: Hogrefe D, Leue S (eds) Proceedings of the 7th WG6.1 International Conference on Formal Description Techniques (FORTE ‘94), vol. 6 of International Federation For Information Processing. Chapman and Hall, Berne, Switzerland, pp 55–70
Cimatti A, Giunchiglia F, Mingardi G, Romano D, Torielli F, Traverso P (1997) Model checking safety critical software with SPIN: an application to a railway interlocking system. In: Langerak R (ed) Proceedings of the 3rd SPIN Workshop (SPIN‘97). Twente University, The Netherlands, pp. 5–17
Clarke E, Grumberg O, Peled D: Model checking. The MIT Press, Cambridge (1999)
Calder M, Miller A: An automatic abstraction technique for verifying featured, parameterised systems. Theor Comput Sci 404(3), 235–255 (2008)
Dill D (1996) The Murϕ verification system. In: Alur R, Henzinger T (eds) Proceedings of the 8th International Conference on Computer Aided Verification (CAV ‘96), vol. 1102 of Lecture Notes in Computer Science. Springer, New Brunswick, pp 390–393
Dixon C, Winfield A, Fisher M, Zeng C: Towards temporal verification of swarm robotic systems. Robot Automomous Syst 60, 1429–1441 (2012)
Fisher M, Dennis L, Webster M: Verification of autonomous systems. Commun ACM 56(9), 84–93 (2013)
Fainekos G, Girard A, Kress-Gazit H, Pappas G: Temporal logic motion planning for dynamic robots. Automatica 45(2), 343–352 (2009)
Guo M, Johansson K, Dimarogonas (2013) Revising motion planning under linear temporal logic specifications in partially known workspaces. In: IEEE international Conference Robotics and Automation
Gallardo M, Merino P, Salmerón A (2016) River basin management with SPIN. In: Proceedings of the 23rd International SPIN symposium on Model Checking of Software (SPIN 2016) (to appear)
Hoffman R, Ireland M, Miller A, Norman G, Veres S (2016) Autonomous agent behaviour modelled in PRISM. In: Proceedings of the 23rd International SPIN symposium on Model Checking of Software (SPIN 2016) (to appear)
Hinton A, Kwiatkowska M, Norman G, Parker D: Prism: a tool for automatic verification of probabilistic systems. LNCS 3920, 441–444 (2006)
Holzmann GJ (2003) Trends in software verification. In: Araki K, Gnesi S, Mandrioli D (eds) Proceedings of the International Symposium on Formal Methods Europe (FME 2006), vol. 2805 of Lecture Notes in Computer Science. Springer, Pisa, Italy, pp 40–50
Holzmann G (2004) The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Pearson Education
(2002) Proceedings of the 5th International Workshop on Hybrid Systems: Computation and Control (HSCC ’02), vol. 2289 of Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, Santa Barbara
Humphrey L (2013) Model checking for verification in uav cooperative control applications. Vol. 444 of Lecture Notes in Computer Science. Springer, New York, pp 69–117.
Jeyaraman S, Tsourdos A, Zbikowski R, White B (2006) Kripke modelling approaches of a multiple robots system with minimalist communication: a formal approach of choice. Int J Syst Sci 37(6):339–349
Kloetzer M, Belta C (2006) Hierarchical abstractions for robotic swarms. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2006), pp 952–957
Kloetzer M, Belta C: Temporal logic planning and control robotic swarms by heirarchical abstractions. IEEE Trans Robot 213(2), 320–330 (2007)
Konur S, Dixon C, Fisher M: Analysing robot swarm behaviour via probabilistic model checking. Robot Auton Syst 60(2), 199–213 (2012)
Kirwan RF (2014) Applying model checking to agent-based learning systems. Ph.D. thesis, University of Glasgow
Kulvicius T, Kolodziejski C, Tamosiunaite T, Porr B, Worgotter F: Behavioral analysis of differential Hebbian learning in closed-loop systems. Biol Cybern 103(4), 255–271 (2010)
Kumar S, Li K (2002) Using model checking to debug device firmware. In: Proceedings of the 5th Symposium on Operating System Design and Implementation (OSDI 2002). USENIX, Boston
Kouvaros P, Lomuscio A (2015) A counter abstraction technique for the verification of robot swarms. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin, pp 2081–2088
Kirwan R, Miller A, Porr B (2016). doi:10.5525/gla.researchdata.310
Kirwan R, Miller A, Porr B, Di Prodi P: Formal modelling of robot behaviour with learning. Neural Comput 25(11), 2976–3019 (2013)
Kripke S: Semantical considerations on modal logics. Acta Philos Fenn 16, 83–94 (1963)
McMillan K (1993) Symbolic model checking. Boston
Merz S (2000) Model checking: a tutorial overview. In: Cassez F Jard C, Rozoy B, Ryan M (eds) Modeling and verification of parallel processes, 4th Summer School, MOVEP, vol. 2067 of Lecture Notes in Computer Science. Springer, Nantes
Milner R (1971) An algebraic definition of simulation between programs. In: Proceedings of the 2nd International Joint Conference on Artificial Intelligence, pp 481–489
Müller-Olm M, Schmidt D, Steffen B (1999) Model-checking: a tutorial introduction. In: Cortesi A, File G (eds) Proceedings of the 6th International Static Analysis Symposium (SAS’99), vol. 1694 of Lecture Notes in Computer Science (LNCS). Springer, Venice, pp 330–354
Porr B, von Ferber C, Wörgötter F (2003) ISO-learning approximates a solution to the inverse-controller problem in an unsupervised behavioural paradigm. Neural Comput 15(4):865–884
Porr B, Wörgötter F: Strongly improved stability and faster convergence of temporal sequence learning by utilising input correlations only. Neural Comput 18(6), 1380–1412 (2006)
Sutton R, Barto A (1987) A temporal-difference model of classical conditioning. In: Proceedings of the Ninth Annual Conference of the Cognitive Science Society. Seattle, Washington, pp 355–378
Stocker R, Dennis L, Dixon C, Fisher M (2012) Verifiying Brahms human-robot teamwork models. In: Proceedings of the 13th European conference on logics in Artificial Intelligence (JELIA-2012), pp 385–397
Sierhuis M (2001) Modeling and simulating work practice. BRAHMS: A multi-agent modeling and simulation language for work system analysis and design. Ph.D. thesis, University of Amsterdam
Sharma O, Lewis J, Miller A, Dearle A, Balasubramaniam D, Morrison R, Sventek J (2009) Towards verifying correctness of wireless sensor network applications using insense and spin. In: Păsrăeanu C (ed) Proceedings of the 16th International SPIN Workshop (SPIN 2009), Lecture Notes in Computer Science. Springer, Grenoble, pp 223–240
Stauner T (2002) Discrete-time refinement of hybrid automata. In: Sta02 [hsc02], pp 407–420
Tiwari A, Khanna G (2002) Series of abstractions for hybrid automata. In: TiwKha02 [hsc02], pp 465–478
Weissman M, Bedenk S, Buckl C, Knoll A (2011) Model checking industrial robot systems. In: Groce A, Musuvathi M (eds) Proceedings of the 18th International SPIN Workshop (SPIN 2009), vol. 6823 of Lecture Notes in Computer Science. Springer, Snowbird, pp 161–176
Webster M, Dixon C, Fisher M, Salem M, Saunders J, Koay J, Dautenhahn K, Saez-Pons J: Toward reliable autonomous robotic assistants through formal verification: a case study. IEEE Trans Human Mach Syst 99, 1–11 (2015)
Yuen C, Tjioe W (2001) Modeling and verifying a price model for congestion control in computer networks using Promela/Spin. In: Dwyer MB (ed) Proceedings of the 8th International SPIN Workshop (SPIN 2001), vol. 2057 of Lecture Notes in Computer Science. Springer, Toronto, Canada, pp 272–287
Author information
Authors and Affiliations
Corresponding author
Additional information
Michael Butler
Ryan Kirwan was supported by the Engineering and Physical Sciences Research Council [grant number EP/P504937/1].
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Kirwan, R., Miller, A. & Porr, B. Model checking learning agent systems using Promela with embedded C code and abstraction. Form Asp Comp 28, 1027–1056 (2016). https://doi.org/10.1007/s00165-016-0382-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00165-016-0382-2