Skip to main content
SpringerLink
Log in
Menu
Find a journal Publish with us
Search
Cart
  1. Home
  2. Formal Aspects of Computing
  3. Article

RiskStructures: A design algebra for risk-aware machines

  • Original Article
  • Open access
  • Published: 26 May 2021
  • volume 33, pages 763–802 (2021)
Download PDF

You have full access to this open access article

Formal Aspects of Computing
RiskStructures: A design algebra for risk-aware machines
Download PDF
  • Mario Gleirscher  ORCID: orcid.org/0000-0002-9445-68631,
  • Radu Calinescu  ORCID: orcid.org/0000-0002-2678-92601 &
  • Jim Woodcock  ORCID: orcid.org/0000-0001-7955-27021 

  • 391 Accesses

  • 4 Citations

  • Explore all metrics

Cite this article

Abstract

Machines, such as mobile robots and delivery drones, incorporate controllers responsible for a task while handling risk (e.g. anticipating and mitigating hazards; preventing and alleviating accidents). We refer to machines with this capability as risk-awaremachines. Risk awareness includes robustness and resilience and complicates monitoring (i.e., introspection, sensing, prediction), decision making, and control. From an engineering perspective, risk awareness adds a range of dependability requirements to system assurance. Such assurance mandates a correct-by-construction approach to controller design, based on mathematical theory.We introduce RiskStructures, an algebraic framework for risk modelling intended to support the design of safety controllers for risk-aware machines. Using the concept of a risk factor as a modelling primitive, this framework provides facilities to construct, examine, and assure these controllers.We prove desirable algebraic properties of these facilities, and demonstrate their applicability by using them to specify key aspects of safety controllers for risk-aware automated driving and collaborative robots.

Article PDF

Download to read the full article text

Use our pre-submission checklist

Avoid common mistakes on your manuscript.

References

  1. Alami R, Albu-Schäffer A, Bicchi A, Bischoff R, Chatila R, De Luca A, De Santis A, Giralt G, Guiochet J, Hirzinger G, Ingrand F, Lippiello V, Mattone R, Powell D, Sen S, Siciliano B, Tonietti G, Villani L (2006) Safe and dependable physical human-robot interaction in anthropic domains: State of the art and challenges. In: Intelligent robots and systems (IROS), IEEE/RSJ international conference

  2. Althoff, D., Kuffner, J.J., Wollherr, D., Buss, M.: Safety assessment of robot trajectories for navigation in uncertainand dynamic environments. Auton Robots 32(3), 285–302 (2011)

    Article  Google Scholar 

  3. Avizienis, A., Laprie, J.-C., Randell, B., Landwehr, C.: Basic concepts and taxonomy of dependable and secure computing. Dependable Secure Comput IEEE Trans 1(1), 11–33 (2004)

    Article  Google Scholar 

  4. Ajoudani, A., Zanchettin, A.M., Ivaldi, S., Albu-Schäffer, A., Kosuge, K., Khatib, O.: Progress and prospects of the human-robot collaboration. Auton Robots 42(5), 957–975 (2017)

    Article  Google Scholar 

  5. Bogdiukiewicz C, Butler M, Hoang TS, Paxton M, Snook J, Waldron X, Wilkinson T (2017) Formal development of policing functions for intelligent systems. In: Software Reliability Engineering (ISSRE), 28th IEEE international symposium

  6. Boudali H, Crouzen P, Stoelinga M (2007) Dynamic fault tree analysis using input/output interactive Markov chains. In: Dependable systems and networks (DSN), 37th annual IEEE/IFIP international conference, pp 708–717

  7. Beer, M., Ferson, S., Kreinovich, V.: Imprecise probabilities in engineering analyses. Mech Syst Signal Process 37(1–2), 4–29 (2013)

    Article  Google Scholar 

  8. Birolini A (2017) Reliability Engineering. Springer, 8th edition

  9. Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press (2008)

    MATH  Google Scholar 

  10. Broy, M., Stølen, K.: Specification and Development of nteractive Systems. Springer (2001)

    Book  MATH  Google Scholar 

  11. Chen, C., Liu, X., Chen, H.-H., Li, M., Zhao, L.: A rear-end collision risk evaluation and control scheme using a Bayesian network model. IEEE Trans Intell Transp Syst 20(1), 264–284 (2019)

    Article  Google Scholar 

  12. Ericson CA (2015) Hazard analysis techniques for system safety. Wiley, 2nd edition

  13. Fraichard, T., Asama, H.: Inevitable collision states - a step towards safer robots? Adv Robotics 18(10), 1001–1024 (2004)

    Article  Google Scholar 

  14. Foster, S., Baxter, J., Cavalcanti, A., Woodcock, J., Zeyda, F.: Unifying semantic foundations for automated verification tools in Isabelle/UTP. Sci Comput Program 197, (2020)

    Article  Google Scholar 

  15. Feyzabadi S, Carpin S (2014) Risk-aware path planning using hierarchical constrained Markov decision processes. In: Automation science and engineering (CASE), IEEE international conference IEEE

  16. Foster S, Gleirscher M, Calinescu R (2020) Towards deductive verification of control algorithms for autonomous marine vehicles. In: Engineering of complex computer systems (ICECCS), 25th international conference, pp 113–118, Singapore

  17. Foot P (1978) The problem of abortion and the doctrine of the double effect. Virtues and Vices and Other Essays in Moral Philosopy, 19. Originally published in 1967

  18. Gleirscher M, Carlan C (2017) Arguing from hazard analysis in safety cases: A modular argument pattern. In: High Assurance Systems Engineering (HASE), 18th international symposium, pp 53–60. IEEE

  19. Gleirscher M, Calinescu R (2020) Safety controller synthesis for collaborative robots. In: Engineering of complex computer systems (ICECCS), 25th international conference, pp 83–92, Singapore

  20. Gleirscher M, Foster S, Woodcock J (2019) New opportunities for integrated formal methods. ACM Comput Surv, 52:117:1–117:36

  21. Gleirscher, M., Kugele, S.: From hazard analysis to hazard mitigation planning: The automated driving case. In: Barrett, C., et al. (eds.) NASA Formal Methods (NFM), 9th international symposium, volume 10227 of LNCS. Springer (2017)

    Google Scholar 

  22. Gleirscher M (2014) Behavioral Safety of Technical Systems. Dissertation, Technische Universität München

  23. Gleirscher, M.: Run-time risk mitigation in automated vehicles: A model for studying preparatory steps. In: Bulwahn, L., Kamali, M., Linker, S. (eds.) Formal Verification of Autonomous Vehicles (FVAV), 1st iFM Workshop. EPTCS (2017)

    Google Scholar 

  24. Gleirscher M (2018) Strukturen für die Gefahrenerkennung und -behandlung in autonomen Maschinen. In: Jürgen B, Petra W (eds), Beiträge zu einer Systemtheorie Sicherheit, acatech DISKUSSION, Chapter 8.4, pp 154–167. Herbert Utz Verlag, München

  25. Gleirscher, M.: Yap: Tool support for deriving safety controllers from hazard analysis and risk assessments. In: Luckuck, M., Farrell, M. (eds.) Formal Methods for Autonomous Systems (FMAS), 2nd Workshop, volume 329 of EPTCS, pp 31–47. Open Publishing Association (2020)

    Google Scholar 

  26. Gleirscher M (2021) Yap Against Perils: Application Guide and User's Manual. University of York and Technical University of Munich

  27. Gleirscher, M., Marmsoler, D.: Formal methods in dependable systems engineering: A survey of professionals from Europe and North America. Empir Softw Eng 25(6), 4473–4546 (2020)

    Article  Google Scholar 

  28. Guiochet J, Martin-Guillerez D, Powell D (2010) Experience with model-based user-centered risk assessment for service robots. In: High Assurance Systems Engineering (HASE), 12th IEEE international symposium

  29. Guiochet J, Powell D, Baudin É, Blanquart J-P(2008) Online safety monitoring using safety modes. In: Technical challenges for dependable robots in human environments, workshop, pp 1–13. Rapport LAAS No. 08339

  30. Harpwood V (2000) Principles of Tort Law. Cavendish, 4th edn

  31. Haddadin, S., Albu-Schäffer, A., Hirzinger, G.: Requirements for safe robots: Measurements, analysis and new insights. Int J Robot Res 28(11–12), 1507–1527 (2009)

    Article  Google Scholar 

  32. Hamdi M, Boudriga N (2003) Algebraic specification of network security risk management. In: ACM workshop on formal methods in security engineering (FMSE). ACM Press

  33. Huang J, Erdogan C, Zhang Y, Moore B, Luo Q, Sundaresan A, Rosu G (2014) ROSRV: Runtime verification for robots. In: Runtime verification, pp 247–254. Springer

  34. Holland, O., Goodman, R.: Robots with internal models: A route to machine consciousness? J Conscious Stud 10(4–5), 77–109 (2003)

    Google Scholar 

  35. Howe, R.D., Matsuoka, Y.: Robotics for surgery. Ann Rev BiomedEng 1(1), 211–240 (1999)

    Google Scholar 

  36. Hoare T (1985) Communicating sequential processes. International series in computer science, Prentice-Hall

  37. Hansen, K.M., Ravn, A.P., Stavridou, V.: From safety analysis to software requirement. IEEE Trans Softw Eng 24(7), 573–84 (1998)

    Article  Google Scholar 

  38. Iamsumang, C., Mosleh, A., Modarres, M.: Monitoring and learning algorithms for dynamic hybrid Bayesian network in on-line system health management applications. Reliab Eng Syst Safety 178, 118–129 (2018)

    Article  Google Scholar 

  39. Ishibuchi, H., Tanaka, H.: Multiobjective programming in optimization of the interval objective function. Eur J Oper Res 48(2), 219–225 (1990)

    Article  MATH  Google Scholar 

  40. Kaplan, S., Garrick, B.J.: On the quantitative definition of risk. Risk Anal 1(1), 11–27 (1981)

    Article  Google Scholar 

  41. Kumamoto, H.: Satisfying safety goals by probabilistic risk assessment. Springer, Reliability engineering (2007)

    MATH  Google Scholar 

  42. Koopman, P., Wagner, M.: Autonomous vehicle safety: An interdisciplinary challenge. IEEE Int Transp Syst Mag 9(1), 90–96 (2017)

    Article  Google Scholar 

  43. Leveson, N.G.: Safeware: system safety and computers. Addison-Wesley (1995)

    Google Scholar 

  44. Leveson, N.G.: A new accident model for engineering safer systems. Safety Sci 42(4), 237–70 (2004)

    Article  Google Scholar 

  45. Leveson, N.G.: Engineering a safer world: systems thinking applied to safety. MIT Press, Engineering systems (2012)

    Book  Google Scholar 

  46. Leitner-Fischer, F., Leue, S.: Probabilistic fault tree synthesis using causality computation. Int J Critical Comput-Based Syst 4(2), 119–43 (2013)

    Article  Google Scholar 

  47. Leveson, N.G., Stolzy, J.L.: Safety analysis using Petri nets. IEEE Trans Softw Eng 13(3), 386–97 (1987)

    Article  Google Scholar 

  48. Leucker, M., Schallhart, C.: A brief account of runtime verification. J Logic Algeb Program 78(5), 293–303 (2009)

    Article  MATH  Google Scholar 

  49. Lund MS, Solhaug B, Stølen K (2011) Model-driven risk analysis: The CORAS approach. Springer

  50. McDermid John, A.: Support for safety cases and safety arguments using SAM. Reliability Engineering & System Safety 43(2), 111–127 (1994)

    Article  Google Scholar 

  51. Machin, M., Guiochet, J., Waeselynck, H., Blanquart, J.-P., Roy, M., Masson, L.: SMOF - a safety monitoring framework for autonomous systems. IEEE Trans Syst Man Cybern: Syst 48(5), 702–715 (2018)

    Article  Google Scholar 

  52. Meredith, P.O., Jin, D., Griffith, D., Chen, F., Roşu, G.: An overview of the MOP runtime verification framework. Int J Softw Tools Technol Trans 14(3), 249–289 (2011)

    Article  Google Scholar 

  53. Mekki-Mokhtar A, Blanquart J-P, Guiochet J, Powell D, Roy M (2012) Safety trigger conditions for critical autonomous systems. In: Dependable Computing (PRDC), 18th IEEE Pacific Rim International symposium. IEEE

  54. Müller J, Sukhatme GS (2014) Risk-aware trajectory generation with application to safe quadrotor landing. In: Intelligent Robots and Systems (IROS), IEEE/RSJ International conference

  55. Netravali AN, Sabnani KK, Viswanathan R (2003) Correct passive testing algorithms and complete fault coverage. In: Formal techniques for networked and distributed systems (FORTE), pp 303–318. Springer

  56. Oliveira, M., Cavalcanti, A., Woodcock, J.: A UTP semantics for Circus. Formal Aspects Comput 21(1–2), 3–32 (2009)

    Article  MATH  Google Scholar 

  57. Ortmeier, F., Reif, W., Schellhorn, G.: Deductive cause-consequence analysis (DCCA). IFAC Proc 38(1), 62–67 (2006)

    Article  Google Scholar 

  58. Pereira, A.A., Binney, J., Hollinger, G.A., Sukhatme, G.S.: Risk-aware path planning for autonomous underwater vehicles using predictive ocean models. J Field Robot 30(5), 741–762 (2013)

    Article  Google Scholar 

  59. Roscoe, A.W.: Understanding concurrent systems. Springer (2010)

    Book  MATH  Google Scholar 

  60. Sanger, T.D.: Risk-aware control. Neural Comput 26(12), 2669–2691 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  61. Sobek, R.P., Chatila, R.G.: Integrated planning and execution control for an autonomous mobile robot. Artif Intell Eng 3(2), 103–113 (1988)

    Article  Google Scholar 

  62. Schneider, S.: Concurrent and real-time systems: the CSP approach. Wiley, New York (1999)

    Google Scholar 

  63. Sheridan, T.B.: Telerobotics, automation, and human supervisory control. The MIT Press (2003)

    Google Scholar 

  64. Simmons, R.G.: Structured control for autonomous robots. IEEE Trans Robot Autom 10(1), 34–43 (1994)

    Article  Google Scholar 

  65. Sorin, A., Larsen, M., Jensen, K., Schultz, U.P.: Rule-based dynamic safety monitoring for mobile robots. J Softw Eng Robot 7(1), 120–141 (2016)

    Google Scholar 

  66. Svedung, I., Rasmussen, J.: Graphic representation of accident scenarios: mapping system structure and the causation of accidents. Safety Sci 40(5), 397–417 (2002)

    Article  Google Scholar 

  67. Shalev-Shwartz, S., Shammah, S., Shashua, A.: On a formal model of safe and scalable self-driving cars. Technical report, Mobileye (2018)

    Google Scholar 

  68. Tretmans J (2008) Model based testing with labelled transition systems. In: Formal methods and testing, pp 1–38. Springer

  69. United Nations General Assembly (1948) Universal declaration of human rights. Technical report research 217 A (III), United Nations General Assembly. [hereinafter ``UDHR'']

  70. Unanue JIA, Papadopoulos Y, Merle G (2018) Explicit modelling and treatment of repair in prediction of dependability. IEEE Trans Depend Secure Comput, pp 1–16

  71. Volk M, Junges S, Katoen J-P (2016) Advancing dynamic fault tree analysis – get succinct state spaces fast and synthesise failure rates. In: Computer safety, reliability, and security (SAFECOMP), 35th international conference, pp 253–265

  72. Warburton, N.: Philosophy: the basics. Taylor & Francis (2012)

    Google Scholar 

Download references

Acknowledgements

Mario Gleirscher was supported in part by the German Research Foundation (DFG) under the Fellowship Grant no. 381212925. Work by Radu Calinescu and Mario Gleirscher was partially supported by the Lloyd's Register Foundation under the Autonomy Assurance International Programme (AAIP) Grant CSI:Cobot. Radu Calinescu was additionally supported by the UKRI Project EP/V026747/1 "Trustworthy Autonomous Systems Node in Resilience". We would like to thank Simon Foster for inspiring discussions on the use of relational specification; Ana Cavalcanti and Cliff Jones for insightful questions about the abstraction, composition, and methodology underlying RiskStructures; James Baxter, AlvaroMiyazawa, and Pedro Ribeiro for enlightening conversations about CSP. We are also thankful to Sam Clark for helpful feedback on an early version of the introductory and closing sections.

Author information

Authors and Affiliations

  1. Computer Science Department, University of York, York, UK

    Mario Gleirscher, Radu Calinescu & Jim Woodcock

Authors
  1. Mario Gleirscher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Radu Calinescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jim Woodcock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mario Gleirscher.

Additional information

Cliff Jones

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Mario Gleirscher was supported in part by the German Research Foundation (DFG) under the Fellowship Grant no. 381212925.

Work by Radu Calinescu and Mario Gleirscher was partially supported by the Lloyd's Register Foundation under the Autonomy Assurance International Programme (AAIP) Grant CSI:Cobot.

Radu Calinescu was additionally supported by the UKRI Project EP/V026747/1 "Trustworthy Autonomous Systems Node in Resilience".

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gleirscher, M., Calinescu, R. & Woodcock, J. RiskStructures: A design algebra for risk-aware machines. Form Asp Comp 33, 763–802 (2021). https://doi.org/10.1007/s00165-021-00545-4

Download citation

  • Received: 15 September 2020

  • Revised: 08 March 2021

  • Accepted: 13 March 2021

  • Published: 26 May 2021

  • Issue Date: August 2021

  • DOI: https://doi.org/10.1007/s00165-021-00545-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

  • Correct construction
  • Formal development
  • Risk awareness
  • Run-time mitigation
  • Safety controllers
  • Robots and autonomous systems
Use our pre-submission checklist

Avoid common mistakes on your manuscript.

Advertisement

search

Navigation

  • Find a journal
  • Publish with us

Discover content

  • Journals A-Z
  • Books A-Z

Publish with us

  • Publish your research
  • Open access publishing

Products and services

  • Our products
  • Librarians
  • Societies
  • Partners and advertisers

Our imprints

  • Springer
  • Nature Portfolio
  • BMC
  • Palgrave Macmillan
  • Apress
  • Your US state privacy rights
  • Accessibility statement
  • Terms and conditions
  • Privacy policy
  • Help and support

Not affiliated

Springer Nature

© 2023 Springer Nature