Investigating Safety of a Radiotherapy Machine Using System Models with Pluggable Checkers

  • Stuart PernsteinerEmail author
  • Calvin Loncaric
  • Emina Torlak
  • Zachary Tatlock
  • Xi Wang
  • Michael D. Ernst
  • Jonathan Jacky
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9780)


Formal techniques for guaranteeing software correctness have made tremendous progress in recent decades. However, applying these techniques to real-world safety-critical systems remains challenging in practice. Inspired by goals set out in prior work, we report on a large-scale case study that applies modern verification techniques to check safety properties of a radiotherapy system in current clinical use. Because of the diversity and complexity of the system’s components (software, hardware, and physical), no single tool was suitable for both checking critical component properties and ensuring that their composition implies critical system properties. This paper describes how we used state-of-the-art approaches to develop specialized tools for verifying safety properties of individual components, as well as an extensible tool for composing those properties to check the safety of the system as a whole. We describe the key design decisions that diverged from previous approaches and that enabled us to practically apply our approach to provide machine-checked guarantees. Our case study uncovered subtle safety-critical flaws in a pre-release of the latest version of the radiotherapy system’s control software.


Case study Safety-critical systems SMT-based verification Lightweight formal methods 



This material is based on research sponsored by DARPA under agreement numbers FA8750-12-2-0107, FA8750-15-C-0010, and FA8750-16-2-0032. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.


  1. 1.
    Alloy: a language and tool for relational models (2014).
  2. 2.
  3. 3.
    Safety case for CNTS prescription safety, August 2015.
  4. 4.
    Babic, D., Hu, A.J.: Calysto: scalable and precise extended static checking. In: ICSE (2008)Google Scholar
  5. 5.
    Barnett, M., Rustan, M., Leino, K., Schulte, W.: The Spec# programming system: an overview. In: Barthe, G., Burdy, L., Huisman, M., Lanet, J.-L., Muntean, T. (eds.) CASSIS 2004. LNCS, vol. 3362, pp. 49–69. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Beringer, L., Petcher, A., Ye, K.Q., Appel, A.W.: Verified correctness and security of openSSL HMAC. In: 24th USENIX Security Symposium (USENIX Security 2015), Washington, D.C., pp. 207–221 (2015). USENIX Association. ISBN:978-1-931971-232
  7. 7.
    Cadar, C., Dunbar, D., Engler, D.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th Symposium on Operating Systems Designand Implementation (OSDI), San Diego, CA, pp. 209–224, December 2008Google Scholar
  8. 8.
    Clarke, E., Kroning, D., Lerda, F.: A tool for checking ANSI-C programs. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 168–176. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Coq development team. Coq Reference Manual, Version 8.4pl5. INRIA, October 2014.
  10. 10.
    Cruanes, S., Hamon, G., Owre, S., Shankar, N.: Tool integration with the evidential tool bus. In: Giacobazzi, R., Berdine, J., Mastroeni, I. (eds.) VMCAI 2013. LNCS, vol. 7737, pp. 275–294. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  11. 11.
    de Moura, L., Bjørner, N.S.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Denney, E., Pai, G.: Evidence arguments for using formal methods in software certification. In: 2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 375–380, November 2013. doi: 10.1109/ISSREW.2013.6688924
  13. 13.
    Denney, E., Pai, G., Pohl, J.: AdvoCATE: an assurance case automation toolset. In: Ortmeier, F., Daniel, P. (eds.) SAFECOMP Workshops 2012. LNCS, vol. 7613, pp. 8–21. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33675-1_2 CrossRefGoogle Scholar
  14. 14.
    Dependability Research Group. Safety cases repository, February 2006.
  15. 15.
    Dolby, J., Vaziri, M., Tip, F.: Finding bugs efficiently with a SAT solver. In: FSE (2007)Google Scholar
  16. 16.
  17. 17.
  18. 18.
    Ernst, M.D., Grossman, D., Jacky, J., Loncaric, C., Pernsteiner, S., Tatlock, Z., Torlak, E., Wang, X.: Toward a dependability case language and workflow for a radiation therapy system. In: Summit on Advances in Programming Languages (SNAPL) (2015)Google Scholar
  19. 19.
    Gacek, A., Backes, J., Cofer, D.D., Slind, K., Whalen, M.: Resolute: an assurance case language for architecture models (2014). CoRR, abs/1409.4629.
  20. 20.
    Galeotti, J.P.: Software verification using alloy. Ph.D. thesis, University of Buenos Aires (2010)Google Scholar
  21. 21.
    Greenwell, W.S., Knight, J.C., Holloway, C.M., Pease, J.J.: A taxonomy of fallacies in system safety arguments. In: International System Safety Conference (2006)Google Scholar
  22. 22.
    Hawblitzel, C., Howell, J., Kapritsos, M., Lorch, J.R., Parno, B., Roberts, M.L., Setty, S., Zill, B.: Ironfleet: proving practical distributed systems correct. In: Proceedings of the 25th Symposium on Operating Systems Principles, SOSP 2015, pp. 1–17. ACM, New York (2015). doi: 10.1145/2815400.2815428, ISBN:978-1-4503-3834-9
  23. 23.
    Holzmann, G.J.: The power of 10: rules for developing safety-critical code. Computer 39(6), 95–97 (2006). doi: 10.1109/MC.2006.212. ISSN:0018-9162CrossRefGoogle Scholar
  24. 24.
    Holzmann, G.J.: Mars code. Commun. ACM 57(2), 64–73 (2014). doi: 10.1145/2560217.2560218. ISSN:0001-0782CrossRefGoogle Scholar
  25. 25.
    Jackson, D.: A direct path to dependable software. Commun. ACM 52(4), 78–88 (2009)CrossRefGoogle Scholar
  26. 26.
    Jackson, D.: Software Abstractions: Logic, Language, and Analysis. MIT Press, Cambridge (2012)Google Scholar
  27. 27.
    Jackson, D., Kang, E.: Property-part diagrams: a dependence notation for software systems. Technical report, Massachusetts Institute of Technology (2009).
  28. 28.
    Jackson, D., Thomas, M.: Software for Dependable Systems: Sufficient Evidence?. National Academy Press, Washington, D.C. (2007). ISBN:0309103940, 9780309103947Google Scholar
  29. 29.
    Jacky, J.: The Clinical Neutron Therapy System.
  30. 30.
    Jacky, J.: Formal safety analysis of the control program for a radiation therapy machine. In: Schlegel, W., Bortfeld, T. (eds.) The Use of Computers in Radiation Therapy, pp. 68–70. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  31. 31.
    Jacky, J.: EPICS-based control system for a radiation therapy machine. In: International Conference on Accelerator and Large Experimental Physics Control Systems (ICALEPCS) (2013)Google Scholar
  32. 32.
    Jacky, J., Risler, R.: Clinical neutron therapy system therapist’s guide. Technical report 99–07-01, University of Washington, Department of Radiation Oncology (2002)Google Scholar
  33. 33.
    Jacky, J., Risler, R.: Clinical neutron therapy system reference manual. Technical report 99–10-01, University of Washington, Department of Radiation Oncology (2002)Google Scholar
  34. 34.
    Jacky, J., Risler, R., Kalet, I., Wootton, P.: Clinical neutron therapy system control system specification part I. Technical report 90–12-01, University of Washington, Department of Radiation Oncology (1990)Google Scholar
  35. 35.
    Jacky, J., Risler, R., Reid, D., Emery, R., Unger, J., Patrick, M.: A control system for a radiation therapy machine. Technical report 2001–05-01, University of Washington, Department of Radiation Oncology (2001)Google Scholar
  36. 36.
    John, K.H., Tiegelkamp, M.: IEC 61131–3: Programming Industrial Automation Systems: Concepts and Programming Languages, Requirements for Programming Systems Decision-Making Aids, 2nd edn. Springer Publishing Company, Incorporated, New York (2010). ISBN:3642120148, 9783642120145zbMATHGoogle Scholar
  37. 37.
    Kelly, T., Weaver, R.: The goal structuring notation - a safety argument notation. In: Proceedings of the Dependable Systems and Networks 2004 Workshop on Assurance Cases, July 2004Google Scholar
  38. 38.
    Lyu, M.R. (ed.): Handbook of Software Reliability Engineering. McGraw-Hill Inc., Hightstown (1996)Google Scholar
  39. 39.
    Near, J.P., Milicevic, A., Kang, E., Jackson, D.: A lightweight code analysis and its role in evaluation of a dependability case. In: Proceedings of the 33rd International Conference of Computer Safety, Reliability and Security, Waikiki, Honolulu, HI, pp. 31–40, May 2011Google Scholar
  40. 40.
    Nipkow, T., Paulson, L.C., Wenzel, M.: Isabelle/HOL – A Proof Assistant for Higher-Order Logic. LNCS, vol. 2283. Springer, Heidelberg (2002)zbMATHGoogle Scholar
  41. 41.
    Owre, S., Rushby, J.M., Shankar, N.: PVS: a prototype verification system. In: Kapur, D. (ed.) CADE 1992. LNCS(LNAI), vol. 607, pp. 748–752. Springer, Heidelberg (1992)Google Scholar
  42. 42.
    Rushby, J.: Formalism in safety cases. In: Safety-Critical Systems Symposium (2010)Google Scholar
  43. 43.
    Rushby, J.: Mechanized support for assurance case argumentation. In: Nakano, Y., Satoh, K., Bekki, D. (eds.) JSAI-isAI 2013. LNCS, vol. 8417, pp. 304–318. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10061-6_20 Google Scholar
  44. 44.
    Torlak, E.: Constraint solver for software engineering: finding models and cores of large relational specifications. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge (2009). AAI0821754Google Scholar
  45. 45.
    Torlak, E., Bodik, R.: Growing solver-aided languages with Rosette. In: Proceedings of the 2013 ACM International Symposium on New Ideas. New Paradigms, and Reflections on Programming & Software, Onward! 2013, Indianapolis, IN, pp. 135–152 (2013)Google Scholar
  46. 46.
    Torlak, E., Bodik, R.: A lightweight symbolic virtual machine for solver-aided host languages. In: Proceedings of the 2014 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), Edinburgh, UK, pp. 530–541, June 2014Google Scholar
  47. 47.
    Torlak, E., Jackson, D.: Kodkod: a relational model finder. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 632–647. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  48. 48.
    Weinstock, C., Goodenough, J., Hudak, J.: Dependability cases. Technical report CMU/SEI-2004-TN-016, Software Engineering Institute, Carnegie Mellon University, Pittsburgh (2004)Google Scholar
  49. 49.
    Xie, Y., Aiken, A.: Saturn: a scalable framework for error detection using Boolean satisfiability. ACM Trans. Program. Lang. Syst. (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Stuart Pernsteiner
    • 1
    Email author
  • Calvin Loncaric
    • 1
  • Emina Torlak
    • 1
  • Zachary Tatlock
    • 1
  • Xi Wang
    • 1
  • Michael D. Ernst
    • 1
  • Jonathan Jacky
    • 2
  1. 1.Department of Computer ScienceUniversity of WashingtonSeattleUSA
  2. 2.Department of Radiation OncologyUniversity of WashingtonSeattleUSA

Personalised recommendations