Skip to main content

TraceVis: Towards Visualization for Deep Statistical Model Checking

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 12479)

Abstract

With the proliferation of neural networks (NN), the need to analyze, and ideally verify, their behavior becomes more and more pressing. Significant progress has been made in the analysis of individual NN decision episodes, but the verification of NNs as part of larger systems remains a grand challenge. Deep statistical model checking (DSMC) is a recent approach addressing that challenge in the context of Markov decision processes (MDP) where a NN represents a policy taking action decisions. The NN determinizes the MDP, resulting in a Markov chain which is analyzed by statistical model checking. Initial results in a Racetrack case study (a simple abstract encoding of driving control) suggest that such a DSMC analysis can be useful for quality assurance in system approval or certification.

Here we explore the use of visualization to support DSMC users (human analysts, domain engineers). We implement an interactive visualization tool, TraceVis, for the Racetrack case study. The tool allows to explore crash probabilities into particular wall segments as a function of start position and velocity. It furthermore supports the in-depth examination of the policy traces generated by DSMC, in aggregate form as well as individually. This demonstrates how visualization can foster the effective analysis of DSMC results, and it forms a first step in combining model checking and visualization in the analysis of NN behavior.

Keywords

  • Statistical Model Checking
  • Neural Networks
  • Visualization

Authors are listed alphabetically.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-83723-5_3
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-83723-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.

References

  1. Barto, A.G., Bradtke, S.J., Singh, S.P.: Learning to act using real-time dynamic programming. Artif. Intell. 72(1–2), 81–138 (1995)

    CrossRef  Google Scholar 

  2. Bogdoll, J., Ferrer Fioriti, L.M., Hartmanns, A., Hermanns, H.: Partial order methods for statistical model checking and simulation. In: Bruni, R., Dingel, J. (eds.) FMOODS/FORTE 2011. LNCS, vol. 6722, pp. 59–74. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21461-5_4

    CrossRef  Google Scholar 

  3. Bonet, B., Geffner, H.: Labeled RTDP: improving the convergence of real-time dynamic programming. In: ICAPS, pp. 12–21 (2003)

    Google Scholar 

  4. Budde, C.E., D’Argenio, P.R., Hartmanns, A., Sedwards, S.: A statistical model checker for nondeterminism and rare events. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10806, pp. 340–358. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89963-3_20

    CrossRef  Google Scholar 

  5. Budde, C.E., Dehnert, C., Hahn, E.M., Hartmanns, A., Junges, S., Turrini, A.: JANI: quantitative model and tool interaction. In: Legay, A., Margaria, T. (eds.) TACAS 2017. LNCS, vol. 10206, pp. 151–168. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54580-5_9

    CrossRef  Google Scholar 

  6. Croce, F., Andriushchenko, M., Hein, M.: Provable robustness of ReLU networks via maximization of linear regions. In: AISTATS, PMLR 89, pp. 2057–2066 (2019)

    Google Scholar 

  7. Dehnert, C., Junges, S., Katoen, J.-P., Volk, M.: A storm is coming: a modern probabilistic model checker. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10427, pp. 592–600. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63390-9_31

    CrossRef  Google Scholar 

  8. Ehlers, R.: Formal verification of piece-wise linear feed-forward neural networks. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 269–286. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68167-2_19

    CrossRef  Google Scholar 

  9. Etienne, L., Devogele, T., Buchin, M., McArdle, G.: Trajectory Box Plot: a new pattern to summarize movements. Int. J. Geograph. Inf. Sci. 30(5), 835–853 (2016). https://doi.org/10.1080/13658816.2015.1081205

  10. Gardner, M.: Mathematical games. Sci. Am. 229, 118–121 (1973)

    CrossRef  Google Scholar 

  11. Gardner, M., Dorling, S.: Artificial neural networks (the multilayer perceptron)–a review of applications in the atmospheric sciences. Atmos. Environ. 32(14), 2627–2636 (1998)

    CrossRef  Google Scholar 

  12. Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.T.: AI2: safety and robustness certification of neural networks with abstract interpretation. In: IEEE Symposium on Security and Privacy 2018, pp. 3–18 (2018)

    Google Scholar 

  13. Gros, T.P., Groß, D., Gumhold, S., Hoffmann, J., Klauck, M., Steinmetz, M.: TraceVis: Visualization for DSMC: tool, demonstration video, data (2020). https://doi.org/10.5281/zenodo.3961196

  14. Gros, T.P., Hermanns, H., Hoffmann, J., Klauck, M., Steinmetz, M.: Deep statistical model checking. In: Gotsman, A., Sokolova, A. (eds.) FORTE 2020. LNCS, vol. 12136, pp. 96–114. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50086-3_6

    CrossRef  Google Scholar 

  15. Gros, T.P., Höller, D., Hoffmann, J., Wolf, V.: Tracking the race between deep reinforcement learning and imitation learning. In: Gribaudo M., Jansen, D.N., Remke, A. (eds.) Proceedings of the 17th International Conference on Quantitative Evaluation of SysTems (QEST). Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59854-9

  16. Gumhold, S.: The computer graphics and visualization framework. https://github.com/sgumhold/cgv. Accessed 18 May 2020

  17. Gumhold, S.: Splatting illuminated ellipsoids with depth correction. In: Ertl, T. (ed.) Proceedings of the Vision, Modeling, and Visualization Conference 2003 (VMV 2003), München, Germany, 19–21 November 2003, pp. 245–252. Aka GmbH (2003)

    Google Scholar 

  18. Hahn, E.M., Li, Y., Schewe, S., Turrini, A., Zhang, L.: iscasMc: a web-based probabilistic model checker. In: Jones, C., Pihlajasaari, P., Sun, J. (eds.) FM 2014. LNCS, vol. 8442, pp. 312–317. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06410-9_22

    CrossRef  Google Scholar 

  19. Hartmanns, A., Hermanns, H.: The modest toolset: an integrated environment for quantitative modelling and verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54862-8_51

    CrossRef  Google Scholar 

  20. Hérault, T., Lassaigne, R., Magniette, F., Peyronnet, S.: Approximate probabilistic model checking. In: Steffen, B., Levi, G. (eds.) VMCAI 2004. LNCS, vol. 2937, pp. 73–84. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24622-0_8

    CrossRef  MATH  Google Scholar 

  21. Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)

    CrossRef  Google Scholar 

  22. Hohman, F., Kahng, M., Pienta, R., Chau, D.H.: Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers. arXiv:1801.06889 [cs, stat], May 2018

  23. Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_1

    CrossRef  Google Scholar 

  24. The JANI specification. http://www.jani-spec.org/. Accessed 28 Feb 2020

  25. Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5

    CrossRef  Google Scholar 

  26. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)

    Google Scholar 

  27. Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_47

    CrossRef  Google Scholar 

  28. Kwiatkowska, M., Norman, G., Parker, D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) SFM 2007. LNCS, vol. 4486, pp. 220–270. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72522-0_6

    CrossRef  Google Scholar 

  29. Li, J., Liu, J., Yang, P., Chen, L., Huang, X., Zhang, L.: Analyzing deep neural networks with symbolic propagation: towards higher precision and faster verification. In: Chang, B.-Y.E. (ed.) SAS 2019. LNCS, vol. 11822, pp. 296–319. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32304-2_15

    CrossRef  Google Scholar 

  30. McMahan, H.B., Gordon, G.J.: Fast exact planning in Markov decision processes. In: ICAPS, pp. 151–160 (2005)

    Google Scholar 

  31. Mirzargar, M., Whitaker, R.T., Kirby, R.M.: Curve Boxplot: generalization of boxplot for ensembles of curves. IEEE Trans. Vis. Comput. Graph. 20(12), 2654–2663 (2014). https://doi.org/10.1109/TVCG.2014.2346455. Conference Name: IEEE Transactions on Visualization and Computer Graphics

  32. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    CrossRef  Google Scholar 

  33. Pineda, L.E., Lu, Y., Zilberstein, S., Goldman, C.V.: Fault-tolerant planning under uncertainty. In: IJCAI, pp. 2350–2356 (2013)

    Google Scholar 

  34. Pineda, L.E., Zilberstein, S.: Planning under uncertainty using reduced models: revisiting determinization. In: ICAPS, pp. 217–225 (2014)

    Google Scholar 

  35. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Google Scholar 

  36. Silver, D., et al.: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362(6419), 1140–1144 (2018)

    MathSciNet  CrossRef  Google Scholar 

  37. Stoll, C., Gumhold, S., Seidel, H.P.: Incremental raycasting of piecewise quadratic surfaces on the GPU. In: 2006 IEEE Symposium on Interactive Ray Tracing, pp. 141–150. IEEE. https://doi.org/10.1109/RT.2006.280225. http://ieeexplore.ieee.org/document/4061556/

  38. Tominski, C., Schumann, H., Andrienko, G., Andrienko, N.: Stacking-based visualization of trajectory attribute data. IEEE Trans. Vis. Comput. Graph. 18(12), 2565–2574 (2012). https://doi.org/10.1109/TVCG.2012.265. Conference Name: IEEE Transactions on Visualization and Computer Graphics

  39. Tukey, J.W.: Mathematics and the picturing of data. In: Proceedings of the International Congress of Mathematicians, Vancouver, 1975, vol. 2, pp. 523–531 (1975)

    Google Scholar 

  40. Wang, J., Gou, L., Shen, H.W., Yang, H.: DQNViz: a visual analytics approach to understand deep Q-networks. IEEE Trans. Vis. Comput. Graph. 25(1), 288–298 (2019). https://doi.org/10.1109/TVCG.2018.2864504. https://ieeexplore.ieee.org/document/8454905/

  41. Wang, J., Hazarika, S., Li, C., Shen, H.W.: Visualization and visual analysis of ensemble data: a survey. IEEE Trans. Vis. Comput. Graph. 25(9), 2853–2872 (2019). https://doi.org/10.1109/TVCG.2018.2853721. Conference Name: IEEE Transactions on Visualization and Computer Graphics

  42. Wicker, M., Huang, X., Kwiatkowska, M.: Feature-guided black-box safety testing of deep neural networks. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10805, pp. 408–426. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89960-2_22

    CrossRef  Google Scholar 

  43. Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 223–235. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45657-0_17

    CrossRef  MATH  Google Scholar 

  44. Zahavy, T., Zrihem, N.B., Mannor, S.: Graying the black box: understanding DQNs. arXiv:1602.02658 [cs], April 2017

Download references

Acknowledgements

This work was partially supported by the ERC Advanced Investigators Grant 695614 (POWVER), by DFG grant 389792660 as part of TRR 248 (see https://perspicuous-computing.science) and by the two Clusters of Excellence CeTI (EXC 2050/1, grant 390696704) and PoL (EXC-2068, grant 390729961) of TU Dresden.

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Timo P. Gros , David Groß , Stefan Gumhold , Jörg Hoffmann , Michaela Klauck or Marcel Steinmetz .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Gros, T.P., Groß, D., Gumhold, S., Hoffmann, J., Klauck, M., Steinmetz, M. (2021). TraceVis: Towards Visualization for Deep Statistical Model Checking. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends. ISoLA 2020. Lecture Notes in Computer Science(), vol 12479. Springer, Cham. https://doi.org/10.1007/978-3-030-83723-5_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-83723-5_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-83722-8

  • Online ISBN: 978-3-030-83723-5

  • eBook Packages: Computer ScienceComputer Science (R0)