Skip to main content
SpringerLink
Log in

Towards a Unifying Logical Framework for Neural Networks

  • Conference paper
  • First Online:
Theoretical Aspects of Computing – ICTAC 2022 (ICTAC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13572))

Included in the following conference series:

  • 393 Accesses

Abstract

Neural networks are increasingly used in safety-critical applications such as medical diagnosis and autonomous driving, which calls for the need for formal specification of their behaviors. In this paper, we use matching logic—a unifying logic to specify and reason about programs and computing systems—to axiomatically define dynamic propagation and temporal operations in neural networks and to formally specify common properties about neural networks. As instances, we use matching logic to formalize a variety of neural networks, including generic feed-forward neural networks with different activation functions and recurrent neural networks. We define their formal semantics and several common properties in matching logic. This way, we obtain a unifying logical framework for specifying neural networks and their properties.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Babak, A., Delong, A., Weirauch, M.T., Frey, B.J.: Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33(8), 831–838 (2015). https://doi.org/10.1038/nbt.3300

    Article  Google Scholar 

  2. Bojarski, M., et al.: End to end learning for self-driving cars. CoRR abs/1604.07316 (2016). http://arxiv.org/abs/1604.07316

  3. Boopathy, A., Weng, T., Chen, P., Liu, S., Daniel, L.: CNN-Cert: an efficient framework for certifying robustness of convolutional neural networks. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, USA, pp. 3240–3247. AAAI Press (2019). https://doi.org/10.1609/aaai.v33i01.33013240

  4. Bunel, R., Turkaslan, I., Torr, P.H.S., Kohli, P., Mudigonda, P.K.: A unified view of piecewise linear neural network verification. In: Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, Montréal, Canada, pp. 4795–4804. Curran Associates Inc. (2018)

    Google Scholar 

  5. Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: Proceedings of the 38th IEEE Symposium on Security and Privacy, San Jose, California, USA, pp. 39–57. IEEE Computer Society (2017). https://doi.org/10.1109/SP.2017.49

  6. Chen, X., Lucanu, D., Roşu, G.: Matching logic explained. J. Logical Algebraic Methods Program. 120, 1–36 (2021). https://doi.org/10.1016/j.jlamp.2021.100638

    Article  MathSciNet  MATH  Google Scholar 

  7. Chen, X., Roşu, G.: Matching \(\mu \)-logic. In: Proceedings of the 34th Annual ACM/IEEE Symposium on Logic in Computer Science, Vancouver, Canada, pp. 1–13. IEEE (2019). https://doi.org/10.1109/LICS.2019.8785675

  8. Clarkson, M.R., Schneider, F.B.: Hyperproperties. J. Comput. Secur. 18(6), 1157–1210 (2010). https://doi.org/10.3233/JCS-2009-0393

    Article  Google Scholar 

  9. Clevert, D., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (ELUs). In: Proceedings of the 4th International Conference on Learning Representations, San Juan, Puerto Rico. OpenReview.net (2016)

    Google Scholar 

  10. Codevilla, F., Müller, M., López, A.M., Koltun, V., Dosovitskiy, A.: End-to-end driving via conditional imitation learning. In: Proceedings of the 2018 IEEE International Conference on Robotics and Automation, Brisbane, Australia, pp. 1–9. IEEE (2018). https://doi.org/10.1109/ICRA.2018.8460487

  11. Dahl, G.E., Stokes, J.W., Deng, L., Yu, D.: Large-scale malware classification using random projections and neural networks. In: Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, British Columbia, Canada, pp. 3422–3426. IEEE (2013). https://doi.org/10.1109/ICASSP.2013.6638293

  12. Dreossi, T., Jha, S., Seshia, S.A.: Semantic adversarial deep learning. In: Chockler, H., Weissenbacher, G. (eds.) CAV 2018. LNCS, vol. 10981, pp. 3–26. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-96145-3_1

    Chapter  Google Scholar 

  13. Dutta, S., Jha, S., Sankaranarayanan, S., Tiwari, A.: Output range analysis for deep feedforward neural networks. In: Dutle, A., Muñoz, C., Narkawicz, A. (eds.) NFM 2018. LNCS, vol. 10811, pp. 121–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77935-5_9

    Chapter  Google Scholar 

  14. Dvijotham, K., Stanforth, R., Gowal, S., Mann, T.A., Kohli, P.: A dual approach to scalable verification of deep networks. In: Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, Monterey, California, USA, pp. 550–559. AUAI Press (2018)

    Google Scholar 

  15. Ehlers, R.: Formal verification of piece-wise linear feed-forward neural networks. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 269–286. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68167-2_19

    Chapter  Google Scholar 

  16. Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990). https://doi.org/10.1207/s15516709cog1402_1

    Article  Google Scholar 

  17. Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., Vechev, M.T.: AI2: safety and robustness certification of neural networks with abstract interpretation. In: Proceedings of the 39th IEEE Symposium on Security and Privacy, San Francisco, California, USA, pp. 3–18. IEEE Computer Society (2018). https://doi.org/10.1109/SP.2018.00058

  18. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Proceedings of the 3rd International Conference on Learning Representations, San Diego, California, USA. OpenReview.net (2015)

    Google Scholar 

  19. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Proceedings of the 30th Annual Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 3315–3323. Curran Associates Inc. (2016)

    Google Scholar 

  20. Hayou, S., Doucet, A., Rousseau, J.: On the selection of initialization and activation function for deep neural networks. CoRR abs/1805.08266 (2018). http://arxiv.org/abs/1805.08266

  21. Huang, X., et al.: A survey of safety and trustworthiness of deep neural networks: verification, testing, adversarial attack and defence, and interpretability. Comput. Sci. Rev. 37, 100270 (2020). https://doi.org/10.1016/j.cosrev.2020.100270

  22. Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_1

    Chapter  Google Scholar 

  23. Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Long Beach, California, USA, pp. 971–980. Curran Associates Inc. (2017)

    Google Scholar 

  24. LeCun, Y., Bengio, Y.: Convolutional Networks for Images, Speech, and Time Series, pp. 255–258. MIT Press, Cambridge (1998)

    Google Scholar 

  25. Liu, W.-W., Song, F., Zhang, T.-H.-R., Wang, J.: Verifying ReLU neural networks from a model checking perspective. J. Comput. Sci. Technol. 35(6), 1365–1381 (2020). https://doi.org/10.1007/s11390-020-0546-7

    Article  Google Scholar 

  26. Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, Georgia, USA, vol. 30. PMLR (2013)

    Google Scholar 

  27. Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, pp. 807–814. Omnipress (2010)

    Google Scholar 

  28. Narayan, S.: The generalized sigmoid activation function: competitive supervised learning. Inf. Sci. 99(1), 69–82 (1997). https://doi.org/10.1016/S0020-0255(96)00200-9

    Article  MathSciNet  Google Scholar 

  29. Papernot, N., McDaniel, P.D., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: Proceedings of the 1st IEEE European Symposium on Security and Privacy, Saarbrücken, Germany, pp. 372–387. IEEE (2016). https://doi.org/10.1109/EuroSP.2016.36

  30. Pnueli, A.: The temporal logic of programs. In: Proceedings of the 18th Annual Symposium on Foundations of Computer Science, Providence, Rhode Island, USA, pp. 46–57. IEEE Computer Society (1977). https://doi.org/10.1109/SFCS.1977.32

  31. Roşu, G.: Finite-trace linear temporal logic: coinductive completeness. Formal Methods Syst. Des. 53(1), 138–163 (2018). https://doi.org/10.1007/s10703-018-0321-3

    Article  MATH  Google Scholar 

  32. Ruan, W., Huang, X., Kwiatkowska, M.: Reachability analysis of deep neural networks with provable guarantees. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, pp. 2651–2659. ijcai.org (2018). https://doi.org/10.24963/ijcai.2018/368

  33. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015). https://doi.org/10.1016/j.neunet.2014.09.003

    Article  Google Scholar 

  34. Seshia, S.A., et al.: Formal specification for deep neural networks. In: Lahiri, S.K., Wang, C. (eds.) ATVA 2018. LNCS, vol. 11138, pp. 20–34. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01090-4_2

    Chapter  Google Scholar 

  35. Shin, E.C.R., Song, D., Moazzezi, R.: Recognizing functions in binaries with neural networks. In: Proceedings of the 24th USENIX Security Symposium, Washington, D.C., USA, pp. 611–626. USENIX Association (2015)

    Google Scholar 

  36. Shriver, D., Elbaum, S., Dwyer, M.B.: DNNV: a framework for deep neural network verification. In: Silva, A., Leino, K.R.M. (eds.) CAV 2021. LNCS, vol. 12759, pp. 137–150. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-81685-8_6

    Chapter  Google Scholar 

  37. Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the 7th International Conference on Document Analysis and Recognition, Edinburgh, Scotland, UK, pp. 958–962. IEEE Computer Society (2003). https://doi.org/10.1109/ICDAR.2003.1227801

  38. Singh, G., Ganvir, R., Püschel, M., Vechev, M.T.: Beyond the single neuron convex barrier for neural network certification. In: Proceedings of the 33rd Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, pp. 15072–15083. Curran Associates Inc. (2019)

    Google Scholar 

  39. Singh, G., Gehr, T., Püschel, M., Vechev, M.T.: An abstract domain for certifying neural networks. Proc. ACM Program. Lang. 3(POPL), 41:1–41:30 (2019). https://doi.org/10.1145/3290354

  40. Singh, G., Gehr, T., Püschel, M., Vechev, M.T.: Boosting robustness certification of neural networks. In: Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA. OpenReview.net (2019)

    Google Scholar 

  41. Sun, B., Sun, J., Dai, T., Zhang, L.: Probabilistic verification of neural networks against group fairness. In: Huisman, M., Păsăreanu, C., Zhan, N. (eds.) FM 2021. LNCS, vol. 13047, pp. 83–102. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-90870-6_5

    Chapter  Google Scholar 

  42. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of the 28th Annual Conference on Neural Information Processing Systems, Montreal, Quebec, Canada, pp. 3104–3112. Curran Associates Inc. (2014)

    Google Scholar 

  43. Tarski, A.: A lattice-theoretical fixpoint theorem and its applications. Pac. J. Math. 5(2), 285–309 (1955). pjm/1103044538

    Google Scholar 

  44. Tjeng, V., Xiao, K.Y., Tedrake, R.: Evaluating robustness of neural networks with mixed integer programming. In: Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA. OpenReview.net (2019)

    Google Scholar 

  45. Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S.: Efficient formal safety analysis of neural networks. In: Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, Montréal, Canada, pp. 6369–6379. Curran Associates Inc. (2018)

    Google Scholar 

  46. Wang, S., Pei, K., Whitehouse, J., Yang, J., Jana, S.: Formal security analysis of neural networks using symbolic intervals. In: Proceedings of the 27th USENIX Security Symposium, Baltimore, Maryland, USA, pp. 1599–1614. USENIX Association (2018)

    Google Scholar 

  47. Weng, T., et al.: Towards fast computation of certified robustness for ReLU networks. In: Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, Stockholmsmässan, Stockholm, Sweden, vol. 80, pp. 5273–5282. PMLR (2018)

    Google Scholar 

  48. Weng, T., et al.: Evaluating the robustness of neural networks: an extreme value theory approach. In: Proceedings of the 6th International Conference on Learning Representations, Vancouver, British Columbia, Canada. OpenReview.net (2018)

    Google Scholar 

  49. Xiang, W., Tran, H., Johnson, T.T.: Output reachable set estimation and verification for multilayer neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5777–5783 (2018). https://doi.org/10.1109/TNNLS.2018.2808470

    Article  MathSciNet  Google Scholar 

  50. You, S., Ding, D., Canini, K.R., Pfeifer, J., Gupta, M.R.: Deep lattice networks and partial monotonic functions. In: Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Long Beach, California, USA, pp. 2981–2989. Curran Associates Inc. (2017)

    Google Scholar 

  51. Zhang, H., Weng, T., Chen, P., Hsieh, C., Daniel, L.: Efficient neural network robustness certification with general activation functions. In: Proceedings of the 32nd Annual Conference on Neural Information Processing Systems, Montréal, Canada, pp. 4944–4953. Curran Associates Inc. (2018)

    Google Scholar 

  52. Zhang, X., Chen, X., Sun, M.: Towards a unifying logical framework for neural networks. Technical report, Peking University and University of Illinois Urbana-Champaign (2022). https://hdl.handle.net/2142/114412

Download references

Acknowledgements

This research was sponsored by the National Natural Science Foundation of China under Grant No. 62172019, 61772038, and CCF-Huawei Formal Verification Innovation Research Plan. The work presented in this paper was supported in part by NSF CNS 16-19275. This material is based upon work supported by the United States Air Force and DARPA under Contract No. FA8750-18-C-0092.

Author information

Authors and Affiliations

  1. Peking University, Beijing, China

    Xiyue Zhang & Meng Sun

  2. University of Illinois Urbana-Champaign, Champaign, USA

    Xiaohong Chen

Authors
  1. Xiyue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaohong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xiaohong Chen or Meng Sun .

Editor information

Editors and Affiliations

  1. TU München, München, Germany

    Helmut Seidl

  2. Southwest University, Chongqing, China

    Zhiming Liu

  3. NASA Ames Research Center, Moffett Field, CA, USA

    Corina S. Pasareanu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, X., Chen, X., Sun, M. (2022). Towards a Unifying Logical Framework for Neural Networks. In: Seidl, H., Liu, Z., Pasareanu, C.S. (eds) Theoretical Aspects of Computing – ICTAC 2022. ICTAC 2022. Lecture Notes in Computer Science, vol 13572. Springer, Cham. https://doi.org/10.1007/978-3-031-17715-6_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17715-6_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17714-9

  • Online ISBN: 978-3-031-17715-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation