Advertisement

Verification of Binarized Neural Networks via Inter-neuron Factoring

(Short Paper)
  • Chih-Hong ChengEmail author
  • Georg Nührenberg
  • Chung-Hao Huang
  • Harald Ruess
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11294)

Abstract

Binarized Neural Networks (BNN) have recently been proposed as an energy-efficient alternative to more traditional learning networks. Here we study the problem of formally verifying BNNs by reducing it to a corresponding hardware verification problem. The main step in this reduction is based on factoring computations among neurons within a hidden layer of the BNN in order to make the BNN verification problem more scalable in practice. The main contributions of this paper include results on the NP-hardness and hardness of PTAS approximability of this essential optimization and factoring step, and we design polynomial-time search heuristics for generating approximate factoring solutions. With these techniques we are able to scale the verification problem to moderately-sized BNNs for embedded devices with thousands of neurons and inputs.

Notes

Acknowledgments

We thank Dr. Ljubo Mercep from Mentor Graphics for indicating to us some recent results on quantized neural networks, Dr. Alan Mishchenko from UC Berkeley for his kind suggestions and support regarding \(\texttt {ABC}\), and Hugo A. Andrade from Xilinx for exchanging the view of BNN.

References

  1. 1.
    Umuroglu, Y., et al.: FINN: a framework for fast, scalable binarized neural network arXiv preprint arXiv:1612.07119 (2017)
  2. 2.
    Ambühl, C., Mastrolilli, M., Svensson, O.: Inapproximability results for maximum edge biclique, minimum linear arrangement, and sparsest cut. SIAM J. Comput. 40(2), 567–596 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Brayton, R., Mishchenko, A.: ABC: an academic industrial-strength verification tool. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV 2010. LNCS, vol. 6174, pp. 24–40. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-14295-6_5CrossRefGoogle Scholar
  4. 4.
    Chen, C., Seff, A., Kornhauser, A., Xiao, J.: Deepdriving: learning affordance for direct perception in autonomous driving. In: ICCV, pp. 2722–2730 (2015)Google Scholar
  5. 5.
    Cheng, C.-H., Nührenberg, G., Ruess, H.: Maximum resilience of artificial neural networks. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 251–268. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-68167-2_18CrossRefGoogle Scholar
  6. 6.
    Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830 (2016)
  7. 7.
    Ehlers, R.: Formal verification of piece-wise linear feed-forward neural networks. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 269–286. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-68167-2_19CrossRefGoogle Scholar
  8. 8.
    Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 3–29. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-63387-9_1CrossRefGoogle Scholar
  9. 9.
    Huval, B., et al. An empirical evaluation of deep learning on highway driving. arXiv preprint arXiv:1504.01716 (2015)
  10. 10.
    Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-63387-9_5CrossRefGoogle Scholar
  11. 11.
    Kim, M., Smaragdis, P.: Bitwise neural networks. arXiv preprint arXiv:1601.06071 (2016)
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  13. 13.
    LeCun, Y.: The MNIST database of handwritten digits (1998). http://yann.lecun.com/exdb/mnist/
  14. 14.
    Lenz, D., Diehl, F., Le, M.T., Knoll, A.: Deep neural networks for Markovian interactive scene prediction in highway scenarios. In: Intelligent Vehicles Symposium IV. IEEE (2017)Google Scholar
  15. 15.
    Lomuscio, A., Maganti, L.: An approach to reachability analysis for feed-forward relu neural networks. arXiv preprint arXiv:1706.07351 (2017)
  16. 16.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CPVR, pp. 3431–3440. IEEE (2015)Google Scholar
  17. 17.
    Narodytska, N., Kasiviswanathan, S.P., Ryzhyk, L., Sagiv, M., Walsh, T.: Verifying properties of binarized deep neural networks. arXiv preprint arXiv:1709.06662 (2014)
  18. 18.
    Peeters, R.: The maximum edge biclique problem is NP-complete. Discret. Appl. Math. 131(3), 651–654 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Pulina, L., Tacchella, A.: An abstraction-refinement approach to verification of artificial neural networks. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV 2010. LNCS, vol. 6174, pp. 243–257. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-14295-6_24CrossRefGoogle Scholar
  20. 20.
    Sermanet, P., Eigen, D., Zhang, X. , Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013)
  21. 21.
    Soos, M.: The Cryptominisat 5 set of solvers at sat competition 2016. In: Sat Competition 2016, p. 28 (2016)Google Scholar
  22. 22.
    Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: The German traffic sign recognition benchmark: a multi-class classification competition. In: IEEE International Joint Conference on Neural Networks, pp. 1453–1460 (2011)Google Scholar
  23. 23.
    Sun, L., Peng, C., Zhan, W., Tomizuka, M.: A fast integrated planning and control framework for autonomous driving via imitation learning (2017). arXiv preprint arXiv:1707.02515
  24. 24.
    Wolf, C., Glaser, J., Kepler, J.: Yosys-a free verilog synthesis suite. In: Proceedings of the 21st Austrian Workshop on Microelectronics (Austrochip) (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Chih-Hong Cheng
    • 1
    Email author
  • Georg Nührenberg
    • 1
  • Chung-Hao Huang
    • 1
  • Harald Ruess
    • 1
  1. 1.fortiss - Landesforschungsinstitut des Freistaats BayernMunichGermany

Personalised recommendations