NeVer: a tool for artificial neural networks verification

  • Luca PulinaEmail author
  • Armando Tacchella


The adoption of Artificial Neural Networks (ANNs) in safety-related applications is often avoided because it is difficult to rule out possible misbehaviors with traditional analytical or probabilistic techniques. In this paper we present NeVer, our tool for checking safety of ANNs. NeVer encodes the problem of verifying safety of ANNs into the problem of satisfying corresponding Boolean combinations of linear arithmetic constraints. We describe the main verification algorithm and the structure of NeVer. We present also empirical results confirming the effectiveness of NeVer on realistic case studies.


Formal methods for adaptive systems Abstraction techniques Algorithms and tools for verification 

Mathematics Subject Classifications (2010)

68Q60 68Q45 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zhang, G.P.: Neural networks for classification: a survey. IEEE Trans. Syst. Man Cybern., Part C Appl. Rev. 30(4), 451–462 (2000)CrossRefGoogle Scholar
  2. 2.
    Smith, D.J., Simpson, K.G.L.: Functional Safety – A Straightforward Guide to Applying IEC 61505 and Related Standards (2nd edn.). Elsevier (2004)Google Scholar
  3. 3.
    Schumann, J., Gupta, P., Nelson, S.: On verification & validation of neural network based controllers. In: Proc. of International Conf. on Engineering Applications of Neural Networks (EANN’03) (2003)Google Scholar
  4. 4.
    Kurd, Z., Kelly, T., Austin, J.: Developing artificial neural networks for safety critical systems. Neural Comput. Appl. 16(1), 11–19 (2007)Google Scholar
  5. 5.
    Clarke, E.M., Emerson, E.A., Sistla, A.P.: Automatic verification of finite-state concurrent systems using temporal logic specifications. ACM Trans. Program. Lang. Syst. (TOPLAS) 8(2), 263 (1986)CrossRefGoogle Scholar
  6. 6.
    Queille, J., Sifakis, J.: Specification and verification of concurrent systems in CESAR. In: International Symposium on Programming, pp. 337–351. Springer (1982)Google Scholar
  7. 7.
    Schubert, T.: High level formal verification of next-generation microprocessors. In: Proceedings of the 40th annual Design Automation Conference. ACM (2003)Google Scholar
  8. 8.
    Ball, T., Cook, B., Levin, V., Rajamani, S.K.: SLAM and static driver verifier: Technology transfer of formal methods inside Microsoft. In: Integrated Formal Methods, pp. 1–20. Springer (2004)Google Scholar
  9. 9.
    Armando, A., Carbone, R., Compagna, L.: LTL model checking for security protocols. In: 20th IEEE Computer Security Foundations Symposium, pp. 385–396 (2007)Google Scholar
  10. 10.
    Alur, R., Henzinger, T.A., Ho, P.: Automatic symbolic verification of embedded systems. In: IEEE Real-Time Systems Symposium, pp. 2–11 (1993)Google Scholar
  11. 11.
    Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. Springer (1999)Google Scholar
  12. 12.
    Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw 2(5), 359–366 (1989)CrossRefGoogle Scholar
  13. 13.
    Pulina, L., Tacchella, A.: An abstraction-refinement approach to verification of artificial neural networks. In: 22nd International Conference on Computer Aided Verification (CAV 2010). Lecture Notes in Computer Science, vol. 6174, pp. 243–257. Springer (2010)Google Scholar
  14. 14.
    Solar-Lezama, A., Jones, C.G., Bodik, R.: Sketching concurrent data structures. In: 2008 ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 136–148. ACM (2008)Google Scholar
  15. 15.
    Vechev, M., Yahav, E., Yorsh, G.G.: Abstraction-guided synthesis of synchronization. In: 37th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pp. 327–338. ACM (2010)Google Scholar
  16. 16.
    Igel, C., Glasmachers, T., Heidrich-Meisner, V.: Shark. J. Mach. Learn. Res. 9, 993–996 (2008)zbMATHGoogle Scholar
  17. 17.
    Franzle, M., Herde, C., Teige, T., Ratschan, S., Schubert, T.: Efficient solving of large non-linear arithmetic constraint systems with complex boolean structure. JSAT, Boolean Modeling and Computation 1, 209–236 (2007)Google Scholar
  18. 18.
    Yeh, I.C.: Modeling of strength of high-performance concrete using artificial neural networks. Cem. Concr. Res. 28(12), 1797–1808 (1998)CrossRefGoogle Scholar
  19. 19.
    Haykin, S.: Neural Networks: a Comprehensive Foundation. Prentice Hall (2008)Google Scholar
  20. 20.
    Mackworth, A.K.: Consistency in networks of relations. Artif. Intell. 8(1), 99–118 (1977)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Van Hentenryck, P.: Numerica: a modeling language for global optimization. In: Fifteenth International Joint Conference on Artificial Intelligence (IJCAI), pp. 1642–1650 (1997)Google Scholar
  22. 22.
    Rossi, F., Van Beek, P., Walsh, T.: Handbook of Constraint Programming. Elsevier Science Ltd (2006)Google Scholar
  23. 23.
    Barichard, V., Hao, J.K.: A population and interval constraint propagation algorithm. In: Evolutionary Multi-Criterion Optimization, Second International Conference (EMO 2003), pp. 88–101. Springer (2003)Google Scholar
  24. 24.
    Marques-Silva, J., Lynce, I., Malik, S.: Conflict-driven Clause Learning SAT Solvers. Handbook of Satisfiability, pp. 131–153. IOS Press, Amsterdam (2009)Google Scholar
  25. 25.
    Barrett, C., Sebastiani, R., Seshia, S.A., Tinelli, C.: Satisfiability Modulo Theories. Handbook of Satisfiability, pp. 825–885. IOS Press, Amsterdam (2009)Google Scholar
  26. 26.
    Jermann, C., Sam-Haroud, D., Trombettoni, G. (eds.): CP Workshop on Interval Analysis, Constraint Propagation, Applications (IntCP 2009) (2009)Google Scholar
  27. 27.
    Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages, pp. 238–252 (1977)Google Scholar
  28. 28.
    Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-guided abstraction refinement for symbolic model checking. J. ACM (JACM) 50(5), 794 (2003)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., Euler, T.: Yale: rapid prototyping for complex data mining tasks. In: 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’06), pp. 935–940. ACM, New York (2006)CrossRefGoogle Scholar
  30. 30.
    Gordeau, R.: Roboop – a robotics object oriented package in C++. (2005)
  31. 31.
    Rabunal, J.R., Dorrado, J.: Artificial Neural Networks in Real-life Applications. Idea Group Pub (2006)Google Scholar
  32. 32.
    Witten, I.H., Frank, E.: Data Mining (2nd edn.). Morgan Kaufmann (2005)Google Scholar
  33. 33.
    Gordon, D.F.: Asimovian adaptive agents. J. Artif. Intell. Res. 13(1), 95–153 (2000)zbMATHGoogle Scholar
  34. 34.
    Pappas, G., Kress-Gazit, H. (eds.): ICRA Workshop on Formal Methods in Robotics and Automation (2009)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.DISTUniversità di GenovaGenovaItaly

Personalised recommendations