Skip to main content
SpringerLink
Log in

Certification of avionic software based on machine learning: the case for formal monotony analysis

  • General
  • Special Issue: FMICS 2022
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

The use of machine learning (ML) in airborne safety-critical systems requires new methods for certification, as the current standards and practices were defined and refined over decades with classical programming in mind and do not support this new development paradigm. This article provides an overview of the main challenges to the demonstration of compliance with regulation requirements raised by the use of ML and focuses on one particular case where the formal verification may become mandatory in future regulations, which is the verification of (partial) monotony properties. For this case, we propose a method to evaluate the monotony property using mixed integer linear programming. Contrary to the existing literature, our analysis provides a lower and upper bound of the space volume where the property does not hold, that we denote “Non-Monotonic Space Coverage”. This work has several advantages: (i) our formulation of the monotony property works on discrete inputs, (ii) the iterative nature of our algorithm allows for refining the analysis as needed, and (iii) from an industrial point of view, the results of this evaluation are valuable for the aeronautical domain, where it can support the certification demonstration. We applied this method to an avionic case study (braking distance estimation using a neural network) where the verification of the monotony property is of paramount interest from a safety perspective.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Amershi, S., Begel, A., Bird, C., et al.: Software engineering for machine learning: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). https://doi.org/10.1109/ICSE-SEIP.2019.00042

    Chapter  Google Scholar 

  2. Biannic, J., Hardier, G., Roos, C., et al.: Surrogate models for aircraft flight control: some off-line and embedded applications. Aerosp. Lab. 12, 1 (2016)

    Google Scholar 

  3. Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: IEEE SP. IEEE Computer Society, pp. 39–57 (2017). https://doi.org/10.1109/SP.2017.49

    Chapter  Google Scholar 

  4. Chen, S., Sun, Y., Li, D., et al.: Runtime safety assurance for learning-enabled control of autonomous driving vehicles. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 8978–8984 (2022). https://doi.org/10.1109/ICRA46639.2022.9812177

    Chapter  Google Scholar 

  5. Cheng, C.H., Nührenberg, G., Ruess, H.: Maximum resilience of artificial neural networks. In: D’Souza, D., Narayan Kumar, K. (eds.) Automated Technology for Verification and Analysis, pp. 251–268. Springer, Berlin (2017). https://doi.org/10.1007/978-3-319-68167-2_18

    Chapter  Google Scholar 

  6. Cofer, D.D., Amundson, I., Sattigeri, R., et al.: Run-time assurance for learning-enabled systems. In: Lee, R., Jha, S., Mavridou, A. (eds.) NASA Formal Methods – 12th International Symposium, NFM 2020, Moffett Field, CA, USA, May 11–15, 2020. Proceedings, Lecture Notes in Computer Science, vol. 12229, pp. 361–368. Springer, Berlin (2020). https://doi.org/10.1007/978-3-030-55754-6_21

    Chapter  Google Scholar 

  7. Damour, M., Grancey, F.D., Gabreau, C., et al.: Towards certification of a reduced footprint ACAS-Xu system: a hybrid L-based solution. In: Proceedings, Computer Safety, Reliability, and Security – 40th International Conference,SAFECOMP 2021, York, UK, September 8-10, 2021, pp. 34–48 (2021). https://doi.org/10.1007/978-3-030-83903-1_3

    Chapter  Google Scholar 

  8. de Moura, L.M., Bjørner, N.S.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) Tools and Algorithms for the Construction and Analysis of Systems, 14th International Conference, TACAS 2008, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2008. Budapest, Hungary, March 29-April 6, 2008. Proceedings, Lecture Notes in Computer Science, vol. 4963, pp. 337–340. Springer, Berlin (2008). https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  9. EASA: CS-25 Amendment 27 (2021). https://www.easa.Europa.eu/downloads/136622/en

  10. EASA: EASA Concept Paper: First usable guidance for Level 1 machine learning applications (2021). https://www.easa.Europa.eu/downloads/134357/en

  11. Feelders, A.J.: Prior knowledge in economic applications of data mining. In: European Conference on Principles of Data Mining and Knowledge Discovery, pp. 395–400. Springer, Berlin (2000). https://doi.org/10.1007/3-540-45372-5_42

    Chapter  Google Scholar 

  12. Gauffriau, A., Malgouyres, F., Ducoffe, M.: Overestimation learning with guarantees (2021). arXiv preprint arXiv:2101.11717

  13. Grossmann, I.E.: Review of nonlinear mixed-integer and disjunctive programming techniques. Optim. Eng. (2002)

  14. Gupta, A., Shukla, N., Marla, L., et al.: How to incorporate monotonicity in deep networks while preserving flexibility? (2019). arXiv preprint arXiv:1909.10662

  15. Gurobi Optimization, LLC: Gurobi Optimizer Reference Manual (2022). https://www.gurobi.com

  16. Hao, J., Ye, W., Jia, L., et al.: Building surrogate models for engineering problems by integrating limited simulation data and monotonic engineering knowledge. Adv. Eng. Inform. 49, 101342 (2021). https://doi.org/10.1016/j.aei.2021.101342

    Article  Google Scholar 

  17. Jian, Z.D., Chang, H.J., Ts, H., et al.: Learning from simulated world – surrogates construction with deep neural network. In: SIMULTECH 2017: Proceedings of the 7th International Conference on Simulation and Modeling Methodologies, Technologies and Applications. SCITEPRESS (2017). https://doi.org/10.5220/0006418100830092

    Chapter  Google Scholar 

  18. Karpf, J.: Inductive modelling in law: example based expert systems in administrative law. In: Proceedings of the 3rd International Conference on Artificial Intelligence and Law, pp. 297–306 (1991). https://doi.org/10.1145/112646.112684

    Chapter  Google Scholar 

  19. Katz, G., Huang, D.A., Ibeling, D., et al.: The marabou framework for verification and analysis of deep neural networks. In: Dillig, I., Tasiran, S. (eds.) Computer Aided Verification, pp. 443–452. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_26

    Chapter  Google Scholar 

  20. Liu, X., Han, X., Zhang, N., et al.: Certified monotonic neural networks. Adv. Neural Inf. Process. Syst. 33, 15427–15438 (2020). https://proceedings.neurips.cc/paper/2020/hash/b139aeda1c2914e3b579aafd3ceeb1bd-Abstract.html

    Google Scholar 

  21. Madry, A., Makelov, A., Schmidt, L., et al.: Towards deep learning models resistant to adversarial attacks. In: ICLR. OpenReview.net (2018). https://openreview.net/forum?id=rJzIBfZAb

  22. Mamalet, F., Jenn, E., Flandin, G., et al.: White Paper Machine Learning in Certified Systems (2021). https://hal.archives-ouvertes.fr/hal-03176080

  23. Marques-Silva, J., Ignatiev, A.: Delivering trustworthy AI through formal XAI. In: Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 – March 1, 2022, pp. 12342–12350. AAAI Press, Menlo Park (2022). https://ojs.aaai.org/index.php/AAAI/article/view/21499

  24. Martin, R.: Assured software – a journey and discussion (2017). https://www.his-2019.co.uk/session/cwe-cve-its-history-and-future

  25. Müller, M.N., Makarchuk, G., Singh, G., et al.: PRIMA: general and precise neural network certification via scalable convex hull approximations. Proc. ACM Program. Lang. 6(POPL), 43 (2022). https://doi.org/10.1145/3498704

    Article  Google Scholar 

  26. Nguyen, A., Martínez, M.R.: MonoNet: towards interpretable models by learning monotonic features (2019). arXiv preprint arXiv:1909.13611

  27. Phillips, P.J., Hahn, C., Fontana, P., et al.: Four principles of explainable artificial intelligence (2021). https://doi.org/10.6028/NIST.IR.8312. https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=933399

  28. Picard, S., Chapdelaine, C., Cappi, C., et al.: Ensuring dataset quality for machine learning certification. In: ISSRE, pp. 275–282 (2020). https://doi.org/10.1109/ISSREW51248.2020.00085

    Chapter  Google Scholar 

  29. Raghunathan, A., Steinhardt, J., Liang, P.S.: Semidefinite relaxations for certifying robustness to adversarial examples. In: Advances in Neural Information Processing Systems, pp. 10877–10887 (2018). https://proceedings.neurips.cc/paper/2018/hash/29c0605a3bab4229e46723f89cf59d83-Abstract.html

    Google Scholar 

  30. Rushby, J.: The interpretation and evaluation of assurance cases. Tech. Rep., (2015) http://www.csl.sri.com/users/rushby/papers/sri-csl-15-1-assurance-cases.pdf

  31. Schweiger, A., Annighoefer, B., Reich, M., et al.: Classification for avionics capabilities enabled by artificial intelligence. In: 2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC), pp. 1–10 (2021). https://doi.org/10.1109/DASC52595.2021.9594364

    Chapter  Google Scholar 

  32. Singh, G., Gehr, T., Püschel, M., et al.: Robustness certification with refinement. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=HJgeEh09KQ

    Google Scholar 

  33. Sudakov, O., Koroteev, D., Belozerov, B., et al.: Artificial neural network surrogate modeling of oil reservoir: a case study. In: International Symposium on Neural Networks, pp. 232–241. Springer, Berlin (2019). https://doi.org/10.1007/978-3-030-22808-8_24

    Chapter  Google Scholar 

  34. Tjeng, V., Xiao, K.Y., Tedrake, R.: Evaluating robustness of neural networks with mixed integer programming. In: ICLR (2019). https://openreview.net/forum?id=HyGIdiRqtm

    Google Scholar 

  35. Tsuzuku, Y., Sato, I., Sugiyama, M.: Lipschitz-margin training: scalable certification of perturbation invariance for deep neural networks. In: NeurIPS, pp. 6542–6551 (2018). https://proceedings.neurips.cc/paper/2018/hash/485843481a7edacbfce101ecb1e4d2a8-Abstract.html

    Google Scholar 

  36. Urban, C., Miné, A.: A review of formal methods applied to machine learning (2021). arXiv preprint arXiv:2104.02466. https://arxiv.org/abs/2104.02466

  37. Urban, C., Christakis, M., Wüstholz, V., et al.: Perfectly parallel fairness certification of neural networks. Proc. ACM Program. Lang. 4(OOPSLA), 185 (2020). https://doi.org/10.1145/3428253

    Article  Google Scholar 

  38. Wang, S., Pei, K., Whitehouse, J., et al.: Formal security analysis of neural networks using symbolic intervals. In: 27th USENIX Security Symposium (USENIX Security, vol. 18, pp. 1599–1614. USENIX Association, Baltimore (2018). https://www.usenix.org/conference/usenixsecurity18/presentation/wang-shiqi

    Google Scholar 

  39. Wang, S., Zhang, H., Xu, K., et al.: Beta-CROWN: efficient bound propagation with per-neuron split constraints for neural network robustness verification. In: Advances in Neural Information Processing Systems (2021). https://proceedings.neurips.cc/paper/2021/hash/fac7fead96dafceaf80c1daffeae82a4-Abstract.html

    Google Scholar 

  40. Weng, T., Zhang, H., Chen, H., et al.: Towards fast computation of certified robustness for relu networks. In: ICML. Proceedings of Machine Learning Research (2018). http://proceedings.mlr.press/v80/weng18a.html

    Google Scholar 

  41. Xiang, W., Tran, H.D., Johnson, T.T.: Output reachable set estimation and verification for multilayer neural networks. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5777–5783 (2018). https://doi.org/10.1109/TNNLS.2018.2808470

    Article  MathSciNet  Google Scholar 

  42. Xu, K., Shi, Z., Zhang, H., et al.: Automatic perturbation analysis for scalable certified robustness and beyond. In: NeurIPS, pp. 1129–1141 (2020). https://proceedings.neurips.cc/paper/2020/hash/0cbc5671ae26f67871cb914d81ef8fc1-Abstract.html

    Google Scholar 

  43. Zhang, H., Weng, T.W., Chen, P.Y., et al.: Efficient neural network robustness certification with general activation functions. In: Advances in Neural Information Processing Systems, pp. 4939–4948 (2018). https://proceedings.neurips.cc/paper/2018/hash/d04863f100d59b3eb688a11f95b0ae60-Abstract.html

    Google Scholar 

  44. Zhang, H., Zhang, P., Hsieh, C.J.: Recurjac: an efficient recursive algorithm for bounding Jacobian matrix of neural networks and its applications. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5757–5764 (2019). https://doi.org/10.1609/aaai.v33i01.33015757

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Airbus Opération S.A.S., Toulouse, France

    Mélanie Ducoffe, Christophe Gabreau & Eric Guillaume Vidot

  2. Institut de Recherche en Informatique de Toulouse, University of Toulouse, Toulouse, France

    Ileana Ober

  3. Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, France

    Iulian Ober

Authors
  1. Mélanie Ducoffe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christophe Gabreau
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ileana Ober
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Iulian Ober
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eric Guillaume Vidot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iulian Ober.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

All authors contributed equally to this work.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ducoffe, M., Gabreau, C., Ober, I. et al. Certification of avionic software based on machine learning: the case for formal monotony analysis. Int J Softw Tools Technol Transfer 26, 189–205 (2024). https://doi.org/10.1007/s10009-024-00741-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-024-00741-6

Keywords

Search

Navigation