Skip to main content
Book cover

International Conference on Runtime Verification

RV 2021: Runtime Verification pp 42–61Cite as

Into the Unknown: Active Monitoring of Neural Networks

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12974)

Abstract

Neural-network classifiers achieve high accuracy when predicting the class of an input that they were trained to identify. Maintaining this accuracy in dynamic environments, where inputs frequently fall outside the fixed set of initially known classes, remains a challenge. The typical approach is to detect inputs from novel classes and retrain the classifier on an augmented dataset. However, not only the classifier but also the detection mechanism needs to adapt in order to distinguish between newly learned and yet unknown input classes. To address this challenge, we introduce an algorithmic framework for active monitoring of a neural network. A monitor wrapped in our framework operates in parallel with the neural network and interacts with a human user via a series of interpretable labeling queries for incremental adaptation. In addition, we propose an adaptive quantitative monitor to improve precision. An experimental evaluation on a diverse set of benchmarks with varying numbers of classes confirms the benefits of our active monitoring framework in dynamic scenarios.

Keywords

  • Monitoring
  • Neural networks
  • Novelty detection

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-88494-9_3
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-88494-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Notes

  1. 1.

    https://github.com/VeriXAI/Into-the-Unknown.

References

  1. Bendale, A., Boult, T.E.: Towards open world recognition. In: CVPR, pp. 1893–1902. IEEE Computer Society (2015). https://doi.org/10.1109/CVPR.2015.7298799

  2. Bendale, A., Boult, T.E.: Towards open set deep networks. In: CVPR, pp. 1563–1572. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.173

  3. Bendre, N., Terashima-Marín, H., Najafirad, P.: Learning from few samples: a survey. CoRR abs/2007.15484 (2020). https://arxiv.org/abs/2007.15484

  4. Chen, Y., Cheng, C., Yan, J., Yan, R.: Monitoring object detection abnormalities via data-label and post-algorithm abstractions. CoRR abs/2103.15456 (2021). https://arxiv.org/abs/2103.15456

  5. Cheng, C., Nührenberg, G., Yasuoka, H.: Runtime monitoring neuron activation patterns. In: DATE, pp. 300–303. IEEE (2019). https://doi.org/10.23919/DATE.2019.8714971

  6. Cohen, G., Afshar, S., Tapson, J., van Schaik, A.: EMNIST: extending MNIST to handwritten letters. In: IJCNN, pp. 2921–2926. IEEE (2017). https://doi.org/10.1109/IJCNN.2017.7966217

  7. Cohn, D.A., Atlas, L.E., Ladner, R.E.: Improving generalization with active learning. Mach. Learn. 15(2), 201–221 (1994). https://doi.org/10.1007/BF00993277

    CrossRef  Google Scholar 

  8. Das, S., Wong, W., Dietterich, T.G., Fern, A., Emmott, A.: Incorporating expert feedback into active anomaly discovery. In: ICDM, pp. 853–858. IEEE Computer Society (2016). https://doi.org/10.1109/ICDM.2016.0102

  9. Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: ICML. JMLR Workshop and Conference Proceedings, vol. 48, pp. 1050–1059. JMLR.org (2016). http://proceedings.mlr.press/v48/gal16.html

  10. Geifman, Y., El-Yaniv, R.: Selective classification for deep neural networks. In: NeurIPS, pp. 4878–4887 (2017). http://papers.nips.cc/paper/7073-selective-classification-for-deep-neural-networks

  11. Guerriero, S., Caputo, B., Mensink, T.: DeepNCM: deep nearest class mean classifiers. In: ICLR. OpenReview.net (2018). https://openreview.net/forum?id=rkPLZ4JPM

  12. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML. PMLR, vol. 70, pp. 1321–1330. PMLR (2017). http://proceedings.mlr.press/v70/guo17a.html

  13. Gupta, A., Carlone, L.: Online monitoring for neural network based monocular pedestrian pose estimation. In: ITSC, pp. 1–8. IEEE (2020). https://doi.org/10.1109/ITSC45102.2020.9294609

  14. Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: ICLR. OpenReview.net (2017). https://openreview.net/forum?id=Hkg4TI9xl

  15. Henzinger, T.A., Lukina, A., Schilling, C.: Outside the box: abstraction-based monitoring of neural networks. In: ECAI. Frontiers in Artificial Intelligence and Applications, vol. 325, pp. 2433–2440. IOS Press (2020). https://doi.org/10.3233/FAIA200375

  16. Ibrahim, S.H., Nassar, M.: Hack the box: fooling deep learning abstraction-based monitors. CoRR abs/2107.04764 (2021). https://arxiv.org/abs/2107.04764

  17. Jolliffe, I.T.: Principal Component Analysis. Springer Series in Statistics, Springer, Heidelberg (1986). https://doi.org/10.1007/978-1-4757-1904-8

    CrossRef  MATH  Google Scholar 

  18. Knorr, E.M., Ng, R.T.: A unified notion of outliers: properties and computation. In: KDD, pp. 219–222. AAAI Press (1997). http://www.aaai.org/Library/KDD/1997/kdd97-044.php

  19. Krizhevsky, A.: Learning multiple layers of features from tiny images. University of Toronto, Technical report (2009)

    Google Scholar 

  20. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    CrossRef  Google Scholar 

  21. Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., Alsaadi, F.E.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11–26 (2017). https://doi.org/10.1016/j.neucom.2016.12.038

    CrossRef  Google Scholar 

  22. Lloyd, S.P.: Least squares quantization in PCM. Trans. Inf. Theory 28(2), 129–136 (1982). https://doi.org/10.1109/TIT.1982.1056489

    CrossRef  MathSciNet  MATH  Google Scholar 

  23. Lu, J., Gong, P., Ye, J., Zhang, C.: Learning from very few samples: a survey. CoRR abs/2009.02653 (2020). https://arxiv.org/abs/2009.02653

  24. Mancini, M., Karaoguz, H., Ricci, E., Jensfelt, P., Caputo, B.: Knowledge is never enough: towards web aided deep open world recognition. In: ICRA, pp. 9537–9543. IEEE (2019). https://doi.org/10.1109/ICRA.2019.8793803

  25. Mandelbaum, A., Weinshall, D.: Distance-based confidence score for neural network classifiers. CoRR abs/1709.09844 (2017). http://arxiv.org/abs/1709.09844

  26. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Elsevier (1989). http://www.sciencedirect.com/science/article/pii/S0079742108605368

  27. Mensink, T., Verbeek, J.J., Perronnin, F., Csurka, G.: Distance-based image classification: generalizing to new classes at near-zero cost. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2624–2637 (2013). https://doi.org/10.1109/TPAMI.2013.83

    CrossRef  Google Scholar 

  28. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191

    CrossRef  Google Scholar 

  29. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks 113, 54–71 (2019). https://doi.org/10.1016/j.neunet.2019.01.012

  30. Pimentel, M.A.F., Clifton, D.A., Clifton, L.A., Tarassenko, L.: A review of novelty detection. Signal Process. 99, 215–249 (2014). https://doi.org/10.1016/j.sigpro.2013.12.026

    CrossRef  Google Scholar 

  31. Rahman, Q.M., Corke, P., Dayoub, F.: Run-time monitoring of machine learning for robotic perception: a survey of emerging trends. IEEE Access 9, 20067–20075 (2021). https://doi.org/10.1109/ACCESS.2021.3055015

    CrossRef  Google Scholar 

  32. Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: CVPR, pp. 5533–5542. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.587

  33. Redko, I., Morvant, E., Habrard, A., Sebban, M., Bennani, Y.: Advances in Domain Adaptation Theory. Elsevier (2019)

    Google Scholar 

  34. Royer, A., Lampert, C.H.: Classifier adaptation at prediction time. In: CVPR, pp. 1401–1409. IEEE Computer Society (2015). https://doi.org/10.1109/CVPR.2015.7298746

  35. Schölkopf, B., Smola, A., Müller, K.-R.: Kernel principal component analysis. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 583–588. Springer, Heidelberg (1997). https://doi.org/10.1007/BFb0020217

    CrossRef  Google Scholar 

  36. Schultheiss, A., Käding, C., Freytag, A., Denzler, J.: Finding the unknown: novelty detection with extreme value signatures of deep neural activations. In: Roth, V., Vetter, T. (eds.) GCPR 2017. LNCS, vol. 10496, pp. 226–238. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66709-6_19

    CrossRef  Google Scholar 

  37. Settles, B.: Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers (2012). https://doi.org/10.2200/S00429ED1V01Y201207AIM018

  38. Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: The German traffic sign recognition benchmark: a multi-class classification competition. In: IJCNN, pp. 1453–1460. IEEE (2011). https://doi.org/10.1109/IJCNN.2011.6033395

  39. Sun, R., Lampert, C.H.: KS(conf): a light-weight test if a multiclass classifier operates outside of its specifications. Int. J. Comput. Vis. 128(4), 970–995 (2020). https://doi.org/10.1007/s11263-019-01232-x

    CrossRef  MathSciNet  Google Scholar 

  40. Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., Liu, C.: A survey on deep transfer learning. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 270–279. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_27

    CrossRef  Google Scholar 

  41. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: IROS, pp. 23–30. IEEE (2017). https://doi.org/10.1109/IROS.2017.8202133

  42. Wagstaff, K.L., Lu, S.: Efficient active learning for new domains. In: Workshop on Real World Experiment Design and Active Learning (2020)

    Google Scholar 

  43. Wu, C., Falcone, Y., Bensalem, S.: Customizable reference runtime monitoring of neural networks using resolution boxes. CoRR abs/2104.14435 (2021). https://arxiv.org/abs/2104.14435

  44. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747 (2017). http://arxiv.org/abs/1708.07747

  45. Zhang, P., Wang, J., Farhadi, A., Hebert, M., Parikh, D.: Predicting failures of vision systems. In: CVPR, pp. 3566–3573. IEEE Computer Society (2014). https://doi.org/10.1109/CVPR.2014.456

  46. Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2016). https://doi.org/10.1109/TPAMI.2015.2502579

    CrossRef  Google Scholar 

  47. Zhao, P., Hoi, S.C.H.: OTL: a framework of online transfer learning. In: ICML, pp. 1231–1238. Omnipress (2010). https://icml.cc/Conferences/2010/papers/219.pdf

Download references

Acknowledgments

We thank Christoph Lampert and Alex Greengold for fruitful discussions. This research was supported in part by the Simons Institute for the Theory of Computing, the Austrian Science Fund (FWF) under grant Z211-N23 (Wittgenstein Award), and the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 754411.

Author information

Authors and Affiliations

  1. Institute of Science and Technology Austria, Klosterneuburg, Austria

    Anna Lukina & Thomas A. Henzinger

  2. University of Konstanz, Konstanz, Germany

    Christian Schilling

Authors
  1. Anna Lukina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Schilling
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas A. Henzinger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Anna Lukina or Christian Schilling .

Editor information

Editors and Affiliations

  1. University of Virginia, Charlottesville, VA, USA

    Ph.D. Lu Feng

  2. Ben-Gurion University of the Negev, Be'er Sheva, Israel

    Dr. Dana Fisman

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Lukina, A., Schilling, C., Henzinger, T.A. (2021). Into the Unknown: Active Monitoring of Neural Networks. In: Feng, L., Fisman, D. (eds) Runtime Verification. RV 2021. Lecture Notes in Computer Science(), vol 12974. Springer, Cham. https://doi.org/10.1007/978-3-030-88494-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-88494-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88493-2

  • Online ISBN: 978-3-030-88494-9

  • eBook Packages: Computer ScienceComputer Science (R0)