Skip to main content

Into the Unknown: Active Monitoring of Neural Networks

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 12974)


Neural-network classifiers achieve high accuracy when predicting the class of an input that they were trained to identify. Maintaining this accuracy in dynamic environments, where inputs frequently fall outside the fixed set of initially known classes, remains a challenge. The typical approach is to detect inputs from novel classes and retrain the classifier on an augmented dataset. However, not only the classifier but also the detection mechanism needs to adapt in order to distinguish between newly learned and yet unknown input classes. To address this challenge, we introduce an algorithmic framework for active monitoring of a neural network. A monitor wrapped in our framework operates in parallel with the neural network and interacts with a human user via a series of interpretable labeling queries for incremental adaptation. In addition, we propose an adaptive quantitative monitor to improve precision. An experimental evaluation on a diverse set of benchmarks with varying numbers of classes confirms the benefits of our active monitoring framework in dynamic scenarios.


  • Monitoring
  • Neural networks
  • Novelty detection

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-88494-9_3
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   64.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-88494-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   84.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.


  1. 1.


  1. Bendale, A., Boult, T.E.: Towards open world recognition. In: CVPR, pp. 1893–1902. IEEE Computer Society (2015).

  2. Bendale, A., Boult, T.E.: Towards open set deep networks. In: CVPR, pp. 1563–1572. IEEE Computer Society (2016).

  3. Bendre, N., Terashima-Marín, H., Najafirad, P.: Learning from few samples: a survey. CoRR abs/2007.15484 (2020).

  4. Chen, Y., Cheng, C., Yan, J., Yan, R.: Monitoring object detection abnormalities via data-label and post-algorithm abstractions. CoRR abs/2103.15456 (2021).

  5. Cheng, C., Nührenberg, G., Yasuoka, H.: Runtime monitoring neuron activation patterns. In: DATE, pp. 300–303. IEEE (2019).

  6. Cohen, G., Afshar, S., Tapson, J., van Schaik, A.: EMNIST: extending MNIST to handwritten letters. In: IJCNN, pp. 2921–2926. IEEE (2017).

  7. Cohn, D.A., Atlas, L.E., Ladner, R.E.: Improving generalization with active learning. Mach. Learn. 15(2), 201–221 (1994).

    CrossRef  Google Scholar 

  8. Das, S., Wong, W., Dietterich, T.G., Fern, A., Emmott, A.: Incorporating expert feedback into active anomaly discovery. In: ICDM, pp. 853–858. IEEE Computer Society (2016).

  9. Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: ICML. JMLR Workshop and Conference Proceedings, vol. 48, pp. 1050–1059. (2016).

  10. Geifman, Y., El-Yaniv, R.: Selective classification for deep neural networks. In: NeurIPS, pp. 4878–4887 (2017).

  11. Guerriero, S., Caputo, B., Mensink, T.: DeepNCM: deep nearest class mean classifiers. In: ICLR. (2018).

  12. Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML. PMLR, vol. 70, pp. 1321–1330. PMLR (2017).

  13. Gupta, A., Carlone, L.: Online monitoring for neural network based monocular pedestrian pose estimation. In: ITSC, pp. 1–8. IEEE (2020).

  14. Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: ICLR. (2017).

  15. Henzinger, T.A., Lukina, A., Schilling, C.: Outside the box: abstraction-based monitoring of neural networks. In: ECAI. Frontiers in Artificial Intelligence and Applications, vol. 325, pp. 2433–2440. IOS Press (2020).

  16. Ibrahim, S.H., Nassar, M.: Hack the box: fooling deep learning abstraction-based monitors. CoRR abs/2107.04764 (2021).

  17. Jolliffe, I.T.: Principal Component Analysis. Springer Series in Statistics, Springer, Heidelberg (1986).

    CrossRef  MATH  Google Scholar 

  18. Knorr, E.M., Ng, R.T.: A unified notion of outliers: properties and computation. In: KDD, pp. 219–222. AAAI Press (1997).

  19. Krizhevsky, A.: Learning multiple layers of features from tiny images. University of Toronto, Technical report (2009)

    Google Scholar 

  20. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    CrossRef  Google Scholar 

  21. Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., Alsaadi, F.E.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11–26 (2017).

    CrossRef  Google Scholar 

  22. Lloyd, S.P.: Least squares quantization in PCM. Trans. Inf. Theory 28(2), 129–136 (1982).

    CrossRef  MathSciNet  MATH  Google Scholar 

  23. Lu, J., Gong, P., Ye, J., Zhang, C.: Learning from very few samples: a survey. CoRR abs/2009.02653 (2020).

  24. Mancini, M., Karaoguz, H., Ricci, E., Jensfelt, P., Caputo, B.: Knowledge is never enough: towards web aided deep open world recognition. In: ICRA, pp. 9537–9543. IEEE (2019).

  25. Mandelbaum, A., Weinshall, D.: Distance-based confidence score for neural network classifiers. CoRR abs/1709.09844 (2017).

  26. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. In: Psychology of Learning and Motivation, vol. 24, pp. 109–165. Elsevier (1989).

  27. Mensink, T., Verbeek, J.J., Perronnin, F., Csurka, G.: Distance-based image classification: generalizing to new classes at near-zero cost. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2624–2637 (2013).

    CrossRef  Google Scholar 

  28. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010).

    CrossRef  Google Scholar 

  29. Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks 113, 54–71 (2019).

  30. Pimentel, M.A.F., Clifton, D.A., Clifton, L.A., Tarassenko, L.: A review of novelty detection. Signal Process. 99, 215–249 (2014).

    CrossRef  Google Scholar 

  31. Rahman, Q.M., Corke, P., Dayoub, F.: Run-time monitoring of machine learning for robotic perception: a survey of emerging trends. IEEE Access 9, 20067–20075 (2021).

    CrossRef  Google Scholar 

  32. Rebuffi, S., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: CVPR, pp. 5533–5542. IEEE Computer Society (2017).

  33. Redko, I., Morvant, E., Habrard, A., Sebban, M., Bennani, Y.: Advances in Domain Adaptation Theory. Elsevier (2019)

    Google Scholar 

  34. Royer, A., Lampert, C.H.: Classifier adaptation at prediction time. In: CVPR, pp. 1401–1409. IEEE Computer Society (2015).

  35. Schölkopf, B., Smola, A., Müller, K.-R.: Kernel principal component analysis. In: Gerstner, W., Germond, A., Hasler, M., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 583–588. Springer, Heidelberg (1997).

    CrossRef  Google Scholar 

  36. Schultheiss, A., Käding, C., Freytag, A., Denzler, J.: Finding the unknown: novelty detection with extreme value signatures of deep neural activations. In: Roth, V., Vetter, T. (eds.) GCPR 2017. LNCS, vol. 10496, pp. 226–238. Springer, Cham (2017).

    CrossRef  Google Scholar 

  37. Settles, B.: Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers (2012).

  38. Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: The German traffic sign recognition benchmark: a multi-class classification competition. In: IJCNN, pp. 1453–1460. IEEE (2011).

  39. Sun, R., Lampert, C.H.: KS(conf): a light-weight test if a multiclass classifier operates outside of its specifications. Int. J. Comput. Vis. 128(4), 970–995 (2020).

    CrossRef  MathSciNet  Google Scholar 

  40. Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., Liu, C.: A survey on deep transfer learning. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 270–279. Springer, Cham (2018).

    CrossRef  Google Scholar 

  41. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: IROS, pp. 23–30. IEEE (2017).

  42. Wagstaff, K.L., Lu, S.: Efficient active learning for new domains. In: Workshop on Real World Experiment Design and Active Learning (2020)

    Google Scholar 

  43. Wu, C., Falcone, Y., Bensalem, S.: Customizable reference runtime monitoring of neural networks using resolution boxes. CoRR abs/2104.14435 (2021).

  44. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747 (2017).

  45. Zhang, P., Wang, J., Farhadi, A., Hebert, M., Parikh, D.: Predicting failures of vision systems. In: CVPR, pp. 3566–3573. IEEE Computer Society (2014).

  46. Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2016).

    CrossRef  Google Scholar 

  47. Zhao, P., Hoi, S.C.H.: OTL: a framework of online transfer learning. In: ICML, pp. 1231–1238. Omnipress (2010).

Download references


We thank Christoph Lampert and Alex Greengold for fruitful discussions. This research was supported in part by the Simons Institute for the Theory of Computing, the Austrian Science Fund (FWF) under grant Z211-N23 (Wittgenstein Award), and the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 754411.

Author information

Authors and Affiliations


Corresponding authors

Correspondence to Anna Lukina or Christian Schilling .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Lukina, A., Schilling, C., Henzinger, T.A. (2021). Into the Unknown: Active Monitoring of Neural Networks. In: Feng, L., Fisman, D. (eds) Runtime Verification. RV 2021. Lecture Notes in Computer Science(), vol 12974. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-88493-2

  • Online ISBN: 978-3-030-88494-9

  • eBook Packages: Computer ScienceComputer Science (R0)