Advertisement

iDropout: Leveraging Deep Taylor Decomposition for the Robustness of Deep Neural Networks

  • Christian SchreckenbergerEmail author
  • Christian Bartelt
  • Heiner Stuckenschmidt
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11877)

Abstract

In this work, we present iDropout, a new method to adjust dropout, from purely randomly dropping inputs to dropping inputs based on a mix based on the relevance of the nodes and some randomness. We use Deep Taylor Decomposition to calculate the respective relevance of the inputs and based on this, we give input nodes with a higher relevance a higher probability to be included than input nodes that seem to have less of an impact. The proposed method does not only seem to increase the performance of a Neural Network, but it also seems to make the network more robust to missing data. We evaluated the approach on artificial data with various settings, e.g. noise in data, number of informative features and on real-world datasets from the UCI Machine Learning Repository.

Keywords

Deep Neural Networks Dropout Explainability Missing data 

Notes

Acknowledgments

This research was supported by the German Federal Ministry for Economic Affairs and Energy (Grant No. 01MD18011D) and the German Federal Ministry of Transport and Digital Infrastructure (Grant No. 16AVF2139F).

References

  1. 1.
    Alber, M., et al.: iNNvestigate neural networks! arXiv:1808.04260 [cs, stat], August 2018
  2. 2.
    Ba, J., Frey, B.: Adaptive dropout for training deep neural networks. In: Advances in Neural Information Processing Systems, pp. 3084–3092 (2013)Google Scholar
  3. 3.
    Bacciu, D., Crecchi, F.: Augmenting recurrent neural networks resilience by dropout. IEEE Trans. Neural Netw. Learn. Syst. 1–7 (2019).  https://doi.org/10.1109/TNNLS.2019.2899744. https://ieeexplore.ieee.org/document/8668686/
  4. 4.
    Bacciu, D., Crecchi, F., Morelli, D.: DropIn: making reservoir computing neural networks robust to missing inputs by dropout. arXiv:1705.02643 [cs, stat], May 2017
  5. 5.
    Dua, D., Graff, C.: UCI machine learning repository (2019). http://archive.ics.uci.edu/ml
  6. 6.
    Garca-Laencina, P.J., Sancho-Gmez, J.L., Figueiras-Vidal, A.R., Verleysen, M.: K nearest neighbours with mutual information for simultaneous classification and missing data imputation. Neurocomputing 72(7–9), 1483–1493 (2009).  https://doi.org/10.1016/j.neucom.2008.11.026. https://linkinghub.elsevier.com/retrieve/pii/S0925231209000149CrossRefGoogle Scholar
  7. 7.
    Guyon, I.: Design of experiments for the NIPS 2003 variable selection benchmark, p. 30 (2003)Google Scholar
  8. 8.
    Keshari, R., Singh, R., Vatsa, M.: Guided dropout. arXiv preprint arXiv:1812.03965 (2018)
  9. 9.
    Li, Y., Parker, L.E.: Nearest neighbor imputation using spatial-temporal correlations in wireless sensor networks. Inf. Fusion 15, 64–79 (2014).  https://doi.org/10.1016/j.inffus.2012.08.007. https://linkinghub.elsevier.com/retrieve/pii/S1566253512000711CrossRefGoogle Scholar
  10. 10.
    Liu, Z.G., Pan, Q., Dezert, J., Martin, A.: Adaptive imputation of missing values for incomplete pattern classification. Pattern Recogn. 52, 85–95 (2016).  https://doi.org/10.1016/j.patcog.2015.10.001. http://arxiv.org/abs/1602.02617CrossRefGoogle Scholar
  11. 11.
    Mahmood, M.A., Seah, W.K., Welch, I.: Reliability in wireless sensor networks: a survey and challenges ahead. Comput. Netw. 79, 166–187 (2015).  https://doi.org/10.1016/j.comnet.2014.12.016. https://linkinghub.elsevier.com/retrieve/pii/S1389128614004708CrossRefGoogle Scholar
  12. 12.
    Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Mller, K.R.: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 65, 211–222 (2017).  https://doi.org/10.1016/j.patcog.2016.11.008. https://linkinghub.elsevier.com/retrieve/pii/S0031320316303582CrossRefGoogle Scholar
  13. 13.
    Montavon, G., Samek, W., Mller, K.R.: Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2018).  https://doi.org/10.1016/j.dsp.2017.10.011. https://linkinghub.elsevier.com/retrieve/pii/S1051200417302385MathSciNetCrossRefGoogle Scholar
  14. 14.
    Morris, A.C., Josifovski, L., Bourlard, H., Cooke, M., Green, P.: A neural network for classification with incomplete data: application to robust ASR. In: Proceedings of ICSLP, p. 4 (2000)Google Scholar
  15. 15.
    Singh, N., Javeed, A., Chhabra, S., Kumar, P.: Missing value imputation with unsupervised kohonen self organizing map. In: Shetty, N.R., Prasad, N.H., Nalini, N. (eds.) Emerging Research in Computing, Information, Communication and Applications, pp. 61–76. Springer, New Delhi (2015).  https://doi.org/10.1007/978-81-322-2550-8_7CrossRefGoogle Scholar
  16. 16.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overtting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Thirukumaran, S., Sumathi, A.: Missing value imputation techniques depth survey and an imputation algorithm to improve the efficiency of imputation. In: Fourth International Conference on Advanced Computing (ICoAC), pp. 1–5. IEEE, Chennai, December 2012.  https://doi.org/10.1109/ICoAC.2012.6416805
  18. 18.
    Wan, L., Zeiler, M., Zhang, S., LeCun, Y., Fergus, R.: Regularization of neural networks using DropConnect. In: International Conference on Machine Learning, p. 9 (2013)Google Scholar
  19. 19.
    Wang, S., et al.: Defensive dropout for hardening deep neural networks under adversarial attacks. In: Proceedings of the International Conference on Computer-Aided Design - ICCAD 2018, pp. 1–8 (2018).  https://doi.org/10.1145/3240765.3264699. http://arxiv.org/abs/1809.05165
  20. 20.
    Li, Y., Parker, L.: A spatial-temporal imputation technique for classification with missing data in a wireless sensor network. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3272–3279. IEEE, Nice, September 2008.  https://doi.org/10.1109/IROS.2008.4650774

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Christian Schreckenberger
    • 1
    • 2
    Email author
  • Christian Bartelt
    • 1
  • Heiner Stuckenschmidt
    • 2
  1. 1.Institute for Enterprise SystemsUniversity of MannheimMannheimGermany
  2. 2.Data and Web Science GroupUniversity of MannheimMannheimGermany

Personalised recommendations