Information-Theoretic Self-compression of Multi-layered Neural Networks

  • Ryotaro KamimuraEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11324)


The present paper aims to propose a new type of model compression method to self-compress multi-layered neural networks into single-layered neural networks. Then, the compressed single-layered neural networks, by the help of mutual information control, can be easily and naturally interpreted with improved generalization. For interpreting the inference mechanism, we have proposed the self-compression method, applied directly to multi-layered neural networks, to produce compressed or collective weights. However, the interpretation of compressed networks tends to be instable or changeable due to the existence of multiple layers, and the need to simplify and stabilize the compressed networks and the corresponding weights has arisen. In this context, the present paper aims not to directly and immediately compress multi-layered networks but to compress multi-layered neural network into single neural networks, and then the single neural networks are forced to be simplified as much as possible by mutual information control, and they are further compressed into non-layered networks. The method was applied to two well-studied data sets: the teaching assistant evaluation data set and the inference of occupancy status. The results confirmed two important points. First, generalization performance could be improved by transferring information from multi-layered neural networks to single-layered networks. Second, compressed or collective connection weights were easily and naturally interpreted.


Self-compression Mutual information control Multi-layered neural networks Single-layered neural networks Interpretation Generalization 


  1. 1.
    Andrews, R., Diederich, J., Tickle, A.B.: Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl. Based Syst. 8(6), 373–389 (1995)CrossRefGoogle Scholar
  2. 2.
    Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One 10(7), e0130140 (2015)CrossRefGoogle Scholar
  3. 3.
    Benítez, J.M., Castro, J.L., Requena, I.: Are artificial neural networks black boxes? IEEE Trans. Neural Networks 8(5), 1156–1164 (1997)CrossRefGoogle Scholar
  4. 4.
    Buciluǎ, C., Caruana, R., Niculescu-Mizil, A.: Model compression. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535–541. ACM (2006)Google Scholar
  5. 5.
    Candanedo, L.M., Feldheim, V.: Accurate occupancy detection of an office room from light, temperature, humidity and co2 measurements using statistical learning models. Energy Build. 112, 28–39 (2016)CrossRefGoogle Scholar
  6. 6.
    Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N.: Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1721–1730. ACM (2015)Google Scholar
  7. 7.
    Che, Z., Purushotham, S., Khemani, R., Liu, Y.: Interpretable deep models for icu outcome prediction. In: AMIA Annual Symposium Proceedings, vol. 2016, p. 371. American Medical Informatics Association (2016)Google Scholar
  8. 8.
    Cheng, Y., Wang, D., Zhou, P., Zhang, T.: A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282 (2017)
  9. 9.
    Dietz, S., Anderson, D., Stern, N., Taylor, C., Zenghelis, D., et al.: Right for the right reasons. World Econ. 8(2), 229–258 (2007)Google Scholar
  10. 10.
    Goodman, B., Flaxman, S.: European union regulations on algorithmic decision-making and a “right to explanation”. arXiv preprint arXiv:1606.08813 (2016)
  11. 11.
    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  12. 12.
    Kamimura, R.: Direct potentiality assimilation for improving multi-layered neural networks. In: Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, pp. 19–23 (2017)Google Scholar
  13. 13.
    Kamimura, R.: Mutual information maximization for improving and interpreting multi-layered neural network. In: Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence (SSCI) (SSCI 2017) (2017)Google Scholar
  14. 14.
    Karpathy, A., Johnson, J., Fei-Fei, L.: Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078 (2015)
  15. 15.
    Lim, T.S., Loh, W.Y., Shih, Y.S.: A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Mach. Learn. 40(3), 203–228 (2000)CrossRefGoogle Scholar
  16. 16.
    Loh, W.Y., Shih, Y.S.: Split selection methods for classification trees. Statistica sinica, pp. 815–840 (1997)Google Scholar
  17. 17.
    Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you?: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144. ACM (2016)Google Scholar
  18. 18.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
  19. 19.
    Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365 (2017)
  20. 20.
    Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
  21. 21.
    Varshney, K.R., Alemzadeh, H.: On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big Data 5(3), 246–255 (2017)CrossRefGoogle Scholar
  22. 22.
    Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.06579 (2015)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.IT Education CenterTokai UniversityHiratsukaJapan

Personalised recommendations