Skip to main content
Book cover

International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems

EXTRAAMAS 2021: Explainable and Transparent AI and Multi-Agent Systems pp 63–82Cite as

Shallow2Deep: Restraining Neural Networks Opacity Through Neural Architecture Search

  • 582 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12688)

Abstract

Recently, the Deep Learning (DL) research community has focused on developing efficient and highly performing Neural Networks (NN). Meanwhile, the eXplainable AI (XAI) research community has focused on making Machine Learning (ML) and Deep Learning methods interpretable and transparent, seeking explainability. This work is a preliminary study on the applicability of Neural Architecture Search (NAS) (a sub-field of DL looking for automatic design of NN structures) in XAI. We propose Shallow2Deep, an evolutionary NAS algorithm that exploits local variability to restrain opacity of DL-systems through NN architectures simplification. Shallow2Deep effectively reduces NN complexity – therefore their opacity – while reaching state-of-the-art performances. Unlike its competitors, Shallow2Deep promotes variability of localised structures in NN, helping to reduce NN opacity. The proposed work analyses the role of local variability in NN architectures design, presenting experimental results that show how this feature is actually desirable.

Keywords

  • Neural Architecture Search
  • Evolutionary algorithm
  • Opacity
  • Interpretability

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-82017-6_5
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-82017-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.

Notes

  1. 1.

    https://eur-lex.europa.eu/eli/reg/2016/679/oj.

  2. 2.

    By “shallow” (resp. “deep”) layers of a NN we mean the inner layers close to the input (resp. output) neurons.

  3. 3.

    https://github.com/AndAgio/Shallow2Deep.

References

  1. Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018). https://doi.org/10.1109/ACCESS.2018.2870052

  2. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), 1–46 (2015). https://doi.org/10.1371/journal.pone.0130140

  3. Arrieta, A.B., et al.: Explainable explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58(December 2019), 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012

  4. Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: quantifying interpretability of deep visual representations. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 3319–3327. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.354

  5. Cai, H., Zhu, L., Han, S.: ProxylessNAS: direct neural architecture search on target task and hardware. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019). https://openreview.net/forum?id=HylVB3AqYm

  6. Calegari, R., Ciatto, G., Omicini, A.: On the integration of symbolic and sub-symbolic techniques for XAI: a survey. Intelligenza Artificiale 14(1), 7–32 (2020). https://doi.org/10.3233/IA-190036

  7. Casale, F.P., Gordon, J., Fusi, N.: Probabilistic neural architecture search. CoRR abs/1902.05116 (2019). http://arxiv.org/abs/1902.05116

  8. Chen, S., Bateni, S., Grandhi, S., Li, X., Liu, C., Yang, W.: DENAS: automated rule generation by knowledge extraction from neural networks. In: Devanbu, P., Cohen, M.B., Zimmermann, T. (eds.) ESEC/FSE 2020: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, 8–13 November 2020, pp. 813–825. ACM (2020). https://doi.org/10.1145/3368089.3409733

  9. Chu, X., Zhang, B., Xu, R.: Multi-objective reinforced evolution in mobile neural architecture search. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12538, pp. 99–113. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66823-5_6

    CrossRef  Google Scholar 

  10. Ciatto, G., Schumacher, M.I., Omicini, A., Calvaresi, D.: Agent-based explanations in AI: towards an abstract framework. In: Calvaresi, D., Najjar, A., Winikoff, M., Främling, K. (eds.) EXTRAAMAS 2020. LNCS (LNAI), vol. 12175, pp. 3–20. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-51924-7_1

    CrossRef  Google Scholar 

  11. Dosilovic, F.K., Brcic, M., Hlupic, N.: Explainable artificial intelligence: a survey. In: Skala, K., et al. (eds.) 41st International Convention on Information and Communication Technology, Electronics and Microelectronics, MIPRO 2018, Opatija, Croatia, 21–25 May 2018, pp. 210–215. IEEE (2018). https://doi.org/10.23919/MIPRO.2018.8400040

  12. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20, 55:1–55:21 (2019). http://jmlr.org/papers/v20/18-598.html

  13. Golovko, V., Egor, M., Brich, A., Sachenko, A.: A shallow convolutional neural network for accurate handwritten digits classification. In: Krasnoproshin, V.V., Ablameyko, S.V. (eds.) PRIP 2016. CCIS, vol. 673, pp. 77–85. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54220-1_8

    CrossRef  Google Scholar 

  14. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Computi. Surv. 51(5) (2018). https://doi.org/10.1145/3236009

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 770–778. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.90

  16. Hecht-Nielsen, R.: Theory of the backpropagation neural network. Neural Netw. 1(Supplement-1), 445–448 (1988). https://doi.org/10.1016/0893-6080(88)90469-8

  17. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017). http://arxiv.org/abs/1704.04861

  18. Janzing, D., Minorics, L., Blöbaum, P.: Feature relevance quantification in explainable AI: a causal problem. In: Chiappa, S., Calandra, R. (eds.) The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS 2020). Proceedings of Machine Learning Research, vol. 108, pp. 2907–2916 (2020). http://proceedings.mlr.press/v108/janzing20a.html

  19. Kaya, Y., Hong, S., Dumitras, T.: Shallow-deep networks: understanding and mitigating network overthinking. In: Chaudhuri, K., Salakhutdinov, R. (eds.) 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Long Beach, CA, USA. Proceedings of Machine Learning Research, vol. 97, pp. 3301–3310 (2019). http://proceedings.mlr.press/v97/kaya19a.html

  20. Li, J., Liang, X., Shen, S., Xu, T., Feng, J., Yan, S.: Scale-aware fast R-CNN for pedestrian detection. IEEE Trans. Multimedia 20(4), 985–996 (2018). https://doi.org/10.1109/TMM.2017.2759508

  21. Lipton, Z.C.: The mythos of model interpretability. Queue 16(3), 31–57 (2018). https://doi.org/10.1145/3236386.3241340

  22. Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 19–35. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_2

    CrossRef  Google Scholar 

  23. Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019). https://openreview.net/forum?id=S1eYHoC5FX

  24. Liu, J., Tripathi, S., Kurup, U., Shah, M.: Pruning algorithms to accelerate convolutional neural networks for edge applications: a survey. CoRR abs/2005.04275 (2020). https://arxiv.org/abs/2005.04275

  25. Luo, R., Tan, X., Wang, R., Qin, T., Chen, E., Liu, T.: Neural architecture search with GBDT. CoRR abs/2007.04785 (2020). https://arxiv.org/abs/2007.04785

  26. Miller, G.F., Todd, P.M., Hegde, S.U.: Designing neural networks using genetic algorithms. In: Schaffer, J.D. (ed.) 3rd International Conference on Genetic Algorithms, Fairfax, VA, USA, pp. 379–384. Morgan Kaufmann, June 1989

    Google Scholar 

  27. Nam, H., Han, B.: Learning multi-domain convolutional neural networks for visual tracking. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 4293–4302. IEEE Computer Society (2016). https://doi.org/10.1109/CVPR.2016.465

  28. Nguyen, A.M., Dosovitskiy, A., Yosinski, J., Brox, T., Clune, J.: Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In: Lee, D.D., Sugiyama, M., von Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016, pp. 3387–3395 (2016). https://proceedings.neurips.cc/paper/2016/hash/5d79099fcdf499f12b79770834c0164a-Abstract.html

  29. Nguyen, A.M., Yosinski, J., Clune, J.: Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. CoRR abs/1602.03616 (2016). http://arxiv.org/abs/1602.03616

  30. O’Shea, K., Nash, R.: An introduction to convolutional neural networks. CoRR abs/1511.08458 (2015). http://arxiv.org/abs/1511.08458

  31. Peng, S., Ji, F., Lin, Z., Cui, S., Chen, H., Zhang, Y.: MTSS: learn from multiple domain teachers and become a multi-domain dialogue expert. In: AAAI Conference on Artificial Intelligence (AAAI-20 Technical Tracks 5), vol. 34, pp. 8608–8615. AAAI Press (2020). https://doi.org/10.1609/aaai.v34i05.6384

  32. Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: AAAI Conference on Artificial Intelligence (AAAI-19, IAAI-19, EAAI-20), vol. 33, pp. 4780–4789. AAAI Press (2019). https://doi.org/10.1609/aaai.v33i01.33014780

  33. Ren, S., He, K., Girshick, R.B., Zhang, X., Sun, J.: Object detection networks on convolutional feature maps. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1476–1481 (2017). https://doi.org/10.1109/TPAMI.2016.2601099

  34. Rolnick, D., Tegmark, M.: The power of deeper networks for expressing natural functions. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, 30 April– 3 May 2018, Conference Track Proceedings. OpenReview.net (2018). https://openreview.net/forum?id=SyProzZAW

  35. Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.: Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Networks Learn. Syst. 28(11), 2660–2673 (2017). https://doi.org/10.1109/TNNLS.2016.2599820

  36. Setiono, R., Leow, W.K., Zurada, J.M.: Extraction of rules from artificial neural networks for nonlinear regression. IEEE Trans. Neural Netw. 13(3), 564–577 (2002). https://doi.org/10.1109/TNN.2002.1000125

  37. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1409.1556

  38. Tan, M., et al.: MnasNet: platform-aware neural architecture search for mobile. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 2820–2828. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00293

  39. Tan, M., Le, Q.V.: EfficientNet: rethinking model scaling for convolutional neural networks. CoRR abs/1905.11946 (2019). http://arxiv.org/abs/1905.11946

  40. Thrun, S.: Extracting rules from artificial neural networks with distributed representations. In: 7th International Conference on Neural Information Processing Systems (NIPS 1994), pp. 505–512. MIT Press (1994)

    Google Scholar 

  41. Tjoa, E., Guan, C.: A survey on explainable artificial intelligence (XAI): towards medical XAI. CoRR abs/1907.07374 (2019). http://arxiv.org/abs/1907.07374

  42. Wistuba, M., Rawat, A., Pedapati, T.: A survey on neural architecture search. CoRR abs/1905.01392 (2019). http://arxiv.org/abs/1905.01392

  43. Wu, B., et al.: FBNet: hardware-aware efficient convnet design via differentiable neural architecture search. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 10734–10742. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.01099

  44. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747 (2017). http://arxiv.org/abs/1708.07747

  45. Yang, Z., et al.: CARS: continuous evolution for efficient neural architecture search. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 1826–1835. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00190

  46. Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: NAS-Bench-101: towards reproducible neural architecture search. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, California, USA, 9–15 June 2019. Proceedings of Machine Learning Research, vol. 97, pp. 7105–7114. PMLR (2019). http://proceedings.mlr.press/v97/ying19a.html

  47. Yosinski, J., Clune, J., Nguyen, A.M., Fuchs, T.J., Lipson, H.: Understanding neural networks through deep visualization. CoRR abs/1506.06579 (2015). http://arxiv.org/abs/1506.06579

  48. Zhang, Q., Wu, Y.N., Zhu, S.: Interpretable convolutional neural networks. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018. pp. 8827–8836. IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00920

  49. Zhang, Q., Yang, Y., Ma, H., Wu, Y.N.: Interpreting CNNs via decision trees. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, 16–20 June 2019, pp. 6261–6270. Computer Vision Foundation/IEEE (2019). https://doi.org/10.1109/CVPR.2019.00642

  50. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. In: 5th International Conference on Learning Representations (ICLR 2017). Toulon, France, 24–26 April 2017. https://openreview.net/forum?id=r1Ue8Hcxg

Download references

Acknowledgements

This paper has been partially supported by (i) the H2020 project “StairwAI” (G.A. 101017142), and (ii) the CHIST-ERA IV project “EXPECTATION” (G.A. CHIST-ERA-19-XAI-005).

Author information

Authors and Affiliations

  1. Alma Mater Studiorum—University of Bologna, Cesena, Italy

    Andrea Agiollo, Giovanni Ciatto & Andrea Omicini

  2. The Research Hub by Electrolux Professional S.p.A., 33170, Pordenone, PN, Italy

    Andrea Agiollo

Authors
  1. Andrea Agiollo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giovanni Ciatto
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Omicini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrea Agiollo .

Editor information

Editors and Affiliations

  1. University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland

    Dr. Davide Calvaresi

  2. University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Amro Najjar

  3. Victoria University of Wellington, Wellington, New Zealand

    Prof. Michael Winikoff

  4. Umeå University, Umeå, Sweden

    Kary Främling

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Agiollo, A., Ciatto, G., Omicini, A. (2021). Shallow2Deep: Restraining Neural Networks Opacity Through Neural Architecture Search. In: Calvaresi, D., Najjar, A., Winikoff, M., Främling, K. (eds) Explainable and Transparent AI and Multi-Agent Systems. EXTRAAMAS 2021. Lecture Notes in Computer Science(), vol 12688. Springer, Cham. https://doi.org/10.1007/978-3-030-82017-6_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82017-6_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82016-9

  • Online ISBN: 978-3-030-82017-6

  • eBook Packages: Computer ScienceComputer Science (R0)