Abstract
The newly appeared \({\cal G}\)-SGD algorithm can only heuristically find the basis path set in a simple neural network, so its generalization to a more practical network is hindered. From the perspective of graph theory, the BasisPathSetSearching problem is formulated to find the basis path set in a complicated fully connected neural network. This paper proposes algorithm DEAH to hierarchically solve the BasisPathSetSearching problem by eliminating the path dependencies. For this purpose, the authors discover the underlying cause of the path dependency between two independent substructures. The path subdivision chain is proposed to effectively eliminate the path dependency, both inside the chain and between chains. The theoretical proofs and the analysis of time complexity are presented for Algorithm DEAH. This paper therefore provides one methodology to find the basis path set in a general and practical neural network.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Wu S, Dimakis A G, and Sanghavi S, Learning distributions generated by one-layer ReLU networks, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 2019.
Wang Y, Liu Y T, and Ma Z M, The scale-invariant space for attention layer in neural network, Neurocomputing, 2020, 392: 1–10.
Neeyshabur B, Salakhutdinov R R, and Srebro N, Path-sgd: Path normalized optimization in deep neural networks, NIPS 15 Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015, 2422–2430.
Zheng S X, Meng Q, Zhang H S, et al., Capacity control of ReLU neural networks by basis-path norm, Thirty-third AAAI Conference on Artificial Intelligence (AAAI2019), 2019.
Meng Q, Zheng S X, Zhang H S, et al., G-SGD: Optimizing ReLU neural networks in its positively scale-invariant space, International Conference of Learning Representations (ICLR2019), 2019.
Rumelhart D E, Hinton G E, and Williams R J, Learning representations by back-propagating errors, Nature, 1986, 323(6088): 533–536.
Fan F, Xiong J, Li M, et al., On interpretability of artificial neural networks: A survey, IEEE Transactions on Radiation and Plasma Medical Sciences, 2021, 5(6): 741–760.
Guan C, Wang X, Zhang Q, et al., Towards a deep and unified understanding of deep neural models in NLP, Proceedings of the 36th International Conference on Machine Learning, Long Beach, California, USA, 2019.
Hooker S, Erhan D, Kidermans P, et al., A Benchmark for Interpretability Methods in Deep Neural Networks, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada, 2019.
Inoue K, Expressive numbers of two or more hidden layer ReLU neural networks, 2019 Seventh International Symposium on Computing and Networking workshops (CANDARW 2019), 2019.
Zhang Q S, Cao R M, Shi F, et al., Interpreting CNN knowledge via an explanatory graph, The Thirty-Second AAAI Conference on Artificial Intelligence, 2018, 4454–4463.
Wu M, Wicker M, Ruan W, et al., A game-based approximate verification of deep neural networks with provable guarantees, Theoretical Computer Science, 2020, 807: 298–329.
Ensign D, Neville S, Paul A, et al., The complexity of explaining neural networks through (group) invariants, Theoretical Computer Science, 2020, 808: 74–85.
Xing R T, Xiao M, Zhang Y Z, et al., Stability and Hopf bifurcation analysis of an (n + m)-neuron double-ring neural network model with multiple time delays, Journal of Systems Science & Complexity, 2021, DOI: https://doi.org/10.1007/s11424-021-0108-2.
Zhu J P, Meng Q, Chen W, et al., Interpreting basis path set in neural networks, Journal of Systems Science and Complexity, 2020, 33(1): 1–13.
Corberan A and Laporte G, Arc Routing Problems, Methods, and Applications, Society for Industrial and Applied Mathematics, 2015.
Jensen J B and Gutin G Z, Digraphs: Theory, Algorithms and Applications (Second Edition), Springer, New York, 2009.
Korte B and Vygen J, Combinatorial Optimization, Theory and Algorithm (Fifth Edition), Springer, New York, 2012.
Babu C S and Diwan A A, Subdivisions of graphs: A generalization of paths and cycles, Discrete Mathematics, 2008, 308(19): 4479–4486.
Bondy J A and Murty U S R, Graph Theory, Section 10.1, 2008.
Dettlaff M, Raczek J, and Yero I G, Edge subdivision and edge multisubdivision versus some domination related parameters in generalized corona graph, Opuscula Mathematica, 2016, 36(5): 575–588.
Chaieb M, Jemai J, and Mellouli K, A hierarchical decomposition framework for modeling combinatorial optimization problems, Procedia Computer Science, 2015, 60: 478–487.
Chang Y, Tang H, Cheng Y, et al., Dynamic hierarchical energy efficient method based on combinatorial optimization for wireless sensor networks, Sensors, 2017, 17(7): 1665.
Ochiai H, Kanazawa T, Tamura K, et al., Combinatorial optimization method based on hierarchical structure in solution space, Electronics and communications in Japan, 2016, 99(18): 25–37.
Racke H, Optimal hierarchical decompositions for congestion minimization in networks, Proceedings of the 40th Annual ACM Symposium on Theory of Computing, 2008, 255–264.
Author information
Authors and Affiliations
Corresponding authors
Additional information
This research was supported by Project for Innovation Team (Cultivation) of Yunnan Province under Grant No. 202005AE160006 and Key Project of Yunnan Provincial Science and Technology Department and Yunnan University under Grant No. 2018FY001014.
This paper was recommended for publication by Editor LI Qizhai.
Rights and permissions
About this article
Cite this article
Zhu, J., Meng, Q., Chen, W. et al. Constructing the Basis Path Set by Eliminating the Path Dependency. J Syst Sci Complex 35, 1944–1962 (2022). https://doi.org/10.1007/s11424-022-0208-7
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11424-022-0208-7