Efficient Interpretation of Deep Learning Models Using Graph Structure and Cooperative Game Theory: Application to ASD Biomarker Discovery

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11492)


Discovering imaging biomarkers for autism spectrum disorder (ASD) is critical to help explain ASD and predict or monitor treatment outcomes. Toward this end, deep learning classifiers have recently been used for identifying ASD from functional magnetic resonance imaging (fMRI) with higher accuracy than traditional learning strategies. However, a key challenge with deep learning models is understanding just what image features the network is using, which can in turn be used to define the biomarkers. Current methods extract biomarkers, i.e., important features, by looking at how the prediction changes if “ignoring” one feature at a time. However, this can lead to serious errors if the features are conditionally dependent. In this work, we go beyond looking at only individual features by using Shapley value explanation (SVE) from cooperative game theory. Cooperative game theory is advantageous here because it directly considers the interaction between features and can be applied to any machine learning method, making it a novel, more accurate way of determining instance-wise biomarker importance from deep learning models. A barrier to using SVE is its computational complexity: \(2^N\) given N features. We explicitly reduce the complexity of SVE computation by two approaches based on the underlying graph structure of the input data: (1) only consider the centralized coalition of each feature; (2) a hierarchical pipeline which first clusters features into small communities, then applies SVE in each community. Monte Carlo approximation can be used for large permutation sets. We first validate our methods on the MNIST dataset and compare to human perception. Next, to insure plausibility of our biomarker results, we train a Random Forest (RF) to classify ASD/control subjects from fMRI and compare SVE results to standard RF-based feature importance. Finally, we show initial results on ranked fMRI biomarkers using SVE on a deep learning classifier for the ASD/control dataset.


  1. 1.
    Goldani, A.A.S., Downs, S.R., Widjaja, F., Lawton, B., Hendren, R.L.: Biomarkers in autism. Front. Psychiatry 5, 100 (2014)CrossRefGoogle Scholar
  2. 2.
    Li, X., Dvornek, N.C., Zhuang, J., Ventola, P., Duncan, J.S.: Brain biomarker interpretation in ASD using deep learning and fMRI. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 206–214. Springer, Cham (2018). Scholar
  3. 3.
    Kaiser, M.D., et al.: Neural signatures of autism. Proc. Natl. Acad. Sci. 107(49), 21223–21228 (2010)CrossRefGoogle Scholar
  4. 4.
    Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, pp. 4765–4774 (2017)Google Scholar
  5. 5.
    Chen, J., Song, L., Wainwright, M.J., Jordan, M.I.: L-shapley and C-shapley: efficient model interpretation for structured data. arXiv preprint arXiv:1808.02610 (2018)
  6. 6.
    Kononenko, I., Strumbelj, E.: An efficient explanation of individual classifications using game theory. J. Mach. Learn. Res. 11(Jan), 1–18 (2010)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Shapley, L.S.: A value for n-person games. Contrib. Theory Games 2(28), 307–317 (1953)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Zintgraf, L.M., Cohen, T.S., Adel, T., Welling, M.: Visualizing deep neural network decisions: prediction difference analysis. arXiv preprint arXiv:1702.04595 (2017)
  9. 9.
    Clauset, A., Newman, M.E., Moore, C.: Finding community structure in very large networks. Phys. Rev. E 70(6), 066111 (2004)CrossRefGoogle Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  11. 11.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  12. 12.
    Tzourio-Mazoyer, N., et al.: Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. Neuroimage 15, 273–289 (2002)CrossRefGoogle Scholar
  13. 13.
    Newman, M.: Networks. Oxford University Press, Oxford (2018)CrossRefGoogle Scholar
  14. 14.
    Li, X., et al.: 2-channel convolutional 3D deep neural network (2CC3D) for fMRI analysis: ASD classification and feature learning. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1252–1255. IEEE (2018)Google Scholar
  15. 15.
    Young, R.C., Biggs, J.T., Ziegler, V.E., Meyer, D.A.: A rating scale for mania: reliability, validity and sensitivity. Br. J. Psychiatry 133(5), 429–435 (1978)CrossRefGoogle Scholar
  16. 16.
    Yarkoni, T., Poldrack, R.A., Nichols, T.E., Van Essen, D.C., Wager, T.D.: Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8(8), 665 (2011)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Biomedical EngineeringYale UniversityNew HavenUSA
  2. 2.Radiology and Biomedical ImagingYale School of MedicineNew HavenUSA
  3. 3.Child Study CenterYale School of MedicineNew HavenUSA
  4. 4.Electrical EngineeringYale UniversityNew HavenUSA
  5. 5.Statistics and Data ScienceYale UniversityNew HavenUSA

Personalised recommendations