Quantum-Chemical Insights from Interpretable Atomistic Neural Networks

  • Kristof T. Schütt
  • Michael Gastegger
  • Alexandre TkatchenkoEmail author
  • Klaus-Robert MüllerEmail author
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11700)


With the rise of deep neural networks for quantum chemistry applications, there is a pressing need for architectures that, beyond delivering accurate predictions of chemical properties, are readily interpretable by researchers. Here, we describe interpretation techniques for atomistic neural networks on the example of Behler–Parrinello networks as well as the end-to-end model SchNet. Both models obtain predictions of chemical properties by aggregating atom-wise contributions. These latent variables can serve as local explanations of a prediction and are obtained during training without additional cost. Due to their correspondence to well-known chemical concepts such as atomic energies and partial charges, these atom-wise explanations enable insights not only about the model but more importantly about the underlying quantum-chemical regularities. We generalize from atomistic explanations to 3d space, thus obtaining spatially resolved visualizations which further improve interpretability. Finally, we analyze learned embeddings of chemical elements that exhibit a partial ordering that resembles the order of the periodic table. As the examined neural networks show excellent agreement with chemical knowledge, the presented techniques open up new venues for data-driven research in chemistry, physics and materials science.



This work was supported by the Federal Ministry of Education and Research (BMBF) for the Berlin Big Data Center BBDC (01IS14013A) and the Berlin Center for Machine Learning (01IS18037A). Additional support was provided by the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement NO 792572. This research was supported by Institute for Information & Communications Technology Promotion and funded by the Korea government (MSIT) (No. 2017-0-00451, No. 2017-0-01779). A.T. acknowledges support from the European Research Council (ERC-CoG grant BeStMo).


  1. 1.
    Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 10(7), e0130140 (2015)CrossRefGoogle Scholar
  2. 2.
    Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., Müller, K.R.: How to explain individual classification decisions. J. Mach. Learn. Res. 11, 1803–1831 (2010)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Bajorath, J.: Integration of virtual and high-throughput screening. Nat. Rev. Drug Discovery 1(11), 882 (2002)CrossRefGoogle Scholar
  4. 4.
    Bartók, A.P., Kondor, R., Csányi, G.: On representing chemical environments. Phys. Rev. B 87(18), 184115 (2013)CrossRefGoogle Scholar
  5. 5.
    Bartók, A.P., Payne, M.C., Kondor, R., Csányi, G.: Gaussian approximation potentials: the accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett. 104(13), 136403 (2010)CrossRefGoogle Scholar
  6. 6.
    Bartók, A.P., Csányi, G.: Gaussian approximation potentials: a brief tutorial introduction. Int. J. Quantum Chem. 115(16), 1051–1057 (2015)CrossRefGoogle Scholar
  7. 7.
    Becke, A.D.: Density-functional exchange-energy approximation with correct asymptotic behavior. Phys. Rev. A 38(6), 3098 (1988)CrossRefGoogle Scholar
  8. 8.
    Behler, J.: Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys. 134(7), 074106 (2011)CrossRefGoogle Scholar
  9. 9.
    Behler, J., Parrinello, M.: Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett. 98(14), 146401 (2007)CrossRefGoogle Scholar
  10. 10.
    Blum, L.C., Reymond, J.L.: 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc. 131, 8732 (2009)CrossRefGoogle Scholar
  11. 11.
    Brockherde, F., Voigt, L., Li, L., Tuckerman, M.E., Burke, K., Müller, K.R.: Bypassing the Kohn-Sham equations with machine learning. Nat. Commun. 8, 872 (2017)CrossRefGoogle Scholar
  12. 12.
    Chen, H., et al.: Carbonophosphates: a new family of cathode materials for Li-Ion batteries identified computationally. Chem. Mater. 24(11), 2009–2016 (2012)CrossRefGoogle Scholar
  13. 13.
    Chmiela, S., Tkatchenko, A., Sauceda, H.E., Poltavsky, I., Schütt, K.T., Müller, K.R.: Machine learning of accurate energy-conserving molecular force fields. Sci. Adv. 3(5), e1603015 (2017)CrossRefGoogle Scholar
  14. 14.
    Hohenberg, P., Kohn, W.: Inhomogeneous electron gas. Phys. Rev. 136, B864–B871 (1964)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Duvenaud, D.K., et al.: Convolutional networks on graphs for learning molecular fingerprints. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) NIPS, pp. 2224–2232 (2015)Google Scholar
  16. 16.
    Eickenberg, M., Exarchakis, G., Hirn, M., Mallat, S.: Solid harmonic wavelet scattering: predicting quantum molecular energy from invariant descriptors of 3D electronic densities. In: Advances in Neural Information Processing Systems 30, pp. 6543–6552. Curran Associates, Inc., Long Beach (2017)Google Scholar
  17. 17.
    Faber, F.A., et al.: Prediction errors of molecular machine learning models lower than hybrid DFT error. J. Chem. Theory Comput. 13(11), 5255–5264 (2017)CrossRefGoogle Scholar
  18. 18.
    Gastegger, M., Behler, J., Marquetand, P.: Machine learning molecular dynamics for the simulation of infrared spectra. Chem. Sci. 8(10), 6924–6935 (2017)CrossRefGoogle Scholar
  19. 19.
    Gastegger, M., Schwiedrzik, L., Bittermann, M., Berzsenyi, F., Marquetand, P.: wACSF-weighted atom-centered symmetry functions as descriptors in machine learning potentials. J. Chem. Phys. 148(24), 241709 (2018)CrossRefGoogle Scholar
  20. 20.
    Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: Proceedings of the 34th International Conference on Machine Learning, pp. 1263–1272 (2017)Google Scholar
  21. 21.
    Hansen, K., et al.: Machine learning predictions of molecular properties: accurate many-body potentials and nonlocality in chemical space. J. Phys. Chem. Lett. 6, 2326 (2015)CrossRefGoogle Scholar
  22. 22.
    Hansen, K., et al.: Assessment and validation of machine learning methods for predicting molecular atomization energies. J. Chem. Theory Comput. 9(8), 3404–3419 (2013)CrossRefGoogle Scholar
  23. 23.
    Hautier, G., Jain, A., Mueller, T., Moore, C., Ong, S.P., Ceder, G.: Designing multielectron lithium-ion phosphate cathodes by mixing transition metals. Chem. Mater. 25(10), 2064–2074 (2013)CrossRefGoogle Scholar
  24. 24.
    Huo, H., Rupp, M.: Unified representation for machine learning of molecules and crystals. arXiv preprint. arXiv:1704.06439 (2017)
  25. 25.
    Kang, K., Meng, Y.S., Bréger, J., Grey, C.P., Ceder, G.: Electrodes with high power and high capacity for rechargeable lithium batteries. Science 311(5763), 977–980 (2006)CrossRefGoogle Scholar
  26. 26.
    Kearnes, S., McCloskey, K., Berndl, M., Pande, V., Riley, P.: Molecular graph convolutions: moving beyond fingerprints. J. Comput. Aided Mol. Des. 30(8), 595–608 (2016)CrossRefGoogle Scholar
  27. 27.
    Kindermans, P.J., et al.: Learning how to explain neural networks: PatternNet and PatternAttribution. In: International Conference on Learning Representations (ICLR) (2018)Google Scholar
  28. 28.
    Kohn, W., Sham, L.J.: Self-consistent equations including exchange and correlation effects. Phys. Rev. 140, A1133–A1138 (1965). Scholar
  29. 29.
    Lee, C., Yang, W., Parr, R.G.: Development of the colle-salvetti correlation-energy formula into a functional of the electron density. Phys. Rev. B 37(2), 785 (1988)CrossRefGoogle Scholar
  30. 30.
    von Lilienfeld, O.A.: First principles view on chemical compound space: gaining rigorous atomistic control of molecular properties. Int. J. Quantum Chem. 113(12), 1676–1689 (2013)CrossRefGoogle Scholar
  31. 31.
    Lubbers, N., Smith, J.S., Barros, K.: Hierarchical modeling of molecular energies using a deep neural network. J. Chem. Phys. 148(24), 241715 (2018)CrossRefGoogle Scholar
  32. 32.
    Montavon, G., et al.: Machine learning of molecular electronic properties in chemical compound space. New J. Phys. 15(9), 095003 (2013)CrossRefGoogle Scholar
  33. 33.
    Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recogn. 65, 211–222 (2017)CrossRefGoogle Scholar
  34. 34.
    Montavon, G., Samek, W., Müller, K.R.: Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 73, 1–15 (2018)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Olivares-Amaya, R., et al.: Accelerated computational discovery of high-performance materials for organic photovoltaics by means of cheminformatics. Energy Environ. Sci. 4, 4849–4861 (2011)CrossRefGoogle Scholar
  36. 36.
    Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1713–1721 (2015)Google Scholar
  37. 37.
    Pronobis, W., Tkatchenko, A., Müller, K.R.: Many-body descriptors for predicting molecular properties with machine learning: analysis of pairwise and three-body interactions in molecules. J. Chem. Theory Comput. 14(6), 2991–3003 (2018). Scholar
  38. 38.
    Ramakrishnan, R., Dral, P.O., Rupp, M., von Lilienfeld, O.A.: Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 140022 (2014)CrossRefGoogle Scholar
  39. 39.
    Ruddigkeit, L., Van Deursen, R., Blum, L.C., Reymond, J.L.: Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17. J. Chem. Inf. Model. 52(11), 2864–2875 (2012)CrossRefGoogle Scholar
  40. 40.
    Rupp, M., Tkatchenko, A., Müller, K.R., Von Lilienfeld, O.A.: Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. 108(5), 058301 (2012)CrossRefGoogle Scholar
  41. 41.
    Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.R.: Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Netw. Learn. Syst. 28(11), 2660–2673 (2017)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Schütt, K.T., Arbabzadah, F., Chmiela, S., Müller, K.R., Tkatchenko, A.: Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8, 13890 (2017)CrossRefGoogle Scholar
  43. 43.
    Schütt, K.T., Glawe, H., Brockherde, F., Sanna, A., Müller, K.R., Gross, E.: How to represent crystal structures for machine learning: towards fast prediction of electronic properties. Phys. Rev. B 89(20), 205118 (2014)CrossRefGoogle Scholar
  44. 44.
    Schütt, K.T., Kindermans, P.J., Sauceda, H.E., Chmiela, S., Tkatchenko, A., Müller, K.R.: SchNet: a continuous-filter convolutional neural network for modeling quantum interactions. In: Advances in Neural Information Processing Systems, vol. 30, pp. 992–1002 (2017)Google Scholar
  45. 45.
    Schütt, K.T., Sauceda, H.E., Kindermans, P.J., Tkatchenko, A., Müller, K.R.: SchNet - a deep learning architecture for molecules and materials. J. Chem. Phys. 148(24), 241722 (2018)CrossRefGoogle Scholar
  46. 46.
    Shoichet, B.K.: Virtual screening of chemical libraries. Nature 432(7019), 862 (2004)CrossRefGoogle Scholar
  47. 47.
    Sifain, A.E., et al.: Discovering a transferable charge assignment model using machine learning. J. Phys. Chem. Lett. 9, 4495–4501 (2018)CrossRefGoogle Scholar
  48. 48.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint. arXiv:1312.6034 (2013)
  49. 49.
    Vollhardt, K.P.C., Schore, N.E.: Organic Chemistry; Palgrave Version: Structure and Function. Palgrave Macmillan, Basingstoke (2014)CrossRefGoogle Scholar
  50. 50.
    Xie, T., Grossman, J.C.: Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120(14), 145301 (2018)CrossRefGoogle Scholar
  51. 51.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). Scholar
  52. 52.
    Zintgraf, L.M., Cohen, T.S., Adel, T., Welling, M.: Visualizing deep neural network decisions: prediction difference analysis. arXiv preprint. arXiv:1702.04595 (2017)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Technische Universität BerlinBerlinGermany
  2. 2.University of LuxembourgLuxembourgLuxembourg
  3. 3.Max-Planck-Institut für InformatikSaarbrückenGermany
  4. 4.Korea UniversitySeongbuk-gu, SeoulKorea

Personalised recommendations