Skip to main content

Message Passing Neural Networks

Part of the Lecture Notes in Physics book series (LNP,volume 968)

Abstract

Supervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science. Luckily, several promising and closely related neural network models invariant to molecular symmetries have already been described in the literature. These models learn a message passing algorithm and aggregation procedure to compute a function of their entire input graph. In this chapter, we describe a general common framework for learning representations on graph data called message passing neural networks (MPNNs) and show how several prior neural network models for graph data fit into this framework. This chapter contains large overlap with Gilmer et al. (International Conference on Machine Learning, pp. 1263–1272, 2017), and has been modified to highlight more recent extensions to the MPNN framework.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-40245-7_10
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-40245-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 10.1

Notes

  1. 1.

    By comparison, the inference time of the neural networks discussed in this work is 300k times faster.

  2. 2.

    As reported in Schütt et al. [15]. The model was trained on a different train/test split with 100k training samples vs 110k used in our experiments.

References

  1. Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, et al., (2016, preprint). arXiv:1609.08144

    Google Scholar 

  2. G. Hinton, L. Deng, D. Yu, G.E. Dahl, A.-R. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T.N. Sainath, et al., IEEE Signal. Proc. Mag. 29(6), 82 (2012)

    ADS  CrossRef  Google Scholar 

  3. A. Krizhevsky, I. Sutskever, G.E. Hinton, Advances in Neural Information Processing Systems (The MIT Press, Cambridge, 2012), pp. 1097–1105

    Google Scholar 

  4. K. Hansen, F. Biegler, R. Ramakrishnan, W. Pronobis, O.A. von Lilienfeld, K.-R. Müller, A. Tkatchenko, J. Phys. Chem. Lett. 6(12), 2326 (2015). https://doi.org/10.1021/acs.jpclett.5b00831

    Google Scholar 

  5. B. Huang, O.A. von Lilienfeld, J. Chem. Phys. 145(16), 161102 (2016). https://doi.org/10.1063/1.4964627

    ADS  CrossRef  Google Scholar 

  6. M. Rupp, A. Tkatchenko, K.-R. Müller, O.A. von Lilienfeld, Phys. Rev. Lett. 108(5), 058301 (2012)

    ADS  CrossRef  Google Scholar 

  7. D. Rogers, M. Hahn, J. Chem. Inf. Model. 50(5), 742 (2010)

    CrossRef  Google Scholar 

  8. G. Montavon, K. Hansen, S. Fazli, M. Rupp, F. Biegler, A. Ziehe, A. Tkatchenko, O.A. von Lilienfeld, K.-R. Müller, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2012), pp. 440–448

    Google Scholar 

  9. J. Behler, M. Parrinello, Phys. Rev. Lett. 98, 146401 (2007). https://doi.org/10.1103/PhysRevLett.98.146401

    ADS  CrossRef  Google Scholar 

  10. S.S. Schoenholz, E.D. Cubuk, D.M. Sussman, E. Kaxiras, A.J. Liu, A structural approach to relaxation in glassy liquids. Nat. Phys. 12(5), 469–471 (2016)

    CrossRef  Google Scholar 

  11. M. Allamanis, M. Brockschmidt, M. Khademi, (2017, preprint). arXiv:1711.00740

    Google Scholar 

  12. T.N. Kipf, M. Welling, ArXiv e-prints (2016)

    Google Scholar 

  13. V. Zambaldi, D. Raposo, A. Santoro, V. Bapst, Y. Li, I. Babuschkin, K. Tuyls, D. Reichert, T. Lillicrap, E. Lockhart, et al. (2018, preprint). arXiv:1806.01830

    Google Scholar 

  14. D.K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, R.P. Adams, Advances in Neural Information Processing Systems (2015), pp. 2224–2232

    Google Scholar 

  15. K.T. Schütt, F. Arbabzadah, S. Chmiela, K.R. Müller, A. Tkatchenko, Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8(1), 1–8 (2017)

    CrossRef  Google Scholar 

  16. F. Faber, L. Hutchison, B. Huang, J. Gilmer, S.S. Schoenholz, G.E. Dahl, O. Vinyals, S. Kearnes, P.F. Riley, O.A. von Lilienfeld, (2017). https://arxiv.org/abs/1702.05532

  17. J. Gilmer, S.S. Schoenholz, P.F. Riley, O. Vinyals, G.E. Dahl, International Conference on Machine Learning (2017), pp. 1263–1272

    Google Scholar 

  18. R. Ramakrishnan, P.O. Dral, M. Rupp, O.A. Von Lilienfeld, Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 140022 (2014)

    CrossRef  Google Scholar 

  19. P.B. Jørgensen, K.W. Jacobsen, M.N. Schmidt (2018, preprint). arXiv:1806.03146

    Google Scholar 

  20. K.T. Schütt, P.-J. Kindermans, H.E.S. Felix, S. Chmiela, A. Tkatchenko, K.-R. Müller, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2017), pp. 991–1001

    Google Scholar 

  21. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio, (2017, preprint). arXiv:1710.10903

    Google Scholar 

  22. P.W. Battaglia, J.B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al., (2018, preprint). arXiv:1806.01261

    Google Scholar 

  23. X. Wang, R. Girshick, A. Gupta, K. He, (2017, preprint). arXiv:1711.07971

    Google Scholar 

  24. S. Kearnes, K. McCloskey, M. Berndl, V. Pande, P. Riley, J. Comput.-Aided Mol. Des. 30(8), 595 (2016)

    ADS  CrossRef  Google Scholar 

  25. Y. Li, D. Tarlow, M. Brockschmidt, R. Zemel, International Conference on Learning Representations, ICLR (2016)

    Google Scholar 

  26. K. Cho, B. Van Merriënboer, D. Bahdanau, Y. Bengio, (2014, preprint). arXiv:1409.1259

    Google Scholar 

  27. P. Battaglia, R. Pascanu, M. Lai, D.J. Rezende, K. Kavukcuoglu, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2016), pp. 4502–4510

    Google Scholar 

  28. J. Bruna, W. Zaremba, A. Szlam, Y. LeCun, (2013, preprint). arXiv:1312.6203

    Google Scholar 

  29. M. Defferrard, X. Bresson, P. Vandergheynst, Advances in Neural Information Processing Systems (Curran Associates, Red Hook, 2016), pp. 3837–3845

    Google Scholar 

  30. O. Vinyals, S. Bengio, M. Kudlur, (2015, preprint). arXiv:1511.06391

    Google Scholar 

  31. L. Ruddigkeit, R. Van Deursen, L.C. Blum, J.-L. Reymond, J. Chem. Inf. Model. 52(11), 2864 (2012)

    CrossRef  Google Scholar 

  32. D. Kingma, J. Ba, (2014, preprint). arXiv:1412.6980

    Google Scholar 

Download references

Acknowledgements

We would like to thank Lukasz Kaiser, Geoffrey Irving, Alex Graves, and Yujia Li for helpful discussions. Thank you to Adrian Roitberg for pointing out an issue with the use of partial charges in an earlier version of this work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Justin Gilmer .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E. (2020). Message Passing Neural Networks. In: Schütt, K., Chmiela, S., von Lilienfeld, O., Tkatchenko, A., Tsuda, K., Müller, KR. (eds) Machine Learning Meets Quantum Physics. Lecture Notes in Physics, vol 968. Springer, Cham. https://doi.org/10.1007/978-3-030-40245-7_10

Download citation