Skip to main content
SpringerLink
Log in

Human Trajectory Prediction via Neural Social Physics

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13694))

Included in the following conference series:

Abstract

Trajectory prediction has been widely pursued in many fields, and many model-based and model-free methods have been explored. The former include rule-based, geometric or optimization-based models, and the latter are mainly comprised of deep learning approaches. In this paper, we propose a new method combining both methodologies based on a new Neural Differential Equation model. Our new model (Neural Social Physics or NSP) is a deep neural network within which we use an explicit physics model with learnable parameters. The explicit physics model serves as a strong inductive bias in modeling pedestrian behaviors, while the rest of the network provides a strong data-fitting capability in terms of system parameter estimation and dynamics stochasticity modeling. We compare NSP with 15 recent deep learning methods on 6 datasets and improve the state-of-the-art performance by 5.56%–70%. Besides, we show that NSP has better generalizability in predicting plausible trajectories in drastically different scenarios where the density is 2–5 times as high as the testing data. Finally, we show that the physics model in NSP can provide plausible explanations for pedestrian behaviors, as opposed to black-box deep learning. Code is available: https://github.com/realcrane/Human-Trajectory-Prediction-via-Neural-Social-Physics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social LSTM: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–971 (2016)

    Google Scholar 

  2. Álvarez León, L.M., Esclarín Monreal, J., Lefébure, M., Sánchez, J.: A PDE model for computing the optical flow. In: CEDYA XVI (1999)

    Google Scholar 

  3. Antonucci, A., Papini, G.P.R., Palopoli, L., Fontanelli, D.: Generating reliable and efficient predictions of human motion: a promising encounter between physics and neural networks. arXiv preprint arXiv:2006.08429 (2020)

  4. Bartoli, F., Lisanti, G., Ballan, L., Del Bimbo, A.: Context-aware trajectory prediction. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 1941–1946. IEEE (2018)

    Google Scholar 

  5. Bendali-Braham, M., Weber, J., Forestier, G., Idoumghar, L., Muller, P.A.: Recent trends in crowd analysis: a review. Mach. Learn. Appl. 4, 100023 (2021)

    Google Scholar 

  6. Bera, A., Manocha, D.: Realtime multilevel crowd tracking using reciprocal velocity obstacles. In: 2014 22nd International Conference on Pattern Recognition, pp. 4164–4169. IEEE (2014)

    Google Scholar 

  7. Bera, A., Randhavane, T., Manocha, D.: Aggressive, tense or shy? identifying personality traits from crowd videos. In: IJCAI, pp. 112–118 (2017)

    Google Scholar 

  8. van den Berg, J., Lin, M., Manocha, D.: Reciprocal velocity obstacles for real-time multi-agent navigation. In: 2008 IEEE International Conference on Robotics and Automation (2008)

    Google Scholar 

  9. Bhattacharyya, A., Hanselmann, M., Fritz, M., Schiele, B., Straehle, C.N.: Conditional flow variational autoencoders for structured sequence prediction. In: 4th Workshop on Bayesian Deep Learning. bayesiandeeplearning. org (2019)

    Google Scholar 

  10. Cai, S., Mao, Z., Wang, Z., Yin, M., Karniadakis, G.E.: Physics-informed neural networks (pinns) for fluid mechanics: a review. Acta Mech. Sinica 37, 1727–1738 (2022)

    Google Scholar 

  11. Chaker, R., Al Aghbari, Z., Junejo, I.N.: Social network model for crowd anomaly detection and localization. Pattern Recogn. 61, 266–281 (2017)

    Article  Google Scholar 

  12. Charalambous, P., Karamouzas, I., Guy, S.J., Chrysanthou, Y.: A data-driven framework for visual crowd analysis. Comput. Graphi. Forum 33, 41–50. Wiley Online Library (2014)

    Google Scholar 

  13. Chen, R.T., Rubanova, Y., Bettencourt, J., Duvenaud, D.K.: Neural ordinary differential equations. In: 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), vol. 33 (2018)

    Google Scholar 

  14. Deo, N., Trivedi, M.M.: Trajectory forecasts in unknown environments conditioned on grid-based plans. arXiv preprint arXiv:2001.00735 (2020)

  15. Ellis, D., Sommerlade, E., Reid, I.: Modelling pedestrian trajectory patterns with gaussian processes. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 1229–1234. IEEE (2009)

    Google Scholar 

  16. Gao, J., Shi, X., Yu, J.J.: Social-dualcvae: multimodal trajectory forecasting based on social interactions pattern aware and dual conditional variational auto-encoder. arXiv preprint arXiv:2202.03954 (2022)

  17. Gong, D., Zhu, Z., Andrew, B., Wang, H.: Fine-grained differentiable physics: a yarn-level model for fabrics. In: International Conference on Learning Representations (2022)

    Google Scholar 

  18. Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2255–2264 (2018)

    Google Scholar 

  19. He, F., Xia, Y., Zhao, X., Wang, H.: Informative scene decomposition for crowd analysis, comparison and simulation guidance. ACM Trans. Graph. 4(39) (2020)

    Google Scholar 

  20. Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)

    Article  Google Scholar 

  21. Hossain, S., Johora, F.T., Müller, J.P., Hartmann, S., Reinhardt, A.: SFMGNet: A physics-based neural network to predict pedestrian trajectories. arXiv (2022)

    Google Scholar 

  22. Ivanovic, B., Pavone, M.: The trajectron: probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2375–2384 (2019)

    Google Scholar 

  23. Karamouzas, I., Sohre, N., Hu, R., Guy, S.J.: Crowd space: a predictive crowd analysis technique. ACM Trans. Graph. 37(6), 1–14 (2018)

    Article  Google Scholar 

  24. Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L.: Physics-informed machine learning. Nat. Rev. Phys. 3, 422–440 (2021)

    Article  Google Scholar 

  25. Kidger, P.: On neural differential equations (2022)

    Google Scholar 

  26. Kim, S., Bera, A., Manocha, D.: Interactive crowd content generation and analysis using trajectory-level behavior learning. In: 2015 IEEE International Symposium on Multimedia (ISM), pp. 21–26. IEEE (2015)

    Google Scholar 

  27. Kreiss, S.: Deep social force. arXiv preprint arXiv:2109.12081 (2021)

  28. Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. Comput. Graph. Forum 26, 655–664. Wiley Online Library (2007)

    Google Scholar 

  29. Li, J., Ma, H., Tomizuka, M.: Conditional generative neural system for probabilistic trajectory prediction. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6150–6156. IEEE (2019)

    Google Scholar 

  30. Liang, J., Lin, M., Koltun, V.: Differentiable cloth simulation for inverse problems. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  31. Liang, J., Jiang, L., Hauptmann, A.: SimAug: learning robust representations from 3d simulation for pedestrian trajectory prediction in unseen cameras. arXiv preprint arXiv:2004.02022 2 (2020)

  32. Liang, J., Jiang, L., Murphy, K., Yu, T., Hauptmann, A.: The garden of forking paths: Towards multi-future trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10508–10518 (2020)

    Google Scholar 

  33. Liang, J., Jiang, L., Niebles, J.C., Hauptmann, A.G., Fei-Fei, L.: Peeking into the future: predicting future person activities and locations in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5725–5734 (2019)

    Google Scholar 

  34. Liu, Y., Yan, Q., Alahi, A.: Social NCE: contrastive learning of socially-aware motion representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15118–15129 (2021)

    Google Scholar 

  35. López, A., Chaumette, F., Marchand, E., Pettré, J.: Character navigation in dynamic environments based on optical flow. Comput. Graphi. Forum 38, 181–192. Wiley Online Library (2019)

    Google Scholar 

  36. Luo, L., et al.: Agent-based human behavior modeling for crowd simulation. Comput. Anim. Virtual Worlds 19 (2008)

    Google Scholar 

  37. Mangalam, K., An, Y., Girase, H., Malik, J.: From goals, waypoints & paths to long term human trajectory forecasting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15233–15242 (2021)

    Google Scholar 

  38. Mangalam, K., Girase, H., Agarwal, S., Lee, K.-H., Adeli, E., Malik, J., Gaidon, A.: It is not the journey but the destination: endpoint conditioned trajectory prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 759–776. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_45

    Chapter  Google Scholar 

  39. Mohamed, A., Qian, K., Elhoseiny, M., Claudel, C.: Social-STGCNN: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14424–14432 (2020)

    Google Scholar 

  40. Narain, R., Golas, A., Curtis, S., Lin, M.C.: Aggregate dynamics for dense crowd simulation. In: ACM SIGGRAPH Asia 2009 papers, pp. 1–8 (2009)

    Google Scholar 

  41. Narang, S., Best, A., Curtis, S., Manocha, D.: Generating pedestrian trajectories consistent with the fundamental diagram based on physiological and psychological factors. PLoS ONE 10(4), e0117856 (2015)

    Article  Google Scholar 

  42. Oliver, N.M., Rosario, B., Pentland, A.P.: A Bayesian computer vision system for modeling human interactions. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 831–843 (2000)

    Article  Google Scholar 

  43. Pellegrini, S., Ess, A., Van Gool, L.: Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 452–465. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15549-9_33

    Chapter  Google Scholar 

  44. Rackauckas, C., et al.: Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385 (2020)

  45. Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019)

    Article  MathSciNet  Google Scholar 

  46. Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 549–565. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_33

    Chapter  Google Scholar 

  47. Sadeghian, A., Kosaraju, V., Gupta, A., Savarese, S., Alahi, A.: TrajNet: towards a benchmark for human trajectory prediction. arXiv preprint (2018)

    Google Scholar 

  48. Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Rezatofighi, H., Savarese, S.: Sophie: an attentive GAN for predicting paths compliant to social and physical constraints. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1349–1358 (2019)

    Google Scholar 

  49. Salzmann, T., Ivanovic, B., Chakravarty, P., Pavone, M.: Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12363, pp. 683–700. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58523-5_40

    Chapter  Google Scholar 

  50. Shen, S., et al.: High-order differentiable autoencoder for nonlinear model reduction. ACM Trans. Graph. 40(4) (2021)

    Google Scholar 

  51. Shen, Y., Henry, J., Wang, H., Ho, E.S.L., Komura, T., Shum, H.P.H.: Data-driven crowd motion control with multi-touch gestures. Comput. Graph. Forum (2018). https://doi.org/10.1111/cgf.13333

    Article  Google Scholar 

  52. Shi, L., Wang, L., Long, C., Zhou, S., Zhou, M., Niu, Z., Hua, G.: SGCN: sparse graph convolution network for pedestrian trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8994–9003 (2021)

    Google Scholar 

  53. Sighencea, B.I., Stanciu, R.I., Căleanu, C.D.: A review of deep learning-based methods for pedestrian trajectory prediction. Sensors 21(22), 7543 (2021)

    Article  Google Scholar 

  54. Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 2 (2015)

    Google Scholar 

  55. Su, T., Meng, Y., Xu, Y.: Pedestrian trajectory prediction via spatial interaction transformer network. In: 2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops), pp. 154–159. IEEE (2021)

    Google Scholar 

  56. Tan, Q., Pan, Z., Manocha, D.: Lcollision: Fast generation of collision-free human poses using learned non-penetration constraints. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 3913–3921 (2021)

    Google Scholar 

  57. Tan, Q., Pan, Z., Smith, B., Shiratori, T., Manocha, D.: N-penetrate: Active learning of neural collision handler for complex 3d mesh deformations. In: International Conference on Machine Learning, pp. 21037–21049. PMLR (2022)

    Google Scholar 

  58. Van Toll, W., Pettré, J.: Algorithms for microscopic crowd simulation: advancements in the 2010s. Comput. Graph. Forum 40(2) (2021)

    Google Scholar 

  59. Vemula, A., Muelling, K., Oh, J.: Social attention: modeling attention in human crowds. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4601–4607. IEEE (2018)

    Google Scholar 

  60. Virtanen, A.: Energy-based pedestrian navigation. In: Proceedings of 20th ITS World Congress, pp. 1–9 (2013)

    Google Scholar 

  61. Wan, Z., Hu, X., He, H., Guo, Y.: A learning based approach for social force model parameter estimation. In: IJCNN, pp. 4058–4064. IEEE (2017)

    Google Scholar 

  62. Wang, H., Ondřej, J., O’Sullivan, C.: Path patterns: analyzing and comparing real and simulated crowds. In: ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games 2016, pp. 49–57 (2016)

    Google Scholar 

  63. Wang, H., Ondřej, J., O’Sullivan, C.: Trending paths: a new semantic-level metric for comparing simulated and real crowd data. IEEE Trans. Visual. Comput. Graph. 99, 1–1 (2016)

    Google Scholar 

  64. Wang, H., O’Sullivan, C.: Globally Continuous and non-Marconian crowd activity analysis from videos. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 527–544. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_32

    Chapter  Google Scholar 

  65. Wang, P.: Understanding social-force model in psychological principles of collective behavior. arXiv preprint arXiv:1605.05146 (2016)

  66. Wang, X., Ma, K.T., Ng, G.W., Grimson, W.E.L.: Trajectory analysis and semantic region modeling using nonparametric hierarchical Bayesian models. Int. J. Comput. Vision 95(3), 287–312 (2011)

    Article  Google Scholar 

  67. Wei, J., Fan, W., Li, Z., Guo, Y., Fang, Y., Wang, J.: Simulating crowd evacuation in a social force model with iterative extended state observer. J. Adv. Transp. 2020 (2020)

    Google Scholar 

  68. Werling, K., Omens, D., Lee, J., Exarchos, I., Liu, C.K.: Fast and feature-complete differentiable physics for articulated rigid bodies with contact. CoRR abs/2103.16021 (2021)

    Google Scholar 

  69. Wolinski, D., J. Guy, S., Olivier, A.H., Lin, M., Manocha, D., Pettré, J.: Parameter estimation and comparative evaluation of crowd simulations. Comput. Graph. Forum 33(2), 303–312 (2014)

    Google Scholar 

  70. Xia, B., Wong, C., Peng, Q., Yuan, W., You, X.: CscNet: contextual semantic consistency network for trajectory prediction in crowded spaces. Pattern Recog. 126,, 108552 (2022)

    Google Scholar 

  71. Zeng, W., Chen, P., Nakamura, H., Iryo-Asano, M.: Application of social force model to pedestrian behavior analysis at signalized crosswalk. Transp. Res. Part C Emerg. Ttechnol. 40, 143–159 (2014)

    Article  Google Scholar 

  72. Zhang, Z., Jimack, P.K., Wang, H.: MeshingNet3D: efficient generation of adapted tetrahedral meshes for computational mechanics. Adv. Eng. Softw. 157, 103021 (2021)

    Google Scholar 

  73. Zhang, Z., Wang, Y., Jimack, P.K., Wang, H.: MeshingNet: a new mesh generation method based on deep learning. In: Krzhizhanovskaya, W., et al. (eds.) ICCS 2020. LNCS, vol. 12139, pp. 186–198. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50420-5_14

    Chapter  Google Scholar 

  74. Zhong, Y.D., Dey, B., Chakraborty, A.: Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint arXiv:1909.12077 (2019)

  75. Zhou, B., Wang, X., Tang, X.: Random field topic model for semantic region analysis in crowded scenes from tracklets. In: CVPR 2011, pp. 3441–3448. IEEE (2011)

    Google Scholar 

  76. Zhou, H., Ren, D., Yang, X., Fan, M., Huang, H.: Sliding sequential CVAE with time variant socially-aware rethinking for trajectory prediction. arXiv preprint arXiv:2110.15016 (2021)

  77. Zubov, K., et al.: Neuralpde: automating physics-informed neural networks (pinns) with error approximations. CoRR abs/2107.09443 (2021)

    Google Scholar 

Download references

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 899739 CrowdDNA.

Author information

Authors and Affiliations

  1. University of Leeds, Leeds, UK

    Jiangbei Yue & He Wang

  2. University of Maryland at College Park, College Park, USA

    Dinesh Manocha

Authors
  1. Jiangbei Yue
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dinesh Manocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. He Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to He Wang .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4708 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yue, J., Manocha, D., Wang, H. (2022). Human Trajectory Prediction via Neural Social Physics. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13694. Springer, Cham. https://doi.org/10.1007/978-3-031-19830-4_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19830-4_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19829-8

  • Online ISBN: 978-3-031-19830-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation