Skip to main content
SpringerLink
Log in

It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12347))

Included in the following conference series:

Abstract

Human trajectory forecasting with multiple socially interacting agents is of critical importance for autonomous navigation in human environments, e.g., for self-driving cars and social robots. In this work, we present Predicted Endpoint Conditioned Network (PECNet) for flexible human trajectory prediction. PECNet infers distant trajectory endpoints to assist in long-range multi-modal trajectory prediction. A novel non-local social pooling layer enables PECNet to infer diverse yet socially compliant trajectories. Additionally, we present a simple “truncation-trick” for improving diversity and multi-modal trajectory prediction performance. We show that PECNet improves state-of-the-art performance on the Stanford Drone trajectory prediction benchmark by \({\sim }20.9\%\) and on the ETH/UCY benchmark by \({\sim }40.8\%\) (Code available at project homepage: https://karttikeya.github.io/publication/htf/).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bennewitz, M., Burgard, W., Thrun, S.: Learning motion patterns of persons for mobile service robots. In: Proceedings of the 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 4, pp. 3601–3606. IEEE (2002)

    Google Scholar 

  2. Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics. Intelligent Robotics and Autonomous Agents Series. MIT Press, Cambridge (2005)

    MATH  Google Scholar 

  3. Baker, C.L., Saxe, R., Tenenbaum, J.B.: Action understanding as inverse planning. Cognition 113(3), 329–349 (2009)

    Article  Google Scholar 

  4. Ziebart, B.D.,et al.: Planning-based prediction for pedestrians. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3931–3936. IEEE (2009)

    Google Scholar 

  5. Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 549–565. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_33

    Chapter  Google Scholar 

  6. Pellegrini, S., Ess, A., Van Gool, L.: Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 452–465. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15549-9_33

    Chapter  Google Scholar 

  7. Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. In: Computer Graphics Forum, vol. 26, pp. 655–664. Wiley Online Library (2007)

    Google Scholar 

  8. Rudenko, A., Palmieri, L., Herman, M., Kitani, K.M., Gavrila, D.M., Arras, K.O.: Human motion trajectory prediction: a survey. arXiv e-prints (2019)

    Google Scholar 

  9. Kruse, E., Wahl, F.M.: Camera-based observation of obstacle motions to derive statistical data for mobile robot motion planning. In: Proceedings of the 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), vol. 1, pp. 662–667. IEEE (1998)

    Google Scholar 

  10. Liao, L., Fox, D., Hightower, J., Kautz, H., Schulz, D.: Voronoi tracking: location estimation using sparse and noisy sensor data. In: Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003),(Cat. No. 03CH37453), vol. 1, pp. 723–728. IEEE (2003)

    Google Scholar 

  11. Bennewitz, M., Burgard, W., Cielniak, G., Thrun, S.: Learning motion patterns of people for compliant robot motion. Int. J. Robot. Res. 24(1), 31–48 (2005)

    Article  Google Scholar 

  12. Tay, M.K.C., Laugier, C.: Modelling smooth paths using Gaussian processes. In: Laugier, C., Siegwart, R. (eds.) Field and Service Robotics. STAR, vol. 42, pp. 381–390. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-75404-6_36

    Chapter  Google Scholar 

  13. Käfer, E., Hermes, C., Wöhler, C., Ritter, H., Kummert, F.: Recognition of situation classes at road intersections. In: 2010 IEEE International Conference on Robotics and Automation, pp. 3960–3965. IEEE (2010)

    Google Scholar 

  14. Aoude, G., Joseph, J., Roy, N., How, J.: Mobile agent trajectory prediction using Bayesian nonparametric reachability trees. In: Infotech@ Aerospace 2011, p. 1512 (2011)

    Google Scholar 

  15. Keller, C.G., Gavrila, D.M.: Will the pedestrian cross? A study on pedestrian path prediction. IEEE Trans. Intell. Transp. Syst. 15(2), 494–506 (2013)

    Article  Google Scholar 

  16. Goldhammer, M., Doll, K., Brunsmann, U., Gensler, A., Sick, B.: Pedestrian’s trajectory forecast in public traffic with artificial neural networks. In: 2014 22nd International Conference on Pattern Recognition, pp. 4110–4115. IEEE (2014)

    Google Scholar 

  17. Xiao, S., Wang, Z., Folkesson, J.: Unsupervised robot learning to predict person motion. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 691–696. IEEE (2015)

    Google Scholar 

  18. Kucner, T.P., Magnusson, M., Schaffernicht, E., Bennetts, V.H., Lilienthal, A.J.: Enabling flow awareness for mobile robots in partially observable environments. IEEE Robot. Autom. Lett. 2(2), 1093–1100 (2017)

    Article  Google Scholar 

  19. Kitani, K.M., Ziebart, B.D., Bagnell, J.A., Hebert, M.: Activity forecasting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 201–214. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33765-9_15

    Chapter  Google Scholar 

  20. Ballan, L., Castaldo, F., Alahi, A., Palmieri, F., Savarese, S.: Knowledge transfer for scene-specific motion prediction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 697–713. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_42

    Chapter  Google Scholar 

  21. Kim, B.D., Kang, C.M., Kim, J., Lee, S.H., Chung, C.C., Choi, J.W.: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 399–404. IEEE (2017)

    Google Scholar 

  22. Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)

    Article  Google Scholar 

  23. Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 935–942. IEEE (2009)

    Google Scholar 

  24. Yamaguchi, K., Berg, A.C., Ortiz, L.E., Berg, T.L.: Who are you with and where are you going? In: CVPR 2011, pp. 1345–1352. IEEE (2011)

    Google Scholar 

  25. Alahi, A., Ramanathan, V., Fei-Fei, L.: Socially-aware large-scale crowd forecasting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2203–2210 (2014)

    Google Scholar 

  26. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  27. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social LSTM: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–971 (2016)

    Google Scholar 

  28. Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P.H.S., Chandraker, M.: Desire: Distant future prediction in dynamic scenes with interacting agents. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 336–345 (2017)

    Google Scholar 

  29. Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2255–2264 (2018)

    Google Scholar 

  30. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  31. Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Rezatofighi, H., Savarese, S.: SoPhie: an attentive GAN for predicting paths compliant to social and physical constraints. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1349–1358 (2019)

    Google Scholar 

  32. Zou, H., Su, H., Song, S., Zhu, J.: Understanding human behaviors in crowds by imitating the decision-making process. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  33. Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Advances in Neural Information Processing Systems, pp. 4565–4573 (2016)

    Google Scholar 

  34. Rehder, E., Kloeden, H.: Goal-directed pedestrian prediction. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 50–58 (2015)

    Google Scholar 

  35. Rehder, E., Wirth, F., Lauer, M., Stiller, C.: Pedestrian prediction by planning using deep neural networks. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–5. IEEE (2018)

    Google Scholar 

  36. Rhinehart, N., McAllister, R., Kitani, K., Levine, S.: PRECOG: PREdiction conditioned on goals in visual multi-agent settings. arXiv preprint arXiv:1905.01296 (2019)

  37. Li, J., Ma, H., Tomizuka, M.: Conditional generative neural system for probabilistic trajectory prediction. arXiv preprint arXiv:1905.01631 (2019)

  38. Bhattacharyya, A., Hanselmann, M., Fritz, M., Schiele, B., Straehle, C.-N.: Conditional flow variational autoencoders for structured sequence prediction. arXiv preprint arXiv:1908.09008 (2019)

  39. Deo, N., Trivedi, M.M.: Trajectory forecasts in unknown environments conditioned on grid-based plans. arXiv preprint arXiv:2001.00735 (2020)

  40. Sadeghian, A., Kosaraju, V., Gupta, A., Savarese, S., Alahi, A.: TrajNet: towards a benchmark for human trajectory prediction. arXiv preprint (2018)

    Google Scholar 

  41. Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)

  42. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  43. Liang, J., Jiang, L., Hauptmann, A.: SimAug: learning robust representations from 3D simulation for pedestrian trajectory prediction in unseen cameras (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, Berkeley, USA

    Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal & Jitendra Malik

  2. Toyota Research Institute, Ann Arbor, USA

    Kuan-Hui Lee & Adrien Gaidon

  3. Stanford University, Stanford, USA

    Ehsan Adeli

Authors
  1. Karttikeya Mangalam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harshayu Girase
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shreyas Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kuan-Hui Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ehsan Adeli
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jitendra Malik
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Adrien Gaidon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karttikeya Mangalam .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pptx 25268 KB)

Supplementary material 2 (mp4 1288 KB)

Supplementary material 3 (mp4 539 KB)

Supplementary material 4 (pdf 24093 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mangalam, K. et al. (2020). It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12347. Springer, Cham. https://doi.org/10.1007/978-3-030-58536-5_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58536-5_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58535-8

  • Online ISBN: 978-3-030-58536-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation