It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12347)


Human trajectory forecasting with multiple socially interacting agents is of critical importance for autonomous navigation in human environments, e.g., for self-driving cars and social robots. In this work, we present Predicted Endpoint Conditioned Network (PECNet) for flexible human trajectory prediction. PECNet infers distant trajectory endpoints to assist in long-range multi-modal trajectory prediction. A novel non-local social pooling layer enables PECNet to infer diverse yet socially compliant trajectories. Additionally, we present a simple “truncation-trick” for improving diversity and multi-modal trajectory prediction performance. We show that PECNet improves state-of-the-art performance on the Stanford Drone trajectory prediction benchmark by \({\sim }20.9\%\) and on the ETH/UCY benchmark by \({\sim }40.8\%\) (Code available at project homepage:


Multimodal trajectory prediction Social pooling 

Supplementary material

504434_1_En_45_MOESM1_ESM.pptx (24.7 mb)
Supplementary material 1 (pptx 25268 KB)

Supplementary material 2 (mp4 1288 KB)

Supplementary material 3 (mp4 539 KB)

504434_1_En_45_MOESM4_ESM.pdf (23.5 mb)
Supplementary material 4 (pdf 24093 KB)


  1. 1.
    Bennewitz, M., Burgard, W., Thrun, S.: Learning motion patterns of persons for mobile service robots. In: Proceedings of the 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 4, pp. 3601–3606. IEEE (2002)Google Scholar
  2. 2.
    Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics. Intelligent Robotics and Autonomous Agents Series. MIT Press, Cambridge (2005)zbMATHGoogle Scholar
  3. 3.
    Baker, C.L., Saxe, R., Tenenbaum, J.B.: Action understanding as inverse planning. Cognition 113(3), 329–349 (2009)CrossRefGoogle Scholar
  4. 4.
    Ziebart, B.D.,et al.: Planning-based prediction for pedestrians. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3931–3936. IEEE (2009)Google Scholar
  5. 5.
    Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 549–565. Springer, Cham (2016). Scholar
  6. 6.
    Pellegrini, S., Ess, A., Van Gool, L.: Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 452–465. Springer, Heidelberg (2010). Scholar
  7. 7.
    Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. In: Computer Graphics Forum, vol. 26, pp. 655–664. Wiley Online Library (2007)Google Scholar
  8. 8.
    Rudenko, A., Palmieri, L., Herman, M., Kitani, K.M., Gavrila, D.M., Arras, K.O.: Human motion trajectory prediction: a survey. arXiv e-prints (2019)Google Scholar
  9. 9.
    Kruse, E., Wahl, F.M.: Camera-based observation of obstacle motions to derive statistical data for mobile robot motion planning. In: Proceedings of the 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), vol. 1, pp. 662–667. IEEE (1998)Google Scholar
  10. 10.
    Liao, L., Fox, D., Hightower, J., Kautz, H., Schulz, D.: Voronoi tracking: location estimation using sparse and noisy sensor data. In: Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003),(Cat. No. 03CH37453), vol. 1, pp. 723–728. IEEE (2003)Google Scholar
  11. 11.
    Bennewitz, M., Burgard, W., Cielniak, G., Thrun, S.: Learning motion patterns of people for compliant robot motion. Int. J. Robot. Res. 24(1), 31–48 (2005)CrossRefGoogle Scholar
  12. 12.
    Tay, M.K.C., Laugier, C.: Modelling smooth paths using Gaussian processes. In: Laugier, C., Siegwart, R. (eds.) Field and Service Robotics. STAR, vol. 42, pp. 381–390. Springer, Heidelberg (2008). Scholar
  13. 13.
    Käfer, E., Hermes, C., Wöhler, C., Ritter, H., Kummert, F.: Recognition of situation classes at road intersections. In: 2010 IEEE International Conference on Robotics and Automation, pp. 3960–3965. IEEE (2010)Google Scholar
  14. 14.
    Aoude, G., Joseph, J., Roy, N., How, J.: Mobile agent trajectory prediction using Bayesian nonparametric reachability trees. In: Infotech@ Aerospace 2011, p. 1512 (2011)Google Scholar
  15. 15.
    Keller, C.G., Gavrila, D.M.: Will the pedestrian cross? A study on pedestrian path prediction. IEEE Trans. Intell. Transp. Syst. 15(2), 494–506 (2013)CrossRefGoogle Scholar
  16. 16.
    Goldhammer, M., Doll, K., Brunsmann, U., Gensler, A., Sick, B.: Pedestrian’s trajectory forecast in public traffic with artificial neural networks. In: 2014 22nd International Conference on Pattern Recognition, pp. 4110–4115. IEEE (2014)Google Scholar
  17. 17.
    Xiao, S., Wang, Z., Folkesson, J.: Unsupervised robot learning to predict person motion. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 691–696. IEEE (2015)Google Scholar
  18. 18.
    Kucner, T.P., Magnusson, M., Schaffernicht, E., Bennetts, V.H., Lilienthal, A.J.: Enabling flow awareness for mobile robots in partially observable environments. IEEE Robot. Autom. Lett. 2(2), 1093–1100 (2017)CrossRefGoogle Scholar
  19. 19.
    Kitani, K.M., Ziebart, B.D., Bagnell, J.A., Hebert, M.: Activity forecasting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7575, pp. 201–214. Springer, Heidelberg (2012). Scholar
  20. 20.
    Ballan, L., Castaldo, F., Alahi, A., Palmieri, F., Savarese, S.: Knowledge transfer for scene-specific motion prediction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 697–713. Springer, Cham (2016). Scholar
  21. 21.
    Kim, B.D., Kang, C.M., Kim, J., Lee, S.H., Chung, C.C., Choi, J.W.: Probabilistic vehicle trajectory prediction over occupancy grid map via recurrent neural network. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 399–404. IEEE (2017)Google Scholar
  22. 22.
    Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)CrossRefGoogle Scholar
  23. 23.
    Mehran, R., Oyama, A., Shah, M.: Abnormal crowd behavior detection using social force model. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 935–942. IEEE (2009)Google Scholar
  24. 24.
    Yamaguchi, K., Berg, A.C., Ortiz, L.E., Berg, T.L.: Who are you with and where are you going? In: CVPR 2011, pp. 1345–1352. IEEE (2011)Google Scholar
  25. 25.
    Alahi, A., Ramanathan, V., Fei-Fei, L.: Socially-aware large-scale crowd forecasting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2203–2210 (2014)Google Scholar
  26. 26.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  27. 27.
    Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social LSTM: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–971 (2016)Google Scholar
  28. 28.
    Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P.H.S., Chandraker, M.: Desire: Distant future prediction in dynamic scenes with interacting agents. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 336–345 (2017)Google Scholar
  29. 29.
    Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2255–2264 (2018)Google Scholar
  30. 30.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  31. 31.
    Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Rezatofighi, H., Savarese, S.: SoPhie: an attentive GAN for predicting paths compliant to social and physical constraints. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1349–1358 (2019)Google Scholar
  32. 32.
    Zou, H., Su, H., Song, S., Zhu, J.: Understanding human behaviors in crowds by imitating the decision-making process. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)Google Scholar
  33. 33.
    Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Advances in Neural Information Processing Systems, pp. 4565–4573 (2016)Google Scholar
  34. 34.
    Rehder, E., Kloeden, H.: Goal-directed pedestrian prediction. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 50–58 (2015)Google Scholar
  35. 35.
    Rehder, E., Wirth, F., Lauer, M., Stiller, C.: Pedestrian prediction by planning using deep neural networks. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–5. IEEE (2018)Google Scholar
  36. 36.
    Rhinehart, N., McAllister, R., Kitani, K., Levine, S.: PRECOG: PREdiction conditioned on goals in visual multi-agent settings. arXiv preprint arXiv:1905.01296 (2019)
  37. 37.
    Li, J., Ma, H., Tomizuka, M.: Conditional generative neural system for probabilistic trajectory prediction. arXiv preprint arXiv:1905.01631 (2019)
  38. 38.
    Bhattacharyya, A., Hanselmann, M., Fritz, M., Schiele, B., Straehle, C.-N.: Conditional flow variational autoencoders for structured sequence prediction. arXiv preprint arXiv:1908.09008 (2019)
  39. 39.
    Deo, N., Trivedi, M.M.: Trajectory forecasts in unknown environments conditioned on grid-based plans. arXiv preprint arXiv:2001.00735 (2020)
  40. 40.
    Sadeghian, A., Kosaraju, V., Gupta, A., Savarese, S., Alahi, A.: TrajNet: towards a benchmark for human trajectory prediction. arXiv preprint (2018)Google Scholar
  41. 41.
    Brock, A., Donahue, J., Simonyan, K.: Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018)
  42. 42.
    Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)Google Scholar
  43. 43.
    Liang, J., Jiang, L., Hauptmann, A.: SimAug: learning robust representations from 3D simulation for pedestrian trajectory prediction in unseen cameras (2020)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of CaliforniaBerkeleyUSA
  2. 2.Toyota Research InstituteAnn ArborUSA
  3. 3.Stanford UniversityStanfordUSA

Personalised recommendations