Skip to main content
SpringerLink
Log in

Large-scale trajectory prediction via relationship-aware adaptive hierarchical graph learning

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

Trajectory prediction is an important task that enables applications in many domains, such as intelligent transportation systems and video analytics. Most existing methods either (i) perform sequential learning of individuals that are unaware of the behavior of others or (ii) incorporate others’ behavior by constructing static graph-based relationships based on geometric attributes such as distances, which may not capture latent relationships between individuals. Furthermore, the task becomes even more challenging with large-scale trajectories such as hundreds of taxis in a city. In this work, we design a Relationship-aware Adaptive Hierarchical Graph Learning approach for large-scale trajectory prediction, or REAHG. We introduce an adaptive graph generation mechanism to dynamically learn a hidden relationship graph with the ability to adapt to new observations, representing the dynamically changing hidden relationships between individuals to avoid the drawback of manually constructed static graphs. To effectively and efficiently utilize the relationship graph, we design an attentional hierarchical graph pooling mechanism. Each layer of the hierarchy represents the representation learned based on the information from different nodes, both neighbor and non-neighbor nodes. To fuse the representation from different layers, we introduce an attention mechanism to learn the importance of different layers, which informs the prediction of future locations. Our extensive evaluation based on three real-world trajectory datasets shows that REAHG outperforms state-of-the-art baselines in trajectory prediction. The code for this project can be found at the following link: https://github.com/DASHLab/REAHG.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social lstm: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 961–971 (2016)

  • Altché, F., Florent, de La Fortelle, A.: An LSTM network for highway trajectory prediction. In: 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp. 353–359 (2017)

  • Bae, I., Jeon, H.-G.: Disentangled multi-relational graph convolutional network for pedestrian trajectory prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence 35, pp. 911–919 (2021)

  • Bai, L., Yao, L., Li, C., Wang, X., Wang, C.: Adaptive graph convolutional recurrent network for traffic forecasting. Adv. Neural Inform. Process. Syst. 33, 17804–17815 (2020)

    Google Scholar 

  • Chen, B., Barzilay, R., Jaakkola, T.: Path-augmented graph transformer network. arXiv preprint arXiv:1905.12712 (2019)

  • Feng, J., Li, Y., Zhang, C., Sun, F., Meng, F., Guo, A., Jin, D.: Deepmove: predicting human mobility with attentional recurrent networks. In: Proceedings of the 2018 World Wide Web Conference, pp. 1459–1468 (2018)

  • Yang, D., Zhang, D., Chen, L., Qu, B.: NationTelescope: monitoring and visualizing large-scale collective behavior in LBSNs. J. Netw. Comput. Appl. 55, 170–180 (2015)

  • Gao, J., Sun, C., Zhao, H., Shen, Y., Anguelov, D., Li, C., Schmid, C.: Vectornet: encoding hd maps and agent dynamics from vectorized representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11525–11533 (2020)

  • Giuliari, F., Hasan, I., Cristani, M., Galasso, F.: Transformer networks for trajectory forecasting. In: 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, pp. 10335–10342 (2021)

  • Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025–1035 (2017)

  • He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., Wang, M.: Lightgcn: simplifying and powering graph convolution network for recommendation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 639–648 (2020)

  • Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput 9(8): 1735–1780 (1997)

    Article  Google Scholar 

  • Huang, X., Li, Y., Wang, Y., Chen, X., Xiao, Y., Zhang, L.: CTS: a cellular-based trajectory tracking system with GPS-level accuracy. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1(4), 1–29 (2018)

  • Jain, A., Zamir, A. R., Savarese, S., Saxena, A.: Structural-rnn: deep learning on spatio-temporal graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5308–5317 (2016)

  • Kamra, N., Zhu, H.,Trivedi, D. K., Zhang, M., Liu, Y.: Multi-agent trajectory prediction with Fuzzy query attention. In: Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Ed.), 33, 22530–22541 (2020) https://proceedings.neurips.cc/paper/2020/file/fe87435d12ef7642af67d9bc82a8b3cd-Paper.pdf

  • Kipf, T.N.: Welling, max: semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

  • Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., Zemel, R.: Neural relational inference for interacting systems. In: International Conference on Machine Learning. PMLR, pp.2688–2697 (2018)

  • Kong, D., Fei, W.: HST-LSTM: a hierarchical spatial-temporal long-short term memory network for location prediction. In IJCAI 18, 2341–2347 (2018)

  • Kosaraju, V., Sadeghian, A., Martín-Martín, R., Reid, I., Rezatofighi, H., Savarese, S.: Social-bigat: multimodal trajectory forecasting using bicycle-gan and graph attention networks. Adv. Neural Inf. Process. Syst. 32, 137–146 (2019)

  • Kraemer, M.U.G., Yang, C.-H., Gutierrez, B., Wu, C.-H., Klein, B., Pigott, D. M., Plessis, D., Louis, F., Nuno, R., Li, R., Hanage, W.: The effect of human mobility and control measures on the COVID-19 epidemic in China. Science 368(6490), 493–497 (2020)

    Article  Google Scholar 

  • Li, Y., Yu, R., Shahabi, C., Liu, Y.: Diffusion convolutional recurrent neural network: data-driven traffic forecasting. In: International Conference on Learning Representations (2018)

  • Li, H., Lin, F., Lu, X., Xu, C., Huang, G., Zhang, J., Mei, Q., Liu, X.: Systematic analysis of fine-grained mobility prediction with on-device contextual data. IEEE Trans. Mobile. Comput. 21, 1096–109 (2020a)

    Article  Google Scholar 

  • Li, L. L., Yang, B., Liang, M., Zeng, W., Ren, M., Segal, S., Urtasun, R.: End-to-end contextual perception and prediction with interaction transformer. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 5784–5791 (2020b)

  • Li, J., Yang, F., Tomizuka, M., Choi, C.: EvolveGraph: multi-agent trajectory prediction with dynamic relational reasoning. In: Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Ed.) Curran Associates, Inc.,33, pp. 19783–19794 (2020c) https://proceedings.neurips.cc/paper_files/paper/2020/file/e4d8163c7a068b65a64c89bd745ec360-Paper.pdf

  • Li, X., Hu, R., Wang, Z., Yamasaki, T.: Location prediction via bi-direction speculation and dual-level association. arXiv preprint arXiv:2106.15070 (2021)

  • Liang, M., Yang, B., Hu, R., Chen, Y., Liao, R., Feng, S., Urtasun, R.: Learning lane graph representations for motion forecasting. In: European Conference on Computer Vision. Springer, pp. 541–556 (2020)

  • Lin, Y., Wan, H., Guo, S., Lin, Y.: Pre-training context and time aware location embeddings from spatial-temporal trajectories for user next location prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence 35, pp. 4241–4248 (2021)

  • Liu, Y., Zhang, J., Fang, L., Jiang, Q., Zhou, B.: Multimodal motion prediction with stacked transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition. pp. 7577–7586 (2021)

  • Liu, Y., Yao, L., Li, B., Wang, X., Sammut, C.: Social graph transformer networks for pedestrian trajectory prediction in complex social scenarios. In: Proceedings of the 31st ACM International Conference on Information and Knowledge Management. 1339–1349 (2022)

  • Luca, M., Barlacchi, G., Lepri, B., Pappalardo, L.: Deep learning for human mobility: a survey on data and models. arXiv preprint arXiv:2012.02825 (2020)

  • Ma, Y., Zhu, X., Zhang, S., Yang, R., Wang, W., Manocha, D.: Trafficpredict: trajectory prediction for heterogeneous traffic-agents. In: Proceedings of the AAAI Conference on Artificial Intelligence 33, 6120–6127 (2019)

  • Mathew, W., Raposo, R., Martins, B.: Predicting future locations with hidden Markov models. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 911–918 (2012)

  • Mohamed A., Qian, K., Elhoseiny, M., Claudel, C.: Social-stgcnn: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14424–14432 (2020)

  • Palumbo, E., Rizzo, G., Troncy, R., Baralis, E.: Predicting your next stop-over from location-based social network data with recurrent neural networks. In: RecTour@ RecSys, pp. 1–8 (2017)

  • Piorkowski, M., Sarafijanovic-Djukic, N., Grossglauser, M.: A parsimonious model of mobile partitioned networks with clustering. In: 2009 First International Communication Systems and Networks and Workshops. IEEE, pp. 1–10 (2009)

  • Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: Factorizing personalized markov chains for next-basket recommendation. In: Proceedings of the 19th International Conference on World Wide Web, pp. 811–820 (2010)

  • Rudenko, A., Palmieri, L., Herman, M., Kitani, K.M., Gavrila, D. M., Arras, K.O.: Human motion trajectory prediction: a survey. Int. J. Robot. Res. 39(8), 895–935 (2020)

    Article  Google Scholar 

  • Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., Battaglia, P.: Learning to simulate complex physics with graph networks. In: International Conference on Machine Learning. PMLR, pp. 8459–8468 (2020)

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 5998–6008 (2017)

  • Walters, R., Li, J., Yu, R.: Trajectory prediction using equivariant continuous convolution. arXiv preprint arXiv:2010.11344 (2020)

  • Xie, X., Yang, Y., Fang, Z., Wang, G., Zhang, F., Zhang, F., Liu, Y., Zhang, D.: coSense: collaborative urban-scale vehicle sensing based on heterogeneous fleets. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2(4), 1–25 (2018)

  • Xu, Y., Xu, J., Zhao, J., Zheng, K., Liu, A., Zhao, L., Zhou, X.: MetaPTP: an adaptive meta-optimized model for personalized spatial trajectory prediction. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2151–2159 (2022)

  • Xue, H., Salim, F.D., Ren, Y., Oliver, N.: MobTCast: leveraging auxiliary trajectory forecasting for human mobility prediction. arXiv preprint arXiv:2110.01401 (2021)

  • Yan, H., Wang, S., Yang, Y., Guo, B., He, T., Zhang, D.: O2-siterec: store site recommendation under the o2o model via multi-graph attention networks. In: 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, pp. 525–538 (2022)

  • Yang, Y., Zhang, F., Zhang, D.: SharedEdge: GPS-free fine-grained travel time estimation in state-level highway systems. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2(1), 1–26 (2018)

  • Yang, Y., Wang, X., Song, M., Yuan, J., Tao, D.: SPAGAN: shortest path graph attention network. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Sarit Kraus (Ed.), pp. 4099–4105 (2019a)

  • Yang, Y., Xie, X., Fang, Z., Zhang, F., Wang, Y., Zhang, D.: Vemo: enabling transparent vehicular mobility modeling at individual levels with full penetration. In: The 25th Annual International Conference on Mobile Computing and Networking, pp. 1–16 (2019b)

  • Yang, Y., Ding, Y., Yuan, D., Wang, G., Xie, X., Liu, Y., He, T., Zhang, D.: Transloc: transparent indoor localization with uncertain human participation for instant delivery. In: Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, pp. 1–14 (2020a)

  • Yang, Y., Fang, Z., Xie, X., Zhang, F., Liu, Y., Zhang, D.: Extending coverage of stationary sensing systems with mobile sensing systems for human mobility modeling. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4(3), 1–21 (2020b)

  • Yang, D., Fankhauser, B., Rosso, P., Cudre-Mauroux, P.: Location prediction over sparse user mobility traces using RNNs: flashback in hidden states. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), pp. 2184–2190 (2020c)

  • Ying, R., You, J., Morris, C., Ren, X., Hamilton, W.L, Leskovec, J.: Hierarchical graph representation learning with differentiable pooling. In: Proceedings of the 32nd International Conference on Neural Information Processing System, pp. 4805–4815 (2018)

  • Yu, C., Ma, X., Ren, J., Zhao, H., Yi, S.: Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, pp. 507–523. Springer (2020)

  • Yuan, J., Zheng, Y., Zhang, C., Xie, W., Xie, X., Sun, G., Huang, Y.: T-drive: driving directions based on taxi trajectories. In: Proceedings of the 18th Sigspatial International Conference on Advances in Geographic Information Systems, pp. 99–108 (2010)

  • Yuan, J., Zheng, Y., Xie, X., Sun, G.: Driving with knowledge from the physical world. In: Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 316–324 (2011)

  • Zhang, J., Shi, X., Xie, J., Ma, H., King, I., Yeung, D.-Y.: GaAN: gated attention networks for learning on large and spatiotemporal graphs. In: Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, Amir Globerson and Ricardo Silva (Ed.), pp. 339–349 (2018)

  • Zhang, P., Ouyang, W., Zhang, P., Xue, J., Zheng, N.: Sr-lstm: state refinement for lstm towards pedestrian trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12085–12094 (2019)

  • Zhao, P., Zhu, H., Liu, Y., Jiajie, X., Li, Z., Zhuang, F., Sheng, V.S., Zhou, X.: Where to go next: a spatio-temporal gated network for next POI recommendation. In: Proceedings of the AAAI Conference on Artificial Intelligence 33, 5877–5884 (2019)

  • Zheng, C., Fan, X., Wang, C., Qi, J.: Gman: a graph multi-attention network for traffic prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence 34, 1234–1241 (2020)

  • Zheng, X., Han, J., Sun, A.: A survey of location prediction on twitter. IEEE Trans. Knowl. Data. Eng. 30(9), 1652–1671 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Lehigh University, Bethlehem, PA, USA

    Hua Yan & Yu Yang

Authors
  1. Hua Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Yang.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, H., Yang, Y. Large-scale trajectory prediction via relationship-aware adaptive hierarchical graph learning. CCF Trans. Pervasive Comp. Interact. 5, 351–366 (2023). https://doi.org/10.1007/s42486-023-00133-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-023-00133-w

Keywords

Search

Navigation