Skip to main content
SpringerLink
Log in

MissFormer: (In-)Attention-Based Handling of Missing Observations for Trajectory Filtering and Prediction

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13017))

Included in the following conference series:

Abstract

In applications such as object tracking, time-series data inevitably carry missing observations. Following the success of deep learning-based models for various sequence learning tasks, these models increasingly replace classic approaches in object tracking applications for inferring the objects’ motion states. While traditional tracking approaches can deal with missing observations, most of their deep counterparts are, by default, not suited for this.

Towards this end, this paper introduces a transformer-based approach for handling missing observations in variable input length trajectory data. The model is formed indirectly by successively increasing the complexity of the demanded inference tasks. Starting from reproducing noise-free trajectories, the model then learns to infer trajectories from noisy inputs. By providing missing tokens, binary-encoded missing events, the model learns to in-attend to missing data and infers a complete trajectory conditioned on the remaining inputs. In the case of a sequence of successive missing events, the model then acts as a pure prediction model. The abilities of the approach are demonstrated on synthetic data and real-world data reflecting prototypical object tracking scenarios.

Fraunhofer IOSB is a member of the Fraunhofer Center for Machine Learning.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social LSTM: human trajectory prediction in crowded spaces. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–971 (2016)

    Google Scholar 

  2. Becker, S., Hug, R., Hübner, W., Arens, M.: RED: a simple but effective baseline predictor for the TrajNet benchmark. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 138–153. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_13

    Chapter  Google Scholar 

  3. Becker, S., Hug, R., Hübner, W., Arens, M., Morris, B.T.: Handling missing observations with an RNN-based prediction-update cycle (2021)

    Google Scholar 

  4. Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, Heidelberg (2006)

    Google Scholar 

  5. Brownlee, J.: Introduction to Time Series Forecasting with Python: How to Prepare Data and Develop Models to Predict the Future (2017)

    Google Scholar 

  6. Che, Z., Purushotham, S., Cho, K., Sontag, D., Liu, Y.: Recurrent neural networks for multivariate time series with missing values. Sci. Rep. (SREP) 8(6085), 1–12 (2018)

    Google Scholar 

  7. Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A., Bengio, Y.: A recurrent latent variable model for sequential data. In: Advances in Neural Information Processing Systems (NeurIPS) (2015)

    Google Scholar 

  8. De Boor, C.: A Practical Guide to Splines, rev Applied mathematical sciences. Springer, Heidelberg (2001)

    Google Scholar 

  9. Dinh, L., Krueger, D., Bengio, Y.: NICE: non-linear independent components estimation. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, Workshop Track Proceedings, San Diego, CA, USA, 7–9 May 2015 (2015)

    Google Scholar 

  10. Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2015)

    Google Scholar 

  11. Giuliari, F., Hasan, I., Cristani, M., Galasso, F.: Transformer networks for trajectory forecasting. In: International Conference on Pattern Recognition (ICPR) (2020)

    Google Scholar 

  12. Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)

    Google Scholar 

  13. Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2018)

    Google Scholar 

  14. Hug, R., Becker, S., Hübner, W., Arens, M.: On the reliability of LSTM-MDL models for pedestrian trajectory prediction. In: Chen, L., Ben Amor, B., Ghorbel, F. (eds.) RFMI 2017. CCIS, vol. 842, pp. 20–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19816-9_2

    Chapter  Google Scholar 

  15. Hug, R., Becker, S., Hübner, W., Arens, M.: A complementary trajectory prediction benchmark. In: ECCV Workshop on Benchmarking Trajectory Forecasting Models (BTFM) (2020)

    Google Scholar 

  16. Hug, R., Becker, S., Hübner, W., Arens, M.: Quantifying the complexity of standard benchmarking datasets for long-term human trajectory prediction. IEEE Access 9, 77693–77704 (2021)

    Article  Google Scholar 

  17. Hug, R., Hübner, W., Arens, M.: Introducing probabilistic bézier curves for n-step sequence prediction. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 06, pp. 10162–10169, April 2020

    Google Scholar 

  18. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Conference Track Proceedings, Banff, AB, Canada, 14–16 April 2014 (2014)

    Google Scholar 

  19. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)

    Google Scholar 

  20. Kothari, P., Kreiss, S., Alahi, A.: Human trajectory forecasting in crowds: a deep learning perspective. arXiv preprint arXiv:2007.03639 (2020)

  21. Kreindler, D., Lumsden, C.J.: The effects of the irregular sample and missing data in time series analysis. Nonlinear Dyn. Psychol. Life Sci. 10(2), 187–214 (2006)

    Google Scholar 

  22. Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. In: Computer Graphic Forum, vol. 26, no. 3, pp. 655–664 (2007)

    Google Scholar 

  23. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019)

    Google Scholar 

  24. Nikhil, N., Morris, B.: Convolutional neural network for trajectory prediction. In: The European Conference on Computer Vision (ECCV) Workshops (2018)

    Google Scholar 

  25. Parveen, S., Green, P.: Speech recognition with missing data using recurrent neural nets. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 1189–1195. MIT Press (2002)

    Google Scholar 

  26. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 8024–8035. Curran Associates, Inc. (2019)

    Google Scholar 

  27. Pellegrini, S., Ess, A., Schindler, K., van Gool, L.: You’ll never walk alone: modeling social behavior for multi-target tracking. In: International Conference on Computer Vision (ICCV), pp. 261–268. IEEE (2009)

    Google Scholar 

  28. Rasouli, A.: Deep learning for vision-based prediction: a survey. arXiv preprint arXiv:2007.00095 (2020)

  29. Rudenko, A., Palmieri, L., Herman, M., Kitani, K.M., Gavrila, D.M., Arras, K.O.: Human motion trajectory prediction: a survey. Int. J. Robot. Res. 39, 895–935 (2020)

    Article  Google Scholar 

  30. Saleh, K.: Pedestrian trajectory prediction using context-augmented transformer networks. arXiv preprint arXiv:2012.01757 (2020)

  31. Salzmann, T., Ivanovic, B., Chakravarty, P., Pavone, M.: Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12363, pp. 683–700. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58523-5_40

    Chapter  Google Scholar 

  32. Schafer, J.L., Graham, J.W.: Missing data: our view of the state of the art. Psychol. Meth. 7(2), 147–177 (2002)

    Article  Google Scholar 

  33. Teknom, K.: Microscopic pedestrian flow characteristics: development of an image processing data collection and simulation model. Ph.D. thesis, Tohoku University (2002)

    Google Scholar 

  34. Tresp, V., Briegel, T.: A solution for missing data in recurrent neural networks with an application to blood glucose prediction. In: International Conference on Neural Information Processing Systems (NeurIPS). pp. 971–977. MIT Press, Cambridge (1997)

    Google Scholar 

  35. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, pp. 5998–6008 (2017)

    Google Scholar 

  36. Vemula, A., Muelling, K., Oh, J.: Social attention: modeling attention in human crowds. In: International Conference on Robotics and Automation (ICRA), pp. 1–7 (2018)

    Google Scholar 

  37. Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning (ICML). Proceedings of Machine Learning Research, Lille, France, vol. 37, pp. 2048–2057. PMLR (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Fraunhofer IOSB, Ettlingen, Germany

    Stefan Becker, Ronny Hug, Wolfgang Huebner & Michael Arens

  2. University of Nevada, Las Vegas, USA

    Brendan Tran Morris

Authors
  1. Stefan Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ronny Hug
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wolfgang Huebner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Arens
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brendan Tran Morris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Becker .

Editor information

Editors and Affiliations

  1. University of Nevada, Reno, NV, USA

    George Bebis

  2. University of Texas at Arlington, Arlington, TX, USA

    Vassilis Athitsos

  3. University of South Carolina, Columbia, SC, USA

    Tong Yan

  4. City University of Hong Kong, Kowloon, Hong Kong

    Manfred Lau

  5. School of Engineering and Computing, University of Durham, Durham, Durham, UK

    Frederick Li

  6. Airbnb, New York, NY, USA

    Conglei Shi

  7. Peking University, Beijing, China

    Xiaoru Yuan

  8. Purdue University, West Lafayette, IN, USA

    Christos Mousas

  9. IST, School of Modeling, Simulation, and Training, Orlando, FL, USA

    Gerd Bruder

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Becker, S., Hug, R., Huebner, W., Arens, M., Morris, B.T. (2021). MissFormer: (In-)Attention-Based Handling of Missing Observations for Trajectory Filtering and Prediction. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2021. Lecture Notes in Computer Science(), vol 13017. Springer, Cham. https://doi.org/10.1007/978-3-030-90439-5_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90439-5_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90438-8

  • Online ISBN: 978-3-030-90439-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation