Abstract
In tasks such as tracking, time-series data inevitably carry missing observations. While traditional tracking approaches can handle missing observations, recurrent neural networks (RNNs) are designed to receive input data in every step. Furthermore, current solutions for RNNs, like omitting the missing data or data imputation, are not sufficient to account for the resulting increased uncertainty. Towards this end, this paper introduces an RNN-based approach that provides a full temporal filtering cycle for motion state estimation. The Kalman filter inspired approach enables to deal with missing observations and outliers. For providing a full temporal filtering cycle, a basic RNN is extended to take observations and the associated belief about its accuracy into account for updating the current state. An RNN prediction model, which generates a parametrized distribution to capture the predicted states, is combined with an RNN update model, which relies on the prediction model output and the current observation. By providing the model with masking information, binary-encoded missing events, the model can overcome limitations of standard techniques for dealing with missing input values. The model abilities are demonstrated on synthetic data reflecting prototypical pedestrian tracking scenarios.
Fraunhofer IOSB is a member of the Fraunhofer Center for Machine Learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social LSTM: human trajectory prediction in crowded spaces. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–971 (2016)
Amirian, J., Hayet, J.B., Pettre, J.: Social ways: learning multi-modal distributions of pedestrian trajectories with GANs. In: Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2964–2972 (2019)
Becker, S., Hug, R., Hübner, W., Arens, M.: RED: a simple but effective baseline predictor for the TrajNet benchmark. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 138–153. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_13
Becker, S., Hug, R., Hübner, W., Arens, M.: An RNN-based IMM filter surrogate. In: Felsberg, M., Forssén, P.-E., Sintorn, I.-M., Unger, J. (eds.) SCIA 2019. LNCS, vol. 11482, pp. 387–398. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20205-7_32
Becker, S.: Dynamic Switching State Systems for Visual Tracking. Ph.D. thesis, Karlsruher Institut für Technologie (KIT) (2020)
Bishop, C.M.: Mixture Density Networks. Technical report, Microsoft Research (1994)
Brownlee, J.: Introduction to time series forecasting with python: how to prepare data and develop models to predict the future. Machine Learning Mastery (2017)
Che, Z., Purushotham, S., Cho, K., Sontag, D., Liu, Y.: Recurrent neural networks for multivariate time series with missing values. Sci. Rep. (SREP) 8, 6085 (2018)
Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A., Bengio, Y.: A recurrent latent variable model for sequential data. In: Advances in Neural Information Processing Systems (NeurIPS), vol. 28, pp. 2980–2988 (2015)
De Boor, C.: A practical guide to splines; rev. ed. Applied mathematical sciences, Springer, Berlin (2001)
Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2625–2634 (2015)
Giuliari, F., Hasan, I., Cristani, M., Galasso, F.: Transformer networks for trajectory forecasting. In: International Conference on Pattern Recognition (ICPR), pp. 10335–10342 (2021)
Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2255–2264 (2018)
Hasan, I., Setti, F., Tsesmelis, T., Bue, A.D., Galasso, F., Cristani, M.: MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6067–6076 (2018)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hug, R., Becker, S., Hübner, W., Arens, M.: On the reliability of LSTM-MDL models for pedestrian trajectory prediction. In: Chen, L., Ben Amor, B., Ghorbel, F. (eds.) RFMI 2017. CCIS, vol. 842, pp. 20–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19816-9_2
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
Kothari, P., Kreiss, S., Alahi, A.: Human trajectory forecasting in crowds: a deep learning perspective. IEEE Transactions on Intelligent Transportation Systems, pp. 1–15 (2021)
Kreindler, D., Lumsden, C.J.: The effects of the irregular sample and missing data in time series analysis. Nonlinear Dyn. Psychol. Life Sci. 10(2), 187–214 (2006)
Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. computer graphic. Forum 26(3), 655–664 (2007)
Lipton, Z.C., Kale, D., Wetzel, R.: Directly modeling missing data in sequences with Rnns: improved classification of clinical time series. In: Proceedings of the 1st Machine Learning for Healthcare Conference, vol. 56, pp. 253–270. PMLR, Children’s Hospital LA, Los Angeles, CA, USA (2016)
Nikhil, N., Morris, B.T.: Convolutional neural network for trajectory prediction. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 186–196. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_16
Parveen, S., Green, P.: Speech recognition with missing data using recurrent neural nets. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 1189–1195. MIT Press (2002)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 8024–8035. Curran Associates, Inc. (2019)
Pellegrini, S., Ess, A., Schindler, K., van Gool, L.: You’ll never walk alone: modeling social behavior for multi-target tracking. In: International Conference on Computer Vision (ICCV), pp. 261–268 (2009)
Rasouli, A.: Deep Learning for Vision-based Prediction: A Survey. arXiv abs/2007.00095 (2020)
Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 549–565. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_33
Rudenko, A., Palmieri, L., Herman, M., Kitani, K.M., Gavrila, D.M., Arras, K.O.: Human motion trajectory prediction: a survey. Int. J. Robot. Res. 39, 895–935 (2020)
Saleh, K.: Pedestrian Trajectory Prediction using Context-Augmented Transformer Networks. arXiv abs/2012.01757 (2020)
Schafer, J.L., Graham, J.W.: Missing data: our view of the state of the art. Psychol. Methods 7(2), 147–177 (2002)
Schneider, N., Gavrila, D.M.: Pedestrian path prediction with recursive bayesian filters: a comparative study. In: Weickert, J., Hein, M., Schiele, B. (eds.) GCPR 2013. LNCS, vol. 8142, pp. 174–183. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40602-7_18
Syed, A., Morris, B.T., et al.: CNN, segmentation or semantic embeddings: evaluating scene context for trajectory prediction. In: George, B. (ed.) ISVC 2020. LNCS, vol. 12510, pp. 706–717. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64559-5_56
Teknom, K.: Microscopic Pedestrian Flow Characteristics: Development of an Image Processing Data Collection and Simulation Model. Ph.D. thesis, Tohoku University (2002)
Tresp, V., Briegel, T.: A solution for missing data in recurrent neural networks with an application to blood glucose prediction. In: International Conference on Neural Information Processing Systems (NeurIPS), pp. 971–977. MIT Press, Cambridge, MA, USA (1997)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning (ICML), vol. 37, pp. 2048–2057. PMLR (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Becker, S., Hug, R., Huebner, W., Arens, M., Morris, B.T. (2021). Handling Missing Observations with an RNN-based Prediction-Update Cycle. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds) Computer Analysis of Images and Patterns. CAIP 2021. Lecture Notes in Computer Science(), vol 13052. Springer, Cham. https://doi.org/10.1007/978-3-030-89128-2_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-89128-2_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89127-5
Online ISBN: 978-3-030-89128-2
eBook Packages: Computer ScienceComputer Science (R0)