Abstract
Predicting the future using deep learning models is a research field of increasing interest. However, there is a lack of established evaluation methods for assessing their predictive abilities. Images and videos are targeted towards human observers, and since humans have individual perceptions of the world, evaluation of videos should take subjectivity into account. In this paper, we present a framework for evaluating predictive models using subjective data. The methodology is based on a mixed methods research design, and is applied in an experiment to measure the realism and accuracy of predictions of a visual traffic environment. Our method is shown to be uncorrelated with the predominant approach for evaluating predictive models, which is a frame-wise comparison between predictions and ground truth. These findings emphasise the importance of using subjective data in the assessment of predictive abilities of models and open up a new direction for evaluating predictive deep learning models.
Supported by the University of Oslo.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Battaglia, M.: Encyclopedia of Survey Research Methods (2008). https://doi.org/10.4135/9781412963947. http://sk.sagepub.com/reference/survey
Bubic, A., von Cramon, D.Y., Schubotz, R.I.: Prediction, cognition and the brain (2010). https://doi.org/10.3389/fnhum.2010.00025
Castelló, J.S.: A comprehensive survey on deep future frame video prediction (2018)
Cox, S.: What is Meaningful Information? - Voice - Two Twelve (2014). http://www.twotwelve.com/voice/what-is-meaningful-information.html
Crowston, K.: Amazon mechanical turk: a research tool for organizations and information systems scholars. In: Bhattacherjee, A., Fitzgerald, B. (eds.) IS&O 2012. IAICT, vol. 389, pp. 210–221. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35142-6_14
Dosovitskiy, A., Ros, G., Codevilla, F., López, A., Koltun, V.: CARLA: an open urban driving simulator. Technical report (2017)
Ha, D., Schmidhuber, J.: World models (2018). https://doi.org/10.5281/zenodo.1207631
Hayes, A.F., Krippendorff, K.: Answering the call for a standard reliability measure for coding data. Commun. Methods Measures 1(1), 77–89 (2007). https://doi.org/10.1080/19312450709336664
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Technical report 8 (1997). http://www7.informatik.tu-muenchen.de/~hochreitwww.idsia.ch/~juergen
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings (2015)
Lee, A.X., Zhang, R., Ebert, F., Abbeel, P., Finn, C., Levine, S.: Stochastic adversarial video prediction (2018). https://doi.org/10.1080/08870440802530798
Lerer, A., Gross, S., Fergus, R.: Learning physical intuition of block towers by example (2016). https://doi.org/10.1016/j.neucom.2015.11.100
Lotter, W., Kreiman, G., Cox, D.: Deep predictive coding networks for video prediction and unsupervised learning, pp. 1–18 (2016). https://doi.org/10.1063/1.1727962. http://arxiv.org/abs/1605.08104
Luc, P., Neverova, N., Couprie, C., Verbeek, J., Lecun, Y.: Predicting deeper into the future of semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision (2017), https://doi.org/10.1109/ICCV.2017.77
Moorthy, A.K., Wang, Z., Bovik, A.C.: Visual perception and quality assessment. In: Optical and Digital Image Processing: Fundamentals and Applications, pp. 419–439 (2011). https://doi.org/10.1002/9783527635245.ch19
van den Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning. In: Advances in Neural Information Processing Systems. vol. 2017-Decem, pp. 6307–6316 (2017)
Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with VQ-VAE-2. In: Wallach, H., Larochelle, H., Beygelzimer, A., d Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 14866–14876. Curran Associates, Inc. (2019). http://papers.nips.cc/paper/9625-generating-diverse-high-fidelity-images-with-vq-vae-2.pdf
Schoonenboom, J., Johnson, R.B.: How to construct a mixed methods research design. KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie 69, 107–131 (2017). https://doi.org/10.1007/s11577-017-0454-1
Strack, F.: “Order effects” in survey research: activation and information functions of preceding questions. In: Schwarz, N., Sudman, S. (eds.) Context Effects in Social and Psychological Research, pp. 23–34. Springer, New York (1992). https://doi.org/10.1007/978-1-4612-2848-6_3
Wichers, N., Villegas, R., Erhan, D., Lee, H.: Hierarchical long-term video prediction without supervision, June 2018. http://arxiv.org/abs/1806.04768
Willig, C.: Introducing Qualitative Research in Psychology, 3rd edn (2013)
Vaughan, J.W.: Making better use of the crowd: how crowdsourcing can advance machine learning research. Technical report (2018). http://jmlr.org/papers/v18/17-234.html
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Gorton, P.R., Ellefsen, K.O. (2021). Evaluating Predictive Deep Learning Models. In: Yildirim Yayilgan, S., Bajwa, I.S., Sanfilippo, F. (eds) Intelligent Technologies and Applications. INTAP 2020. Communications in Computer and Information Science, vol 1382. Springer, Cham. https://doi.org/10.1007/978-3-030-71711-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-71711-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-71710-0
Online ISBN: 978-3-030-71711-7
eBook Packages: Computer ScienceComputer Science (R0)