International Journal of Computer Vision

, Volume 126, Issue 9, pp 902–919 | Cite as

Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

  • Matthias MüllerEmail author
  • Vincent Casser
  • Jean Lahoud
  • Neil Smith
  • Bernard Ghanem


We present a photo-realistic training and evaluation simulator (Sim4CV) ( with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool.


Simulator Unreal Engine 4 Object tracking Autonomous driving Deep learning Imitation learning 



This work was supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the VCC funding.

Supplementary material

11263_2018_1073_MOESM1_ESM.mp4 (21.9 mb)
Supplementary material 1 (mp4 22407 KB)


  1. Andersson, O., Wzorek, M., & Doherty, P. (2017). Deep learning quadcopter control via risk-aware active learning. In Thirty-first AAAI conference on artificial intelligence (AAAI), San Francisco, February 4–9, Accepted.Google Scholar
  2. Babenko, B., Yang, M. H., & Belongie, S. (2010). Visual tracking with online multiple instance learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8), 1619–1632. Scholar
  3. Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45), 18,327–18,332.,,
  4. Bojarski, M., Testa, D. D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., et al. (2016). End to end learning for self-driving cars. arXiv:1604.07316.
  5. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., et al. (2016). Openai gym. arXiv:1606.01540.
  6. Chen, C., Seff, A., Kornhauser, A., & Xiao, J. (2015). Deepdriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the 2015 IEEE international conference on computer vision (ICCV), IEEE Computer Society, Washington, DC, USA, ICCV ’15 (pp. 2722–2730).
  7. Collins, R., Zhou, X., & Teh, S. K. (2005). An open source tracking testbed and evaluation web site. In IEEE international workshop on performance evaluation of tracking and surveillance (PETS 2005), January 2005.Google Scholar
  8. Danelljan, M., Hager, G., Shahbaz Khan, F., & Felsberg, M. (2015). Learning spatially regularized correlation filters for visual tracking. In The IEEE international conference on computer vision (ICCV) Google Scholar
  9. Danelljan, M., Robinson, A., Shahbaz Khan, F., & Felsberg, M. (2016). Beyond correlation filters: Learning continuous convolution operators for visual tracking (pp. 472–488). Cham: Springer. Scholar
  10. De Souza, C., Gaidon, A., Cabon, Y., & Lopez Pena, A.(2017). Procedural generation of videos to train deep action recognition networks. In IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  11. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., & Koltun, V. (2017). CARLA: An open urban driving simulator. In Proceedings of the 1st annual conference on robot learning (pp. 1–16).Google Scholar
  12. Fu, C., Carrio, A., Olivares-Mendez, M., Suarez-Fernandez, R., & Campoy, P. (2014). Robust real-time vision-based aircraft tracking from unmanned aerial vehicles. In 2014 ieee international conference on robotics and automation (ICRA) (pp. 5441–5446).
  13. Furrer, F., Burri, M., Achtelik, M., & Siegwart, R. (2016). RotorS—A modular gazebo MAV simulator framework (Vol. 625, pp. 595–625)., Studies in computational intelligence Cham: Springer.Google Scholar
  14. Gaidon, A., Wang, Q., Cabon, Y., & Vig, E. (2016). Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4340–4349).Google Scholar
  15. Gaszczak, A., Breckon, TP., & Han, J. (2011). Real-time people and vehicle detection from UAV imagery. In J. Röning, D. P. Casasent, & E. L. Hall (Eds.), IST/SPIE electronic imaging, international society for optics and photonics (Vol. 7878, pp. 78,780B-1-13).
  16. Ha, S., & Liu, C. K. (2014). Iterative training of dynamic skills inspired by human coaching techniques. ACM Transactions on Graphics, 34(1), 1:1–1:11. Scholar
  17. Hamalainen, P., Eriksson, S., Tanskanen, E., Kyrki, V., & Lehtinen, J. (2014). Online motion synthesis using sequential monte carlo. ACM Transactions on Graphics, 33(4), 51:1–52:12. Scholar
  18. Hamalainen, P., Rajamaki, J., & Liu, C. K. (2015). Online control of simulated humanoids using particle belief propagation. ACM Transactions on Graphics, 34(4), 81:1–81:13. Scholar
  19. Hejrati, M., & Ramanan, D. (2014). Analysis by synthesis: 3D object recognition by object reconstruction. In 2014 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2449–2456).
  20. Ju, E., Won, J., Lee, J., Choi, B., Noh, J., & Choi, M. G. (2013). Data-driven control of flapping flight. ACM Transactions on Graphics, 32(5), 151:1–151:12. Scholar
  21. Kendall, A., Salvapantula, N., & Stol, K. (2014). On-board object tracking control of a quadcopter with monocular vision. In 2014 international conference on unmanned aircraft systems (ICUAS) (pp. 404–411).
  22. Kim, D. K., & Chen, T. (2015). Deep neural network for real-time autonomous indoor navigation. arXiv:1511.04668.
  23. Koutník, J., Cuccu, G., Schmidhuber, J., & Gomez, F. (2013). Evolving large-scale neural networks for vision-based reinforcement learning. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, ACM, New York, NY, USA, GECCO ’13 (pp. 1061–1068).
  24. Koutník, J., Schmidhuber, J., & Gomez, F. (2014). Online evolution of deep convolutional network for vision-based reinforcement learning (pp. 260–269). Cham: Springer. Scholar
  25. Kristan, M., Pflugfelder, R., Leonardis, A., Matas, J., Čehovin, L., Nebehay, G., et al. (2014). The visual object tracking vot2014 challenge results. In Computer Vision—ECCV 2014 Workshops (pp. 191–217). Springer.Google Scholar
  26. Lerer, A., Gross, S., & Fergus, R. (2016). Learning physical intuition of block towers by example. ArXiv:1603.01312v1.
  27. Li, A., Lin, M., Wu, Y., Yang, M. H., & Yan, S. (2016). NUS-PRO: A new visual tracking challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 335–349. Scholar
  28. Liang, P., Blasch, E., & Ling, H. (2015). Encoding color information for visual tracking: Algorithms and benchmark. IEEE Transactions on Image Processing, 24(12), 5630–5644. Scholar
  29. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., et al. (2016). Continuous control with deep reinforcement learning. arXiv:1509.02971.
  30. Lim, H., & Sinha, S. N. (2015). Monocular localization of a moving person onboard a quadrotor mav. In 2015 IEEE international conference on robotics and automation (ICRA) (pp. 2182–2189).
  31. Marín, J., Vázquez, D., Gerónimo, D., & López, A. M. (2010). Learning appearance in virtual scenarios for pedestrian detection. In 2010 IEEE computer society conference on computer vision and pattern recognition (pp. 137–144).
  32. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., et al. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning (pp. 1928–1937).Google Scholar
  33. Movshovitz-Attias, Y., Sheikh, Y., Naresh Boddeti, V., & Wei, Z. (2014). 3D pose-by-detection of vehicles via discriminatively reduced ensembles of correlation filters. In Proceedings of the British machine vision conference. BMVA Press.
  34. Mueller, M., Sharma, G., Smith, N., & Ghanem, B. (2016a). Persistent aerial tracking system for UAVs. In 2016 IEEE/RSJ international conference intelligent robots and systems (IROS).Google Scholar
  35. Mueller, M., Smith, N., & Ghanem, B. (2016b). A Benchmark and simulator for UAV tracking (pp. 445–461). Cham: Springer. Scholar
  36. Mueller, M., Smith, N., & Ghanem, B. (2017). Context-aware correlation filter tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR).Google Scholar
  37. Muller, U., Ben, J., Cosatto, E., Flepp, B., & Cun, Y. L. (2006). Off-road obstacle avoidance through end-to-end learning. In Y. Weiss, P. B. Schölkopf, & J. C. Platt (Eds.), Advances in neural information processing systems (Vol. 18, pp. 739–746). MIT Press. Accessed 1 June 2017.
  38. Naseer, T., Sturm, J., & Cremers, D. (2013). Followme: Person following and gesture recognition with a quadrocopter. In 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 624–630).
  39. Nussberger, A., Grabner, H., & Van Gool, L. (2014). Aerial object tracking from an airborne platform. In 2014 international conference on unmanned aircraft systems (ICUAS) (pp. 1284–1293).
  40. Papon, J., & Schoeler, M. (2015). Semantic pose using deep networks trained on synthetic RGB-D. arXiv:1508.00835.
  41. Pepik, B., Stark, M., Gehler, P., & Schiele, B. (2012). Teaching 3D geometry to deformable part models. In 2012 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3362–3369).
  42. Pestana, J., Sanchez-Lopez, J., Campoy, P., & Saripalli, S. (2013). Vision based GPS-denied object tracking and following for unmanned aerial vehicles. In 2013 IEEE international symposium on safety, security, and rescue robotics (SSRR) (pp. 1–6).
  43. Pollard, T., & Antone, M. (2012). Detecting and tracking all moving objects in wide-area aerial video. In 2012 IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW) (pp. 15–22).
  44. Pomerleau, D. A. (1989). ALVINN: An autonomous land vehicle in a neural network. In D. S. Touretzky (Ed.), Advances in neural information processing systems (Vol. 1, pp. 305–313). Morgan-Kaufmann. Accessed 1 June 2017.
  45. Portmann, J., Lynen, S., Chli, M., & Siegwart, R. (2014). People detection and tracking from aerial thermal views. In 2014 IEEE international conference on robotics and automation (ICRA) (pp. 1794–1800).
  46. Prabowo, Y. A., Trilaksono, B. R., & Triputra, F. R. (2015). Hardware in-the-loop simulation for visual servoing of fixed wing UAV. In 2015 international conference on electrical engineering and informatics (ICEEI) (pp. 247–252).
  47. Prokaj, J., & Medioni, G. (2014). Persistent tracking for wide area aerial surveillance. In 2014 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1186–1193).
  48. Qadir, A., Neubert, J., Semke, W., & Schultz, R. (2011). On-board visual tracking with unmanned aircraft system (UAS), American Institute of Aeronautics and Astronautics, chap on-board visual tracking with unmanned aircraft system (UAS). Infotech@Aerospace Conferences.
  49. Qiu, W., Zhong, F., Zhang, Y., Qiao, S., Xiao, Z., Kim, T. S., et al. (2017). Unrealcv: Virtual worlds for computer vision. In ACM multimedia open source software competition.Google Scholar
  50. Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016). Playing for data: Ground truth from computer games (pp. 102–118). Cham: Springer. Scholar
  51. Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. (2016). The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR.Google Scholar
  52. Sadeghi, F., & Levine, S. (2016). CAD2RL: Real single-image flight without a single real image. arXiv:1611.04201.
  53. Shah, S., Dey, D., Lovett, C., & Kapoor, A. (2017). Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics. arXiv:1705.05065.
  54. Shah, U., Khawad, R., & Krishna, K. M. (2016). Deepfly: Towards complete autonomous navigation of MAVs with monocular camera. In Proceedings of the Tenth Indian conference on computer vision, graphics and image processing, ACM, New York, NY, USA, ICVGIP ’16 (pp. 59:1–59:8).
  55. Smeulders, A. W. M., Chu, D. M., Cucchiara, R., Calderara, S., Dehghan, A., & Shah, M. (2014). Visual tracking: An experimental survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7), 1442–1468. Scholar
  56. Smolyanskiy, N., Kamenev, A., Smith, J., & Birchfield, S. (2017). Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness. arXiv:1705.02550.
  57. Tan, J., Gu, Y., Liu, C. K., & Turk, G. (2014). Learning bicycle stunts. ACM Transactions on Graphics, 33(4), 50:1–50:12. Scholar
  58. Trilaksono, B. R., Triadhitama, R., Adiprawita, W., Wibowo, A., & Sreenatha, A. (2011). Hardware-in-the-loop simulation for visual target tracking of octorotor UAV. Aircraft Engineering and Aerospace Technology, 83(6), 407–419. Scholar
  59. Wu, Y., Lim, J., & Yang, M. H. (2013). Online object tracking: A benchmark. In 2013 IEEE conference on computer vision and pattern recognition (pp. 2411–2418). IEEE.
  60. Wymann, B., Dimitrakakis, C., Sumner, A., Espié, E., Guionneau, C., & Coulom, R. (2014). TORCS, the open racing car simulator. Accessed 1 June 2017.
  61. Zhang, J., Ma, S., & Sclaroff, S. (2014). MEEM: Robust tracking via multiple experts using entropy minimization. In Proceedings of the European conference on computer vision (ECCV).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Electrical Engineering, Visual Computing CenterKing Abdullah University of Science and Technology (KAUST)ThuwalSaudi Arabia

Personalised recommendations