Realtime Time Synchronized Event-Based Stereo

  • Alex Zihao ZhuEmail author
  • Yibo Chen
  • Kostas Daniilidis
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11210)


In this work, we propose a novel event based stereo method which addresses the problem of motion blur for a moving event camera. Our method uses the velocity of the camera and a range of disparities to synchronize the positions of the events, as if they were captured at a single point in time. We represent these events using a pair of novel time synchronized event disparity volumes, which we show remove motion blur for pixels at the correct disparity in the volume, while further blurring pixels at the wrong disparity. We then apply a novel matching cost over these time synchronized event disparity volumes, which both rewards similarity between the volumes while penalizing blurriness. We show that our method outperforms more expensive, smoothing based event stereo methods, by evaluating on the Multi Vehicle Stereo Event Camera dataset.


Event cameras Stereo depth estimation 3D computer vision 



Thanks to Tobi Delbruck and the team at iniLabs for providing and supporting the DAVIS-346b cameras. We also gratefully appreciate support through the following grants: NSF-IIS-1703319, NSF-IIP-1439681 (I/UCRC), ARL RCTA W911NF-10-2-0016, and the DARPA FLA program.


  1. 1.
    Besse, F., Rother, C., Fitzgibbon, A., Kautz, J.: PMBP: patchmatch belief propagation for correspondence field estimation. Int. J. Comput. Vis. 110(1), 2–13 (2014)CrossRefGoogle Scholar
  2. 2.
    Camuñas-Mesa, L.A., Serrano-Gotarredona, T., Ieng, S.H., Benosman, R.B., Linares-Barranco, B.: On the use of orientation filters for 3D reconstruction in event-driven stereo vision. Front. Neurosci. 8, 48 (2014)Google Scholar
  3. 3.
    Camuñas-Mesa, L.A., Serrano-Gotarredona, T., Ieng, S.H., Benosman, R., Linares-Barranco, B.: Event-driven stereo visual tracking algorithm to solve object occlusion. IEEE Trans. Neural Netw. Learn. Syst. 29, 4223–4237 (2017)Google Scholar
  4. 4.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70(1), 41–54 (2006)CrossRefGoogle Scholar
  5. 5.
    Gallego, G., Scaramuzza, D.: Accurate angular velocity estimation with an event camera. IEEE Robot. Autom. Lett. 2(2), 632–639 (2017)CrossRefGoogle Scholar
  6. 6.
    Geiger, A., Roser, M., Urtasun, R.: Efficient large-scale stereo matching. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6492, pp. 25–38. Springer, Heidelberg (2011). Scholar
  7. 7.
    Kogler, J., Humenberger, M., Sulzbachner, C.: Event-based stereo matching approaches for frameless address event stereo data. In: Bebis, G. (ed.) ISVC 2011. LNCS, vol. 6938, pp. 674–685. Springer, Heidelberg (2011). Scholar
  8. 8.
    Marr, D., Poggio, T.: Cooperative computation of stereo disparity. In: Vaina, L. (ed.) From the Retina to the Neocortex, pp. 239–243. Birkhäuser, Boston (1976). Scholar
  9. 9.
    Mitrokhin, A., Fermuller, C., Parameshwara, C., Aloimonos, Y.: Event-based moving object detection and tracking. arXiv preprint arXiv:1803.04523 (2018)
  10. 10.
    Mueggler, E., Forster, C., Baumli, N., Gallego, G., Scaramuzza, D.: Lifetime estimation of events from dynamic vision sensors. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 4874–4881. IEEE (2015)Google Scholar
  11. 11.
    Piatkowska, E., Belbachir, A.N., Gelautz, M.: Cooperative and asynchronous stereo vision for dynamic vision sensors. Meas. Sci. Technol. 25(5), 055108 (2014)CrossRefGoogle Scholar
  12. 12.
    Piatkowska, E., Kogler, J., Belbachir, N., Gelautz, M.: Improved cooperative stereo matching for dynamic vision sensors with ground truth evaluation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 370–377. IEEE (2017)Google Scholar
  13. 13.
    Rebecq, H., Gallego, G., Mueggler, E., et al.: EMVS: event-based multi-view stereo—3D reconstruction with an event camera in real-time. Int. J. Comput. Vis. 1–21 (2017).
  14. 14.
    Rebecq, H., Horstschaefer, T., Scaramuzza, D.: Real-time visual-inertial odometry for event cameras using keyframe-based nonlinear optimization. In: British Machine Vision Conference (BMVC), vol. 3 (2017)Google Scholar
  15. 15.
    Rogister, P., Benosman, R., Ieng, S.H., Lichtsteiner, P., Delbruck, T.: Asynchronous event-based binocular stereo matching. IEEE Trans. Neural Netw. Learn. Syst. 23(2), 347–353 (2012)CrossRefGoogle Scholar
  16. 16.
    Schraml, S., Nabil Belbachir, A., Bischof, H.: Event-driven stereo matching for real-time 3D panoramic vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 466–474 (2015)Google Scholar
  17. 17.
    Xie, Z., Chen, S., Orchard, G.: Event-based stereo depth estimation using belief propagation. Front. Neurosci. 11, 535 (2017)CrossRefGoogle Scholar
  18. 18.
    Xie, Z., Zhang, J., Wang, P.: Event-based stereo matching using semiglobal matching. Int. J. Adv. Robot. Syst. 15(1) (2018). Scholar
  19. 19.
    Zhu, A.Z., Atanasov, N., Daniilidis, K.: Event-based feature tracking with probabilistic data association. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 4465–4470. IEEE (2017)Google Scholar
  20. 20.
    Zhu, A.Z., Thakur, D., Ozaslan, T., Pfrommer, B., Kumar, V., Daniilidis, K.: The multi vehicle stereo event camera dataset: an event camera dataset for 3D perception. IEEE Robot. Autom. Lett. 3, 2032–2039 (2018)CrossRefGoogle Scholar
  21. 21.
    Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: EV-FlowNet: self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898 (2018)
  22. 22.
    Zou, D., et al.: Context-aware event-driven stereo matching. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 1076–1080. IEEE (2016)Google Scholar
  23. 23.
    Zou, D., et al.: Robust dense depth map estimation from sparse DVS stereos. In: British Machine Vision Conference (BMVC), vol. 3 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of PennsylvaniaPhiladelphiaUSA

Personalised recommendations