Skip to main content
SpringerLink
Log in

Secrets of Event-Based Optical Flow

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13678))

Included in the following conference series:

Abstract

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event_based_optical_flow.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Akolkar, H., Ieng, S.H., Benosman, R.: Real-time high speed motion prediction using fast aperture-robust event-driven visual flow. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 361–372 (2022). https://doi.org/10.1109/TPAMI.2020.3010468

  2. Bardow, P., Davison, A.J., Leutenegger, S.: Simultaneous optical flow and intensity estimation from an event camera. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 884–892 (2016). https://doi.org/10.1109/CVPR.2016.102

  3. Benosman, R., Clercq, C., Lagorce, X., Ieng, S.H., Bartolozzi, C.: Event-based visual flow. IEEE Trans. Neural Netw. Learn. Syst. 25(2), 407–417 (2014). https://doi.org/10.1109/TNNLS.2013.2273537

  4. Benosman, R., Ieng, S.H., Clercq, C., Bartolozzi, C., Srinivasan, M.: Asynchronous frameless event-based optical flow. Neural Netw. 27, 32–37 (2012). https://doi.org/10.1016/j.neunet.2011.11.001

  5. Brebion, V., Moreau, J., Davoine, F.: Real-time optical flow for vehicular perception with low- and high-resolution event cameras. IEEE Trans. Intell. Transport. Syst. 23, 1–13 (2021). https://doi.org/10.1109/TITS.2021.3136358

  6. Brosch, T., Tschechne, S., Neumann, H.: On event-based optical flow detection. Front. Neurosci. 9, 137 (2015). https://doi.org/10.3389/fnins.2015.00137

  7. Cannici, M., Ciccone, M., Romanoni, A., Matteucci, M.: A differentiable recurrent surface for asynchronous event-based data. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 136–152. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_9

  8. Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Deterministic edge-preserving regularization in computed imaging. IEEE Trans. Image Process. 6(2), 298–311 (1997). https://doi.org/10.1109/83.551699

  9. Delmerico, J., Cieslewski, T., Rebecq, H., Faessler, M., Scaramuzza, D.: Are we ready for autonomous drone racing? the UZH-FPV drone racing dataset. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 6713–6719 (2019). https://doi.org/10.1109/ICRA.2019.8793887

  10. Ding, Z., et al.: Spatio-temporal recurrent networks for event-based optical flow estimation. In: AAAI Conference on Artificial intelligence, vol. 36(1), pp. 525–533 (2022)

    Google Scholar 

  11. Evans, L.C.: Partial Differential Equations. Graduate Studies in Mathematics. American Mathematical Society (2010)

    Google Scholar 

  12. Gallego, G., et al.: Event-based vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 154–180 (2022). https://doi.org/10.1109/TPAMI.2020.3008413

  13. Gallego, G., Gehrig, M., Scaramuzza, D.: Focus is all you need: Loss functions for event-based vision. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12272–12281 (2019). https://doi.org/10.1109/CVPR.2019.01256

  14. Gallego, G., Rebecq, H., Scaramuzza, D.: A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3867–3876 (2018). https://doi.org/10.1109/CVPR.2018.00407

  15. Gallego, G., Scaramuzza, D.: Accurate angular velocity estimation with an event camera. IEEE Robot. Autom. Lett. 2(2), 632–639 (2017). https://doi.org/10.1109/LRA.2016.2647639

  16. Gehrig, D., Loquercio, A., Derpanis, K.G., Scaramuzza, D.: End-to-end learning of representations for asynchronous event-based data. In: Conference on Computer Vision (ICCV), pp. 5632–5642 (2019). https://doi.org/10.1109/ICCV.2019.00573

  17. Gehrig, M., Aarents, W., Gehrig, D., Scaramuzza, D.: DSEC: A stereo event camera dataset for driving scenarios. IEEE Robot. Autom. Lett. 6(3), 4947–4954 (2021). https://doi.org/10.1109/LRA.2021.3068942

  18. Gehrig, M., Millhäusler, M., Gehrig, D., Scaramuzza, D.: E-RAFT: Dense optical flow from event cameras. In: International Conference on 3D Vision (3DV), pp. 197–206 (2021). https://doi.org/10.1109/3DV53792.2021.00030

  19. Gonzalez, R.C., Woods, R.E.: Digital Image Processing. Pearson Education (2009)

    Google Scholar 

  20. Gu, C., Learned-Miller, E., Sheldon, D., Gallego, G., Bideau, P.: The spatio-temporal Poisson point process: A simple model for the alignment of event camera data. In: International Conference on Computer Vision (ICCV), pp. 13495–13504 (2021). https://doi.org/10.1109/ICCV48922.2021.01324

  21. Hagenaars, J.J., Paredes-Valles, F., de Croon, G.C.H.E.: Self-supervised learning of event-based optical flow with spiking neural networks. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 34, pp. 7167–7179 (2021). https://doi.org/10.48550/arXiv.2106.01862

  22. Kim, H., Kim, H.J.: Real-time rotational motion estimation with contrast maximization over globally aligned events. IEEE Robot. Autom. Lett. 6(3), 6016–6023 (2021). https://doi.org/10.1109/LRA.2021.3088793

  23. Lagorce, X., Orchard, G., Gallupi, F., Shi, B.E., Benosman, R.: HOTS: A hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1346–1359 (2017). https://doi.org/10.1109/TPAMI.2016.2574707

  24. Lee, C., Kosta, A.K., Zhu, A.Z., Chaney, K., Daniilidis, K., Roy, K.: Spike-flownet: Event-based optical flow estimation with energy-efficient hybrid neural networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 366–382. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_22

  25. Li, H., Li, G., Shi, L.: Super-resolution of spatiotemporal event-stream image. Neurocomputing 335, 206–214 (2019). https://doi.org/10.1016/j.neucom.2018.12.048

  26. Liu, M., Delbruck, T.: Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors. In: British Machine Vision Conference (BMVC), pp. 1–12 (2018)

    Google Scholar 

  27. Mitrokhin, A., Fermuller, C., Parameshwara, C., Aloimonos, Y.: Event-based moving object detection and tracking. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–9 (2018). https://doi.org/10.1109/IROS.2018.8593805

  28. Mueggler, E., Rebecq, H., Gallego, G., Delbruck, T., Scaramuzza, D.: The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. Int. J. Robot. Research 36(2), 142–149 (2017). https://doi.org/10.1177/0278364917691115

  29. Nagata, J., Sekikawa, Y., Aoki, Y.: Optical flow estimation by matching time surface with event-based cameras. Sensors 21(4) (2021). https://doi.org/10.3390/s21041150

  30. Nunes, U.M., Demiris, Y.: Robust event-based vision model estimation by dispersion minimisation. IEEE Trans. Pattern Anal. Mach. Intell. (2021). https://doi.org/10.1109/TPAMI.2021.3130049

  31. Orchard, G., Benosman, R., Etienne-Cummings, R., Thakor, N.V.: A spiking neural network architecture for visual motion estimation. In: IEEE Biomedical Circuits and Systems Conference (BioCAS), pp. 298–301 (2013). https://doi.org/10.1109/biocas.2013.6679698

  32. Parameshwara, C.M., Sanket, N.J., Singh, C.D., Fermüller, C., Aloimonos, Y.: 0-MMS: Zero-shot multi-motion segmentation with a monocular event camera. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 9594–9600 (2021). https://doi.org/10.1109/ICRA48506.2021.9561755

  33. Paredes-Valles, F., de Croon, G.C.H.E.: Back to event basics: Self-supervised learning of image reconstruction for event cameras via photometric constancy. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3445–3454 (2021). https://doi.org/10.1109/CVPR46437.2021.00345

  34. Peng, X., Gao, L., Wang, Y., Kneip, L.: Globally-optimal contrast maximisation for event cameras. IEEE Trans. Pattern Anal. Mach. Intell. 44(7), 3479–3495 (2022). https://doi.org/10.1109/TPAMI.2021.3053243

  35. Posch, C., Serrano-Gotarredona, T., Linares-Barranco, B., Delbruck, T.: Retinomorphic event-based vision sensors: Bioinspired cameras with spiking output. Proc. IEEE 102(10), 1470–1484 (2014). https://doi.org/10.1109/jproc.2014.2346153

  36. Rebecq, H., Gallego, G., Mueggler, E., Scaramuzza, D.: EMVS: Event-based multi-view stereo–3D reconstruction with an event camera in real-time. Int. J. Comput. Vis. 126(12), 1394–1414 (2018). https://doi.org/10.1007/s11263-017-1050-6

  37. Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

  38. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992). https://doi.org/10.1016/0167-2789(92)90242-F

  39. Seok, H., Lim, J.: Robust feature tracking in dvs event stream using Bezier mapping. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1647–1656 (2020). https://doi.org/10.1109/WACV45572.2020.9093607

  40. Sethian, J.: Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science. Cambridge University Press, Cambridge Monographs on Applied and Computational Mathematics (1999)

    Google Scholar 

  41. Shiba, S., Aoki, Y., Gallego, G.: Event collapse in contrast maximization frameworks. Sensors 22(14), 1–20 (2022). https://doi.org/10.3390/s22145190

  42. Stoffregen, T., Gallego, G., Drummond, T., Kleeman, L., Scaramuzza, D.: Event-based motion segmentation by motion compensation. In: International Conference on Computer Vision (ICCV), pp. 7243–7252 (2019). https://doi.org/10.1109/ICCV.2019.00734

  43. Stoffregen, T., Kleeman, L.: Simultaneous optical flow and segmentation (SOFAS) using Dynamic Vision Sensor. In: Australasian Conference on Robotics and Automation (ACRA) (2017)

    Google Scholar 

  44. Stoffregen, T., Kleeman, L.: Event cameras, contrast maximization and reward functions: an analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12292–12300 (2019). https://doi.org/10.1109/CVPR.2019.01258

  45. Stoffregen, T., et al.: Reducing the sim-to-real gap for event cameras. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12372, pp. 534–549. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58583-9_32

  46. Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106(2), 115–137 (2013). https://doi.org/10.1007/s11263-013-0644-x

  47. Taverni, G., et al.: Front and back illuminated Dynamic and Active Pixel Vision Sensors comparison. IEEE Trans. Circuits Syst. II 65(5), 677–681 (2018). https://doi.org/10.1109/TCSII.2018.2824899

  48. Teed, Z., Deng, J.: RAFT: Recurrent all pairs field transforms for optical flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 402–419. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_24

  49. Trucco, E., Verri, A.: Introductory Techniques for 3-D Computer Vision. Prentice Hall PTR, Upper Saddle River, NJ, USA (1998)

    Google Scholar 

  50. Ye, C., Mitrokhin, A., Parameshwara, C., Fermüller, C., Yorke, J.A., Aloimonos, Y.: Unsupervised learning of dense optical flow, depth and egomotion with event-based sensors. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5831–5838 (2020). https://doi.org/10.1109/IROS45743.2020.9341224

  51. Zhang, Z., Yezzi, A., Gallego, G.: Image reconstruction from events. Why learn it? arXiv e-prints (2021). https://doi.org/10.48550/arXiv.2112.06242

  52. Zhou, Y., Gallego, G., Lu, X., Liu, S., Shen, S.: Event-based motion segmentation with spatio-temporal graph cuts. IEEE Trans. Neural Netw. Learn. Syst. 1–13 (2021). https://doi.org/10.1109/TNNLS.2021.3124580

  53. Zhu, A.Z., Atanasov, N., Daniilidis, K.: Event-based feature tracking with probabilistic data association. In: IEEE International Conference on Intelligent Robots and Systems (ICRA), pp. 4465–4470 (2017). https://doi.org/10.1109/ICRA.2017.7989517

  54. Zhu, A.Z., Atanasov, N., Daniilidis, K.: Event-based visual inertial odometry. In: IEEE Conf. Comput. Vis. Pattern Recog. (CVPR). pp. 5816–5824 (2017). https://doi.org/10.1109/CVPR.2017.616

  55. Zhu, A.Z., Thakur, D., Ozaslan, T., Pfrommer, B., Kumar, V., Daniilidis, K.: The multivehicle stereo event camera dataset: An event camera dataset for 3D perception. IEEE Robot. Autom. Lett. 3(3), 2032–2039 (2018). https://doi.org/10.1109/lra.2018.2800793

  56. Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: EV-FlowNet: Self-supervised optical flow estimation for event-based cameras. In: Robotics: Science and Systems (RSS) (2018). https://doi.org/10.15607/RSS.2018.XIV.062

  57. Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: Unsupervised event-based learning of optical flow, depth, and egomotion. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 989–997 (2019). https://doi.org/10.1109/CVPR.2019.00108

Download references

Acknowledgements

We thank Prof. A. Yezzi and Dr. A. Zhu for useful discussions. Funded by the German Academic Exchange Service (DAAD), Research Grant - Bi-nationally Supervised Doctoral Degrees/Cotutelle, 2021/22 (57552338). Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy - EXC 2002/1 “Science of Intelligence” - project number 390523135.

Author information

Authors and Affiliations

  1. Department of Electronics and Electrical Engineering, Faculty of Science and Technology, Keio University, Kanagawa, Japan

    Shintaro Shiba & Yoshimitsu Aoki

  2. Department of EECS, Technische Universität Berlin, Berlin, Germany

    Shintaro Shiba & Guillermo Gallego

  3. Einstein Center Digital Future and SCIoI Excellence Cluster, Berlin, Germany

    Guillermo Gallego

Authors
  1. Shintaro Shiba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yoshimitsu Aoki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guillermo Gallego
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shintaro Shiba .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2159 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shiba, S., Aoki, Y., Gallego, G. (2022). Secrets of Event-Based Optical Flow. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation