Advertisement

Polarized Optical-Flow Gyroscope

Conference paper
  • 778 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12361)

Abstract

We merge by generalization two principles of passive optical sensing of motion. One is common spatially resolved imaging, where motion induces temporal readout changes at high-contrast spatial features, as used in traditional optical-flow. The other is the polarization compass, where axial rotation induces temporal readout changes due to the change of incoming polarization angle, relative to the camera frame. The latter has traditionally been modeled for uniform objects. This merger generalizes the brightness constancy assumption and optical-flow, to handle polarization. It also generalizes the polarization compass concept to handle arbitrarily textured objects. This way, scene regions having partial polarization contribute to motion estimation, irrespective of their texture and non-uniformity. As an application, we derive and demonstrate passive sensing of differential ego-rotation around the camera optical axis.

Keywords

Low level vision Self-calibration Bio-inspired 

Notes

Acknowledgements

We thank M. Sheinin, A. Levis, A. Vainiger, T. Loeub, V. Holodovsky, M. Fisher, Y. Gat, and O. Elezra for fruitful discussions. We thank I. Czerninski, O. Shubi, D. Yegudin, and I. Talmon for technical support. Yoav Schechner is the Mark and Diane Seiden Chair in Science at the Technion. He is a Landau Fellow - supported by the Taub Foundation. His work was conducted in the Ollendorff Minerva Center. Minvera is funded through the BMBF. This work is supported by the Israel Science Foundation (ISF fund 542/16).

Supplementary material

504471_1_En_22_MOESM1_ESM.pdf (1.1 mb)
Supplementary material 1 (pdf 1102 KB)

References

  1. 1.
    Technology Image Sensor: Polarization Products Sony Semiconductor Solutions Group. https://www.sony-semicon.co.jp/e/products/IS/polarization/technology.html. Accessed 16 July 2020
  2. 2.
    Agrawal, P., Ratnoo, A., Ghose, D.: Inverse optical flow based guidance for UAV navigation through urban canyons. Aerosp. Sci. Technol. 68, 163–178 (2017)Google Scholar
  3. 3.
    Barta, A., Horváth, G.: Underwater binocular imaging of aerial objects versus the position of eyes relative to the flat water surface. JOSA A 20(12), 2370 (2003)Google Scholar
  4. 4.
    Ben-Ezra, M.: Segmentation with invisible keying signal. In: CVPR, pp. 32–37. IEEE (2000)Google Scholar
  5. 5.
    Berezhnyy, I., Dogariu, A.: Time-resolved Mueller matrix imaging polarimetry. Opt. Express 12(19), 4635–4649 (2004)Google Scholar
  6. 6.
    Bomzon, Z., Biener, G., Kleiner, V., Hasman, E.: Spatial Fourier-transform polarimetry using space-variant subwavelength metal-stripe polarizers. Opt. Lett. 26(21), 1711–1713 (2001)Google Scholar
  7. 7.
    Brines, M.L.: Dynamic patterns of skylight polarization as clock and compass. J. Theor. Biol. 86(3), 507–512 (1980)Google Scholar
  8. 8.
    Brines, M.L., Gould, J.L.: Skylight polarization patterns and animal orientation. J. Exp. Biol. 96(1), 69–91 (1982)Google Scholar
  9. 9.
    Carey, S.J., Barr, D.R., Dudek, P.: Low power high-performance smart camera system based on SCAMP vision sensor. J. Syst. Architect. 59(10), 889–899 (2013)Google Scholar
  10. 10.
    Chahl, J.S., Srinivasan, M.V., Zhang, S.W.: Landing strategies in honeybees and applications to uninhabited airborne vehicles. Int. J. Rob. Res. 23(2), 101–110 (2004)Google Scholar
  11. 11.
    Collin, C., Pattanaik, S., LiKamWa, P., Bouatouch, K.: Computation of polarized subsurface BRDF for rendering. In: Proceedings of Graphics Interface, pp. 201–208. Canadian Information Processing Society (2014)Google Scholar
  12. 12.
    Courroux, S., Chevobbe, S., Darouich, M., Paindavoine, M.: Use of wavelet for image processing in smart cameras with low hardware resources. J. Syst. Architect. 59(10), 826–832 (2013)Google Scholar
  13. 13.
    Cronin, T.W.: Polarization vision and its role in biological signaling. Integr. Comp. Biol. 43(4), 549–558 (2003)Google Scholar
  14. 14.
    Dahlberg, A.R., Pust, N.J., Shaw, J.A.: Effects of surface reflectance on skylight polarization measurements at the Mauna Loa Observatory. Opt. Express 19(17), 16008–16021 (2011)Google Scholar
  15. 15.
    Diamant, Y., Schechner, Y.Y.: Overcoming visual reverberations. In: CVPR, pp. 1–8. IEEE (2008)Google Scholar
  16. 16.
    Diner, D.J., et al.: First results from a dual photoelastic-modulator-based polarimetric camera. Appl. Opt. 49(15), 2929–2946 (2010)Google Scholar
  17. 17.
    Dupeyroux, J., Diperi, J., Boyron, M., Viollet, S., Serres, J.: A novel insect-inspired optical compass sensor for a hexapod walking robot. In: IROS, pp. 3439–3445. IEEE/RSJ (2017)Google Scholar
  18. 18.
    Dupeyroux, J., Serres, J., Viollet, S.: A hexapod walking robot mimicking navigation strategies of desert ants cataglyphis. In: Vouloutsi, V., et al. (eds.) Living Machines 2018. LNCS (LNAI), vol. 10928, pp. 145–156. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-95972-6_16CrossRefGoogle Scholar
  19. 19.
    Dupeyroux, J., Viollet, S., Serres, J.R.: An ant-inspired celestial compass applied to autonomous outdoor robot navigation. Rob. Auton. Syst. 117, 40–56 (2019)Google Scholar
  20. 20.
    Evangelista, C., Kraft, P., Dacke, M., Labhart, T., Srinivasan, M.: Honeybee navigation: critically examining the role of the polarization compass. Philos. Trans. Roy. Soc. B Biol. Sci. 369(1636), 20130037 (2014)Google Scholar
  21. 21.
    Fan, C., Hu, X., Lian, J., Zhang, L., He, X.: Design and calibration of a novel camera-based bio-inspired polarization navigation sensor. IEEE Sens. J. 16(10), 3640–3648 (2016)Google Scholar
  22. 22.
    Farid, H., Adelson, E.H.: Separating reflections from images by use of independent component analysis. JOSA A 16(9), 2136–2145 (1999)Google Scholar
  23. 23.
    Fernandes, J., Postula, A., Srinivasan, M., Thurrowgood, S.: Insect inspired vision for micro aerial vehicle navigation. In: ACRA, pp. 1–8. ARAA (2011)Google Scholar
  24. 24.
    Foster, J.J., Sharkey, C.R., Gaworska, A.V., Roberts, N.W., Whitney, H.M., Partridge, J.C.: Bumblebees learn polarization patterns. Curr. Biol. 24(12), 1415–1420 (2014)Google Scholar
  25. 25.
    Garcia, M., Edmiston, C., Marinov, R., Vail, A., Gruev, V.: Bio-inspired color-polarization imager for real-time in situ imaging. Optica 4(10), 1263 (2017)Google Scholar
  26. 26.
    Gill, P.R., Kellam, M., Tringali, J., Vogelsang, T., Erickson, E., Stork, D.G.: Computational diffractive imager with low-power image change detection. In: Computational Optical Sensing and Imaging, pp. CM3E-2. OSA (2015)Google Scholar
  27. 27.
    Goddard, S.M., Forward, R.B.: The role of the underwater polarized light pattern, in sun compass navigation of the grass shrimp, Palaemonetes Vulgaris. J. Compa. Physiol. A Neuroethol. Sens. Neural Behavi. Physiol. 169(4), 479–491 (1991)Google Scholar
  28. 28.
    Gruev, V., Van der Spiegel, J., Engheta, N.: Dual-tier thin film polymer polarization imaging sensor. Opt. Express 18(18), 19292–19303 (2010)Google Scholar
  29. 29.
    Guan, L., Liu, S., Qi Li, S., Lin, W., Yuan Zhai, L., Kui Chu, J.: Study on polarized optical flow algorithm for imaging bionic polarization navigation micro sensor. Optoelectron. Lett. 14(3), 220–225 (2018)Google Scholar
  30. 30.
    Guan, L., et al.: Study on displacement estimation in low illumination environment through polarized contrast-enhanced optical flow method for polarization navigation applications. Optik 210, 164513 (2020)Google Scholar
  31. 31.
    Guarnera, G.C., Peers, P., Debevec, P., Ghosh, A.: Estimating surface normals from spherical stokes reflectance fields. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012. LNCS, vol. 7584, pp. 340–349. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33868-7_34CrossRefGoogle Scholar
  32. 32.
    Guo, Q., et al.: Compact single-shot metalens depth sensors inspired by eyes of jumping spiders. Proc. Natl. Acad. Sci. 116(46), 22959–22965 (2019)Google Scholar
  33. 33.
    Gupta, M., Narasimhan, S.G., Schechner, Y.Y.: On controlling light transport in poor visibility environments. In: CVPR, pp. 1–8. IEEE (2008)Google Scholar
  34. 34.
    Gutierrez, D., Narasimhan, S.G., Jensen, H.W., Jarosz, W.: Scattering. In: ACM Siggraph Asia Courses, p. 18 (2008)Google Scholar
  35. 35.
    Hooper, B.A., Baxter, B., Piotrowski, C., Williams, J., Dugan, J.: An airborne imaging multispectral polarimeter. In: Oceans, vol. 2, p. 7. IEEE/MTS (2009)Google Scholar
  36. 36.
    Horisaki, R., Choi, K., Hahn, J., Tanida, J., Brady, D.J.: Generalized sampling using a compound-eye imaging system for multi-dimensional object acquisition. Opt. Express 18(18), 19367–19378 (2010)Google Scholar
  37. 37.
    Horstmeyer, R., Euliss, G., Athale, R., Levoy, M.: Flexible multimodal camera using a light field architecture. In: ICCP, pp. 1–8. IEEE (2009)Google Scholar
  38. 38.
    Horváth, G.: Polarized Light and Polarization Vision in Animal Sciences, vol. 2. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-642-54718-8CrossRefGoogle Scholar
  39. 39.
    Ihrke, I., Kutulakos, K.N., Lensch, H.P., Magnor, M., Heidrich, W.: State of the art in transparent and specular object reconstruction. In: Eurographics STAR - State of the Art Report (2008)Google Scholar
  40. 40.
    Joos, F., Buenzli, E., Schmid, H.M., Thalmann, C.: Reduction of polarimetric data using Mueller calculus applied to Nasmyth instruments. In: Observatory Operations: Strategies, Processes, and Systems II, vol. 7016, p. 70161I. SPIE (2008)Google Scholar
  41. 41.
    Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Polarized 3D: high-quality depth sensing with polarization cues. In: ICCV, pp. 3370–3378. IEEE (2015)Google Scholar
  42. 42.
    Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Depth sensing using geometrically constrained polarization normals. Int. J. Comput. Vision 125(1–3), 34–51 (2017)MathSciNetGoogle Scholar
  43. 43.
    Kaftory, R., Schechner, Y.Y., Zeevi, Y.Y.: Variational distance-dependent image restoration. In: CVPR, pp. 1–8. IEEE (2007)Google Scholar
  44. 44.
    Kattawar, G.W., Adams, C.N.: Stokes vector calculations of the submarine light field in an atmosphere-ocean with scattering according to a Rayleigh phase matrix: effect of interface refractive index on radiance and polarization. Limnol. Oceanogr. 34(8), 1453–1472 (1989)Google Scholar
  45. 45.
    Kong, N., Tai, Y.W., Shin, J.S.: A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 209–221 (2013)Google Scholar
  46. 46.
    Lanman, D., Wetzstein, G., Hirsch, M., Heidrich, W., Raskar, R.: Polarization fields: dynamic light field display using multi-layer LCDs. ACM Trans. Graph. 30, 186 (2011)Google Scholar
  47. 47.
    Lerner, A., Sabbah, S., Erlick, C., Shashar, N.: Navigation by light polarization in clear and turbid waters. Philos. Trans. Roy. Soc. B Biol. Sci. 366(1565), 671–679 (2011)Google Scholar
  48. 48.
    Levis, A., Schechner, Y.Y., Davis, A.B., Loveridge, J.: Multi-view polarimetric scattering cloud tomography and retrieval of droplet size. arXiv preprint arXiv:2005.11423 (2020)
  49. 49.
    Maeda, T., Kadambi, A., Schechner, Y.Y., Raskar, R.: Dynamic heterodyne interferometry. In: ICCP, pp. 1–11. IEEE (2018)Google Scholar
  50. 50.
    McGuire, K., De Croon, G., De Wagter, C., Tuyls, K., Kappen, H.: Efficient optical flow and stereo vision for velocity estimation and obstacle avoidance on an autonomous pocket drone. IEEE Rob. Autom. Lett. 2(2), 1070–1076 (2017)Google Scholar
  51. 51.
    Miyazaki, D., Saito, M., Sato, Y., Ikeuchi, K.: Determining surface orientations of transparent objects based on polarization degrees in visible and infrared wavelengths. JOSA A 19(4), 687–694 (2002)Google Scholar
  52. 52.
    Mohan, A., Woo, G., Hiura, S., Smithwick, Q., Raskar, R.: Bokode: imperceptible visual tags for camera based interaction from a distance. ACM Trans. Graph. 28, 98 (2009)Google Scholar
  53. 53.
    Monjur, M., Spinoulas, L., Gill, P.R., Stork, D.G.: Panchromatic diffraction gratings for miniature computationally efficient visual-bar-position sensing. Sens. Transducers 194(11), 127 (2015)Google Scholar
  54. 54.
    Mujat, M., Baleine, E., Dogariu, A.: Interferometric imaging polarimeter. JOSA A 21(11), 2244–2249 (2004)Google Scholar
  55. 55.
    Pijnacker Hordijk, B.J., Scheper, K.Y., De Croon, G.C.: Vertical landing for micro air vehicles using event-based optical-flow. J. Field Rob. 35(1), 69–90 (2018)Google Scholar
  56. 56.
    Reddy, D., Veeraraghavan, A., Chellappa, R.: P2C2: programmable pixel compressive camera for high speed imaging. In: CVPR, pp. 329–336. IEEE (2011)Google Scholar
  57. 57.
    Schechner, Y.Y.: Inversion by P4: polarization-picture post-processing. Philos. Trans. Roy. Soc. B Biol. Sci. 366(1565), 638–648 (2011)Google Scholar
  58. 58.
    Schechner, Y.Y.: Self-calibrating imaging polarimetry. In: ICCP, pp. 1–10. IEEE (2015)Google Scholar
  59. 59.
    Schechner, Y.Y., Diner, D.J., Martonchik, J.V.: Spaceborne underwater imaging. In: ICCP, pp. 1–8. IEEE (2011)Google Scholar
  60. 60.
    Schechner, Y.Y., Nayar, S.K.: Generalized mosaicing: polarization panorama. IEEE Trans. Pattern Anal. Mach. Intell. 27(4), 631–636 (2005)Google Scholar
  61. 61.
    Schilling, K., Schechner, Y., Koren, I.: Cloudct-computed tomography of clouds by a small satellite formation. In: Proceedings of the 12th IAA Symposium on Small Satellites for Earth Observation (2019)Google Scholar
  62. 62.
    Shantaiya, S., Verma, K., Mehta, K.: Multiple object tracking using Kalman filter and optical flow. Eur. J. Adv. Eng. Technol. 2(2), 34–39 (2015)Google Scholar
  63. 63.
    Shashar, N.: Transmission of linearly polarized light in seawater: implications for polarization signaling. J. Exp. Biol. 207(20), 3619–3628 (2004)Google Scholar
  64. 64.
    Shashar, N., et al.: Underwater linear polarization: physical limitations to biological functions. Philos. Trans. Roy. Soc. B Biol. Sci. 366(1565), 649–654 (2011)Google Scholar
  65. 65.
    Shashar, N., Rutledge, P., Cronin, T.: Polarization vision in cuttlefish in a concealed communication channel? J. Exp. Biol. 199(9), 2077–2084 (1996)Google Scholar
  66. 66.
    Shashar, N., Hagan, R., Boal, J.G., Hanlon, R.T.: Cuttlefish use polarization sensitivity in predation on silvery fish. Vis. Res. 40, 71–75 (2000)Google Scholar
  67. 67.
    Shashar, N., Milbury, C., Hanlon, R.: Polarization vision in cephalopods: neuroanatomical and behavioral features that illustrate aspects of form and function. Mar. Freshw. Behav. Physiol. 35, 57–68 (2002)Google Scholar
  68. 68.
    Shen, C., et al.: Optical flow sensor/INS/magnetometer integrated navigation system for MAV in GPS-denied environment. J. Sens. 2016, 1–10 (2016)Google Scholar
  69. 69.
    Smith, F., Stewart, D.: Robot and insect navigation by polarized skylight. In: BIOSIGNALS, pp. 183–188 (2014)Google Scholar
  70. 70.
    Song, X., Seneviratne, L.D., Althoefer, K.: A Kalman filter-integrated optical flow method for velocity sensing of mobile robots. IEEE/ASME Trans. Mechatron. 16(3), 551–563 (2010)Google Scholar
  71. 71.
    Srinivasan, M.V.: Honeybees as a model for the study of visually guided flight, navigation, and biologically inspired robotics. Physiol. Rev. 91(2), 413–460 (2011)Google Scholar
  72. 72.
    Stork, D.G.: Optical elements as computational devices for low-power sensing and imaging. In: Imaging Systems and Applications, pp. ITu4E-4. OSA (2017)Google Scholar
  73. 73.
    Stork, D.G., Gill, P.R.: Special-purpose optics to reduce power dissipation in computational sensing and imaging systems. In: Sensors, pp. 1–3. IEEE (2017)Google Scholar
  74. 74.
    Tanaka, K., Mukaigawa, Y., Matsushita, Y., Yagi, Y.: Descattering of transmissive observation using parallel high-frequency illumination. In: ICCP, pp. 1–8. IEEE (2013)Google Scholar
  75. 75.
    Tessens, L., Morbée, M., Philips, W., Kleihorst, R., Aghajan, H.: Efficient approximate foreground detection for low-resource devices. In: International Conference on Distributed Smart Cameras, pp. 1–8. ACM/IEEE (2009)Google Scholar
  76. 76.
    Thurrowgood, S., Moore, R.J., Soccol, D., Knight, M., Srinivasan, M.V.: A biologically inspired, vision-based guidance system for automatic landing of a fixed-wing aircraft. J. Field Rob. 31(4), 699–727 (2014)Google Scholar
  77. 77.
    Treeaporn, V., Ashok, A., Neifeld, M.A.: Increased field of view through optical multiplexing. Opt. Express 18(21), 22432–22445 (2010)Google Scholar
  78. 78.
    Treibitz, T., Schechner, Y.Y.: Polarization: Beneficial for visibility enhancement? In: CVPR, pp. 525–532. IEEE (2009)Google Scholar
  79. 79.
    Tyo, J.S., Goldstein, D.L., Chenault, D.B., Shaw, J.A.: Review of passive imaging polarimetry for remote sensing applications. Appl. Opt. 45(22), 5453–5469 (2006)Google Scholar
  80. 80.
    Umeyama, S., Godin, G.: Separation of diffuse and specular components of surface reflection by use of polarization and statistical analysis of images. IEEE Trans. Pattern Anal. Mach. Intell. 5, 639–647 (2004)Google Scholar
  81. 81.
    Wang, D., Liang, H., Zhu, H., Zhang, S.: A bionic camera-based polarization navigation sensor. Sensors 14(7), 13006–13023 (2014)Google Scholar
  82. 82.
    Wehner, R.D.: Polarized-light navigation by insects. Sci. Am. 235(1), 106–115 (1976)Google Scholar
  83. 83.
    Xu, Z., Yue, D.K., Shen, L., Voss, K.J.: Patterns and statistics of in-water polarization under conditions of linear and nonlinear ocean surface waves. J. Geophys. Res. Oceans 116(12), 1–14 (2011)Google Scholar
  84. 84.
    Yuan, W., Dana, K., Varga, M., Ashok, A., Gruteser, M., Mandayam, N.: Computer vision methods for visual MIMO optical system. In: CVPR Workshops, pp. 37–43. IEEE (2011)Google Scholar
  85. 85.
    Zallat, J., Torzynski, M., Lallement, A.: Double-pass self-spectral-calibration of a polarization state analyzer. Opt. Lett. 37(3), 401–403 (2012)Google Scholar
  86. 86.
    Zhang, L., Hancock, E.R., Atkinson, G.A.: Reflection component separation using statistical analysis and polarisation. In: Vitrià, J., Sanches, J.M., Hernández, M. (eds.) IbPRIA 2011. LNCS, vol. 6669, pp. 476–483. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-21257-4_59CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Viterbi Faculty of Electrical EngineeringTechnion - Israel Institute of TechnologyHaifaIsrael

Personalised recommendations