Simultaneous 3D Reconstruction for Water Surface and Underwater Scene

  • Yiming QianEmail author
  • Yinqiang Zheng
  • Minglun Gong
  • Yee-Hong Yang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11207)


This paper presents the first approach for simultaneously recovering the 3D shape of both the wavy water surface and the moving underwater scene. A portable camera array system is constructed, which captures the scene from multiple viewpoints above the water. The correspondences across these cameras are estimated using an optical flow method and are used to infer the shape of the water surface and the underwater scene. We assume that there is only one refraction occurring at the water interface. Under this assumption, two estimates of the water surface normals should agree: one from Snell’s law of light refraction and another from local surface structure. The experimental results using both synthetic and real data demonstrate the effectiveness of the presented approach.


3D reconstruction Water surface Underwater imaging 



We thank NSERC, Alberta Innovates and the University of Alberta for the financial support. Yinqiang Zheng is supported by ACT-I, JST and Microsoft Research Asia through the 2017 Collaborative Research Program (Core13).


  1. 1.
    Adamson, A., Alexa, M.: Ray tracing point set surfaces. In: Shape Modeling International, 2003, pp. 272–279. IEEE (2003)Google Scholar
  2. 2.
    Agrawal, A., Ramalingam, S., Taguchi, Y., Chari, V.: A theory of multi-layer flat refractive geometry. In: Computer Vision and Pattern Recognition (CVPR), IEEE Conference on 2012, pp. 3346–3353. IEEE (2012)Google Scholar
  3. 3.
    Alterman, M., Schechner, Y.Y., Swirski, Y.: Triangulation in random refractive distortions. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–10. IEEE (2013)Google Scholar
  4. 4.
    Asano, Y., Zheng, Y., Nishino, K., Sato, I.: Shape from water: bispectral light absorption for depth recovery. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 635–649. Springer, Cham (2016). Scholar
  5. 5.
    Belden, J.: Calibration of multi-camera systems with refractive interfaces. Exp. Fluids 54(2), 1463 (2013)CrossRefGoogle Scholar
  6. 6.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). Scholar
  7. 7.
    Chang, Y.J., Chen, T.: Multi-view 3D reconstruction for scenes under the refractive plane with known vertical direction. In: IEEE International Conference on Computer Vision (ICCV), pp. 351–358. IEEE (2011)Google Scholar
  8. 8.
    Chuang, Y.Y., Zongker, D.E., Hindorff, J., Curless, B., Salesin, D.H., Szeliski, R.: Environment matting extensions: towards higher accuracy and real-time capture. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 121–130. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  9. 9.
    Dagum, L., Menon, R.: OpenMP: an industry standard API for shared-memory programming. IEEE Comput. Sci. Eng. 5(1), 46–55 (1998)CrossRefGoogle Scholar
  10. 10.
    Ding, Y., Li, F., Ji, Y., Yu, J.: Dynamic fluid surface acquisition using a camera array. In: IEEE International Conference on Computer Vision (ICCV), pp. 2478–2485. IEEE (2011)Google Scholar
  11. 11.
    Efros, A., Isler, V., Shi, J., Visontai, M.: Seeing through water. In: Advances in Neural Information Processing Systems, pp. 393–400 (2005)Google Scholar
  12. 12.
    Ferreira, R., Costeira, J.P., Santos, J.A.: Stereo reconstruction of a submerged scene. In: Marques, J.S., Pérez de la Blanca, N., Pina, P. (eds.) IbPRIA 2005. LNCS, vol. 3522, pp. 102–109. Springer, Heidelberg (2005). Scholar
  13. 13.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. In: Readings in computer vision, pp. 726–740. Elsevier (1987)Google Scholar
  14. 14.
    Gregson, J., Ihrke, I., Thuerey, N., Heidrich, W.: From capture to simulation: connecting forward and inverse problems in fluids. ACM Trans. Graph. (TOG) 33(4), 139 (2014)CrossRefzbMATHGoogle Scholar
  15. 15.
    Guennebaud, G., Jacob, B., et al.: Eigen v3. (2010)
  16. 16.
    Han, K., Wong, K.Y.K., Liu, M.: A fixed viewpoint approach for dense reconstruction of transparent objects. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4001–4008 (2015)Google Scholar
  17. 17.
    Jähne, B., Klinke, J., Waas, S.: Imaging of short ocean wind waves: a critical theoretical review. JOSA A 11(8), 2197–2209 (1994)CrossRefGoogle Scholar
  18. 18.
    Ji, Y., Ye, J., Yu, J.: Reconstructing gas flows using light-path approximation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2507–2514 (2013)Google Scholar
  19. 19.
    Katzir, G., Intrator, N.: Striking of underwater prey by a reef heron, egretta gularis schistacea. J. Comp. Physiol. A 160(4), 517–523 (1987)CrossRefGoogle Scholar
  20. 20.
    Kay, T.L., Kajiya, J.T.: Ray tracing complex scenes. In: ACM SIGGRAPH Computer Graphics, vol. 20, pp. 269–278. ACM (1986)Google Scholar
  21. 21.
    Kim, J., Reshetouski, I., Ghosh, A.: Acquiring axially-symmetric transparent objects using single-view transmission imaging. In: 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  22. 22.
    Kudela, L., Frischmann, F., Yossef, O.E., Kollmannsberger, S., Yosibash, Z., Rank, E.: Image-based mesh generation of tubular geometries under circular motion in refractive environments. Mach. Vis. Appl. 29(5), 719–733 (2018). Scholar
  23. 23.
    Kutulakos, K.N., Steger, E.: A theory of refractive and specular 3D shape by light-path triangulation. Int. J. Comput. Vis. 76(1), 13–29 (2008)CrossRefGoogle Scholar
  24. 24.
    Morris, N.J., Kutulakos, K.N.: Dynamic refraction stereo. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1518–1531 (2011)CrossRefGoogle Scholar
  25. 25.
    Mulsow, C.: A flexible multi-media bundle approach. Int. Arch. Photogram. Remote Sens. Spat. Inf. Sci 38, 472–477 (2010)Google Scholar
  26. 26.
    Murase, H.: Surface shape reconstruction of a nonrigid transparent object using refraction and motion. IEEE Trans. Pattern Anal. Mach. Intell. 14(10), 1045–1052 (1992)CrossRefGoogle Scholar
  27. 27.
    Murez, Z., Treibitz, T., Ramamoorthi, R., Kriegman, D.J.: Photometric stereo in a scattering medium. IEEE Trans. Pattern Anal. Mach. Intell. 39(9), 1880–1891 (2017)CrossRefGoogle Scholar
  28. 28.
    Qian, Y., Gong, M., Hong Yang, Y.: 3D reconstruction of transparent objects with position-normal consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4369–4377 (2016)Google Scholar
  29. 29.
    Qian, Y., Gong, M., Yang, Y.H.: Frequency-based environment matting by compressive sensing. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3532–3540 (2015)Google Scholar
  30. 30.
    Qian, Y., Gong, M., Yang, Y.H.: Stereo-based 3D reconstruction of dynamic fluid surfaces by global optimization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1269–1278 (2017)Google Scholar
  31. 31.
    Rusu, R.B.: Semantic 3D object maps for everyday manipulation in human living environments. Ph.D. thesis, Computer Science department, Technische Universitaet Muenchen, Germany, October 2009Google Scholar
  32. 32.
    Saito, H., Kawamura, H., Nakajima, M.: 3D shape measurement of underwater objects using motion stereo. In: Proceedings of the 1995 IEEE IECON 21st International Conference on Industrial Electronics, Control, and Instrumentation, vol. 2, pp. 1231–1235. IEEE (1995)Google Scholar
  33. 33.
    Sedlazeck, A., Koch, R.: Calibration of housing parameters for underwater stereo-camera rigs. In: BMVC, pp. 1–11. Citeseer (2011)Google Scholar
  34. 34.
    Shan, Q., Agarwal, S., Curless, B.: Refractive height fields from single and multiple images. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 286–293. IEEE (2012)Google Scholar
  35. 35.
  36. 36.
    Tanaka, K., Mukaigawa, Y., Kubo, H., Matsushita, Y., Yagi, Y.: Recovering transparent shape from time-of-flight distortion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4387–4395 (2016)Google Scholar
  37. 37.
    Tian, Y., Narasimhan, S.G.: Seeing through water: image restoration using model-based tracking. In: IEEE 12th International Conference on Computer Vision, pp. 2303–2310. IEEE (2009)Google Scholar
  38. 38.
    Westaway, R.M., Lane, S.N., Hicks, D.M.: Remote sensing of clear-water, shallow, gravel-bed rivers using digital photogrammetry. Photogram. Eng. Remote Sens. 67(11), 1271–1282 (2001)Google Scholar
  39. 39.
    Wetzstein, G., Raskar, R., Heidrich, W.: Hand-held schlieren photography with light field probes. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–8. IEEE (2011)Google Scholar
  40. 40.
    Wu, B., Zhou, Y., Qian, Y., Gong, M., Huang, H.: Full 3D reconstruction of transparent objects. ACM Trans. Graph. (Proc. SIGGRAPH) 37(4), 103:1–103:11 (2018)Google Scholar
  41. 41.
    Xue, T., Rubinstein, M., Wadhwa, N., Levin, A., Durand, F., Freeman, W.T.: Refraction wiggles for measuring fluid depth and velocity from video. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 767–782. Springer, Cham (2014). Scholar
  42. 42.
    Yau, T., Gong, M., Yang, Y.H.: Underwater camera calibration using wavelength triangulation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2499–2506. IEEE (2013)Google Scholar
  43. 43.
    Ye, J., Ji, Y., Li, F., Yu, J.: Angular domain reconstruction of dynamic 3D fluid surfaces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 310–317. IEEE (2012)Google Scholar
  44. 44.
    Zhang, M., Lin, X., Gupta, M., Suo, J., Dai, Q.: Recovering scene geometry under wavy fluid via distortion and defocus analysis. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 234–250. Springer, Cham (2014). Scholar
  45. 45.
    Zhang, X., Cox, C.S.: Measuring the two-dimensional structure of a wavy water surface optically: a surface gradient detector. Exp. Fluids 17(4), 225–237 (1994)CrossRefGoogle Scholar
  46. 46.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar
  47. 47.
    Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. (TOMS) 23(4), 550–560 (1997)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.University of AlbertaEdmontonCanada
  2. 2.National Institute of InformaticsTokyoJapan
  3. 3.Memorial University of NewfoundlandSt. John’sCanada

Personalised recommendations