Light Field Segmentation Using a Ray-Based Graph Structure

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9911)


In this paper, we introduce a novel graph representation for interactive light field segmentation using Markov Random Field (MRF). The greatest barrier to the adoption of MRF for light field processing is the large volume of input data. The proposed graph structure exploits the redundancy in the ray space in order to reduce the graph size, decreasing the running time of MRF-based optimisation tasks. Concepts of free rays and ray bundles with corresponding neighbourhood relationships are defined to construct the simplified graph-based light field representation. We then propose a light field interactive segmentation algorithm using graph-cuts based on such ray space graph structure, that guarantees the segmentation consistency across all views. Our experiments with several datasets show results that are very close to the ground truth, competing with state of the art light field segmentation methods in terms of accuracy and with a significantly lower complexity. They also show that our method performs well on both densely and sparsely sampled light fields.


Light field Segmentation Markov Random Field 


  1. 1.
    Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. 2(11), 1–11 (2005)Google Scholar
  2. 2.
    Lumsdaine, A., Georgiev, T.: The focused plenoptic camera. In: ICCP, pp. 1–8. IEEE (2009)Google Scholar
  3. 3.
    Zhang, C., Chen, T.: A self-reconfigurable camera array. In: SIGGRAPH Sketches, p. 151. ACM (2004)Google Scholar
  4. 4.
    Wilburn, B., Joshi, N., Vaish, V., Levoy, M., Horowitz, M.: High-speed videography using a dense camera array. In: CVPR, vol. 2, p. II-294. IEEE (2004)Google Scholar
  5. 5.
    Ng, R.: Fourier slice photography. TOG 24, 735–744 (2005). ACMCrossRefGoogle Scholar
  6. 6.
    Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: ICCV, December 2013Google Scholar
  7. 7.
    Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D light fields. In: CVPR, pp. 41–48. IEEE (2012)Google Scholar
  8. 8.
    Bishop, T.E., Zanetti, S., Favaro, P.: Light field superresolution. In: ICCP, pp. 1–9. IEEE (2009)Google Scholar
  9. 9.
    Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. PAMI 36(3), 606–619 (2014)CrossRefGoogle Scholar
  10. 10.
    Hochbaum, D.S., Singh, V.: An efficient algorithm for co-segmentation. In: ICCV, pp. 269–276. IEEE (2009)Google Scholar
  11. 11.
    Djelouah, A., Franco, J.S., Boyer, E., Clerc, F., Pérez, P.: Multi-view object segmentation in space and time. In: ICCV, pp. 2640–2647 (2013)Google Scholar
  12. 12.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. PAMI 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  13. 13.
    Boykov, Y., Funka-Lea, G.: Graph cuts and efficient ND image segmentation. IJCV 70(2), 109–131 (2006)CrossRefGoogle Scholar
  14. 14.
    Mihara, H., Funatomi, T., Tanaka, K., Kubo, H., Nagahara, H., Mukaigawa, Y.: 4D light-field segmentation with spatial and angular consistencies. In: ICCP (2016)Google Scholar
  15. 15.
    Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: VMV Workshop, pp. 225–226 (2013)Google Scholar
  16. 16.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV 47(1–3), 7–42 (2002)CrossRefMATHGoogle Scholar
  17. 17.
    Andrew, A.: The (new) stanford light field archive. Accessed 3 Aug 2016
  18. 18.
    Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4D light fields. In: CVPR, pp. 1011–1018. IEEE (2013)Google Scholar
  19. 19.
    Jarabo, A., Masia, B., Gutierrez, D.: Efficient propagation of light field edits. In: SIACG (2011)Google Scholar
  20. 20.
    Seitz, S.M., Kutulakos, K.N.: Plenoptic image editing. IJCV 48(2), 115–129 (2002)CrossRefMATHGoogle Scholar
  21. 21.
    Berent, J., Dragotti, P.L.: Unsupervised extraction of coherent regions for image based rendering. In: BMVC, pp. 1–10 (2007)Google Scholar
  22. 22.
    Dragotti, P.L., Brookes, M.: Efficient segmentation and representation of multi-view images. In: SEAS-DTC Workshop, Edinburgh (2007)Google Scholar
  23. 23.
    Berent, J., Dragotti, P.L.: Plenoptic manifolds-exploiting structure and coherence in multiview images. Sig. Process. Mag. 24, 34–44 (2007)CrossRefGoogle Scholar
  24. 24.
    Rother, C., Minka, T., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching-incorporating a global constraint into MRFS. In: CVPR, vol. 1, pp. 993–1000. IEEE (2006)Google Scholar
  25. 25.
    Mukherjee, L., Singh, V., Peng, J.: Scale invariant cosegmentation for image groups. In: CVPR, pp. 1881–1888. IEEE (2011)Google Scholar
  26. 26.
    Reinbacher, C., Rüther, M., Bischof, H.: Fast variational multi-view segmentation through backprojection of spatial constraints. Image Vis. Comput. 30(11), 797–807 (2012)CrossRefGoogle Scholar
  27. 27.
    Campbell, N.D., Vogiatzis, G., Hernández, C., Cipolla, R.: Automatic object segmentation from calibrated images. In: CVMP, pp. 126–137. IEEE (2011)Google Scholar
  28. 28.
    Sormann, M., Zach, C., Karner, K.: Graph cut based multiple view segmentation for 3D reconstruction. In: 3DPVT, pp. 1085–1092. IEEE (2006)Google Scholar
  29. 29.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: SIGGRAPH, pp. 43–54. ACM (1996)Google Scholar
  30. 30.
    Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? PAMI 26(2), 147–159 (2004)CrossRefGoogle Scholar
  31. 31.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. PAMI 26(9), 1124–1137 (2004)CrossRefMATHGoogle Scholar
  32. 32.
    Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.M.: Scene segmentation by color and depth information and its applications. University of Padova (2010)Google Scholar
  33. 33.
    Mutto, C.D., Zanuttigh, P., Cortelazzo, G.M.: Fusion of geometry and color information for scene segmentation. J-STSP 6(5), 505–521 (2012)Google Scholar
  34. 34.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using iterated graph cuts. TOG 23, 309–314 (2004). ACMCrossRefGoogle Scholar
  35. 35.
    Bilmes, J.A., et al.: A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. ICSI 4(510), 126 (1998)Google Scholar
  36. 36.
    Harville, M., Gordon, G., Woodfill, J.: Foreground segmentation using adaptive mixture models in color and depth. In: Workshop on Detection and Recognition of Events in Video, pp. 3–11. IEEE (2001)Google Scholar
  37. 37.
    Hasnat, M.A., Alata, O., Trémeau, A.: Unsupervised RGB-D image segmentation using joint clustering and region merging. J-STSP 6(5), 505–521 (2012)Google Scholar
  38. 38.
    Drazic, V., Sabater, N.: A precise real-time stereo algorithm. In: IVCNZ, pp. 138–143. ACM (2012)Google Scholar
  39. 39.
    Yang, T., Zhang, Y., Yu, J., Li, J., Ma, W., Tong, X., Yu, R., Ran, L.: All-in-focus synthetic aperture imaging. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 1–15. Springer, Heidelberg (2014)Google Scholar
  40. 40.
    Vineet, V., Narayanan, P.: CUDA cuts: Fast graph cuts on the gpu. In: CVPR, pp. 1–8. IEEE (2008)Google Scholar
  41. 41.
    Bishop, T.E., Favaro, P.: Plenoptic depth estimation from multiple aliased views. In: ICCV Workshops, pp. 1622–1629. IEEE (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Matthieu Hog
    • 1
    • 2
  • Neus Sabater
    • 1
  • Christine Guillemot
    • 2
  1. 1.Technicolor R&IRennesFrance
  2. 2.InriaRennesFrance

Personalised recommendations