Multiple View Object Cosegmentation Using Appearance and Stereo Cues

  • Adarsh Kowdle
  • Sudipta N. Sinha
  • Richard Szeliski
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7576)


We present an automatic approach to segment an object in calibrated images acquired from multiple viewpoints. Our system starts with a new piecewise planar layer-based stereo algorithm that estimates a dense depth map that consists of a set of 3D planar surfaces. The algorithm is formulated using an energy minimization framework that combines stereo and appearance cues, where for each surface, an appearance model is learnt using an unsupervised approach. By treating the planar surfaces as structural elements of the scene and reasoning about their visibility in multiple views, we segment the object in each image independently. Finally, these segmentations are refined by probabilistically fusing information across multiple views. We demonstrate that our approach can segment challenging objects with complex shapes and topologies, which may have thin structures and non-Lambertian surfaces. It can also handle scenarios where the object and background color distributions overlap significantly.


object cosegmentation multiview segmentation multiview stereo 


  1. 1.
    Bai, X., Wang, J., Simons, D., Sapiro, G.: Video SnapCut: robust video object cutout using localized classifiers. In: SIGGRAPH (2009)Google Scholar
  2. 2.
    Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: iCoseg: Interactive co-segmentation with intelligent scribble guidance. In: CVPR (2010)Google Scholar
  3. 3.
    Birchfield, S., Tomasi, C.: Multiway cut for stereo and motion with slanted surfaces. In: ICCV (1999)Google Scholar
  4. 4.
    Bleyer, M., Rother, C., Kohli, P.: Surface stereo with soft segmentation. In: CVPR (2010)Google Scholar
  5. 5.
    Bleyer, M., Rother, C., Kohli, P., Scharstein, D., Sinha, S.: Object stereo - joint stereo matching and object segmentation. In: CVPR (2011)Google Scholar
  6. 6.
    Boykov, Y., Veksler, O., Zabih, R.: Efficient approximate energy minimization via graph cuts. PAMI 20(12), 1222–1239 (2001)CrossRefGoogle Scholar
  7. 7.
    Brown, M., Lowe, D.G.: Unsupervised 3d object recognition and reconstruction in unordered datasets. In: 3DIM, pp. 56–63 (2005)Google Scholar
  8. 8.
    Campbell, N., Vogiatzis, G., Hernandez, C., Cipolla, R.: Automatic 3d object segmentation in multiple views using volumetric graph-cuts. Image and Vision Computing 28, 14–25 (2010)CrossRefGoogle Scholar
  9. 9.
    Campbell, N., Vogiatzis, G., Hernandez, C., Cipolla, R.: Automatic object segmentation from calibrated images. In: CVMP (2011)Google Scholar
  10. 10.
    Cohen-Steiner, D., Alliez, P., Desbrun, M.: Variational shape approximation. ACM Trans. Graph. 23, 905–914 (2004)CrossRefGoogle Scholar
  11. 11.
    Criminisi, A., Cross, G., Blake, A., Kolmogorov, V.: Bilayer segmentation of live video. In: CVPR, pp. 53–60 (2006)Google Scholar
  12. 12.
    Criminisi, A., Sharp, T., Blake, A.: GeoS: Geodesic Image Segmentation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 99–112. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  13. 13.
    Furukawa, Y., Curless, B., Seitz, S., Szeliski, R.: Manhattan-world stereo. In: CVPR (2009)Google Scholar
  14. 14.
    Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards internet-scale multi-view stereo. In: CVPR (2010)Google Scholar
  15. 15.
    Gallup, D., Frahm, J.-M., Pollefeys, M.: Piecewise planar and non-planar stereo for urban scene reconstruction. In: CVPR (2010)Google Scholar
  16. 16.
    Goldberger, J., Gordon, S., Greenspan, H.: An efficient image similarity measure based on approximations of kl-divergence between two gaussian mixtures. In: ICCV (2003)Google Scholar
  17. 17.
    Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. PAMI 30(2), 328–341 (2008)CrossRefGoogle Scholar
  18. 18.
    Hochbaum, D.S., Singh, V.: An efficient algorithm for co-segmentation. In: ICCV (2009)Google Scholar
  19. 19.
    Kowdle, A., Batra, D., Chen, W., Chen, T.: iModel: Interactive co-segmentation for object of interest 3d modeling. In: ECCV - RMLE Workshop (2010)Google Scholar
  20. 20.
    Lee, W., Wontack, W., Boyer, E.: Silhouette segmentation in multiple views. PAMI (2010)Google Scholar
  21. 21.
    Lempitsky, V.S., Rother, C., Roth, S., Blake, A.: Fusion moves for markov random field optimization. PAMI 32(8), 1392–1405 (2010)CrossRefGoogle Scholar
  22. 22.
    Mukherjee, L., Singh, V., Dyer, C.: Half-integrality based algorithms for cosegmentation of images. In: CVPR (2009)Google Scholar
  23. 23.
    Quan, L., Wang, J., Tan, P., Yuan, L.: Image-based modeling by joint segmentation. IJCV 75, 135–150 (2007)CrossRefGoogle Scholar
  24. 24.
    Rissanen, J.: Modeling by shortest data description. Automatica 14, 465–471 (1978)zbMATHCrossRefGoogle Scholar
  25. 25.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: interactive foreground extraction using iterated graph cuts. In: SIGGRAPH (2004)Google Scholar
  26. 26.
    Rother, C., Minka, T., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching - incorporating a global constraint into mrfs. In: CVPR (2006)Google Scholar
  27. 27.
    Simon, I., Seitz, S.M.: Scene Segmentation Using the Wisdom of Crowds. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 541–553. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  28. 28.
    Sinha, S., Steedly, D., Szeliski, R.: Piecewise planar stereo for image-based rendering. In: ICCV (2009)Google Scholar
  29. 29.
    Snavely, N., Seitz, S., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. In: SIGGRAPH (2006)Google Scholar
  30. 30.
    Sormann, M., Zach, C., Karner, K.: Graph cut based multiple view segmentation for 3d reconstruction. 3DPVT 0, 1085–1092 (2006)Google Scholar
  31. 31.
    Tao, H., Sawhney, H., Kumar, R.: A global matching framework for stereo computation. In: ICCV, pp. 532–539 (2001)Google Scholar
  32. 32.
    Vicente, S., Rother, C., Kolmogorov, V.: Object cosegmentation. In: CVPR (2011)Google Scholar
  33. 33.
    Weingarten, J.W., Gruener, G., Siegwart, R.: Probabilistic plane fitting in 3d and an application to robotic mapping. In: ICRA, pp. 927–932 (2004)Google Scholar
  34. 34.
    Xiao, J., Chen, J., Yeung, D.-Y., Quan, L.: Structuring Visual Words in 3D for Arbitrary-View Object Localization. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 725–737. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  35. 35.
    Xiao, J., Wang, J., Tan, P., Quan, L.: Joint affinity propagation for multiple view segmentation. In: ICCV (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Adarsh Kowdle
    • 1
  • Sudipta N. Sinha
    • 2
  • Richard Szeliski
    • 2
  1. 1.Cornell UniversityIthacaUSA
  2. 2.Microsoft ResearchRedmondUSA

Personalised recommendations