Skip to main content
SpringerLink
Log in

Co-segmentation for space-time co-located collections

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We present a co-segmentation technique for space-time co-located image collections. These prevalent collections capture various dynamic events, usually by multiple photographers, and may contain multiple co-occurring objects which are not necessarily part of the intended foreground object, resulting in ambiguities for traditional co-segmentation techniques. Thus, to disambiguate what the common foreground object is, we introduce a weakly supervised technique, where we assume only a small seed, given in the form of a single segmented image. We take a distributed approach, where local belief models are propagated and reinforced with similar images. Our technique progressively expands the foreground and background belief models across the entire collection. The technique exploits the power of the entire set of image without building a global model, and thus successfully overcomes large variability in appearance of the common foreground object. We demonstrate that our method outperforms previous co-segmentation techniques on challenging space-time co-located collections, including dense benchmark datasets which were adapted for our novel problem setting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Arpa, A., Ballan, L., Sukthankar, R., Taubin, G., Pollefeys, M., Raskar, R.: Crowdcam: instantaneous navigation of crowd images using angled graph. In: International Conference on 3D Vision-3DV 2013, pp. 422–429. IEEE (2013)

  2. Basha, T., Moses, Y., Avidan, S.: Photo sequencing. In: Computer Vision–ECCV 2012, pp. 654–667. Springer (2012)

  3. Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: icoseg: Interactive co-segmentation with intelligent scribble guidance. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3169–3176. IEEE (2010)

  4. Campbell, N.D., Vogiatzis, G., Hernández, C., Cipolla, R.: Automatic 3d object segmentation in multiple views using volumetric graph-cuts. Image Vis. Comput. 28(1), 14–25 (2010)

    Article  Google Scholar 

  5. Chang, K.Y., Liu, T.L., Lai, S.H.: From co-saliency to co-segmentation: An efficient and fully unsupervised energy minimization model. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2129–2136. IEEE (2011)

  6. Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)

    Article  Google Scholar 

  7. Chiu, W.C., Fritz, M.: Multi-class video co-segmentation with a generative multi-video model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 321–328 (2013)

  8. Djelouah, A., Franco, J.S., Boyer, E., Pérez, P., Drettakis, G.: Cotemporal multi-view video segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 360–369. IEEE (2016)

  9. Faktor, A., Irani, M.: Co-segmentation by composition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1297–1304 (2013)

  10. Fan, Q., Zhong, F., Lischinski, D., Cohen-Or, D., Chen, B.: Jumpcut: non-successive mask transfer and interpolation for video cutout. ACM Trans. Gr. (TOG) 34(6), 195 (2015)

    Google Scholar 

  11. Fu, H., Xu, D., Lin, S., Liu, J.: Object-based rgbd image co-segmentation with mutex constraint. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4428–4436 (2015)

  12. Gang, Z., Long, Q.: Silhouette extraction from multiple images of an unknown background. In: Proceedings of the Asian Conference of Computer Vision, Citeseer (2004)

  13. HaCohen, Y., Shechtman, E., Goldman, D.B., Lischinski, D.: Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Gr. (TOG) 30(4), 70 (2011)

    Google Scholar 

  14. Heskes, T.: Convexity arguments for efficient minimization of the Bethe and Kikuchi free energies. J. Artif. Intell. Res. 26(1), 153–190 (2006)

    Article  MathSciNet  Google Scholar 

  15. Kim, G., Xing, E.P.: On multiple foreground cosegmentation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 837–844. IEEE (2012)

  16. Kim, G., Xing, E.P.: Jointly aligning and segmenting multiple web photo streams for the inference of collective photo storylines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 620–627 (2013)

  17. Kim, G., Xing, E.P., Fei-Fei, L., Kanade, T.: Distributed cosegmentation via submodular optimization on anisotropic diffusion. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 169–176. IEEE (2011)

  18. Kuettel, D., Guillaumin, M., Ferrari, V.: Segmentation propagation in imagenet. In: Computer Vision–ECCV 2012, pp. 459–473. Springer (2012)

  19. Maerki, N., Perazzi, F., Wang, O., Sorkine-Hornung, A.: Bilateral space video segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  20. Mustafa, A., Hilton, A.: Semantically coherent co-segmentation and reconstruction of dynamic scenes. In: CVPR 2017 Proceedings (2017)

  21. Ning, J., Zhang, L., Zhang, D., Wu, C.: Interactive image segmentation by maximal similarity based region merging. Pattern Recogn. 43(2), 445–456 (2010)

    Article  Google Scholar 

  22. Perazzi, F., Pont-Tuset, J., McWilliams, B., Gool, L.V., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Computer Vision and Pattern Recognition (2016)

  23. Pont-Tuset, J., Arbelaez, P., Barron, J.T., Marques, F., Malik, J.: Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 128–140 (2017)

    Article  Google Scholar 

  24. Ramakanth, S.A., Babu, R.V.: Seamseg: Video object segmentation using patch seams. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 376–383. IEEE (2014)

  25. Rother, C., Minka, T., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching-incorporating a global constraint into mrfs. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 993–1000. IEEE (2006)

  26. Rubinstein, M., Joulin, A., Kopf, J., Liu, C.: Unsupervised joint object discovery and segmentation in internet images. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1939–1946. IEEE (2013)

  27. Rubio, J.C., Serrat, J., López, A., Paragios, N.: Unsupervised co-segmentation through region matching. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 749–756. IEEE (2012)

  28. Vicente, S., Rother, C., Kolmogorov, V.: Object cosegmentation. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2217–2224. IEEE (2011)

  29. Wainwright, M.J., Jaakkola, T.S., Willsky, A.S.: A new class of upper bounds on the log partition function. Trans. Inf. Theory 51(7), 2313–2335 (2005)

    Article  MathSciNet  Google Scholar 

  30. Zhang, D., Javed, O., Shah, M.: Video object co-segmentation by regulated maximum weight cliques. In: European Conference on Computer Vision, pp. 551–566. Springer (2014)

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering School, Tel Aviv University, Tel Aviv, Israel

    Hadar Averbuch-Elor

  2. Facebook, Seattle, WA, USA

    Johannes Kopf

  3. Faculty of Industrial Engineering and Management, Technion, Haifa, Israel

    Tamir Hazan

  4. Computer Science School, Tel Aviv University, Tel Aviv, Israel

    Daniel Cohen-Or

Authors
  1. Hadar Averbuch-Elor
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Johannes Kopf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tamir Hazan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Cohen-Or
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hadar Averbuch-Elor.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 94927 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Averbuch-Elor, H., Kopf, J., Hazan, T. et al. Co-segmentation for space-time co-located collections. Vis Comput 34, 1761–1772 (2018). https://doi.org/10.1007/s00371-017-1467-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-017-1467-5

Keywords

Search

Navigation