Advertisement

Co-segmentation for space-time co-located collections

  • Hadar Averbuch-Elor
  • Johannes Kopf
  • Tamir Hazan
  • Daniel Cohen-Or
Original Article
  • 405 Downloads

Abstract

We present a co-segmentation technique for space-time co-located image collections. These prevalent collections capture various dynamic events, usually by multiple photographers, and may contain multiple co-occurring objects which are not necessarily part of the intended foreground object, resulting in ambiguities for traditional co-segmentation techniques. Thus, to disambiguate what the common foreground object is, we introduce a weakly supervised technique, where we assume only a small seed, given in the form of a single segmented image. We take a distributed approach, where local belief models are propagated and reinforced with similar images. Our technique progressively expands the foreground and background belief models across the entire collection. The technique exploits the power of the entire set of image without building a global model, and thus successfully overcomes large variability in appearance of the common foreground object. We demonstrate that our method outperforms previous co-segmentation techniques on challenging space-time co-located collections, including dense benchmark datasets which were adapted for our novel problem setting.

Keywords

Image co-segmentation Foreground extraction Non-rigid and deformable motion analysis Belief propagation 

Supplementary material

371_2017_1467_MOESM1_ESM.pdf (92.7 mb)
Supplementary material 1 (pdf 94927 KB)

References

  1. 1.
    Arpa, A., Ballan, L., Sukthankar, R., Taubin, G., Pollefeys, M., Raskar, R.: Crowdcam: instantaneous navigation of crowd images using angled graph. In: International Conference on 3D Vision-3DV 2013, pp. 422–429. IEEE (2013)Google Scholar
  2. 2.
    Basha, T., Moses, Y., Avidan, S.: Photo sequencing. In: Computer Vision–ECCV 2012, pp. 654–667. Springer (2012)Google Scholar
  3. 3.
    Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: icoseg: Interactive co-segmentation with intelligent scribble guidance. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3169–3176. IEEE (2010)Google Scholar
  4. 4.
    Campbell, N.D., Vogiatzis, G., Hernández, C., Cipolla, R.: Automatic 3d object segmentation in multiple views using volumetric graph-cuts. Image Vis. Comput. 28(1), 14–25 (2010)CrossRefGoogle Scholar
  5. 5.
    Chang, K.Y., Liu, T.L., Lai, S.H.: From co-saliency to co-segmentation: An efficient and fully unsupervised energy minimization model. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2129–2136. IEEE (2011)Google Scholar
  6. 6.
    Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.M.: Global contrast based salient region detection. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 569–582 (2015)CrossRefGoogle Scholar
  7. 7.
    Chiu, W.C., Fritz, M.: Multi-class video co-segmentation with a generative multi-video model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 321–328 (2013)Google Scholar
  8. 8.
    Djelouah, A., Franco, J.S., Boyer, E., Pérez, P., Drettakis, G.: Cotemporal multi-view video segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 360–369. IEEE (2016)Google Scholar
  9. 9.
    Faktor, A., Irani, M.: Co-segmentation by composition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1297–1304 (2013)Google Scholar
  10. 10.
    Fan, Q., Zhong, F., Lischinski, D., Cohen-Or, D., Chen, B.: Jumpcut: non-successive mask transfer and interpolation for video cutout. ACM Trans. Gr. (TOG) 34(6), 195 (2015)Google Scholar
  11. 11.
    Fu, H., Xu, D., Lin, S., Liu, J.: Object-based rgbd image co-segmentation with mutex constraint. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4428–4436 (2015)Google Scholar
  12. 12.
    Gang, Z., Long, Q.: Silhouette extraction from multiple images of an unknown background. In: Proceedings of the Asian Conference of Computer Vision, Citeseer (2004)Google Scholar
  13. 13.
    HaCohen, Y., Shechtman, E., Goldman, D.B., Lischinski, D.: Non-rigid dense correspondence with applications for image enhancement. ACM Trans. Gr. (TOG) 30(4), 70 (2011)Google Scholar
  14. 14.
    Heskes, T.: Convexity arguments for efficient minimization of the Bethe and Kikuchi free energies. J. Artif. Intell. Res. 26(1), 153–190 (2006)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Kim, G., Xing, E.P.: On multiple foreground cosegmentation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 837–844. IEEE (2012)Google Scholar
  16. 16.
    Kim, G., Xing, E.P.: Jointly aligning and segmenting multiple web photo streams for the inference of collective photo storylines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 620–627 (2013)Google Scholar
  17. 17.
    Kim, G., Xing, E.P., Fei-Fei, L., Kanade, T.: Distributed cosegmentation via submodular optimization on anisotropic diffusion. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 169–176. IEEE (2011)Google Scholar
  18. 18.
    Kuettel, D., Guillaumin, M., Ferrari, V.: Segmentation propagation in imagenet. In: Computer Vision–ECCV 2012, pp. 459–473. Springer (2012)Google Scholar
  19. 19.
    Maerki, N., Perazzi, F., Wang, O., Sorkine-Hornung, A.: Bilateral space video segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  20. 20.
    Mustafa, A., Hilton, A.: Semantically coherent co-segmentation and reconstruction of dynamic scenes. In: CVPR 2017 Proceedings (2017)Google Scholar
  21. 21.
    Ning, J., Zhang, L., Zhang, D., Wu, C.: Interactive image segmentation by maximal similarity based region merging. Pattern Recogn. 43(2), 445–456 (2010)CrossRefzbMATHGoogle Scholar
  22. 22.
    Perazzi, F., Pont-Tuset, J., McWilliams, B., Gool, L.V., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Computer Vision and Pattern Recognition (2016)Google Scholar
  23. 23.
    Pont-Tuset, J., Arbelaez, P., Barron, J.T., Marques, F., Malik, J.: Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans. Pattern Anal. Mach. Intell. 39(1), 128–140 (2017)CrossRefGoogle Scholar
  24. 24.
    Ramakanth, S.A., Babu, R.V.: Seamseg: Video object segmentation using patch seams. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 376–383. IEEE (2014)Google Scholar
  25. 25.
    Rother, C., Minka, T., Blake, A., Kolmogorov, V.: Cosegmentation of image pairs by histogram matching-incorporating a global constraint into mrfs. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 993–1000. IEEE (2006)Google Scholar
  26. 26.
    Rubinstein, M., Joulin, A., Kopf, J., Liu, C.: Unsupervised joint object discovery and segmentation in internet images. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1939–1946. IEEE (2013)Google Scholar
  27. 27.
    Rubio, J.C., Serrat, J., López, A., Paragios, N.: Unsupervised co-segmentation through region matching. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 749–756. IEEE (2012)Google Scholar
  28. 28.
    Vicente, S., Rother, C., Kolmogorov, V.: Object cosegmentation. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2217–2224. IEEE (2011)Google Scholar
  29. 29.
    Wainwright, M.J., Jaakkola, T.S., Willsky, A.S.: A new class of upper bounds on the log partition function. Trans. Inf. Theory 51(7), 2313–2335 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Zhang, D., Javed, O., Shah, M.: Video object co-segmentation by regulated maximum weight cliques. In: European Conference on Computer Vision, pp. 551–566. Springer (2014)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Electrical Engineering SchoolTel Aviv UniversityTel AvivIsrael
  2. 2.FacebookSeattleUSA
  3. 3.Faculty of Industrial Engineering and ManagementTechnionHaifaIsrael
  4. 4.Computer Science SchoolTel Aviv UniversityTel AvivIsrael

Personalised recommendations