Advertisement

On the Usage of the Trifocal Tensor in Motion Segmentation

Conference paper
  • 493 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12365)

Abstract

Motion segmentation, i.e., the problem of clustering data in multiple images based on different 3D motions, is an important task for reconstructing and understanding dynamic scenes. In this paper we address motion segmentation in multiple images by combining partial results coming from triplets of images, which are obtained by fitting a number of trifocal tensors to correspondences. We exploit the fact that the trifocal tensor is a stronger model than the fundamental matrix, as it provides fewer but more reliable matches over three images than fundamental matrices provide over the two. We also consider an alternative solution which merges partial results coming from both triplets and pairs of images, showing the strength of three-frame segmentation in a combination with two-frame segmentation. Our real experiments on standard as well as new datasets demonstrate the superior accuracy of the proposed approaches when compared to previous techniques .

Keywords

Motion segmentation Structure from motion Multi-model fitting Trifocal tensor 

Notes

Acknowledgements

This research was supported by the European Regional Development Fund under IMPACT No. CZ.02.1.01/0.0/0.0/15 003/0000468, R4I 4.0 No. CZ.02.1.01/0.0/0.0/15 003/0000470, EU H2020 ARtwin No. 856994, and EU H2020 SPRING No. 871245 Projects.

Supplementary material

504476_1_En_31_MOESM1_ESM.pdf (67.4 mb)
Supplementary material 1 (pdf 69016 KB)

References

  1. 1.
    Arrigoni, F., Fusiello, A.: Synchronization problems in computer vision with closed-form solutions. Int. J. Comput. Vis. 128, 26–52 (2020)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Arrigoni, F., Pajdla, T.: Motion segmentation via synchronization. In: IEEE International Conference on Computer Vision Workshops (ICCVW) (2019)Google Scholar
  3. 3.
    Arrigoni, F., Pajdla, T.: Robust motion segmentation from pairwise matches. In: Proceedings of the International Conference on Computer Vision (2019)Google Scholar
  4. 4.
    Barath, D., Matas, J.: Multi-class model fitting by energy minimization and mode-seeking. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 229–245. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01270-0_14CrossRefGoogle Scholar
  5. 5.
    Chin, T.J., Suter, D., Wang, H.: Multi-structure model selection via kernel optimisation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3586–3593 (2010)Google Scholar
  6. 6.
    Delong, A., Osokin, A., Isack, H.N., Boykov, Y.: Fast approximate energy minimization with label costs. Int. J. Comput. Vis. 96(1), 1–27 (2012)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Elhamifar, E., Vidal, R.: Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2765–2781 (2013)CrossRefGoogle Scholar
  8. 8.
    Fischler, M., Bolles, R.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Morgan Kaufmann Readings Ser. 24, 726–740 (1987)Google Scholar
  9. 9.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2012)Google Scholar
  10. 10.
    Hartley, R., Vidal, R.: The multibody trifocal tensor: motion segmentation from 3 perspective views. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I-769-I-775, June 2004.  https://doi.org/10.1109/CVPR.2004.1315109
  11. 11.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)CrossRefGoogle Scholar
  12. 12.
    Hartley, R.: Lines and points in three views and the trifocal tensor. Int. J. Comput. Vis. 22(2), 125–140 (1997)CrossRefGoogle Scholar
  13. 13.
    Holland, P.W., Welsch, R.E.: Robust regression using iteratively reweighted least-squares. Commun. Stat. Theory Methods 6(9), 813–827 (1977)CrossRefGoogle Scholar
  14. 14.
    Isack, H., Boykov, Y.: Energy-based geometric multi-model fitting. Int. J. Comput. Vis. 97(2), 123–147 (2012)CrossRefGoogle Scholar
  15. 15.
    Ji, P., Li, H., Salzmann, M., Dai, Y.: Robust motion segmentation with unknown correspondences. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 204–219. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10599-4_14CrossRefGoogle Scholar
  16. 16.
    Ji, P., Li, H., Salzmann, M., Zhong, Y.: Robust multi-body feature tracker: a segmentation-free approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  17. 17.
    Ji, P., Salzmann, M., Li, H.: Shape interaction matrix revisited and robustified: efficient subspace clustering with corrupted and incomplete data. In: Proceedings of the International Conference on Computer Vision, pp. 4687–4695 (2015)Google Scholar
  18. 18.
    Julià, L.F., Monasse, P.: A critical review of the trifocal tensor estimation. In: Paul, M., Hitoshi, C., Huang, Q. (eds.) PSIVT 2017. LNCS, vol. 10749, pp. 337–349. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-75786-5_28CrossRefGoogle Scholar
  19. 19.
    Kim, J.B., Kim, H.J.: Efficient region-based motion segmentation for a video monitoring system. Pattern Recogn. Lett. 24(1), 113–128 (2003)CrossRefGoogle Scholar
  20. 20.
    Kuang, D., Yun, S., Park, H.: SymNMF: nonnegative low-rank approximation of a similarity matrix for graph clustering. J. Global Optim. 62(3), 545–574 (2014).  https://doi.org/10.1007/s10898-014-0247-2MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logistics Q. 2(2), 83–97 (1955)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Lai, T., Wang, H., Yan, Y., Chin, T.J., Zhao, W.L.: Motion segmentation via a sparsity constraint. IEEE Trans. Intell. Transp. Syst. 18(4), 973–983 (2017)CrossRefGoogle Scholar
  23. 23.
    Li, Z., Guo, J., Cheong, L.F., Zhou, S.Z.: Perspective motion segmentation via collaborative clustering. In: Proceedings of the International Conference on Computer Vision, pp. 1369–1376 (2013)Google Scholar
  24. 24.
    Lin, Z., Chen, M., Ma, Y.: The augmented Lagrange multiplier method for exact recovery of corrupted low-rank matrices. eprint arXiv:1009.5055 (2010)
  25. 25.
    Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intel. 26(5), 171–184 (2013)CrossRefGoogle Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004).  https://doi.org/10.1023/B:VISI.0000029664.99615.94CrossRefGoogle Scholar
  27. 27.
    Magri, L., Fusiello, A.: Robust multiple model fitting with preference analysis and low-rank approximation. In: Proceedings of the British Machine Vision Conference, pp. 20.1-20.12. BMVA Press, September 2015Google Scholar
  28. 28.
    Magri, L., Fusiello, A.: Multiple models fitting as a set coverage problem. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3318–3326, June 2016Google Scholar
  29. 29.
    Olsson, C., Enqvist, O.: Stable structure from motion for unordered image collections. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 524–535. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-21227-7_49CrossRefGoogle Scholar
  30. 30.
    Ozden, K.E., Schindler, K., Van Gool, L.: Multibody structure-from-motion in practice. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1134–1141 (2010)CrossRefGoogle Scholar
  31. 31.
    Pachauri, D., Kondor, R., Singh, V.: Solving the multi-way matching problem by permutation synchronization. In: Advances in Neural Information Processing Systems 26, pp. 1860–1868. Curran Associates, Inc. (2013)Google Scholar
  32. 32.
    Pavan, A., Tangwongsan, K., Tirthapura, S., Wu, K.L.: Counting and sampling triangles from a graph stream. Proc. VLDB Endowment 6(14), 1870–1881 (2013)CrossRefGoogle Scholar
  33. 33.
    Rao, S., Tron, R., Vidal, R., Ma, Y.: Motion segmentation in the presence of outlying, incomplete, or corrupted trajectories. Pattern Anal. Mach. Intell. 32(10), 1832–1845 (2010)CrossRefGoogle Scholar
  34. 34.
    Rubino, C., Del Bue, A., Chin, T.J.: Practical motion segmentation for urban street view scenes. In: Proceedings of the IEEE International Conference on Robotics and Automation (2018)Google Scholar
  35. 35.
    Sabzevari, R., Scaramuzza, D.: Monocular simultaneous multi-body motion segmentation and reconstruction from perspective views. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 23–30 (2014)Google Scholar
  36. 36.
    Saputra, M.R.U., Markham, A., Trigoni, N.: Visual SLAM and structure from motion in dynamic environments: a survey. ACM Comput. Surveys 51(2), 37:1–37:36 (2018)Google Scholar
  37. 37.
    Schindler, K., Suter, D., Wang, H.: A model-selection framework for multibody structure-and-motion of image sequences. Int. J. Comput. Vis. 79(2), 159–177 (2008)CrossRefGoogle Scholar
  38. 38.
    Shen, Y., Huang, Q., Srebro, N., Sanghavi, S.: Normalized spectral map synchronization. In: Advances in Neural Information Processing Systems 29, pp. 4925–4933. Curran Associates, Inc. (2016)Google Scholar
  39. 39.
    Toldo, R., Fusiello, A.: Robust multiple structures estimation with J-Linkage. In: Proceedings of the European Conference on Computer Vision, pp. 537–547 (2008)Google Scholar
  40. 40.
    Torr, P.H.S., Zisserman, A.: Concerning Bayesian motion segmentation, model averaging, matching and the trifocal tensor. In: Burkhardt, H., Neumann, B. (eds.) ECCV 1998. LNCS, vol. 1406, pp. 511–527. Springer, Heidelberg (1998).  https://doi.org/10.1007/BFb0055687CrossRefGoogle Scholar
  41. 41.
    Torr, P.H.S., Zisserman, A., Murray, D.W.: Motion clustering using the trilinear constraint over three views. In: Europe-China Workshop on Geometric Modelling and Invariants for Computer Vision, pp. 118–125. Springer (1995)Google Scholar
  42. 42.
    Tron, R., Vidal, R.: A benchmark for the comparison of 3-D motion segmentation algorithms. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)Google Scholar
  43. 43.
    Tron, R., Zhou, X., Esteves, C., Daniilidis, K.: Fast multi-image matching via density-based clustering. In: Proceedings of the International Conference on Computer Vision, pp. 4077–4086 (2017)Google Scholar
  44. 44.
    Vidal, R., Ma, Y., Sastry, S.: Generalized principal component analysis (GPCA). IEEE Trans. Pattern Anal. Mach. Intell. 27(12), 1945–1959 (2005)CrossRefGoogle Scholar
  45. 45.
    Von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17(4), 395–416 (2007)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Wang, Y., Liu, Y., Blasch, E., Ling, H.: Simultaneous trajectory association and clustering for motion segmentation. IEEE Signal Process. Lett. 25(1), 145–149 (2018)CrossRefGoogle Scholar
  47. 47.
    Xu, X., Cheong, L.F., Li, Z.: Motion segmentation by exploiting complementary geometric models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2859–2867 (2018)Google Scholar
  48. 48.
    Yan, J., Pollefeys, M.: A general framework for motion segmentation: independent, articulated, rigid, non-rigid, degenerate and nondegenerate. In: Proceedings of the European Conference on Computer Vision, pp. 94–106 (2006)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.DISI, University of TrentoTrentoItaly
  2. 2.DEIB, Politecnico di MilanoMilanItaly
  3. 3.Czech Institute of Informatics, Robotics and Cybernetics (CIIRC)Czech Technical University in PraguePragueCzech Republic

Personalised recommendations