Abstract
We address the problem of segmenting moving rigid objects based on two-view image correspondences under a perspective camera model. While this is a well understood problem, existing methods scale poorly with the number of correspondences. In this paper we propose a fast segmentation algorithm that scales linearly with the number of correspondences and show that on benchmark datasets it offers the best trade-off between error and computational time: it is at least one order of magnitude faster than the best method (with comparable or better accuracy), with the ratio growing up to three orders of magnitude for larger number of correspondences. We approach the problem from an algebraic perspective by exploiting the fact that all points belonging to a given object lie in the same quadratic surface. The proposed method is based on a characterization of each surface in terms of the Christoffel polynomial associated with the probability that a given point belongs to the surface. This allows for efficiently segmenting points “one surface at a time” in \(\mathcal {O}(\text {number of points)}\).
This work was supported in part by NSF grants IIS–1814631 and CNS–2038493, ONR grant N00014-21-1-2431 and U.S. DHS grant 22STESE00001-01-00.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For a singular \(\textbf{L} \doteq \textbf{U}\begin{bmatrix} \boldsymbol{\Sigma } &{} \textbf{0} \\ \textbf{0} &{} \textbf{0} \end{bmatrix}\textbf{U}^T\), \(\textbf{L}^{-1}\doteq \textbf{U}\begin{bmatrix} \boldsymbol{\Sigma }^{-1} &{} \textbf{0} \\ \textbf{0} &{} \textbf{0} \end{bmatrix}\textbf{U}^T\).
References
Kim, J.B., Kim, H.J.: Efficient region-based motion segmentation for a video monitoring system. Pattern Recogn. Lett. 24(1–3), 113–128 (2003)
Ess, A., Mueller, T., Grabner, H., Van Gool, L.: Segmentation-based urban traffic scene understanding. In: BMVC, vol. 1, p. 2. Citeseer (2009)
Weinland, D., Ronfard, R., Boyer, E.: A survey of vision-based methods for action representation, segmentation and recognition. Comput. Vis. Image Underst. 115(2), 224–241 (2011)
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE (2012)
Saputra, M.R.U., Markham, A., Trigoni, N.: Visual SLAM and structure from motion in dynamic environments: a survey. ACM Comput. Surv. (CSUR) 51(2), 1–36 (2018)
Ji, P., Li, H., Salzmann, M., Dai, Y.: Robust motion segmentation with unknown correspondences. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 204–219. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_14
Wang, Y., Liu, Y., Blasch, E., Ling, H.: Simultaneous trajectory association and clustering for motion segmentation. IEEE Signal Process. Lett. 25(1), 145–149 (2017)
Gear, C.W.: Multibody grouping from motion images. Int. J. Comput. Vis. 29(2), 133–150 (1998)
Elhamifar, E., Vidal, R.: Sparse subspace clustering. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2790–2797. IEEE (2009)
Kanatani, K.: Motion segmentation by subspace separation and model selection. In: Proceedings Eighth IEEE International Conference on computer Vision, ICCV 2001, vol. 2, pp. 586–591. IEEE (2001)
Yan, J., Pollefeys, M.: A general framework for motion segmentation: independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 94–106. Springer, Heidelberg (2006). https://doi.org/10.1007/11744085_8
Liu, G., Lin, Z., Shuicheng Yan, J., Sun, Y.Y., Ma, Y.: Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 171–184 (2012)
Elhamifar, E., Vidal, R.: Sparse subspace clustering: algorithm, theory, and applications. IEEE Trans. Pattern Anal. Mach. Intell. 35(11), 2765–2781 (2013)
Liu, G., Yan, S.: Latent low-rank representation for subspace segmentation and feature extraction. In: 2011 International Conference on Computer Vision, pp. 1615–1622. IEEE (2011)
Vidal, R., Ma, Y., Sastry, S.: Generalized principal component analysis (GPCA). IEEE Trans. PAMI 27(12), 1945–1959 (2005)
Rao, S., Tron, R., Vidal, R., Ma, Y.: Motion segmentation in the presence of outlying, incomplete, or corrupted trajectories. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1832–1845 (2009)
Zappella, L., Provenzi, E., Lladó, X., Salvi, J.: Adaptive motion segmentation algorithm based on the principal angles configuration. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6494, pp. 15–26. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19318-7_2
Keuper, M., Andres, B., Brox, T.: Motion trajectory segmentation via minimum cost multicuts. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3271–3279 (2015)
Costeira, J.P., Kanade, T.: A multibody factorization method for independently moving objects. Int. J. Comput. Vision 29(3), 159–179 (1998)
Gruber, A., Weiss, Y.: Multibody factorization with uncertainty and missing data using the EM algorithm. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004, vol. 1, p. I. IEEE (2004)
Vidal, R., Tron, R., Hartley, R.: Multiframe motion segmentation with missing data using powerfactorization and GPCA. Int. J. Comput. Vision 79(1), 85–105 (2008)
Sugaya, Y., Kanatani, K.: Geometric structure of degeneracy for multi-body motion segmentation. In: Comaniciu, D., Mester, R., Kanatani, K., Suter, D. (eds.) SMVP 2004. LNCS, vol. 3247, pp. 13–25. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30212-4_2
Flores-Mangas, F., Jepson, A.D.: Fast rigid motion segmentation via incrementally-complex local models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2259–2266 (2013)
Tron, R., Vidal, R.: A benchmark for the comparison of 3-d motion segmentation algorithms. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)
Schindler, K., U, J., Wang, H.: Perspective n-view multibody structure-and-motion through model selection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 606–619. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_47
Dragon, R., Rosenhahn, B., Ostermann, J.: Multi-scale clustering of frame-to-frame correspondences for motion segmentation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 445–458. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_32
Li, Z., Guo, J., Cheong, L.-F., Zhou, S.Z.: Perspective motion segmentation via collaborative clustering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1369–1376 (2013)
Xu, X., Cheong, L.F., Li, Z.: Motion segmentation by exploiting complementary geometric models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2859–2867 (2018)
Arrigoni, F., Pajdla, T.: Motion segmentation via synchronization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)
Arrigoni, F., Pajdla, T.: Robust motion segmentation from pairwise matches. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 671–681 (2019)
Arrigoni, F., Magri, L., Pajdla, T.: On the usage of the trifocal tensor in motion segmentation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 514–530. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_31
Arrigoni, F., Ricci, E., Pajdla, T.: Multi-frame motion segmentation by combining two-frame results. Int. J. Comput. Vision 130(3), 696–728 (2022)
Torr, P.H.S.: Geometric motion segmentation and model selection. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 356(1740), 1321–1340 (1998)
Vidal, R., Soatto, S., Ma, Y., Sastry, S.: Segmentation of dynamic scenes from the multibody fundamental matrix. In: ECCV Workshop on Vision and Modeling of Dynamic Scenes (2002)
Li, H.: Two-view motion segmentation from linear programming relaxation. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)
Jian, Y.-D., Chen, C.-S.: Two-view motion segmentation with model selection and outlier removal by RANSAC-enhanced Dirichlet process mixture models. Int. J. Comput. Vision 88(3), 489–501 (2010)
Jung, H., Ju, J., Kim, J.: Rigid motion segmentation using randomized voting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1210–1217 (2014)
Poling, B., Lerman, G.: A new approach to two-view motion segmentation using global dimension minimization. Int. J. Comput. Vision 108(3), 165–185 (2014)
Chum, O., Matas, J., Kittler, J.: Locally optimized RANSAC. In: Michaelis, B., Krell, G. (eds.) DAGM 2003. LNCS, vol. 2781, pp. 236–243. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45243-0_31
Kanazawa, Y., Kawakami, H.: Detection of planar regions with uncalibrated stereo using distributions of feature points. In: BMVC, pp. 1–10. Citeseer (2004)
Tordoff, B.J., Murray, D.W.: Guided-MLESAC: faster image transform estimation by using matching priors. IEEE Trans. Pattern Anal. Mach. Intell. 27(10), 1523–1535 (2005)
Chum, O., Matas, J.: Matching with PROSAC-progressive sample consensus. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 1, pp. 220–226. IEEE (2005)
Brahmachari, A.S., Sarkar, S.: Blogs: balanced local and global search for non-degenerate two view epipolar geometry. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1685–1692. IEEE (2009)
McIlroy, P., Rosten, E., Taylor, S., Drummond, T.: Deterministic sample consensus with multiple match hypotheses. In: BMVC, pp. 1–11. Citeseer (2010)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Vincent, E., Laganiére, R.: Detecting planar homographies in an image pair. In: Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis, ISPA 2001. Conjunction with 23rd International Conference on Information Technology Interfaces (IEEE Cat.), pp. 182–187. IEEE (2001)
Zuliani, M., Kenney, C.S., Manjunath, B.S.: The multiRANSAC algorithm and its application to detect planar homographies. In: IEEE International Conference on Image Processing, vol. 3, pp. III-153. IEEE (2005)
Magri, L., Fusiello, A.: Multiple model fitting as a set coverage problem. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3318–3326 (2016)
Zhang, W., Kǒsecká, J.: Nonparametric estimation of multiple structures with outliers. In: Vidal, R., Heyden, A., Ma, Y. (eds.) WDV 2005–2006. LNCS, vol. 4358, pp. 60–74. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-70932-9_5
Toldo, R., Fusiello, A.: Robust multiple structures estimation with J-linkage. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5302, pp. 537–547. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88682-2_41
Chin, T.-J., Suter, D., Wang, H.: Multi-structure model selection via kernel optimisation. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3586–3593. IEEE (2010)
Magri, L., Fusiello, A.: T-linkage: a continuous relaxation of j-linkage for multi-model fitting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3954–3961 (2014)
Pham, T.T., Chin, T.-J., Yu, J., Suter, D.: The random cluster model for robust geometric fitting. IEEE Trans. Pattern Anal. Mach. Intell. 36(8), 1658–1671 (2014)
Magri, L., Fusiello, A.: Robust multiple model fitting with preference analysis and low-rank approximation. In: BMVC, vol. 20, p. 12 (2015)
Torr, P.H., Nasuto, S.J., Bishop, J.M.: NAPSAC: high noise, high dimensional robust estimation-it’s in the bag. In: British Machine Vision Conference (BMVC) (2002)
Chin, T.-J., Yu, J., Suter, D.: Accelerated hypothesis generation for multistructure data via preference analysis. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 625–638 (2011)
Tennakoon, R., Sadri, A., Hoseinnezhad, R., Bab-Hadiashar, A.: Effective sampling: fast segmentation using robust geometric model fitting. IEEE Trans. Image Process. 27(9), 4182–4194 (2018)
Raguram, R., Frahm, J.-M., Pollefeys, M.: A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 500–513. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88688-4_37
Kim, T., Yu, W.: Performance evaluation of RANSAC family. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 1–12 (2009)
Kluger, F., Brachmann, E., Ackermann, H., Rother, C., Yang, M.Y., Rosenhahn, B.: CONSAC: robust multi-model fitting by conditional sample consensus. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4634–4643 (2020)
Ivashechkin, M., Barath, D., Matas, J.: VSAC: efficient and accurate estimator for H and F. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15243–15252 (2021)
Tennakoon, R., Suter, D., Zhang, E., Chin, T.-J., Bab-Hadiashar, A.: Consensus maximisation using influences of monotone boolean functions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2866–2875 (2021)
Magri, L., Fusiello, A.: Fitting multiple heterogeneous models by multi-class cascaded t-linkage. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7460–7468 (2019)
Xiao, G., Wang, H., Lai, T., Suter, D.: Hypergraph modelling for geometric model fitting. Pattern Recogn. 60, 748–760 (2016)
Lin, S., Xiao, G., Yan, Y., Suter, D., Wang, H.: Hypergraph optimization for multi-structural geometric model fitting. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8730–8737 (2019)
Magri, L., Leveni, F., Boracchi, G.: Multilink: multi-class structure recovery via agglomerative clustering and model selection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1853–1862 (2021)
Wang, H., Xiao, G., Yan, Y., Suter, D.: Searching for representative modes on hypergraphs for robust geometric model fitting. IEEE Trans. Pattern Anal. Mach. Intell. 41(3), 697–711 (2018)
Barath, D., Matas, J.: Progressive-X: efficient, anytime, multi-model fitting algorithm. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3780–3788 (2019)
Barath, D., Rozumny, D., Eichhardt, I., Hajder, L., Matas, J.: Progressive-X+: clustering in the consensus space. arXiv preprint arXiv:2103.13875 (2021)
Vidal, R., Soatto, S., Ma, Y., Sastry, S.: An algebraic geometric approach to the identification of a class of linear hybrid systems. In: 42nd IEEE International Conference on Decision and Control, vol. 1, pp. 167–172 (2003)
Xu, Y.: On orthogonal polynomials in several variables. Spec. Funct. Q-Ser. Related Topics Fields Inst. Res. Math. Sci. Commun. Ser. 14, 247–270 (1997)
Pauwels, E., Lasserre, J.B.: Sorting out typicality with the inverse moment matrix SOS polynomial. In: Advances in Neural Information Processing Systems, pp. 190–198 (2016)
Sznaier, M., Camps, O.: SOS-RSC: a sum-of-squares polynomial approach to robustifying subspace clustering algorithms. In: IEEE CVPR, pp. 8033–8041 (2018)
Wong, H.S., Chin, T.-J., Yu, J., Suter, D.: Dynamic and hierarchical multi-structure geometric model fitting. In: 2011 International Conference on Computer Vision, pp. 1044–1051. IEEE (2011)
Wang, H., Chin, T.-J., Suter, D.: Simultaneously fitting and segmenting multiple-structure data with outliers. IEEE Trans. Pattern Anal. Mach. Intell. 34(6), 1177–1192 (2011)
Tiwari, L., Anand, S.: DGSAC: density guided sampling and consensus. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 974–982. IEEE (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ozbay, B., Camps, O., Sznaier, M. (2022). Fast Two-View Motion Segmentation Using Christoffel Polynomials. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13690. Springer, Cham. https://doi.org/10.1007/978-3-031-20056-4_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-20056-4_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20055-7
Online ISBN: 978-3-031-20056-4
eBook Packages: Computer ScienceComputer Science (R0)