Skip to main content
SpringerLink
Log in

Minimal Basis Subspace Representation: A Unified Framework for Rigid and Non-rigid Motion Segmentation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Motion segmentation and non-rigid structure from motion are two challenging computer vision problems that have attracted numerous research interests. While the previous works handle these two problems separately, we present a general motion segmentation framework in this paper for solving these two seemingly different problems in a unified manner. At the heart of our general motion segmentation framework is a model selection mechanism based on finding the minimal basis subspace representation, by seeking the joint sparse representation of the data matrix. However, such formulation is NP-hard and we solve the convex proxy instead. Unlike other compressive sensing related works, this convex proxy solution is insufficient for our problem. The convex relaxation artefacts and noise yield multiple subspace representations, making identification of the exact number of motion subspaces challenging. We solve for the right number of subspaces by transforming this problem into a Facility Location problem with global cost and solve the factor graph formulation using max product belief propagation message passing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. The figure of \(77.56~\%\) reported in Liu et al. (2012) is based on 156 sequences.

  2. Note that the error for SPF reported in Table 9, is different from the SPF reported in Dai et al. (2012). The error in Table 9 is based on the latest implementation by the author with the SVD sign ambiguity fixed.

References

  • Akhter, I., Sheikh, Y., & Khan, S. (2009). In defense of orthonormality constraints for nonrigid structure from motion. In CVPR (pp. 1534–1541).

  • Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2012). Trajectory basis for nonrigid structure from motion—project page. http://cvlab.lums.edu.pk/nrsfm/.

  • Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Nonrigid structure from motion in trajectory space. In NIPS.

  • Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2010). Trajectory space: A dual representation for nonrigid structure from motion. PAMI, 33(7), 1442–1456.

    Article  Google Scholar 

  • Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3d shape from image streams. CVPR, 2, 690–696.

    Google Scholar 

  • Chin, T., Suter, D., & Wang, H. (2010). Multi-structure model selection via kernel optimisation. In CVPR (pp. 3586–3593).

  • Chin, T., Wang, H., & Suter, D. (2009). The ordered residual kernel for robust motion subspace clustering. In NIPS.

  • Chung, F. (1997). Spectral graph theory. Providence: Amer Mathematical Society.

    MATH  Google Scholar 

  • CMU graphics lab motion capture database (2003). http://mocap.cs.cmu.edu/.

  • Dai, Y., Li, H., & He, M. (2012). A simple prior-free method for non-rigid structure-from-motion factorization. In CVPR (pp. 2018–2025).

  • Del Bue, A., Xavier, J., Agapito, L., & Paladini, M. (2012). Bilinear modeling via augmented lagrange multipliers (balm). PAMI, 34(8), 1496–1508.

    Article  Google Scholar 

  • Elhamifar, E., & Vidal, R. (2009). Sparse subspace clustering. In CVPR (pp. 2790–2797).

  • Elhamifar, E., & Vidal, R. (2013). Sparse subspace clustering: Algorithm, theory, and applications. PAMI, 35(11), 2765–2781.

    Article  Google Scholar 

  • Fayad, J., Agapito, L., & Del Bue, A. (2010). Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In ECCV (pp. 297–310).

  • Fayad, J., Russell, C., & Agapito, L. (2011). Automated articulated structure and 3d shape recovery from point correspondences. In ICCV (pp. 431–438).

  • Fragkiadaki, K., Salas, M., Arbelaez, P., & Malik, J. (2014). Grouping-based low-rank trajectory completion and 3d reconstruction. In NIPS (pp. 55–63).

  • Frey, B. J., & Dueck, D. (2007). Clustering by passing messages between data points. Science, 315, 972–976.

    Article  MathSciNet  MATH  Google Scholar 

  • Givoni, I. (2011). Beyond affinity propagation: Message passing algorithms for clustering. PhD Thesis, University of Toronto.

  • Givoni, I., Chung, C., & Frey, B. (2012). Hierarchical affinity propagation. arXiv preprint arXiv:1202.3722.

  • Givoni, I., & Frey, B. (2009). A binary variable model for affinity propagation. Neural Computation, 21(6), 1589–1600.

    Article  MathSciNet  MATH  Google Scholar 

  • Gotardo, P. F., & Martinez, A. M. (2011). CSF code download. http://cbcsl.ece.ohio-state.edu/downloads.html.

  • Gotardo, P. F., & Martinez, A. M. (2011). Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. PAMI, 33(10), 2051–2065.

    Article  Google Scholar 

  • Hurley, N., & Rickard, S. (2009). Comparing measures of sparsity. Information Theory, IEEE Transactions on, 55(10), 4723–4741.

    Article  MathSciNet  Google Scholar 

  • Ji, P., Li, H., Salzmann, M., & Dai, Y. (2014). Robust motion segmentation with unknown correspondences. In ECCV (pp. 204–219).

  • Kanatani, K., & Matsunaga, C. (2002). Estimating the number of independent motions for multibody motion segmentation. In Asian Conference on Computer Vision (pp. 7–12).

  • Kundu, A., Krishna, K.M., & Jawahar, C. (2011). Realtime multibody visual slam with a smoothly moving monocular camera. In ICCV (pp. 2080–2087).

  • Lauer, F., & Schnorr, C. (2009). Spectral clustering of linear subspaces for motion segmentation. In ICCV (pp. 678–685).

  • Lazic, N., Frey, B., & Aarabi, P. (2010). Solving the uncapacitated facility location problem using message passing algorithms. AISTATS, 9, 429–436.

    Google Scholar 

  • Lazic, N., Givoni, I., Frey, B., & Aarabi, P. (2009). Floss: Facility location for subspace segmentation. In ICCV (pp. 825–832).

  • Lee, C.M., & Cheong, L.F. (2013). Minimal basis facility location for subspace segmentation. In ICCV.

  • Lee, M., Cho, J., Choi, C.H., & Oh, S. (2013). Procrustean normal distribution for non-rigid structure from motion. In CVPR (pp. 1280–1287).

  • Li, H. (2007). Two-view motion segmentation from linear programming relaxation. In CVPR (pp. 1–8).

  • Liu, G. (2011). Low-rank representation matlab code. https://sites.google.com/site/guangcanliu/.

  • Liu, G., Lin, Z., Yan, S., Sun, J., Yu, Y., & Ma, Y. (2012). Robust recovery of subspace structures by low-rank representation. PAMI, 34(11), 171–184.

    Google Scholar 

  • Liu, G., Lin, Z., & Yu, Y. (2010). Robust subspace segmentation by low-rank representation. In ICML (pp. 663–670).

  • Ma, Y., Yang, A. Y., Derksen, H., & Fossum, R. (2008). Estimation of subspace arrangements with applications in modeling and segmenting mixed data. SIAM, 50(3), 413–458.

    Article  MathSciNet  MATH  Google Scholar 

  • Nadler, B., & Galun, M. (2007). Fundamental limitations of spectral clustering. In NIPS (vol. 19, p. 1017).

  • Ng, A., Jordan, M., & Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. NIPS, 2, 849–856.

    Google Scholar 

  • Nocedal, J., & Wright, S. (2006). Numerical optimization. Berlin: Springer.

    MATH  Google Scholar 

  • Paladini, M., Bartoli, A., & Agapito, L. (2010). Sequential non-rigid structure-from-motion with the 3d-implicit low-rank shape model. In ECCV (pp. 15–28).

  • Paladini, M., Del Bue, A., Stošic, M., Dodig, M., Xavier, J., & Agapito, L. (2009). Factorization for non-rigid and articulated structure using metric projections. In CVPR (pp. 2898–2905).

  • Russell, C., Fayad, J., & Agapito, L. (2011). Energy based multiple model fitting for non-rigid structure from motion. In CVPR (pp. 3009–3016).

  • Russell, C., Yu, R., & Agapito, L. (2014). Video pop-up: Monocular 3d reconstruction of dynamic scenes. In ECCV (pp. 583–598). Springer, Berlin.

  • Sabzevari, R., & Scaramuzza, D. (2014). Monocular simultaneous multi-body motion segmentation and reconstruction from perspective views. In Robotics and Automation (ICRA), 2014 IEEE International Conference on (pp. 23–30).

  • Schindler, K., James, U., & Wang, H. (2006). Perspective n-view multibody structure-and-motion through model selection. In ECCV (pp. 606–619).

  • Soltanolkotabi, M., & Candes, E. (2011). A geometric analysis of subspace clustering with outliers. Annals of Statistics, 2195–2238.

  • Szeliski, R., & Kang, S. B. (1997). Shape ambiguities in structure from motion. PAMI, 19(5), 506–512.

    Article  Google Scholar 

  • Taylor, J., Jepson, A. D., & Kutulakos, K.N. (2010). Non-rigid structure from locally-rigid motion. In CVPR (pp. 2761–2768).

  • Tomasi, C., & Kanade, T. (1992). Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2), 137–154.

    Article  Google Scholar 

  • Torresani, L., Hertzmann, A., & Bregler, C. Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors—project page. http://www.cs.dartmouth.edu/~lorenzo/nrsfm.html.

  • Torresani, L., Hertzmann, A., & Bregler, C. (2008). Nonrigid structure-from-motion: Estimating shape and motion with hierarchical priors. PAMI, 30(5), 878–892.

    Article  Google Scholar 

  • Tresadern, P., & Reid, I. (2005). Articulated structure from motion by factorization. CVPR, 2, 1110–1115.

  • Tron, R., & Vidal, R. (2007). A benchmark for the comparison of 3-d motion segmentation algorithms. In CVPR (pp. 1–8).

  • Vidal, R., Ma, Y., & Piazzi, J. (2004). A new gpca algorithm for clustering subspaces by fitting, differentiating and dividing polynomials. In CVPR (vol. 1, pp. I–510).

  • Vidal R., Ma, Y., & Sastry, S. (2005). Generalized principal component analysis (gpca). Pattern Analysis and Machine Intelligence, IEEE Transactions on 27(12), 1945–1959. doi:10.1109/TPAMI.2005.244.

  • Von Luxburg, U. (2007). A tutorial on spectral clustering. Statistics and Computing, 17(4), 395–416.

    Article  MathSciNet  Google Scholar 

  • Xiao, J., Chai, J. x., & Kanade, T. (2004). A closed-form solution to non-rigid shape and motion recovery. In ECCV (pp. 573–587).

  • Yan, J., & Pollefeys, M. (2006). A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate. In ECCV (pp. 94–106).

  • Yan, J., & Pollefeys, M. (2008). A factorization-based approach for articulated nonrigid shape, motion and kinematic chain recovery from video. PAMI, 30(5), 865–877.

    Article  Google Scholar 

Download references

Acknowledgments

We like to express our gratitude to Inmar Givoni for her help and guidance, as well as Rui Yu for providing the Messi dataset. The support of the Singapore PSF Grant 1321202075 is gratefully acknowledged.

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering Department, National University of Singapore, 4 Engineering Drive 3, Singapore, 117576, Singapore

    Choon-Meng Lee & Loong-Fah Cheong

Authors
  1. Choon-Meng Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Loong-Fah Cheong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Choon-Meng Lee.

Additional information

Communicated by J. Kosecka.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 302 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, CM., Cheong, LF. Minimal Basis Subspace Representation: A Unified Framework for Rigid and Non-rigid Motion Segmentation. Int J Comput Vis 121, 209–233 (2017). https://doi.org/10.1007/s11263-016-0928-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-016-0928-z

Keywords

Search

Navigation