Skip to main content
SpringerLink
Log in

Organic Priors in Non-rigid Structure from Motion

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13662))

Included in the following conference series:

Abstract

This paper advocates the use of organic priors in classical non-rigid structure from motion (NRSfM). By organic priors, we mean invaluable intermediate prior information intrinsic to the NRSfM matrix factorization theory. It is shown that such priors reside in the factorized matrices, and quite surprisingly, existing methods generally disregard them. The paper’s main contribution is to put forward a simple, methodical, and practical method that can effectively exploit such organic priors to solve NRSfM. The proposed method does not make assumptions other than the popular one on the low-rank shape and offers a reliable solution to NRSfM under orthographic projection. Our work reveals that the accessibility of organic priors is independent of the camera motion and shape deformation type. Besides that, the paper provides insights into the NRSfM factorization—both in terms of shape and motion—and is the first approach to show the benefit of single rotation averaging for NRSfM. Furthermore, we outline how to effectively recover motion and non-rigid 3D shape using the proposed organic prior based approach and demonstrate results that outperform prior-free NRSfM performance by a significant margin. Finally, we present the benefits of our method via extensive experiments and evaluations on several benchmark datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    See, however, C. Tomasi and T. Kanade, pp. 137–154, IJCV (1992) for the original matrix factorization theory for shape and motion estimation, although devoted to the rigid SfM problem [49].

  2. 2.

    pinv() symbolizes Moore-Penrose inverse of a matrix, also known as pseudoinverse.

  3. 3.

    Familiarity with [13, 28] gives a good insight on our paper’s novelty.

  4. 4.

    filter if sample is too far to the reference rotation after registration.

  5. 5.

    After registration, if samples are filtered out due to its distance from the reference rotation (more than \(\delta \)), then per frame rotations is less than K.

  6. 6.

    For more discussion on partial sum minimization of singular values, cf. the supplementary material. For a comprehensive theory refer to [40].

  7. 7.

    With \(\mathbf {W=RS}\) theory, even GT rotation cannot provide GT shape, cf. [13] Table 3.

References

  1. Aanæs, H., Kahl, F.: Estimation of deformable structure and motion. In: Proceedings of the Vision and Modelling of Dynamic Scenes Workshop, vol. 2, pp. 3. Citeseer (2002)

    Google Scholar 

  2. Akhter, I., Sheikh, Y., Khan, S.: In defense of orthonormality constraints for nonrigid structure from motion. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 1534–1541. IEEE (2009)

    Google Scholar 

  3. Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. In: Advances in Neural Information Processing Systems, pp. 41–48 (2008)

    Google Scholar 

  4. Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Trajectory space: a dual representation for nonrigid structure from motion. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1442–1456 (2011)

    Article  Google Scholar 

  5. Ansari, M.D., Golyanik, V., Stricker, D.: Scalable dense monocular surface reconstruction. In: 2017 International Conference on 3D Vision (3DV), pp. 78–87. IEEE (2017)

    Google Scholar 

  6. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S., Sayd, P.: Coarse-to-fine low-rank structure-from-motion. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

    Google Scholar 

  7. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends® Mach. Learn. 3(1), 1–122 (2011)

    MATH  Google Scholar 

  8. Brand, M.: A direct method for 3D factorization of nonrigid motion observed in 2D. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), vol. 2, pp. 122–128. IEEE (2005)

    Google Scholar 

  9. Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 690–696. IEEE (2000)

    Google Scholar 

  10. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry Of Non-rigid Shapes. Springer Science & Business Media, Heidelberg (2008)

    MATH  Google Scholar 

  11. Cabral, R., De la Torre, F., Costeira, J.P., Bernardino, A.: Unifying nuclear norm and bilinear factorization approaches for low-rank matrix decomposition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2488–2495 (2013)

    Google Scholar 

  12. Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: Computer Vision and Pattern Recognition (CVPR), pp. 2018–2025. IEEE (2012)

    Google Scholar 

  13. Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. Int. J. Comput. Vision 107(2), 101–122 (2014). https://doi.org/10.1007/s11263-013-0684-2

    Article  MathSciNet  MATH  Google Scholar 

  14. Fayad, J., Agapito, L., Del Bue, A.: Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 297–310. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15561-1_22

    Chapter  Google Scholar 

  15. Garg, R., Roussos, A., Agapito, L.: Dense variational reconstruction of non-rigid surfaces from monocular video. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1272–1279 (2013)

    Google Scholar 

  16. Garg, R., Roussos, A., Agapito, L.: A variational approach to video registration with subspace constraints. Int. J. Comput. Vision 104(3), 286–314 (2013). https://doi.org/10.1007/s11263-012-0607-7

    Article  MathSciNet  MATH  Google Scholar 

  17. Golyanik, V., Jonas, A., Stricker, D., Theobalt, C.: Intrinsic dynamic shape prior for dense non-rigid structure from motion. In: 2020 International Conference on 3D Vision (3DV), pp. 692–701. IEEE (2020)

    Google Scholar 

  18. Gotardo, P.F., Martinez, A.M.: Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 2051–2065 (2011)

    Article  Google Scholar 

  19. Gotardo, P.F., Martinez, A.M.: Kernel non-rigid structure from motion. In: IEEE International Conference on Computer Vision, pp. 802–809. IEEE (2011)

    Google Scholar 

  20. Gotardo, P.F., Martinez, A.M.: Non-rigid structure from motion with complementary rank-3 spaces. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3065–3072. IEEE (2011)

    Google Scholar 

  21. Graßhof, S., Brandt, S.S.: Tensor-based non-rigid structure from motion. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3011–3020 (2022)

    Google Scholar 

  22. Hartley, R., Aftab, K., Trumpf, J.: L1 rotation averaging using the weiszfeld algorithm. In: CVPR 2011, pp. 3041–3048. IEEE (2011)

    Google Scholar 

  23. Hartley, R., Trumpf, J., Dai, Y., Li, H.: Rotation averaging. Int. J. Comput. Vision 103(3), 267–305 (2013). https://doi.org/10.1007/s11263-012-0601-0

    Article  MathSciNet  MATH  Google Scholar 

  24. Iglesias, J.P., Olsson, C., Valtonen Örnhag, M.: Accurate optimization of weighted nuclear norm for non-rigid structure from motion. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12372, pp. 21–37. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58583-9_2

    Chapter  Google Scholar 

  25. Jensen, S.H.N., Doest, M.E.B., Aanæs, H., Del Bue, A.: A benchmark and evaluation of non-rigid structure from motion. Int. J. Comput. Vision (IJCV) 129(4), 882–899 (2021). https://doi.org/10.1007/s11263-020-01406-y

    Article  Google Scholar 

  26. Kumar, S.: Jumping manifolds: geometry aware dense non-rigid structure from motion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5346–5355 (2019)

    Google Scholar 

  27. Kumar, S.: Non-rigid structure from motion. Ph.D. thesis, College of Engineering & Computer Science, The Australian National University (2019)

    Google Scholar 

  28. Kumar, S.: Non-rigid structure from motion: prior-free factorization method revisited. In: Winter Conference on Applications of Computer Vision (WACV 2020), pp. 51–60 (2020)

    Google Scholar 

  29. Kumar, S., Cherian, A., Dai, Y., Li, H.: Scalable dense non-rigid structure-from-motion: a Grassmannian perspective. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 254–263 (2018)

    Google Scholar 

  30. Kumar, S., Dai, Y., Li, H.: Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion. Pattern Recogn. 71, 428–443 (2017)

    Article  Google Scholar 

  31. Kumar, S., Dai, Y., Li, H.: Multi-body non-rigid structure-from-motion. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 148–156. IEEE (2016)

    Google Scholar 

  32. Kumar, S., Dai, Y., Li, H.: Monocular dense 3D reconstruction of a complex dynamic scene from two perspective frames. In: IEEE International Conference on Computer Vision, pp. 4649–4657 (2017)

    Google Scholar 

  33. Kumar, S., Dai, Y., Li, H.: Superpixel soup: monocular dense 3D reconstruction of a complex dynamic scene. IEEE Trans. Pattern Anal. Mach. Intell. 43(5), 1705–1717 (2019)

    Article  Google Scholar 

  34. Kumar, S., Ghorakavi, R.S., Dai, Y., Li, H.: Dense depth estimation of a complex dynamic scene without explicit 3D motion estimation. arXiv preprint arXiv:1902.03791 (2019)

  35. Kumar, S., Van Gool, L., de Oliveira, C.E., Cherian, A., Dai, Y., Li, H.: Dense non-rigid structure from motion: a manifold viewpoint. arXiv preprint arXiv:2006.09197 (2020)

  36. Lee, M., Cho, J., Choi, C.H., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1280–1287 (2013)

    Google Scholar 

  37. Lee, M., Cho, J., Oh, S.: Consensus of non-rigid reconstructions. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4670–4678 (2016)

    Google Scholar 

  38. Lee, S.H., Civera, J.: Robust single rotation averaging. arXiv preprint arXiv:2004.00732 (2020)

  39. Matthews, I., Xiao, J., Baker, S.: 2D vs. 3D deformable face models: representational power, construction, and real-time fitting. Int. J. Comput. Vision 75(1), 93–113 (2007). https://doi.org/10.1007/s11263-007-0043-2

    Article  Google Scholar 

  40. Oh, T.H., Tai, Y.W., Bazin, J.C., Kim, H., Kweon, I.S.: Partial sum minimization of singular values in robust PCA: algorithm and applications. IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 744–758 (2016)

    Article  Google Scholar 

  41. Ornhag, M.V., Iglesias, J.P., Olsson, C.: Bilinear parameterization for non-separable singular value penalties. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3897–3906 (2021)

    Google Scholar 

  42. Paladini, M., Del Bue, A., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898–2905. IEEE (2009)

    Google Scholar 

  43. Rabaud, V., Belongie, S.: Re-thinking non-rigid structure from motion. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)

    Google Scholar 

  44. Recht, B., Fazel, M., Parrilo, P.A.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  45. Russell, C., Fayad, J., Agapito, L.: Energy based multiple model fitting for non-rigid structure from motion. In: CVPR 2011, pp. 3009–3016. IEEE (2011)

    Google Scholar 

  46. Salzmann, M., Fua, P.: Deformable surface 3D reconstruction from monocular images. Synth. Lect. Comput. Vision 2(1), 1–113 (2010)

    Article  Google Scholar 

  47. Sidhu, V., Tretschk, E., Golyanik, V., Agudo, A., Theobalt, C.: Neural dense non-rigid structure from motion with latent space constraints. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 204–222. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_13

    Chapter  Google Scholar 

  48. Taylor, J., Jepson, A.D., Kutulakos, K.N.: Non-rigid structure from locally-rigid motion. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2761–2768. IEEE (2010)

    Google Scholar 

  49. Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vision 9(2), 137–154 (1992). https://doi.org/10.1007/BF00129684

    Article  Google Scholar 

  50. Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 878–892 (2008)

    Article  Google Scholar 

  51. Torresani, L., Hertzmann, A., Bregler, C.: Learning non-rigid 3D shape from 2D motion. In: Advances in Neural Information Processing Systems, pp. 1555–1562 (2004)

    Google Scholar 

  52. Valmadre, J., Sridharan, S., Denman, S., Fookes, C., Lucey, S.: Closed-form solutions for low-rank non-rigid reconstruction. In: 2015 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–6. IEEE (2015)

    Google Scholar 

  53. Xiao, J., Chai, J., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 573–587. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_46

    Chapter  Google Scholar 

  54. Zhu, Y., Huang, D., De La Torre, F., Lucey, S.: Complex non-rigid motion 3D reconstruction by union of subspaces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1542–1549 (2014)

    Google Scholar 

Download references

Acknowledgement

The authors thank Google for their generous gift (ETH Zürich Foundation, 2020-HS-411).

Author information

Authors and Affiliations

  1. ETH Zürich Switzerland, Zurich, Switzerland

    Suryansh Kumar & Luc Van Gool

  2. KU Leuven Belgium, Leuven, Belgium

    Luc Van Gool

Authors
  1. Suryansh Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luc Van Gool
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Suryansh Kumar .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2239 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kumar, S., Van Gool, L. (2022). Organic Priors in Non-rigid Structure from Motion. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13662. Springer, Cham. https://doi.org/10.1007/978-3-031-20086-1_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20086-1_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20085-4

  • Online ISBN: 978-3-031-20086-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation