Skip to main content

MHR-Net: Multiple-Hypothesis Reconstruction of Non-Rigid Shapes from 2D Views

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13662))

Included in the following conference series:

Abstract

We propose MHR-Net, a novel method for recovering Non-Rigid Shapes from Motion (NRSfM). MHR-Net aims to find a set of reasonable reconstructions for a 2D view, and it also selects the most likely reconstruction from the set. To deal with the challenging unsupervised generation of non-rigid shapes, we develop a new Deterministic Basis and Stochastic Deformation scheme in MHR-Net. The non-rigid shape is first expressed as the sum of a coarse shape basis and a flexible shape deformation, then multiple hypotheses are generated with uncertainty modeling of the deformation part. MHR-Net is optimized with reprojection loss on the basis and the best hypothesis. Furthermore, we design a new Procrustean Residual Loss, which reduces the rigid rotations between similar shapes and further improves the performance. Experiments show that MHR-Net achieves state-of-the-art reconstruction accuracy on Human3.6M, SURREAL and 300-VW datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Note that in general \(\delta _{res}\) cannot be reduce to exactly zero for all pairs simultaneously.

  2. 2.

    Please refer to the supplementary material or [40] for details.

References

  1. Agudo, A.: Unsupervised 3d reconstruction and grouping of rigid and non-rigid categories. IEEE TPAMI (2020)

    Google Scholar 

  2. Agudo, A., Agapito, L., Calvo, B., Montiel, J.M.M.: Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: CVPR (2014)

    Google Scholar 

  3. Agudo, A., Moreno-Noguer, F.: Learning shape, motion and elastic models in force space. In: ICCV (2015)

    Google Scholar 

  4. Agudo, A., Moreno-Noguer, F.: Recovering pose and 3d deformable shape from multi-instance image ensembles. In: ACCV (2016)

    Google Scholar 

  5. Agudo, A., Moreno-Noguer, F.: DUST: dual union of spatio-temporal subspaces for monocular multiple object 3d reconstruction. In: CVPR, pp. 1513–1521 (2017)

    Google Scholar 

  6. Agudo, A., Moreno-Noguer, F., Calvo, B., Montiel, J.M.M.: Sequential non-rigid structure from motion using physical priors. IEEE TPAMI (2016)

    Google Scholar 

  7. Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Trajectory space: a dual representation for nonrigid structure from motion. IEEE TPAMI 33(7), 1442–1456 (2011)

    Article  Google Scholar 

  8. Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S.I., Sayd, P.: Coarse-to-fine low-rank structure-from-motion. In: CVPR (2008)

    Google Scholar 

  9. Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3d shape from image streams. In: CVPR, pp. 2690–2696 (2000)

    Google Scholar 

  10. Bue, A.D.: Adaptive non-rigid registration and structure from motion from image trajectories. IJCV 103(2), 226–239 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Bue, A.D., Smeraldi, F., Agapito, L.: Non-rigid structure from motion using ranklet-based tracking and non-linear optimization. Image Vis. Comput. 25(3), 297–310 (2007)

    Article  Google Scholar 

  12. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230, 000 3d facial landmarks). In: ICCV (2017)

    Google Scholar 

  13. Cha, G., Lee, M., Oh, S.: Reconstruct as far as you can: consensus of non-rigid reconstruction from feasible regions. IEEE TPAMI, pp. 1–1 (2019)

    Google Scholar 

  14. Chenxin, X., Siheng, C., Maosen, L., Ya, Z.: Invariant teacher and equivariant student for unsupervised 3d human pose estimation. In: AAAI, pp. 3013–3021 (2021)

    Google Scholar 

  15. Chhatkuli, A., Pizarro, D., Collins, T., Bartoli, A.: Inextensible non-rigid shape-from-motion by second-order cone programming. In: CVPR, pp. 1719–1727 (2016)

    Google Scholar 

  16. Dai, Y., Li, H.: Rank minimization or nuclear-norm minimization: are we solving the right problem? In: International Conference on Digital Image Computing: Techniques and Applications (2014)

    Google Scholar 

  17. Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: CVPR (2012)

    Google Scholar 

  18. Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. IJCV 107(2), 101–122 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  19. Fayad, J., Agapito, L., Bue, A.D.: Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences. In: ECCV, pp. 297–310 (2010)

    Google Scholar 

  20. Fragkiadaki, K., Salas, M., Arbeláez, P.A., Malik, J.: Grouping-based low-rank trajectory completion and 3d reconstruction. In: NeurIPS, pp. 55–63 (2014)

    Google Scholar 

  21. Gotardo, P.F.U., Martinez, A.M.: Computing smooth time trajectories for camera and deformable shape in structure from motion with occlusion. IEEE TPAMI 33(10), 2051–2065 (2011)

    Article  Google Scholar 

  22. Gotardo, P.F.U., Martinez, A.M.: Kernel non-rigid structure from motion. In: ICCV, pp. 802–809 (2011)

    Google Scholar 

  23. Gotardo, P.F.U., Martínez, A.M.: Non-rigid structure from motion with complementary rank-3 spaces. In: CVPR (2011)

    Google Scholar 

  24. Iglesias, J.P., Olsson, C., Örnhag, M.V.: Accurate optimization of weighted nuclear norm for non-rigid structure from motion. In: CVPR (2020)

    Google Scholar 

  25. Ijaz, A., Yaser, S., Sohaib, K.: In defense of orthonormality constraints for nonrigid structure from motion. In: CVPR (2009)

    Google Scholar 

  26. Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6m: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE TPAMI 36(7), 1325–1339 (2014)

    Google Scholar 

  27. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)

    Google Scholar 

  28. Kong, C., Lucey, S.: Deep non-rigid structure from motion with missing data. IEEE TPAMI, pp. 1–1 (2020)

    Google Scholar 

  29. Kong, C., Zhu, R., Kiani, H., Lucey, S.: Structure from category: a generic and prior-less approach. In: 3DV, pp. 296–304 (2016)

    Google Scholar 

  30. Kudo, Y., Ogaki, K., Matsui, Y., Odagiri, Y.: Unsupervised adversarial learning of 3d human pose from 2d joint locations. CoRR abs/1803.08244 (2018)

    Google Scholar 

  31. Kumar, S.: Jumping manifolds: Geometry aware dense non-rigid structure from motion. In: CVPR, pp. 5346–5355 (2019)

    Google Scholar 

  32. Kumar, S., Cherian, A., Dai, Y., Li, H.: Scalable dense non-rigid structure-from-motion: a grassmannian perspective. In: CVPR, pp. 254–263 (2018)

    Google Scholar 

  33. Kumar, S., Dai, Y., Li, H.: Spatio-temporal union of subspaces for multi-body non-rigid structure-from-motion. Pattern Recogn. 71, 428–443 (2017)

    Article  Google Scholar 

  34. Lee, M., Cho, J., Choi, C.H., Oh, S.: Procrustean normal distribution for non-rigid structure from motion. IEEE TPAMI 39(7), 1388–1400 (2017)

    Article  Google Scholar 

  35. Lee, S., Purushwalkam, S., Cogswell, M., Ranjan, V., Crandall, D.J., Batra, D.: Stochastic multiple choice learning for training diverse deep ensembles. In: NeurIPS (2016)

    Google Scholar 

  36. Li, C., Lee, G.H.: Generating multiple hypotheses for 3d human pose estimation with mixture density network. In: CVPR (2019)

    Google Scholar 

  37. Li, C., Lee, G.H.: Weakly supervised generative network for multiple 3d human pose hypotheses. In: BMVC (2020)

    Google Scholar 

  38. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM TOG 34(6), 248:1–248:16 (2015)

    Google Scholar 

  39. Martinez, J., Hossain, R., Romero, J., Little, J.J.: A simple yet effective baseline for 3d human pose estimation. In: ICCV (2017)

    Google Scholar 

  40. Novotny, D., Ravi, N., Graham, B., Neverova, N., Vedaldi, A.: C3DPO: canonical 3d pose networks for non-rigid structure from motion. In: ICCV (2019)

    Google Scholar 

  41. Paladini, M., Bartoli, A., Agapito, L.: Sequential non-rigid structure-from-motion with the 3d-implicit low-rank shape model. In: ECCV (2010)

    Google Scholar 

  42. Paladini, M., Bue, A.D., Stosic, M., Dodig, M., Xavier, J., Agapito, L.: Factorization for non-rigid and articulated structure using metric projections. In: CVPR (2009)

    Google Scholar 

  43. Parashar, S., Pizarro, D., Bartoli, A.: Isometric non-rigid shape-from-motion with riemannian geometry solved in linear time. IEEE TPAMI 40(10), 2442–2454 (2018)

    Article  Google Scholar 

  44. Parashar, S., Salzmann, M., Fua, P.: Local non-rigid structure-from-motion from diffeomorphic mappings. In: CVPR (2020)

    Google Scholar 

  45. Park, H.S., Shiratori, T., Matthews, I.A., Sheikh, Y.: 3d reconstruction of a moving point from a series of 2d projections. In: ECCV (2010)

    Google Scholar 

  46. Park, S., Lee, M., Kwak, N.: Procrustean regression: a flexible alignment-based framework for nonrigid structure estimation. IEEE TIP 27(1), 249–264 (2018)

    MathSciNet  MATH  Google Scholar 

  47. Park, S., Lee, M., Kwak, N.: Procrustean regression networks: learning 3d structure of non-rigid objects from 2d annotations. In: ECCV, pp. 1–18 (2020)

    Google Scholar 

  48. Pavllo, D., Feichtenhofer, C., Grangier, D., Auli, M.: 3d human pose estimation in video with temporal convolutions and semi-supervised training. In: CVPR (2019)

    Google Scholar 

  49. Probst, T., Paudel, D.P., Chhatkuli, A., Gool, L.V.: Incremental non-rigid structure-from-motion with unknown focal length. In: ECCV (2018)

    Google Scholar 

  50. Rabaud, V.C., Belongie, S.J.: Re-thinking non-rigid structure from motion. In: CVPR (2008)

    Google Scholar 

  51. Rupprecht, C., Laina, I., DiPietro, R.S., Baust, M.: Learning in an uncertain world: representing ambiguity through multiple hypotheses. In: ICCV (2017)

    Google Scholar 

  52. Russell, C., Fayad, J., Agapito, L.: Energy based multiple model fitting for non-rigid structure from motion. In: CVPR (2011)

    Google Scholar 

  53. Salzmann, M., Pilet, J., Ilic, S., Fua, P.: Surface deformation models for nonrigid 3d shape recovery. IEEE TPAMI 29(8), 1481–1487 (2007)

    Article  Google Scholar 

  54. Schonemann, P.H.: A generalized solution of the orthogonal procrustes problem. Psychometrika (1966)

    Google Scholar 

  55. Sharma, S., Varigonda, P.T., Bindal, P., Sharma, A., Jain, A.: Monocular 3d human pose estimation by generation and ordinal ranking. In: ICCV (2019)

    Google Scholar 

  56. Shen, J., Zafeiriou, S., Chrysos, G.G., Kossaifi, J., Tzimiropoulos, G., Pantic, M.: The first facial landmark tracking in-the-wild challenge: benchmark and results. In: ICCVW (2015)

    Google Scholar 

  57. Sidhu, V., Tretschk, E., Golyanik, V., Agudo, A., Theobalt, C.: Neural dense non-rigid structure from motion with latent space constraints. In: ECCV, pp. 204–222 (2020)

    Google Scholar 

  58. Sminchisescu, C., Triggs, B.: Covariance scaled sampling for monocular 3d body tracking. In: CVPR (2001)

    Google Scholar 

  59. Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: NeurIPS (2015)

    Google Scholar 

  60. Taylor, J., Jepson, A.D., Kutulakos, K.N.: Non-rigid structure from locally-rigid motion. In: CVPR, pp. 2761–2768 (2010)

    Google Scholar 

  61. Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE TPAMI 30(5), 878–892 (2008)

    Article  Google Scholar 

  62. Valmadre, J., Lucey, S.: Deterministic 3d human pose estimation using rigid structure. In: ECCV (2010)

    Google Scholar 

  63. Varol, G., et al.: Learning from synthetic humans. In: CVPR (2017)

    Google Scholar 

  64. Vicente, S., Agapito, L.: Soft inextensibility constraints for template-free non-rigid reconstruction. In: ECCV, pp. 426–440 (2012)

    Google Scholar 

  65. Wang, C., Kong, C., Lucey, S.: Distill knowledge from NRSFM for weakly supervised 3d pose learning. In: ICCV (2019)

    Google Scholar 

  66. Wang, C., Lucey, S.: PAUL: procrustean autoencoder for unsupervised lifting. In: CVPR (2021)

    Google Scholar 

  67. Wehrbein, T., Rudolph, M., Rosenhahn, B., Wandt, B.: Probabilistic monocular 3d human pose estimation with normalizing flows. In: ICCV (2021)

    Google Scholar 

  68. Xiao, J., Chai, J., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. IJCV 67(2), 233–246 (2006)

    Article  MATH  Google Scholar 

  69. Zeng, H., Dai, Y., Yu, X., Wang, X., Yang, Y.: PR-RRN: pairwise-regularized residual-recursive networks for non-rigid structure-from-motion. In: ICCV (2021)

    Google Scholar 

  70. Zhu, Y., Huang, D., Torre, F.D.L., Lucey, S.: Complex non-rigid motion 3d reconstruction by union of subspaces. In: CVPR, pp. 1542–1549 (2014)

    Google Scholar 

Download references

Acknowledgement

This work was partially done when Haitian Zeng interned at Baidu Research. This work was partially supported by ARC DP200100938. We thank Dr. Sungheon Park for sharing the SURREAL dataset. We thank all reviewers and area chairs for their valuable feedback.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yi Yang .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 266 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zeng, H., Yu, X., Miao, J., Yang, Y. (2022). MHR-Net: Multiple-Hypothesis Reconstruction of Non-Rigid Shapes from 2D Views. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13662. Springer, Cham. https://doi.org/10.1007/978-3-031-20086-1_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20086-1_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20085-4

  • Online ISBN: 978-3-031-20086-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics