Symmetric Non-rigid Structure from Motion for Category-Specific Object Structure Estimation

  • Yuan GaoEmail author
  • Alan L. Yuille
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9906)


Many objects, especially these made by humans, are symmetric, e.g. cars and aeroplanes. This paper addresses the estimation of 3D structures of symmetric objects from multiple images of the same object category, e.g. different cars, seen from various viewpoints. We assume that the deformation between different instances from the same object category is non-rigid and symmetric. In this paper, we extend two leading non-rigid structure from motion (SfM) algorithms to exploit symmetry constraints. We model the both methods as energy minimization, in which we also recover the missing observations caused by occlusions. In particularly, we show that by rotating the coordinate system, the energy can be decoupled into two independent terms, which still exploit symmetry, to apply matrix factorization separately on each of them for initialization. The results on the Pascal3D+ dataset show that our methods significantly improve performance over baseline methods.


Symmetry Non-rigid structure from motion 



We would like to thank Ehsan Jahangiri, Cihang Xie, Weichao Qiu, Xuan Dong, Siyuan Qiao for giving feedbacks on the manuscript. This work was supported by ARO 62250-CS and ONR N00014-15-1-2356.

Supplementary material

419974_1_En_26_MOESM1_ESM.pdf (250 kb)
Supplementary material 1 (pdf 249 KB)


  1. 1.
    Tomasi, C., Kanade, T.: Shape and motion from image streams under orthography: a factorization method. Int. J. Comput. Vis. 9(2), 137–154 (1992)CrossRefGoogle Scholar
  2. 2.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  3. 3.
    Torresani, L., Hertzmann, A., Bregler, C.: Learning non-rigid 3D shape from 2D motion. In: NIPS (2003)Google Scholar
  4. 4.
    Xiao, J., Chai, J., Kanade, T.: A closed-form solution to non-rigid shape and motion recovery. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 573–587. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  5. 5.
    Torresani, L., Hertzmann, A., Bregler, C.: Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30, 878–892 (2008)CrossRefGoogle Scholar
  6. 6.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Trajectory space: a dual representation for nonrigid structure from motion. IEEE Trans. Pattern Anal. Mach. Intell. 33(7), 1442–1456 (2011)CrossRefGoogle Scholar
  7. 7.
    Gotardo, P., Martinez, A.: Computing smooth timetrajectories for camera and deformable shape in structure from motion with occlusion. IEEE Trans. Pattern Anal. Mach. Intell. 33, 2051–2065 (2011)CrossRefGoogle Scholar
  8. 8.
    Hamsici, O.C., Gotardo, P.F.U., Martinez, A.M.: Learning spatially-smooth mappings in non-rigid structure from motion. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 260–273. Springer, Heidelberg (2012)Google Scholar
  9. 9.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. In: CVPR (2012)Google Scholar
  10. 10.
    Dai, Y., Li, H., He, M.: A simple prior-free method for non-rigid structure-from-motion factorization. Int. J. Comput. Vis. 107, 101–122 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Ma, J., Zhao, J., Ma, Y., Tian, J.: Non-rigid visible and infrared face registration via regularized gaussian fields criterion. Pattern Recogn. 48(3), 772–784 (2015)CrossRefGoogle Scholar
  12. 12.
    Ma, J., Zhao, J., Tian, J., Tu, Z., Yuille, A.L.: Robust estimation of nonrigid transformation for point set registration. In: CVPR (2013)Google Scholar
  13. 13.
    Ma, J., Zhao, J., Tian, J., Bai, X., Tu, Z.: Regularized vector field learning with sparse approximation for mismatch removal. Pattern Recogn. 46(12), 3519–3532 (2013)CrossRefzbMATHGoogle Scholar
  14. 14.
    Agudo, A., Agapito, L., Calvo, B., Montiel, J.: Good vibrations: a modal analysis approach for sequential non-rigid structure from motion. In: CVPR, pp. 1558–1565 (2014)Google Scholar
  15. 15.
    Rosen, J.: Symmetry Discovered: Concepts and Applications in Nature and Science. Dover Publications, Mineola (2011)zbMATHGoogle Scholar
  16. 16.
    Hong, W., Yang, A.Y., Huang, K., Ma, Y.: On symmetry and multiple-view geometry: structure, pose, and calibration from a single image. Int. J. Comput. Vis. 60, 241–265 (2004)CrossRefGoogle Scholar
  17. 17.
    Gordon, G.G.: Shape from symmetry. In: Proceedings of SPIE (1990)Google Scholar
  18. 18.
    Kontsevich, L.L.: Pairwise comparison technique: a simple solution for depth reconstruction. JOSA A 10(6), 1129–1135 (1993)CrossRefGoogle Scholar
  19. 19.
    Vetter, T., Poggio, T.: Symmetric 3D objects are an easy case for 2D object recognition. Spat. Vis. 8, 443–453 (1994)CrossRefGoogle Scholar
  20. 20.
    Mukherjee, D.P., Zisserman, A., Brady, M.: Shape from symmetry: detecting and exploiting symmetry in affine images. Philos. Trans. Phys. Sci. Eng. 351, 77–106 (1995)CrossRefzbMATHGoogle Scholar
  21. 21.
    Thrun, S., Wegbreit, B.: Shape from symmetry. In: ICCV (2005)Google Scholar
  22. 22.
    Li, Y., Pizlo, Z.: Reconstruction of shapes of 3D symmetric objects by using planarity and compactness constraints. In: Proceedings of SPIE-IS&T Electronic Imaging (2007)Google Scholar
  23. 23.
    Vicente, S., Carreira, J., Agapito, L., Batista, J.: Reconstructing PASCAL VOC. In: CVPR (2014)Google Scholar
  24. 24.
    Kar, A., Tulsiani, S., Carreira, J., Malik, J.: Category-specific object reconstruction from a single image. In: CVPR (2015)Google Scholar
  25. 25.
    Akhter, I., Sheikh, Y., Khan, S.: In defense of orthonormality constraints for nonrigid structure from motion. In: CVPR (2009)Google Scholar
  26. 26.
    Gao, Y., Yuille, A.L.: Exploiting symmetry and/or Manhattan properties for 3D object structure estimation from single and multiple images (2016). arXiv preprint arXiv:1607.07129
  27. 27.
    Xiang, Y., Mottaghi, R., Savarese, S.: Beyond PASCAL: a benchmark for 3D object detection in the wild. In: WACV (2014)Google Scholar
  28. 28.
    Grossmann, E., Santos-Victor, J.: Maximum likehood 3D reconstruction from one or more images under geometric constraints. In: BMVC (2002)Google Scholar
  29. 29.
    Grossmann, E., Santos-Victor, J.: Least-squares 3D reconstruction from one or more views and geometric clues. Comput. Vis. Image Underst. 99(2), 151–174 (2005)CrossRefGoogle Scholar
  30. 30.
    Grossmann, E., Ortin, D., Santos-Victor, J.: Single and multi-view reconstruction of structured scenes. In: ACCV (2002)Google Scholar
  31. 31.
    Kontsevich, L.L., Kontsevich, M.L., Shen, A.K.: Two algorithms for reconstructing shapes. Optoelectron. Instrum. Data Process. 5, 76–81 (1987)Google Scholar
  32. 32.
    Bregler, C., Hertzmann, A., Biermann, H.: Recovering non-rigid 3D shape from image streams. In: CVPR (2000)Google Scholar
  33. 33.
    Hong, J.H., Fitzgibbon, A.: Secrets of matrix factorization: approximations, numerics, manifold optimization and random restarts. In: ICCV (2015)Google Scholar
  34. 34.
    Olsen, S.I., Bartoli, A.: Implicit non-rigid structure-from-motion with priors. J. Math. Imaging Vis. 31(2–3), 233–244 (2008)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Akhter, I., Sheikh, Y., Khan, S., Kanade, T.: Nonrigid structure from motion in trajectory space. In: NIPS (2008)Google Scholar
  36. 36.
    Ceylan, D., Mitra, N.J., Zheng, Y., Pauly, M.: Coupled structure-from-motion and 3D symmetry detection for urban facades. ACM Trans. Graph. 33, 2 (2014)CrossRefzbMATHGoogle Scholar
  37. 37.
    Morris, D.D., Kanatani, K., Kanade, T.: Gauge fixing for accurate 3D estimation. In: CVPR (2001)Google Scholar
  38. 38.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  39. 39.
    Bourdev, L., Maji, S., Brox, T., Malik, J.: Detecting people using mutually consistent poselet activations. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part VI. LNCS, vol. 6316, pp. 168–181. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  40. 40.
    Schönemann, P.H.: A generalized solution of the orthogonal procrustes problem. Psychometrika 31, 1–10 (1966)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Chen, X., Yuille, A.L.: Articulated pose estimation by a graphical model with image dependent pairwise relations. In: NIPS, pp. 1736–1744 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.City University of Hong KongKowloon TongHong Kong
  2. 2.UCLALos AngelesUSA
  3. 3.John Hopkins UniversityBaltimoreUSA

Personalised recommendations