Skip to main content
SpringerLink
Log in

From Images to Depths and Back

  • Chapter
Dense Image Correspondences for Computer Vision

  • 2045 Accesses

Abstract

This chapter describes what is possibly the earliest use of dense correspondence estimation for transferring semantic information between images of different scenes. The method described in this chapter was designed for non-parametric, “example-based” depth estimation of objects appearing in single photos. It consults a database of example 3D geometries and associated appearances, searching for those which look similar to the object in the photo. This is performed at the pixel level, in similar spirit to the more recent methods described in the following chapters. Those newer methods, however, use robust, generic dense correspondence estimation engines. By contrast, the method described here uses a hard-EM optimization to optimize a well-defined target function over the similarity of appearance/depth pairs in the database to appearance/estimated-depth pairs of a query photo. Results are presented demonstrating how depths associated with diverse reference objects may be assigned to different objects appearing in query photos. Going beyond visible shape, we show that the method can be employed for the surprising task of estimating shapes of occluded objects’ backsides. This, so long as the reference database contains examples of mappings from appearances to backside shapes. Finally, we show how the duality of appearance and shape may be exploited in order to “paint colors” on query shapes (“colorize” them) by simply reversing the matching from appearances to depths.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Assa, J., Wolf, L.: Diorama construction from a single image. In: Eurographics, pp. 599–608 (2007)

    Google Scholar 

  2. Atick, J., Griffin, P., Redlich, A.: Statistical approach to shape from shading: reconstruction of three-dimensional face surfaces from single two-dimensional images. Neural Comput. 8(6), 1321–1340 (1996)

    Article  Google Scholar 

  3. Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: Proceedings of Conference on Computer Vision Pattern Recognition, pp. 334–341. IEEE, Providence (2012)

    Google Scholar 

  4. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of ACM SIGGRAPH Conference on Computer Graphics, pp. 187–194. ACM/Addison-Wesley, New York (1999)

    Google Scholar 

  5. Burt, P., Adelson, E.: The laplacian pyramid as a compact image code. IEEE Trans. Commun. 30, 532–540 (1983)

    Article  Google Scholar 

  6. Chen, Y., Kim, T.K., Cipolla, R.: Inferring 3D shapes and deformations from single views. In: European Conference on Computer Vision, pp. 300–313. Springer, Heidelberg (2010)

    Google Scholar 

  7. Chen, X., Guo, Y., Zhou, B., Zhao, Q.: Deformable model for estimating clothed and naked human shapes from a single image. Vis. Comput. 29(11), 1187–1196 (2013)

    Article  Google Scholar 

  8. Cipolla, R., Fletcher, G., Giblin, P.: Surface geometry from cusps of apparent contours. In: Proceedings of International Conference on Computer Vision, pp. 858–863 (1995)

    Google Scholar 

  9. Cole, F., Isola, P., Freeman, W.T., Durand, F., Adelson, E.H.: Shapecollage: occlusion-aware, example-based shape interpretation. In: European Conference on Computer Vision, pp. 665–678. Springer, Heidelberg (2012)

    Google Scholar 

  10. Criminisi, A., Reid, I., Zisserman, A.: Single view metrology. Int. J. Comput. Vis. 40(2), 123–148 (2000)

    Article  MATH  Google Scholar 

  11. Cyberware: http://www.cyberware.com/

  12. Delage, E., Lee, H., Ng, A.: Automatic single-image 3D reconstructions of indoor manhattan world scenes. In: Proceedings of the International Symposium of Robotics Research (ISRR), pp. 305–321 (2005)

    Google Scholar 

  13. Dovgard, R., Basri, R.: Statistical symmetric shape from shading for 3D structure recovery of faces. In: European Conference on Computer Vision, vol. 2, pp. 99–113. Springer, Berlin/Heidelberg (2004)

    Google Scholar 

  14. Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. arXiv preprint (2014) [arXiv:1411.4734]

    Google Scholar 

  15. Gupta, A., Satkin, S., Efros, A.A., Hebert, M.: From 3D scene geometry to human workspace. In: Proceedings of Conference on Computer Vision Pattern Recognition, pp. 1961–1968. IEEE, Providence (2011)

    Google Scholar 

  16. Han, J., Zhou, K., Wei, L., Gong, M., Bao, H., Zhang, X., Guo, B.: Fast example-based surface texture synthesis via discrete optimization. Vis. Comput. 22, 918–925 (2006)

    Article  Google Scholar 

  17. Hassner, T.: Viewing real-world faces in 3D. In: Proceedings of International Conference on Computer Vision, pp. 3607–3614. IEEE, Sydney (2013). Available: www.openu.ac.il/home/hassner/projects/poses

  18. Hassner, T., Basri, R.: Automatic depth-map colorization. In: Eurographics (short), pp. 73–76 (2006)

    Google Scholar 

  19. Hassner, T., Basri, R.: Example based 3D reconstruction from single 2D images. In: Beyond Patches Workshop at CVPR, p. 15 (2006)

    Google Scholar 

  20. Hassner, T., Basri, R.: Single view depth estimation from examples. arXiv preprint (2013) [arXiv:1304.3915]

    Google Scholar 

  21. Hassner, T., Zelnik-Manor, L., Leifman, G., Basri, R.: Minimal-cut model composition. In: International Conference on Shape Modeling and Applications (SMI’ 05), pp. 72–81 (2005)

    Google Scholar 

  22. Hassner, T., Mayzels, V., Zelnik-Manor, L.: On sifts and their scales. In: Proceedings of Conference on Computer Vision Pattern Recognition, pp. 1522–1528. IEEE, Providence (2012)

    Google Scholar 

  23. Hassner, T., Assif, L., Wolf, L.: When standard RANSAC is not enough: cross-media visual matching with hypothesis relevancy. Mach. Vis. Appl. 25(4), 971–983 (2014)

    Article  Google Scholar 

  24. Hassner, T., Harel, S., Paz, E., Enbar, R.: Effective face frontalization in unconstrained images. In: Proceedings of Conference on Computer Vision Pattern Recognition (2015)

    Book  Google Scholar 

  25. Hoiem, D., Efros, A., Hebert, M.: Automatic photo pop-up. ACM Trans. Graph. 24(3), 577–584 (2005)

    Article  Google Scholar 

  26. Hoiem, D., Efros, A., Hebert, M.: Geometric context from a single image. In: Proceedings of International Conference on Computer Vision, pp. 654–661. IEEE Computer Society, Beijing (2005)

    Google Scholar 

  27. Horn, B.: Obtaining Shape from Shading Information. The Psychology of Computer Vision. McGraw-Hill, New York (1975)

    Google Scholar 

  28. Karsch, K., Liu, C., Kang, S.B.: Depth extraction from video using non-parametric sampling. In: European Conference on Computer Vision, pp. 775–788. Springer, Heidelberg (2012)

    Google Scholar 

  29. Kearns, M., Mansour, Y., Ng, A.: An information-theoretic analysis of hard and soft assignment methods for clustering. In: Proceedings of the NATO Advanced Study Institute on Learning in Graphical Models, pp. 495–520. Kluwer Academic, Norwell (1998)

    Google Scholar 

  30. Kemelmacher, I., Basri, R.: Molding face shapes by example. In: European Conference on Computer Vision, p. 2006 (277–288)

    Google Scholar 

  31. Kemelmacher-Shlizerman, I., Seitz, S.: Face reconstruction in the wild. In: Proceedings of International Conference on Computer Vision, pp. 1746–1753. IEEE, Washington (2011)

    Google Scholar 

  32. Kraevoy, V., Sheffer, A.: Cross-parameterization and compatible remeshing of 3D models. ACM Trans. Graph. 23(3), 861–869 (2004)

    Article  Google Scholar 

  33. Leifman, G., Tal, A.: Mesh colorization. Comput. Graph. Forum 31(2), 421–430 (2012)

    Article  Google Scholar 

  34. Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.: Sift flow: dense correspondence across different scenes. In: European Conference on Computer Vision, pp. 28–42 (2008). Available: people.csail.mit.edu/celiu/ECCV2008/

  35. Liu, C., Yuen, J., Torralba, A.: Sift flow: dense correspondence across scenes and its applications. Trans. Pattern Anal. Mach. Intell. 33(5), 978–994 (2011). Available: people.csail.mit.edu/celiu/SIFTflow/

    Article  Google Scholar 

  36. Liu, F., Shen, C., Lin, G., Reid, I.: Learning depth from single monocular images using deep convolutional neural fields. arXiv preprint (2015) [arXiv:1502.07411]

    Google Scholar 

  37. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  38. Oswald, M.R., Toppe, E., Cremers, D.: Fast and globally optimal single view reconstruction of curved objects. In: Proceedings of Conference on Computer Vision Pattern Recognition, pp. 534–541. IEEE, Washington (2012)

    Google Scholar 

  39. Panagopoulos, A., Hadap, S., Samaras, D.: Reconstructing shape from dictionaries of shading primitives. In: Asian Conference on Computer Vision, pp. 80–94. Springer, Heidelberg (2013)

    Google Scholar 

  40. Praun, E., Sweldens, W., Schröder, P.: Consistent mesh parameterizations. In: Proceedings of ACM SIGGRAPH Conference on Computer Graphics, pp. 179–184. ACM, New York (2001)

    Google Scholar 

  41. Qiu, W., Wang, X., Bai, X., Yuille, A., Tu, Z.: Scale-space sift flow. In: Proceedings of Winter Conference on Applications of Computer Vision, pp. 1112–1119. IEEE, Steamboat Springs (2014)

    Google Scholar 

  42. Romdhani, S., Vetter, T.: Efficient, robust and accurate fitting of a 3D morphable model. In: Proceedings of International Conference on Computer Vision, p. 59 (2003)

    Google Scholar 

  43. Saxena, A., Sun, M., Ng, A.: Make3d: learning 3-D scene structure from a single still image. Trans. Pattern Anal. Mach. Intell. 31(5), 824–840 (2009)

    Article  Google Scholar 

  44. Schwing, A.G., Urtasun, R.: Efficient exact inference for 3D indoor scene understanding. In: European Conference on Computer Vision, pp. 299–313. Springer, Heidelberg (2012)

    Google Scholar 

  45. Tau, M., Hassner, T.: Dense correspondences across scenes and scales. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1 (2015)

    Google Scholar 

  46. Thomas, A., Ferrari, V., Leibe, B., Tuytelaars, T., Gool, L.: Shape-from-recognition: recognition enables meta-data transfer. Comput. Vis. Image Underst. 113(12), 1222–1234 (2009)

    Article  Google Scholar 

  47. Toucan virtual museum. Available: http://www.toucan.co.jp/indexE.html

  48. Turk, G.: Texture synthesis on surfaces. In: Proceedings of ACM SIGGRAPH Conference on Computer Graphics, pp. 347–354. ACM, New York (2001)

    Google Scholar 

  49. USF: DARPA Human-ID 3D Face Database: Courtesy of Prof. Sudeep Sarkar. University of South Florida, Tampa. http://marthon.csee.usf.edu/HumanID/

  50. Vedaldi, A., Fulkerson, B.: VLFeat: an open and portable library of computer vision algorithms. In: Proceedings of International Conference on Multimedia, pp. 1469–1472 (2010). Available: www.vlfeat.org/

  51. Wei, L.Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of ACM SIGGRAPH Conference on Computer Graphics, pp. 479–488. ACM/Addison-Wesley, New York (2000)

    Google Scholar 

  52. Wei, L.Y., Levoy, M.: Texture synthesis over arbitrary manifold surfaces. In: Proceedings of ACM SIGGRAPH Conference on Computer Graphics, pp. 355–360. ACM, New York (2001)

    Google Scholar 

  53. Witkin, A.: Recovering surface shape and orientation from texture. Artif. Intell. 17(1–3), 17–45 (1981)

    Article  Google Scholar 

  54. Xiong, Y., Chakrabarti, A., Basri, R., Gortler, S.J., Jacobs, D.W., Zickler, T.: From shading to local shape. arXiv preprint (2014) [arXiv:1310.2916]

    Google Scholar 

  55. Ying, L., Hertzmann, A., Biermann, H., Zorin, D.: Texture and shape synthesis on surfaces. In: Proceedings of the 12th Eurographics Workshop on Rendering Techniques, pp. 301–312. Springer, Heidelberg (2001)

    Google Scholar 

  56. Zhou, K., Wang, X., Tong, Y., Desbrun, M., Guo, B., Shum, H.Y.: Texturemontage: seamless texturing of arbitrary surfaces from multiple images. ACM Trans. Graph. 24(3), 1148–1155 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, The Open University of Israel, Raanana, Israel

    Tal Hassner

  2. The Weizmann Institute, Rehovot, Israel

    Ronen Basri

Authors
  1. Tal Hassner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ronen Basri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tal Hassner .

Editor information

Editors and Affiliations

  1. The Open University of Israel, Raanana, Israel

    Tal Hassner

  2. Google Research, Cambridge, Massachusetts, USA

    Ce Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Hassner, T., Basri, R. (2016). From Images to Depths and Back. In: Hassner, T., Liu, C. (eds) Dense Image Correspondences for Computer Vision. Springer, Cham. https://doi.org/10.1007/978-3-319-23048-1_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-23048-1_8

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-23047-4

  • Online ISBN: 978-3-319-23048-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation