The Visual Computer

, Volume 34, Issue 5, pp 605–616 | Cite as

PhotoSketch: a photocentric urban 3D modeling system

  • George Wolberg
  • Siavash Zokai
Original Article


Online mapping services from Google, Apple, and Microsoft are exceedingly popular applications for exploring 3D urban cities. Their explosive growth provides impetus for photorealistic 3D modeling of urban scenes. Although classical algorithms such as multiview stereo and laser range scanners are traditional sources for detailed 3D models of existing structures, they generate heavyweight models that are not appropriate for the streaming data that these navigation applications leverage. Instead, lightweight models as produced by interactive image-based tools are better suited for this domain. The contribution of this work is that it merges the benefits of multiview geometry, an intuitive sketching interface, and dynamic texture mapping to produce lightweight photorealistic 3D models of buildings. We present experimental results from urban scenes using our PhotoSketch system.


Image-based modeling Phototextured 3D models Structure and motion Multiview geometry 3D photography Camera calibration 



This work was supported by a grant from the US Department of Energy (DE-NA0002492).


  1. 1.
  2. 2.
  3. 3.
  4. 4.
    Arikan, M., Schwärzler, M., Flöry, S., Wimmer, M., Maierhofer, S.: O-snap: optimization-based snapping for modeling architecture. ACM Trans. Graph. 32(1), 6:1–6:15 (2013)CrossRefzbMATHGoogle Scholar
  5. 5.
    Atkinson, K.B.: Close Range Photogrammetry and Machine Vision. Whittles Publishing, Dunbeath, UK (2003)Google Scholar
  6. 6.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110, 346–359 (2008)CrossRefGoogle Scholar
  7. 7.
    Chen, T., Zhu, Z., Shamir, A., Hu, S.M., Cohen-Or, D.: 3-sweep: extracting editable objects from a single photo. ACM Trans. Graph. 32(6), 195:1–195:10 (2013)Google Scholar
  8. 8.
    Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: a hybrid geometry-and image-based approach. Comput. Graph. Proc. SIGGRAPH ’96 30, 11–20 (1996)CrossRefGoogle Scholar
  9. 9.
    El-Hakim, S., Whiting, E., Gonzo, L.: 3D modeling with reusable and integrated building blocks. In: Conference on Optical 3D Measurement Techniques (2005)Google Scholar
  10. 10.
    Faugeras, O., Laveau, S., Robert, L., Csurka, G., Zeller, C.: 3D reconstruction of urban scenes from sequences of images. In: Gruen, A., Kuebler, O., Agouris, P. (eds.) Automatic Extraction of Man-Made Objects from Aerial and Space Images. Birkhauser, Basel, Switzerland (1995)Google Scholar
  11. 11.
    Faugeras, O., Luong, Q.T., Papadopoulou, T.: The Geometry of Multiple Images. MIT Press, Cambridge, MA (2001)zbMATHGoogle Scholar
  12. 12.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1–8 (2007)Google Scholar
  13. 13.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  14. 14.
    van den Hengel, A., Dick, A.R., Thormählen, T., Ward, B., Torr, P.H.: Videotrace: rapid interactive scene modelling from video. ACM Trans. Graph. Proc. SIGGRAPH ’07 26(3), 86 (2007)CrossRefGoogle Scholar
  15. 15.
    Hou, F., Qin, H., Qi, Y.: Procedure-based component and architecture modeling from a single image. Vis. Comput. 32(2), 151–166 (2016)CrossRefGoogle Scholar
  16. 16.
    Li, M., Nan, L., Liu, S.: Fitting boxes to Manhattan scenes using linear integer programming. Int. J. Digit. Earth 9, 806–817 (2016)CrossRefGoogle Scholar
  17. 17.
    Li, W., Wolberg, G., Zokai, S.: Lightweight 3d modeling of urban buildings from range data. In: 3DIMPVT, pp. 124–131 (2011)Google Scholar
  18. 18.
    Liu, L., Stamos, I., Yu, G., Wolberg, G., Zokai, S.: Multiview geometry for texture mapping 2D images onto 3D range data. IEEE Conference Computer Vision and Pattern Recognition (CVPR) pp. 2293–2300 (2006)Google Scholar
  19. 19.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  20. 20.
    Luhmann, T., Robson, S., Kyle, S., Harley, I.: Close Range Photogrammetry: Principles, Techniques and Applications. Wiley, Hoboken (2006)Google Scholar
  21. 21.
    Ma, Y., Soatto, S., Kosecka, J., Sastry, S.: An Invitation to 3-D Vision: From Images to Geometric Models. Springer, Berlin (2004)CrossRefzbMATHGoogle Scholar
  22. 22.
    Mathias, M., Martinović, A., Weissenberg, J., Van Gool, L.: Procedural 3d building reconstruction using shape grammars and detectors. In: 3DIMPVT, pp. 304–311 (2011)Google Scholar
  23. 23.
    Moulon, P., Monasse, P., Marlet, R., Others: Openmvg. an open multiple view geometry library.
  24. 24.
    Müller, P., Wonka, P., Haegler, S., Ulmer, A., Gool, L.V.: Procedural modeling of buildings. ACM Trans. Graph. Proc. SIGGRAPH ’06 25(3), 614–623 (2006)CrossRefGoogle Scholar
  25. 25.
    Müller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of facades. ACM Trans. Graph. 26(3), 85 (2007)CrossRefGoogle Scholar
  26. 26.
    Musialski, P., Wonka, P., Aliaga, D.G., Wimmer, M., van Gool, L., Purgathofer, W.: A survey of urban reconstruction. Comput. Graph. Forum 32(6), 146–177 (2013)CrossRefGoogle Scholar
  27. 27.
    Nan, L., Jiang, C., Ghanem, B., Wonka, P.: Template assembly for detailed urban reconstruction. Comput. Graph. Forum 34, 217–228 (2015)CrossRefGoogle Scholar
  28. 28.
    Parish, Y.I.H., Müller, P.: Procedural modeling of cities. Comput. Graph. (Proc. SIGGRAPH ’01) pp. 301–308 (2001)Google Scholar
  29. 29.
    Remondino, F., El-Hakim, S.: Image-based 3d modelling : a review. Photogramm. Rec. 21, 269–291 (2006)CrossRefGoogle Scholar
  30. 30.
    Samavati, F., Runions, A.: Interactive 3D content modeling for digital earth. Vis. Comput. 32(10), 1293–1309 (2016)CrossRefGoogle Scholar
  31. 31.
    Sinha, S.N., Steedly, D., Szeliski, R., Agrawala, M., Pollefeys, M.: Interactive 3D architectural modeling from unordered photo collections. In: SIGGRAPH Asia ’08, pp. 159:1–159:10 (2008)Google Scholar
  32. 32.
  33. 33.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. ACM Trans. Graph. Proc. SIGGRAPH ’06 25(3), 835–846 (2006)CrossRefGoogle Scholar
  34. 34.
    Stamos, I., Allen, P.K.: Automatic registration of 2D with 3D imagery in urban environments. In: Proceedings of International Conference On Computer Vision (ICCV) pp. 731–737 (2001)Google Scholar
  35. 35.
    Stamos, I., Allen, P.K.: Geometry and texture recovery of scenes of large scale. Comput. Vis. Image Underst. 88(2), 94–118 (2002)CrossRefzbMATHGoogle Scholar
  36. 36.
    Stamos, I., Liu, L., Chen, C., Wolberg, G., Yu, G., Zokai, S.: Integrating automated range registration with multiview geometry for the photorealistic modeling of large-scale scenes. Int. J. Comput. Vis. 78(2), 237–260 (2007)Google Scholar
  37. 37.
    Toldo, R., Gherardi, R., Farenzena, M., Fusiello, A.: Samantha: Structure-and-motion pipeline on a hierarchical cluster tree.
  38. 38.
    Vanegas, C.A., Aliaga, D.G., Beneš, B.: Building reconstruction using Manhattan-world grammars. Comput. Vis. Pattern Recognit. 0, 358–365 (2010)Google Scholar
  39. 39.
    Verdie, Y., Lafarge, F., Alliez, P.: Lod generation for urban scenes. ACM Trans. Graph. 34(3), 30:1–30:14 (2015)CrossRefzbMATHGoogle Scholar
  40. 40.
    Werner, T., Schaffalitzky, F., Zisserman, A.: Automated architecture reconstruction from close-range photogrammetry. In: Proceedings of CIPA 2001 International SymposiumGoogle Scholar
  41. 41.
    Wu, C.: Visualsfm.
  42. 42.
    Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Schematic surface reconstruction. In: Proceedings of IEEE CVPR pp. 1498–1505 (2012)Google Scholar
  43. 43.
    Wu, F., Yan, D.M., Dong, W., Zhang, X., Wonka, P.: Inverse procedural modeling of facade layouts. ACM Trans. Graph. 33(4), 121:1–121:10 (2014)CrossRefGoogle Scholar
  44. 44.
    Zheng, Y., Chen, X., Cheng, M.M., Zhou, K., Hu, S.M., Mitra, N.J.: Interactive images: cuboid proxies for smart image manipulation. ACM Trans. Graph. 31(4), 99:1–99:11 (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  1. 1.City College of New YorkCUNYNew YorkUSA
  2. 2.Brainstorm Technology LLCNew YorkUSA

Personalised recommendations