Visual Urban Perception with Deep Semantic-Aware Network

  • Yongchao Xu
  • Qizheng Yang
  • Chaoran CuiEmail author
  • Cheng Shi
  • Guangle Song
  • Xiaohui Han
  • Yilong YinEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11296)


Visual urban perception has received a lot attention for its importance in many fields. In this paper we transform it into a ranking task by pairwise comparison of images, and use deep neural networks to predict the specific perceptual score of each image. Distinguished from existing researches, we highlight the important role of object semantic information in visual urban perception through the attribute activation maps of images. Base on this concept, we combine the object semantic information with the generic features of images in our method. In addition, we use the visualization techniques to obtain the correlations between objects and visual perception attributes from the well trained neural network, which further proves the correctness of our conjecture. The experimental results on large-scale dataset validate the effectiveness of our method.


Visual urban perception Object semantic information Deep neural network 



This work is supported by the National Natural Science Foundation of China (61573219, 61701281, 61876098), Shandong Provincial Natural Science Foundation (ZR2017QF009), and the Fostering Project of Dominant Discipline and Talent Team of Shandong Province Higher Education Institutions.


  1. 1.
    Can, G., Benkhedda, Y., Gatica-Perez, D.: Ambiance in social media venues: visual cue interpretation by machines and crowds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2363–2372 (2018)Google Scholar
  2. 2.
    Chollet, F., et al.: Keras (2015)Google Scholar
  3. 3.
    Cohen, D.A., Mason, K., Bedimo, A., Scribner, R., Basolo, V., Farley, T.A.: Neighborhood physical conditions and health. Am. J. Public Health 93(3), 467–471 (2003)CrossRefGoogle Scholar
  4. 4.
    Deza, A., Parikh, D.: Understanding image virality. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1818–1826 (2015)Google Scholar
  5. 5.
    Dhar, S., Ordonez, V., Berg, T.L.: High level describable attributes for predicting aesthetics and interestingness. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1657–1664. IEEE (2011)Google Scholar
  6. 6.
    Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes Paris look like Paris? ACM Trans. Graph. 31(4), 101:1–101:9 (2012)CrossRefGoogle Scholar
  7. 7.
    Dubey, A., Naik, N., Parikh, D., Raskar, R., Hidalgo, C.A.: Deep learning the city: quantifying urban perception at a global scale. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 196–212. Springer, Cham (2016). Scholar
  8. 8.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  9. 9.
    He, S., Yoshimura, Y., Helfer, J., Hack, G., Ratti, C., Nagakura, T.: Quantifying memories: mapping urban perception. arXiv preprint arXiv:1806.04054 (2018)
  10. 10.
    Isola, P., Xiao, J., Torralba, A., Oliva, A.: What makes an image memorable? (2011)Google Scholar
  11. 11.
    Joshi, D., et al.: Aesthetics and emotions in images. IEEE Signal Process. Mag. 28(5), 94–115 (2011)CrossRefGoogle Scholar
  12. 12.
    Kao, Y., He, R., Huang, K.: Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Process. 26(3), 1482–1495 (2017)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Law, S., Paige, B., Russell, C.: Take a look around: using street view and satellite images to estimate house prices. arXiv preprint arXiv:1807.07155 (2018)
  14. 14.
    Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
  15. 15.
    Liu, X., Chen, Q., Zhu, L., Xu, Y., Lin, L.: Place-centric visual urban perception with deep multi-instance regression. In: Proceedings of the 2017 ACM on Multimedia Conference, pp. 19–27. ACM (2017)Google Scholar
  16. 16.
    Milam, A., Furr-Holden, C., Leaf, P.: Perceived school and neighborhood safety, neighborhood violence and academic achievement in urban school children. Urban Rev. 42(5), 458–467 (2010)CrossRefGoogle Scholar
  17. 17.
    Naik, N., Philipoom, J., Raskar, R., Hidalgo, C.: Street score-predicting the perceived safety of one million streetscapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 779–785 (2014)Google Scholar
  18. 18.
    Naik, N., Raskar, R., Hidalgo, C.A.: Cities are physical too: using computer vision to measure the quality and impact of urban appearance. Am. Econ. Rev. 106(5), 128–132 (2016)CrossRefGoogle Scholar
  19. 19.
    Nasar, J.L.: The evaluative image of the city. J. Am. Plan. Assoc. 56(1), 41–53 (1990)CrossRefGoogle Scholar
  20. 20.
    Ordonez, V., Berg, T.L.: Learning high-level judgments of urban perception. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 494–510. Springer, Cham (2014). Scholar
  21. 21.
    Piro, F.N., Nœss, Ø., Claussen, B.: Physical activity among elderly people in a city population: the influence of neighbourhood level violence and self perceived safety. J. Epidemiol. Community Health 60(7), 626–632 (2006)CrossRefGoogle Scholar
  22. 22.
    Porzi, L., Rota Bulò, S., Lepri, B., Ricci, E.: Predicting and understanding urban perception with convolutional neural networks. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 139–148. ACM (2015)Google Scholar
  23. 23.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Salesses, P., Schechtner, K., Hidalgo, C.A.: The collaborative image of the city: mapping the inequality of urban perception. PloS one 8(7), e68400 (2013)CrossRefGoogle Scholar
  25. 25.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)
  26. 26.
    Wilson, J.Q.: Broken windows: the police and neighborhood safety James Q. Wilson and George L. Kelling. Criminological perspectives: essential readings 400 (2003)Google Scholar
  27. 27.
    Wu, Z., Fu, Y., Jiang, Y.G., Sigal, L.: Harnessing object and scene semantics for large-scale video understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3112–3121 (2016)Google Scholar
  28. 28.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)Google Scholar
  29. 29.
    Zhou, B., Liu, L., Oliva, A., Torralba, A.: Recognizing city identity via attribute analysis of geo-tagged images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 519–534. Springer, Cham (2014). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Computer Science and TechnologyShandong UniversityJinanChina
  2. 2.School of SoftwareShandong UniversityJinanChina
  3. 3.School of Computer Science and TechnologyShandong University of Finance and EconomicsJinanChina
  4. 4.Shandong Computer Science Center (National Supercomputer Center in Jinan), Shandong Provincial Key Laboratory of Computer NetworksQilu University of Technology (Shandong Academy of Sciences)JinanChina

Personalised recommendations