Skip to main content
SpringerLink
Log in

Visual Urban Perception with Deep Semantic-Aware Network

  • Conference paper
  • First Online:
MultiMedia Modeling (MMM 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11296))

Included in the following conference series:

Abstract

Visual urban perception has received a lot attention for its importance in many fields. In this paper we transform it into a ranking task by pairwise comparison of images, and use deep neural networks to predict the specific perceptual score of each image. Distinguished from existing researches, we highlight the important role of object semantic information in visual urban perception through the attribute activation maps of images. Base on this concept, we combine the object semantic information with the generic features of images in our method. In addition, we use the visualization techniques to obtain the correlations between objects and visual perception attributes from the well trained neural network, which further proves the correctness of our conjecture. The experimental results on large-scale dataset validate the effectiveness of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Can, G., Benkhedda, Y., Gatica-Perez, D.: Ambiance in social media venues: visual cue interpretation by machines and crowds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2363–2372 (2018)

    Google Scholar 

  2. Chollet, F., et al.: Keras (2015)

    Google Scholar 

  3. Cohen, D.A., Mason, K., Bedimo, A., Scribner, R., Basolo, V., Farley, T.A.: Neighborhood physical conditions and health. Am. J. Public Health 93(3), 467–471 (2003)

    Article  Google Scholar 

  4. Deza, A., Parikh, D.: Understanding image virality. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1818–1826 (2015)

    Google Scholar 

  5. Dhar, S., Ordonez, V., Berg, T.L.: High level describable attributes for predicting aesthetics and interestingness. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1657–1664. IEEE (2011)

    Google Scholar 

  6. Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes Paris look like Paris? ACM Trans. Graph. 31(4), 101:1–101:9 (2012)

    Article  Google Scholar 

  7. Dubey, A., Naik, N., Parikh, D., Raskar, R., Hidalgo, C.A.: Deep learning the city: quantifying urban perception at a global scale. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 196–212. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_12

    Chapter  Google Scholar 

  8. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  9. He, S., Yoshimura, Y., Helfer, J., Hack, G., Ratti, C., Nagakura, T.: Quantifying memories: mapping urban perception. arXiv preprint arXiv:1806.04054 (2018)

  10. Isola, P., Xiao, J., Torralba, A., Oliva, A.: What makes an image memorable? (2011)

    Google Scholar 

  11. Joshi, D., et al.: Aesthetics and emotions in images. IEEE Signal Process. Mag. 28(5), 94–115 (2011)

    Article  Google Scholar 

  12. Kao, Y., He, R., Huang, K.: Deep aesthetic quality assessment with semantic information. IEEE Trans. Image Process. 26(3), 1482–1495 (2017)

    Article  MathSciNet  Google Scholar 

  13. Law, S., Paige, B., Russell, C.: Take a look around: using street view and satellite images to estimate house prices. arXiv preprint arXiv:1807.07155 (2018)

  14. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)

  15. Liu, X., Chen, Q., Zhu, L., Xu, Y., Lin, L.: Place-centric visual urban perception with deep multi-instance regression. In: Proceedings of the 2017 ACM on Multimedia Conference, pp. 19–27. ACM (2017)

    Google Scholar 

  16. Milam, A., Furr-Holden, C., Leaf, P.: Perceived school and neighborhood safety, neighborhood violence and academic achievement in urban school children. Urban Rev. 42(5), 458–467 (2010)

    Article  Google Scholar 

  17. Naik, N., Philipoom, J., Raskar, R., Hidalgo, C.: Street score-predicting the perceived safety of one million streetscapes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 779–785 (2014)

    Google Scholar 

  18. Naik, N., Raskar, R., Hidalgo, C.A.: Cities are physical too: using computer vision to measure the quality and impact of urban appearance. Am. Econ. Rev. 106(5), 128–132 (2016)

    Article  Google Scholar 

  19. Nasar, J.L.: The evaluative image of the city. J. Am. Plan. Assoc. 56(1), 41–53 (1990)

    Article  Google Scholar 

  20. Ordonez, V., Berg, T.L.: Learning high-level judgments of urban perception. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 494–510. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_32

    Chapter  Google Scholar 

  21. Piro, F.N., Nœss, Ø., Claussen, B.: Physical activity among elderly people in a city population: the influence of neighbourhood level violence and self perceived safety. J. Epidemiol. Community Health 60(7), 626–632 (2006)

    Article  Google Scholar 

  22. Porzi, L., Rota Bulò, S., Lepri, B., Ricci, E.: Predicting and understanding urban perception with convolutional neural networks. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 139–148. ACM (2015)

    Google Scholar 

  23. Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  24. Salesses, P., Schechtner, K., Hidalgo, C.A.: The collaborative image of the city: mapping the inequality of urban perception. PloS one 8(7), e68400 (2013)

    Article  Google Scholar 

  25. Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034 (2013)

  26. Wilson, J.Q.: Broken windows: the police and neighborhood safety James Q. Wilson and George L. Kelling. Criminological perspectives: essential readings 400 (2003)

    Google Scholar 

  27. Wu, Z., Fu, Y., Jiang, Y.G., Sigal, L.: Harnessing object and scene semantics for large-scale video understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3112–3121 (2016)

    Google Scholar 

  28. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)

    Google Scholar 

  29. Zhou, B., Liu, L., Oliva, A., Torralba, A.: Recognizing city identity via attribute analysis of geo-tagged images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 519–534. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10578-9_34

    Chapter  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (61573219, 61701281, 61876098), Shandong Provincial Natural Science Foundation (ZR2017QF009), and the Fostering Project of Dominant Discipline and Talent Team of Shandong Province Higher Education Institutions.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong University, Jinan, China

    Yongchao Xu & Cheng Shi

  2. School of Software, Shandong University, Jinan, China

    Qizheng Yang & Yilong Yin

  3. School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China

    Chaoran Cui & Guangle Song

  4. Shandong Computer Science Center (National Supercomputer Center in Jinan), Shandong Provincial Key Laboratory of Computer Networks, Qilu University of Technology (Shandong Academy of Sciences), Jinan, China

    Xiaohui Han

Authors
  1. Yongchao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qizheng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chaoran Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cheng Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guangle Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaohui Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yilong Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Chaoran Cui or Yilong Yin .

Editor information

Editors and Affiliations

  1. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Ioannis Kompatsiaris

  2. EURECOM, Sophia Antipolis, France

    Benoit Huet

  3. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Vasileios Mezaris

  4. Dublin City University, Dublin, Ireland

    Cathal Gurrin

  5. National Chiao Tung University, Hsinchu, Taiwan

    Wen-Huang Cheng

  6. Information Technologies Institute, Centre for Research and Technology Hellas, Thessaloniki, Greece

    Stefanos Vrochidis

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Y. et al. (2019). Visual Urban Perception with Deep Semantic-Aware Network. In: Kompatsiaris, I., Huet, B., Mezaris, V., Gurrin, C., Cheng, WH., Vrochidis, S. (eds) MultiMedia Modeling. MMM 2019. Lecture Notes in Computer Science(), vol 11296. Springer, Cham. https://doi.org/10.1007/978-3-030-05716-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-05716-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-05715-2

  • Online ISBN: 978-3-030-05716-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation