Abstract
Computer vision research involving street and road detection methods usually focuses on driving assistance and autonomous vehicle systems. In this context, street segmentation occurs in real-time, based on images centered on the street. This work, on the other hand, uses street segmentation for urban planning research to classify pavement types of a city or region, which is particularly important for developing countries. For this application, it is needed a dataset with images from various locations for each street. These images are not necessarily centered on the street and include challenges that are not common in street segmentation datasets, such as mixed pavement types and the presence of faults and holes on the street. We implemented a multi-class version of a state-of-the-art segmentation algorithm and adapted it to perform street pavement classification, handling navigation along streets and angle variation to increase the accuracy of the classification. A data augmentation approach is also proposed to use preliminary results from the test instances as new ground truth and increase the amount of training data. A dataset with more than 300,000 images from 773 streets from a Brazilian city was built. Our approach achieved a precision of 0.93, showing the feasibility of the proposed application.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
The dataset (https://beautifulcities.herokuapp.com/cidade/1) is available online, as it may be useful for future works related to autonomous vehicles and urban planning in developing countries. Our dataset contains images with specific characteristics commonly present on urban scenes in developing countries, which distinguishes this from other urban scene databases. An application example built with real images is also available in the same link, where one can see our classification results and also navigate through the streets used in our experiments using Google Street View.
Code availability
The source used by or produced in our research is also available online (https://github.com/rgalvaomesquita/KittiSeg). The first author can be contacted in case of any doubts about the source code.
References
Bileschi SM (2006) StreetScenes : towards scene understanding in still images. Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science
Caltagirone L, Bellone M, Svensson L, Wahde M (2019) LIDAR-camera fusion for road detection using fully convolutional neural networks. Robot Auton Syst 111:125–131. https://doi.org/10.1016/j.robot.2018.11.002
Chen Z, Chen Z (2017) RBNet: a deep neural network for unified road and road boundary detection. Int Conf Neural Inf Process 677–687
Chen Z, Zhang J, Tao D (2019) Progressive LiDAR adaptation for road detection. IEEE/CAA J Automatica Sinica 6(3):693–702. https://doi.org/10.1109/JAS.2019.1911459
Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S and Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Diallo S, Schults F, Wildman W (2020) Minding morality: ethical artificial societies for public policy modeling. AI Soc 36:49–57. https://doi.org/10.1007/s00146-020-01028-5
Dubey A, Naik N, Parikh D, Raskar R, Hidalgo CA (2016) Deep learning the city: quantifying urban perception at a global scale. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer vision – ECCV 2016. Springer International Publishing, pp 196–212
Fan R, Wang H, Cai P, Liu M (2020) SNE-RoadSeg: incorporating surface normal information into semantic segmentation for accurate freespace detection. ECCV. https://doi.org/10.1007/978-3-030-58577-8_21
Fritsch J, Kuehnl T and Geiger A (2013) A new performance measure and evaluation benchmark for road detection algorithms. In: International Conference on Intelligent Transportation Systems (ITSC)
Garrick D (2021) San Diego reverses decades-old policies that kept dirt streets in low-income areas unpaved. https://www.sandiegouniontribune.com/news/politics/story/2021-02-09/san-diego-reverses-decades-old-policies-that-have-kept-dirt-streets-in-low-income-areas-unpaved. Accessed 23 Jan 2021
Gehrig S, Schneider N, Stalder R, Franke U (2017) Stereo vision during adverse weather—using priors to increase robustness in real-time stereo vision. Image vis Comput 68:28–39. https://doi.org/10.1016/j.imavis.2017.07.008
Geiger A (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 3354–3361. http://dl.acm.org/citation.cfm?id=2354409.2354978
Ghilardi MC, Junior JJ, Manssour I (2018) Crosswalk localization from low resolution satellite images to assist visually impaired people. IEEE Comput Graphics Appl 38(1):30–46. https://doi.org/10.1109/MCG.2016.50
Google LLC. Google products page. https://about.google/intl/ALL_us/products/. Accessed 06 Apr 2022
Gruyer D, Belaroussi R, Revilloud M (2016) Accurate lateral positioning from map data and road marking detection. Expert Syst Appl 43:1–8. https://doi.org/10.1016/j.eswa.2015.08.015
Han X, Lu J, Zhao C, You S, Li H (2018) Semisupervised and weakly supervised road detection based on generative adversarial networks. IEEE Signal Process Lett 25(4):551–555
Handbook by the U.S. Environmental Protection Agency (2006) https://www.epa.gov/sites/production/files/2015-10/documents/environmentallysensitivemaintenance_dirtgravelroads.pdf. Accessed 23 Jan 2021
Harbaš I, Prentašić P, Subašić M (2018) Detection of roadside vegetation using Fully Convolutional Networks. Image vis Comput 74:1–9. https://doi.org/10.1016/j.imavis.2018.03.008
Harvey C, Aultman-Hall L, Hurley SE, Troy A (2015) Effects of skeletal streetscape design on perceived safety. Landscape Urban Plann 142:18–28. https://doi.org/10.1016/j.landurbplan.2015.05.007
He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR, abs/1512.0. http://arxiv.org/abs/1512.03385
Ilarri S, Stojanovic D, Ray C (2015) Semantic management of moving objects: a vision towards smart mobility. Expert Syst Appl 42(3):1418–1435. https://doi.org/10.1016/j.eswa.2014.08.057
Kauer T, Joglekar S, Redi M, Aiello LM and Quercia D (2018) Mapping and visualizing deep-learning urban beautification. In: IEEE Computer Graphics and Applications 38(5):70–83. http://dblp.uni-trier.de/db/journals/cga/cga38.html#KauerJRAQ18
Kaur B, Bhattacharya J (2019) A convolutional feature map-based deep network targeted towards traffic detection and classification. Expert Syst Appl 124:119–129. https://doi.org/10.1016/j.eswa.2019.01.014
Keizer K, Lindenberg S, Steg L (2008) The spreading of disorder. Science (new York N.y.) 322(5098):1681–1685
Kim T, Muller J-P (1996) Automated urban area building extraction from high resolution stereo imagery. Image vis Comput 14(2):115–130. https://doi.org/10.1016/0262-8856(96)85600-9
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. IEEE Conf Comput vis Pattern Recogn (CVPR) 2015:3431–3440. https://doi.org/10.1109/CVPR.2015.7298965
Lundgren B (2020) Safety requirements vs. crashing ethically: what matters most for policies on autonomous vehicles. AI Soc 36:405–411. https://doi.org/10.1007/s00146-020-00964-6
Lynch K (1960) The image of the city. MIT Press
Maddern W, Pascoe G, Linegar C, Newman P (2017) 1 year, 1000 km: the Oxford RobotCar dataset. Int J Robot Res 36(1):3–15. https://doi.org/10.1177/0278364916679498
Muñoz-Bulnes J, Fernandez C, Parra I, Fernández-Llorca D, Sotelo MA (2017) Deep fully convolutional networks with random data augmentation for enhanced generalization in road detection. In: Workshop on deep learning for autonomous driving on IEEE 20th international conference on intelligent transportation systems, pp 366–371. https://doi.org/10.1109/ITSC.2017.8317901
Naik N, Philipoom J and Raskar R (2014) Streetscore—predicting the perceived safety of one million streetscapes. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 779–785
Ng W-Y, Chau C-K, Powell G, Leung T-M (2015) Preferences for street configuration and street tree planting in urban Hong Kong. Urban Forest Urban Green 14(1):30–38. https://doi.org/10.1016/j.ufug.2014.11.002
O’Rourke J, Aggarwal A, Maddila S, Baldwin M (1986) An optimal algorithm for finding minimal enclosing triangles. J Algorithms 7:258–269
Park M, Hagishima A, Tanimoto J, Narita K (2012) Effect of urban vegetation on outdoor thermal environment: field measurement at a scale model site. Build Environ 56:38–46. https://doi.org/10.1016/j.buildenv.2012.02.015
Rother C, Kolmogorov V and Blake A (2004) “GrabCut”: interactive foreground extraction using iterated graph cuts. ACM SIGGRAPH 2004 Papers, 309–314. https://doi.org/10.1145/1186562.1015720
Sakurada K, Okatani T, Kitani KM (2016) Hybrid macro–micro visual analysis for city-scale state estimation. Comput vis Image Understand 146:86–98. https://doi.org/10.1016/j.cviu.2016.02.017
Sakurada K, Tetsuka D, Okatani T (2017) Temporal city modeling using street level imagery. Comput vis Image Understand 157:55–71. https://doi.org/10.1016/j.cviu.2017.01.012
Salesses P, Schechtner K, Hidalgo CA (2013) The collaborative image of the city: mapping the inequality of urban perception. PLoS One 8(7):1–12. https://doi.org/10.1371/journal.pone.0068400
Scruton R (2011) Beauty: a very short introduction. Oxford University Press
Scruton R (2013) The aesthetics of architecture. Princeton University Press
Shen Q, Zeng W, Ye Y, Arisona SM, Schubiger-Banz S, Burkhard RA, Qu H (2018) StreetVizor: visual exploration of human-scale urban forms based on street views. IEEE Trans Visu Comput Graphics 24:1004–1013
Simonyan K and Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1. http://arxiv.org/abs/1409.1556
Sun J-Y, Kim S-W, Lee S-W, Kim Y-W, Ko S-J (2019) Reverse and boundary attention network for road segmentation. IEEE/CVF Int Conf Comput vis Workshop (ICCVW) 2019:876–885. https://doi.org/10.1109/ICCVW.2019.00116
Suzuki S and Abe K (1985) Topological structural analysis of digitized binary images by border following. Comput Vis Graph Image Proc 30(1):32–46. http://dblp.uni-trier.de/db/journals/cvgip/cvgip30.html#SuzukiA85
Teichmann M, Weber M, Zoellner M, Cipolla R and Urtasun R (2016) MultiNet: real-time joint semantic reasoning for autonomous driving. ArXiv Preprint http://arxiv.org/1612.07695
Wang H, Fan R, Sun Y, Liu M (2020) Applying surface normal information in drivable area and road anomaly detection for ground mobile robots. IEEE/RSJ Int Conf Intell Robot Syst (IROS) 2020:2706–2711. https://doi.org/10.1109/IROS45743.2020.9341340
Wang H, Fan R, Sun Y, Liu M (2021) Dynamic fusion module evolves drivable area and road anomaly detection: a benchmark and algorithms. IEEE Trans Cybernet. https://doi.org/10.1109/TCYB.2021.3064089
Wilson JQ and Kelling GL (1982) Broken windows. The Atlantic Monthly 29–38
Yang F, Choi W, Lin Y (2016) Exploit all the layers: fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. IEEE Conf Comput vis Pattern Recogn (CVPR) 2016:2129–2137
Zeiler MD, Fergus R (2014) Visualizing an understanding convolutional networks. In: Computer vision, ECCV 2014–13th European conference. Lecture notes in computer science (PART 1), vol 8689. Springer, Cham, pp 818–833. https://doi.org/10.1007/978-3-319-10590-1_53
Zhou D, Frémont V, Quost B, Dai Y, Li H (2017) Moving object detection and segmentation in urban environments from a moving platform. Image vis Comput 68:76–87. https://doi.org/10.1016/j.imavis.2017.07.006
Funding
This work is partially supported by INES (www.ines.org.br), CNPq Grant 465614/2014-0, CAPES grant 88887.136410/2017-00, and FACEPE grants APQ-0399-1.03/17 and PRONEX APQ/0388-1.03/14.
Author information
Authors and Affiliations
Contributions
Not applicable.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Ethical approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
de Mesquita, R.G., Ren, T.I., Mello, C.A.B. et al. Street pavement classification based on navigation through street view imagery. AI & Soc (2022). https://doi.org/10.1007/s00146-022-01520-0
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00146-022-01520-0