Analysis Method of Façade Color in the Urban Street Using Deep Learning

. Color planning has become a signiﬁcant issue in urban development, and an overall cognition of the urban color identities will help to design a better urban environment. However, the previous measurement and analysis methods for the facade color in the urban street are limited to manual collection, which is challenging to carry out on a city scale. Recent emerging dataset street view image and deep learning have revealed the possibility to overcome the previous limits, thus bringing forward a research paradigm shift. In the experimental part, we disassemble the goal into three steps: ﬁrstly, capturing the street view images with coordinate information through the API provided by the street view service; then extracting facade images and cleaning up invalid data by using the deep-learning segmentation method; ﬁnally, calculating the dominant color based on the data on the Munsell Color System. Results can show whether the color status satisﬁes the requirements of its urban plan for façade color in the street. This method can help to realize the reﬁned measurement of façade color using open source data, and has good universality in practice.


Introduction
Color is people's first visual impression of the city, which can influence spatial perception in the human-made environment through different color compositions. Besides, color can reveal the history, culture, and the specific identities of regions [1]. Therefore, the city managers attached great importance to the planning of urban color and issued a series of rules. For example, in 1845, Turin published a guide to urban color, which served as a reference [2]. In the 1960s, Paris identified beige as the dominant color of the city. Tokyo requires that the color of building exterior walls should be light in the 1970s [6]. In China, mostly located in the eastern part of the country, conduct urban color planning which allows continuing the urban color context, in the way of historical heritage by the dominant color, but also to guide the development of new construction zones [3]. Nanjing, one of the most important cities on eastern China, organize experts to discuss the urban color guidelines and identified light green as the dominant color of the city in 2004 [4]. Most of these urban color plans and regulations regard building color controlling as the most critical implementation program, because buildings are larger and more complex than the products created by human beings.
Currently, urban planners have established a complete workflow of facade color investigation [5]. First, researchers collect pictures and geographic coordinates of street facades by using digital cameras and GPS devices. Then, referring to the Munsell Color System to obtain color information such as the hue, saturation, and value of the façade image they collected. Finally, the results can show the facade color distribution and the dominant color of the city, meanwhile, planners make suggestions on the design or renovation of the city continuously. However, this workflow is limited to the manual collection at small quantity, so it usually takes some representative cases instead of the granularity analysis to ensure work efficiency in a large-scale practice [6]. Some urban designers and researchers were restricted to these empirical studies, getting refined results in the time specified by the project [7]. How to solve the large-scale, refined, and efficient acquisition of facade colors has become a significant obstacle to the further deepening and popularization of urban color planning.

Literature Review
Academic discussions and applications of large-scale measurement and quantitative analysis of building façade color have revolved around three categories.

Urban Color Planning
In the 1960s, Professor Lenclos Jean-Philippe proposed the color geography theory [8], in his points, the color of the city is formed by the specific geography, climate, and the relevant factors such as the national culture, history, and customs of the people living here. Therefore, the designers consider the characteristics of the city while planning the urban color. In this context, the first modern urban color planning was born in Tokyo in the 1970s. It mainly conducted a color survey across Tokyo [9], and while preserving the style of historical buildings, the planning guided the development of new buildings.
The planning of urban colors requires a quantitative definition of color system, and then the work can be carried out on a systematic and repeatable way. For example, Dinant suggest a standard color chart as a guidance for integral urban planning in Belgium [3]. In Greece, some researchers made studies of rehabilitation of contemporary Greek streetscape, by using regions of the NCS (Natural Color System) solid as color palettes [10]. In Tianjin (China), the authority used color charts based on the Munsell color classification system as a decision support tool [11], which is also a common practice in Chinese cities. These methods can investigate the current status of urban color and provide primary data for sustainable development of cities.

Façade Color Measurements
The current facade color measurement method uses digital cameras as the tools for collecting data. Investigators use digital cameras to obtain photographs of building façades in specific areas and distinguish whether the current status met the requirements of urban color planning. For example, Lu, et al. (2010) carried out field investigations on one avenue based on the color system of architecture design code (GB/T15608-1995) in Shenzhen to analysis the distribution of the dominant colors of the building façade [12]. Luan Nguyen (2017) presented a user-oriented protocol to characterize chromatic attributes of different class areas [3].
However, due to the consumption of time and labor in field investigations, almost all urban designers and researchers at the time could only conduct small-scale empirical research. They were unable to find a connection between theoretical study and practical operation and deepened their logical reasoning through induction and deduction, rather than exploring based on the principles of large-scale investigations.

Quantitative Analysis of Visual Quality in Urban Street
With the development of semantic segmentation algorithms and urban datasets represented by Cityscapes, it is no longer difficult to use a deep convolutional neural network architecture to achieve different feature dictation and high-precision segmentation results [13,14]. The combination of street view data and machine learning in urban measurement can solve the problem in large scale with high refinement, and it is used in many aspects, such as identification of the neighborhood safety through street view pictures [15], the extraction of green street visibility [16], and the measurement of visual quality in street space [17]. The two considerations include in this study are the classic urban design theory and existing deep learning algorithms.
The combination of machine learning and street view images has changed the situation in which street data was difficult to obtain and challenging to use efficiently. Compared to traditional methods, we can quickly get large number of street view photos, then providing scientific suggestions to the visual quality of urban space. The method can solve the limitation of small samples by manual collection of building façade color analysis in the traditional way, which is possible to use in a wide range.

Study Area and Workflow
Our research is conducted within the main districts of Nanjing, which is about 156.2 km 2 . In Chinese history, Nanjing had long been a major center of culture, education, research, politics, economy, transport networks, and tourism being a multi-functional metropolis [18]. The study area located in the center of Nanjing [19]. The workflow of main measurement method can be seen in Fig. 1. The detail process in each section will be explained in the following section.

Street View Data Acquisition
The pictures in this study were extracted from the street view platform which can provide street view photos with a large coverage. The methods of street view download are the following four steps: perpendicular to the road direction from street view service (viewing angle 90°, horizontal angle 0°, picture size 800 × 500 pixels) for each sampling points (Fig. 2).

Building Façade Segmentation and Data Cleaning
We used a deep neural network model, namely, PSPNet, to segment the building façade from street view images. Unlike the previous methods of extracting building element with RGB values, the network structure of the Pyramid Scene Parsing Network (PSP-Net) was widely employed where spatial statistics provide a good descriptor for overall scene interpretation. A single PSPNet yields the new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes [13]. The building segmentation results of Baidu street view images via the proposed trained PSPNet are shown in Fig. 3. Then, the proportions of building façade area can be measured by inputting street view images into the trained model to generate segmentation results, as Eq. (1) shows. However, there is a low proportion of buildings in some street view images, and these building images cannot reflect the color characteristics of the building. To improve accuracy of the experiment, we need to clean up the photos with a low proportion of the building.
P bldg is the proportion of building in a respective scene location, which is averaged by the values of street view images of the left and right side. N i is the total number of pixels in image i, and NW i is the number of building pixels in image i. By calculating the P bldg of each sampling point, we delete the pictures, an example like Fig. 3(c2) shows, whose building proportion is less than 10%.

Façade Color Calculation
Due to the color deviation, not all the colors in street view images correspond to the colors of the building materials. Therefore, using the architectural standard color cards (GSB16-1517-2002) with 258 colors as a reference, we merge the raw color data to standard color. The merge method is to calculate the HSV value of the street view color (c siv ) and replace them with the closest architectural standard color (c std ) by calculating the Euclidean distance. The color distance is the closeness between two points in the HSV cone (the length of the hypotenuse is R, the radius of the bottom circle is r, and the height is h). In the HSV color space model (Fig. 4), the center of the ground is the origin and H = 0 is the positive direction of the x-axis. Then according to Eq. (2), we were able define the three-dimensional coordinates (x, y, z) of the color point (H, S, V) as: After calculating the color distance, we can count the proportion of each color in these pictures. Two of the most important factors affecting the dominant color are the proportion and the saturation contrast. Therefore, following is the selection way of dominant color: Firstly, the dominant color should be the largest part of the building facade. Secondly, when the color proportions are equal, the color with high saturation is used as the dominant color. Finally, when the perception of color is the same and the chroma is the same, the darker one is the dominant color.

Results
We finally obtained 77, 448 valid data in 110, 686 total images. Table 1 shows the dominant color statistics of street view pictures on both sides of the street at each sampling point, including the H (Hue), S (Saturation), V (Value), and geographic coordinate. By calculating and obtaining the quantitative color information of building facades at the city scale, we can not only do a large-scale analysis but also make targeted analysis for a specific area. Combined with the land use plan of the Nanjing City (2011-2020) (Fig. 5(a)) and the Nanjing Historical City Preservation Plan (2010-2020) (Fig. 5(b)), we can illustrate that the downtown blocks with different functions and the control of historic conservation.   Most proportions of the historic style area, the dominant color is low saturation (Colors with a saturation of less than 20% accounted for 73% of the total.), like Fig. 6(b) show. In the historic conservation area, the lightness of dominant color of façade usually higher than 60%, and such strict control are mainly distributed in The Confucius Temple, The East Zhonghua Gate Historical Block, Sipailou Campus, and other historic districts, like Fig. 6(c) show.
Compared with the traditional way. Street view images are obtained as open source, which has a large amount of acquisition and efficiency. In the building recognition stage, the semantic segmentation model we trained by using the Cityscapes Dataset and the IoU accuracy of the building is 90.6%. In the stage of color calculation and statistics, the color distribution of buildings calculated by the HSV model can be visually expressed on the spatial map.

Discussion and Conclusion
This study proposed a framework for measurement and analysis of façade color in the urban street, including using open platforms to collect data and quantitatively analysis. The first was segmented the façades in street view images automatically with the support of PSPNet, the second was visualized in QGIS, while the last was achieved quantitatively through statistics. With façade color as the benchmark, the paper concludes the strengths and weaknesses of the large-scale measurement and quantitative analysis method of façade color in the urban street.
Empirical research on Nanjing's commercial and historical districts shows that Nanjing has gained a reputation for its cultural, historical, and architectural design values. The overall assessment is that color planning is highly consistent with the image of the city. In general, most of Nanjing's historic districts have retained their ancient form, structure, and color, despite being rebuilt in a modern way. Modernization challenges many traditional styles and characteristics of the inner city, and downtown is still a place that attracts the attention of developers.
Despite the contribution of this study in quantitatively measuring the façade color in large scale, it had several limitations that require further exploration. For example, street view photos are affected by outdoor light environment, causing the saturation and lightness to be very different from the actual data. The color calibration of street view images can improve the accuracy of the calculation results. In addition, how to use it to design a better visual environment is a significant problem in future work. The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.