Road marking retroreflectivity study via a visual algorithm

The retroreflectivity (Rl) of road markings is important and should be inspected and maintained throughout their service life. The specifications are provided by European nations, the United States, and many other countries. Although acceptance tests ensure the good Rl quality of newly placed road markings, the RL values of all in-service road markings are rather difficult to inspect by using currently available devices. This study, therefore, aims to determine the relationship between Rl and corresponding image brightness of yellow road markings to evaluate their visibility by analyzing recorded images captured at night. An integrated algorithm was developed to analyze recorded images continuously for identifying road marking brightness 30 m away from a vehicle. Field experiments on three types of road marking materials were performed and repeated at four separate locations. The findings provide a promising direction for using the image brightness of road markings to predict their field Rl. However, limitations of this study are discussed and suggestions for future direction are presented.


Introduction
Road markings play an important role in ensuring road safety. Therefore, the retroreflectivity (RL) and completeness of road markings should be maintained to an acceptable level throughout their entire service life to fulfill their functions. Related specifications are provided for newly painted or in-service road markings in most countries or regions. Although several materials, such as thermoplastic, paint, and preformed materials, are commonly used for road markings, they generally do not differ in RL quality requirements.
RL indicates the coefficient of retroreflected luminance, which is measured in compliance with an international standard. The minimum accepted road marking reflectivity index increases with roadway speed limit to provide drivers with sufficient safety. The United States Manual on Uniform Traffic Control Devices (MUTCD) specifies the lowest maintained levels of dry nighttime visibility RL of 50 and 100 mcd/m 2 /lx for yellow road markings in roadways with speed limits of 35 to 50 and above 50 mph, respectively [1].
Other specifications or standards, such as the American Society of Testing Materials (ASTM) D7942, specifies that the RL requirement at 180 days after placement should be above 200 mcd/m 2 /lx, and the maintenance minimum RL should be no less than 125 mcd/m 2 /lx [2]. The Central, East and West Nippon Expressway Company Limited and the Hong Kong Highways Department set the minimum RL values of yellow road markings to be 80 mcd/m 2 /lx [3,4]. European nations also laid down specifications CEN 1436 for yellow road marking RL, with the minimum levels of R1 (the lowest requirement) and R4 (the highest standard) set at 80 and 200 mcd/m 2 /lx, respectively [5]. Taiwan's specification, CNS 15834, of road marking follows the EN1436; thus, the specified RL numbers of yellow marking R1 to R4 of both standards are exactly the same [6]. Although these mentioned specifications also indicate the required RL for white road markings, only yellow road markings are considered in the current study due to research constraints.
Devices have been developed to measure road marking RL, and the commonly used measuring devices include two general types: handheld and mobile retroreflectometers. The former is usually handled manually to measure road marking RL from one location to another or mounted with moving wheels and pushed manually at a walking speed. The latter is mounted at one side of a vehicle, and the measurement is conducted while the vehicle runs up to 120 km/h. Fig. 1 illustrates the various types of road marking retroreflectometers. Although the efficiency of the mobile type is much higher than that of the handheld type in terms of measuring speed, the high cost of the mobile type limits its wide adoption. Beside the instrumental measurements, the MUTCD suggests that the trained inspectors can provide stable evaluation through visual inspection; however, the training procedures and inspection vehicles should be standardized [1]. Benz et al. also stated that manual visual inspection of road marking retroreflectivity is a highly efficient, low cost, no device needed, and low traffic impact method [7]. However, the variation among subjective judgements may induce the inaccurate results.
In 2006, Horberry et al. applied driving simulation analysis to study the possible safety benefits by enhancing road markings with large glass beads. It was found that the enhanced roadway markings tend to help drivers receive better marking's visibility and keep the vehicle within the traffic lane [8]. Autonomous vehicle technology has rapidly developed in recent years with corresponding progress in image processing and machine learning algorithms, which apply to automatic road marking recognition. Therefore, the visibility of road marking has become more and more important for the autonomous vehicle industry [9]. Considering the possible positive correlation between the concepts of road marking RL and image characteristics, this study aims to study the feasibility of using road marking image brightness captured from video recording devices to analyze RL and increase the efficiency of the RL evaluating of in-service road markings.

Data acquisition systems
In this study, two types of data were collected for algorithm development. The following sections describe the data features and corresponding devices adopted for data acquisition.

Road marking retroreflectivity, RL
The first type of data included RL, which were collected by devices manufactured in compliance with international specifications/standards, such as CEN 1436 (RL) [5], ASTM E1710 (RL) [10], and ASTM E2177 (RL wet) [11]. Fig. 2 illustrates the standard geometry of RL measurement. The simulation assumes that a driver's eye height and vehicle headlight height are 1.2 m and 0.65 m above the ground, respectively. The regulated visibility of surface road marking at 30 m in front of a vehicle should be considered. Therefore, the geometric locations among the driver, headlights, and road marking create a driver's observation angle of 2.29° and an illumination angle of 1.24°. The coefficient of retroreflected luminance is calculated on the basis of the given circumstances with vehicle headlight only. Various brands of RL-measuring devices have been designed, manufactured, and sold commercially. In the current study, a portable handheld device that meets the specifications of CEN 1436 (RL) [5], ASTM E1710 (RL) [10], and ASTM E2177 (RL wet) [11] was adopted for RL data collection. Table 1 provides the general technical specifications of the device.

Road marking images
The second type of data included road marking images, which were collected using video cameras. Imaging techniques were applied for road marking recognition, brightness detection, and distance measurement.

Image processing algorithm
Methodologies for image recognition and identification have been developed for more than three decades, and the algorithms are continuously improved over the years. In general, two major types of image processing algorithms are used, namely, traditional and machine learning methods. The three typical traditional image processing algorithms are threshold segmentation, edge detection, and regional growth method. Studies in various fields have applied one or more of these algorithms to solve research problems successfully. The traditional methods are typically used to identify and recognize a single object in a given region. Targeting objects under certain circumstances, such as those with sizes and/or angles that vary significantly, and the simultaneous targeting of multiple types of objects are both considered relatively difficult. However, these problems have been overcome in recent years, given the rapid improvements in machine learning techniques.

Distance measurement algorithm
Video images have been used for decades for distance detection. In a past study, the images from a single camera were used to analyze 3-dimensional information; however, the data obtained did not satisfy the accuracy of computing distance [16]. Pollefeys et al. highlighted that a single camera image yields a relative scale factor of distance rather than the actual distance [17]. Olaverri-Monreal et al. used two nearby cameras to capture an image of the same object and compute the distance through triangulation [18].
With a stereo camera, two two-dimensional (2D) photos can be reconstructed into a three-dimensional (3D) photo. The distance calculated between two cameras and an object is much more accurate than that obtained from a single camera. In this study, a dual-camera Raspberry Pi module was adopted and installed on a vehicle's rearview mirror to capture road marking images. The distance measurement theory by dual-camera images proposed by Olaverri-Monreal [18] is applied to calculate the distance between the cameras and the target road marking. Only the area where the images of both cameras overlap can be calculated. The distance between a vehicle and any specific road marking point within an overlapping area was computed by triangulation analysis using Eq. (1) [18]. Fig. 3 displays a close-up picture of the dual-camera Raspberry Pi module, and Table 2 provides basic information on this module.
where, Dobject : distance between the cameras and the object; Dcameras: distance between dual cameras; Pxh: horizontal pixel resolution (pixel number); PxL: pixel number at the left picture; PxR: pixel number at the right picture; ϕ0: horizontal field of view angle.
In our system, the distance between cameras (Dcameras) is 6 cm, the horizontal field of view (ϕ0) is 60°, the horizontal pixel resolution (Pxh) is 1280, and the horizontal pixel difference to the same object in both pictures in pixels is PxL−PxR.  brightness and grey level, which is typically nonlinear and close to gamma distribution.

Integrated algorithm for the road marking image analysis
where, Vout: output image pixel intensity after gamma correction; Vin: input image pixel intensity before gamma correction; A: In this study, thousands of road marking images were captured in a totally dark environment with only vehicle headlights. This experiment was conducted at the National Taiwan University Palm Boulevard, a four-lane undivided straight road of approximately 450 m. The center yellow road markings were reconstructed with three types of materials. Data collection was executed after midnight, and all streetlights were turned off. Fig. 5(a) and 5(c) show the typical raw image collected in this study and its corresponding pixel intensity histogram distributions and cumulative frequencies. As can be seen, the pixel intensity histogram concentrates on a bandwidth between 0.0 and 0.2. This result commonly occurs in all of the raw images acquired in this study. This bandwidth is too narrow to sufficiently identify road marking images from the dark background. Therefore, Eq. (2) was applied to the raw images, and the value was tested from 0.1 to 1.0 at an increment interval of 0.1 to investigate the optimum pixel intensity distribution for further study. The results indicate that  = 0.5 provides the best enhancement of pixel intensity. As shown in Fig. 5(b) and 5(d), the distinguishable pixel intensity distribution increases the efficiency of the following analysis steps.

Mask R-CNN.
After gamma correction was applied on each raw image, mask R-CNN was performed for image processing. Mask R-CNN is the latest development of CNN. It uses R-CNN as the base and adds object masks to image processing to increase the efficiency and accuracy of target image recognition and identification. Images with or without road markings are processed and classified into two groups. In this study, a total of 6415 images captured during the experiment, and Table 3 shows the numbers of images used in the training and testing. A confusion matrix [19,20] was adopted to evaluate the precision of analyzed results. Precision is defined in Eq. (3), and the terms of TP and FP are defined below. As indicated by the results, the images with and without road markings are successfully identified with precision rates of 83% and 92%, respectively. The overall precision rate is 87%. In general, the mask R-CNN method provides good results for identifying road marking images, which are retained for further analysis. (3) In the equation above, TP is true positive, and FP is false positive.

Brightness and distance detection
For the image identification, the brightness levels of the suspected road marking pixels were investigated. Fig. 6 displays the brightness histogram of a typical road marking image. Only the pixels of road marking area are retained after applying the mask R-CNN on the raw image; thus, more than 70% of the pixel brightness intensity is concentrated between 200 and 225. The average brightness of the retained pixels was computed to represent the specific road marking brightness. The distance between the vehicle and the farthest point within the mask was then calculated using the distance measurement algorithm and Eq. (1). Therefore, the distance and brightness of each road marking image were paired for further analysis. It should be mentioned that the calculated brightness is based on the experimental device used in this study. A variety of factors, such as brand and model of the dual-lens module or exposure time setting, may affect the capture image brightness. Further study should focus on how these factors will affect the results. However, all parameters are fixed in the following analyses.

Field experiments and data analysis
The field experiments were conducted along the 450 m-long National Taiwan University Palm Boulevard. All the experiments were executed after midnight with the streetlights off and a specific traffic control condition to study the relationship between the road markings' RL and their corresponding image brightness levels. Thus, test vehicle headlights were the only light source for illuminating road markings as the video images were captured.
Three types of road marking materials were selected and placed for the field experiments. Two of which, A and B, were thermoplastic materials with different types of beads and anti-skid additives, and C was a preformed pavement marking tape. The major difference between the beads used in types A and B was the reflective index (RI). Type A includes mixed conventional beads (RI = 1.5) and high-RI beads (RI = 1.64) internally and high-RI external drop-off beads. Type B is a widely used commercial product that uses conventional beads internally and externally. Fig.  7 shows the construction of thermoplastic type and preformed tape. Table 4 provides the specifications of the three materials. Every type of road marking was placed at four separate locations along the test road, and the RL tests were conducted using the handheld retroreflectometer for 46 test points.
During the field experiment, videos were recorded by a dual-lens camera in totally dark environments with only vehicle headlights. The researchers videotaped the setup while a vehicle was moving from 150 m away toward each type of road marking at an average running speed of 40 km/h (25 mi/h, 11.1 m/s). The camera taping frequency was 30 fps. Thus, each frame covering range had approximately 37 cm variation. For every test run, all placed road markings were covered with black tapes, except for the targeted one.
Given that the testing runs always started far away from the target, the first few hundreds of picture frames did not include any road marking image. When the vehicle gradually approached the target, the frames then started to include the road marking images from far to near. Fig. 8 shows the typical analyzed results of the relationship between paired distance and brightness of one driving test. As indicated by the results, the average brightness of the road marking pixels tends to increase as the vehicle gets closer to the analyzed target.
Three types of road markings were tested separately, and each type was tested in four placed locations, resulting in 12 test runs. Similar to the data in Fig. 8, the relationship between paired distance and brightness resulting from the 12 test runs clearly show that the brightness of the road markings increases as the vehicle gets closer to the analyzed target. This finding is consistent regardless of the road marking type.
Meanwhile, Type C has the highest brightness at any giving distance, indicating that it can be seen more easily at night than the two other types. The sequence of brightness of the three road marking types is C > A > B. Type B which uses conventional beads internally and externally has the lowest performance, in terms of image brightness and measured RL, given that the conventional beads have low RI. It should be mentioned that the RL requirement is not yet regulated in existing road marking acceptance specifications in Taiwan, so type B occupies the most market share for its low cost. In order to analyze the correlation between image brightness and road marking RL, the brightness values at a distance of 30 m of each test, the same distance applied to determine the RL simulated by the retroreflectometer, are marked for further analysis.
The RL measurements were conducted simultaneously with image tests. Table 5 provides the paired measured RL and image brightness values of the three types of test road markings placed at four locations. The values of brightness represent the images collected by the dual-lens camera and analyzed by the developed algorithm 30 m ahead of the test vehicle. Once again, Type C provides the highest RL and image brightness.
From the 12 sets of RL measurements and image brightness values, data can be easily clustered into three road marking groups,     141  126  50  67  310  166  118  129  38  69  321  154  139  121  50  72  318  150  119  130  63  66  309  171 A, B, and C, as shown in Fig. 9. This result also indicates that the performance of each road marking in terms of RL is very consistent at four locations. An exponential regression equation is shown in Eq. (4). The correlation coefficient of this non-linear equation is 0.93, which provides a relatively good explanation between these two variables. The corresponding brightness values that match the five classes of RL, which we determined using Eq. (4), are presented in Table 6. For example, a yellow road marking image with a brightness of ≥95 indicates a corresponding RL of ≥80 mcd/m 2 /lx.
In the equation above, Y: predicted road marking RL, mcd/m 2 /lx; X: computed road marking brightness.

Discussions and conclusions
The RL of road markings is a key parameter that indicates their visibility at dry nighttime, particularly when no other light except vehicle headlights resource are available. The construction Fig. 9. Relationship between yellow road marking brightness and RL. acceptance specifications of required RL ensure the good quality of newly placed road markings. However, the entire population of inservice road markings is rather difficult to inspect by using currently available devices. This study aims to establish a reliable relationship between RL and the image brightness of a road marking so that the latter can be used as an efficient indicator to screen the RL of in-service road network markings. The field experiments were conducted to collect road marking images at night using the dual-lens camera. The integrated algorithm was developed for analyzing the image brightness of the road markings at a distance 30 m ahead of a vehicle being driven. The designated 30 m is exactly the same distance as the simulated RL measured by certified devices.
The results show a relatively high correlation between these two parameters, providing a promising research direction of using road marking image brightness to forecast the field RL. However, given the constraints of the research scale, several restrictions and limitations exist in the study, which require further investigation. These are listed below.
1. Additional types of road marking materials should be included. In this study, only three types of yellow road markings were used. Although they were easily distinguishable, additional types of road markings can better provide further information to regression curves in terms of image brightness and RL. Furthermore, white lane line road markings should also be analyzed in future studies. 2. A testing vehicle was used to capture the road marking images during the field experiments. The analyzed results were based on the given vehicle's headlights luminance.
Given that the luminance of vehicle headlights varies by vehicle brand and model, the illumination of road markings may thus differ with various vehicles. Therefore, further studies should focus on how headlight luminance affects image brightness and distance relation and the consequent relationship between road marking RL and image brightness. 3. A dual-lens camera with 6 cm lateral distance between lenses was adopted for road marking image recording. Theoretically, the lateral distance may influence the accuracy of longitudinal distance calculation of an image. In addition, the quality and image exposure condition of dual-lens may affect the calculated brightness. It is suggested that to use a wide dual-lens camera and specify the required specifications for image collection in the future study.

Declarations
Funding: This research project is sponsored by the Ministry of Science and Technology (MOST), Taiwan, (Project No. MOST 107-2221-E-002-039-MY3).
Conflicts of interest: To the best of our knowledge, the named authors have no conflict of interest/competing interests. This article is licensed under a Creative Commons Atribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons license, unless indicated otherwisein a credit line to the material. If the material is not included in the article's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to optain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/