Computer-vision-guided semi-autonomous concrete crack repair for infrastructure maintenance using a robotic arm

Engineering inspection and maintenance technologies play an important role in safety, operation, maintenance and management of buildings. In project construction control, supervision of engineering quality is a difficult task. To address such inspection and maintenance issues, this study presents a computer-vision-guided semi-autonomous robotic system for identification and repair of concrete cracks, and humans can make repair plans for this system. Concrete cracks are characterized through computer vision, and a crack feature database is established. Furthermore, a trajectory generation and coordinate transformation method is designed to determine the robotic execution coordinates. In addition, a knowledge base repair method is examined to make appropriate decisions on repair technology for concrete cracks, and a robotic arm is designed for crack repair. Finally, simulations and experiments are conducted, proving the feasibility of the repair method proposed. The result of this study can potentially improve the performance of on-site automatic concrete crack repair, while addressing such issues as high accident rate, low efficiency, and big loss of skilled workers.


Introduction
Cracks on the surfaces of concrete engineering structures are among the earliest indicators of structural deterioration. Structures suffer from fatigue stress and cyclic loading (Tedeschi & Benedetto, 2017). As a result of external loads, minute cracks on concrete surfaces may produce interconnected passageways, which will worsen the safety of structures (Algaifi et al., 2018). Thus, civil engineers face the challenge of reducing the harm caused by deteriorating structures. In this regard, intelligent technology for unmanned detection and repair is necessary.
As one of the potentially useful technologies, computer vision is increasingly implemented in automated recognition of concrete cracks (Dan & Dan, 2021). Surface condition deficiencies are often evaluated by combining computer-vision detection and surveying equipment (Shamsabadi, et al., 2022). As a result, computer-vision-based concrete-crack detection is becoming a type of non-destructive testing technique (Kim et al., 2022), with many methodologies used to determine the existence and location of cracks. Although some studies have focused on extracting such basic information as length, width, and depth (Cha et al., 2017), such information is not enough in making decisions on crack repair behavior.
Conventional repair materials are classified under various criteria (Tsiatas & Robinson, 1795). Crack epoxy injection is one of the common methods to reconstruct the lost strength properties (Ahmad et al., 2013). Based on the properties of the epoxy injected, this type of repair proves effective on new cracks that arise outside *Correspondence: charleschou@163.com the region of previously repaired cracks. With regard to repair techniques for concrete cracks, such information as length, width, and depth is not enough crack remedy. Therefore, researchers have started to pay attention to the use of bacteria to repair cracks in concrete (Maedeh & Mehdi et al., 2020;Zhou et al., 2019). A review of the existing researches reveals that most of the research on concrete-crack repair techniques is mainly on materials, and few studies have been conducted by using robots.
A robot system for construction quality assessment has been used to optimize the autonomous visual inspection function, so as to cut labor cost and improve accuracy (Liu et al., 2017). An automated integration system has been developed for remote inspection and repair without direct human intervention. In addition, a semi-autonomous robotic system has been proposed for inspection and repair of pavements and bridges, while improving the security of properties and inspectors (Sutter et al., 2018). However, these repair platforms are semi-autonomous and pre-programmed. In contrast, this study will design a computer-vision-guided semi-autonomous robotic system for concrete crack inspection and repair, with the help of human decisions.

Computer-vision-guided recognition of concrete cracks
Cracks on concrete surfaces of engineering structures are among the earliest indicators of structural deterioration as a result of fatigue and other negative factors (Mohan & Poobal, 2018), leading to weakened material integrity. Therefore, observing concrete cracking is crucial for characterizing the safety of structures. Most existing approaches to crack detection are empirical, with human's visual observation becoming the common method of crack detection. However, this and other similar conventional methods to characterize and inspect cracks are time-consuming and error-prone. So an innovative method is necessary. Recently, the developments in digital facility and methods have widened the initial field of concrete crack recognition (Valença et al., 2012). The purpose of crack inspection may vary, depending on parameters to be inspected. Crack detection may be delivered based on length, width, depth, and direction of cracks (Pantoja-Rosero et al., 2022). The major advantage of the computer vision technique lies in that it can provide more accurate results than traditional manual methods (Shanaka et al., 2022). Some of the conundrums in computer vision recognition are related with different shapes, irregular cracks, and various noises. By virtue of superior performance of computer vision, many types of image processing and recognition methods have been proposed, such as integrated algorithm, morphological approach, percolation-based method, and practical technique (Wang et al., 2010), most of which are used to determine whether cracks exist and where they are located. Spatial wavelet transformation has also been proposed to detect and localize cracks by amplifying weak perturbation signal at crack locations (Mardasi et al., 2018). Liu et al. (2019) have used U-net fully convolutional networks to detect concrete cracks based on computer vision. Liu et al. (2021) have adopted the integration of Convolutional Neural Network and Active Contour Model to perform crack segmentation. With deep learning in frequency domain, Zhang et al. (2020) try to detect cracks on concrete bridge decks in real time.

Repair techniques for concrete cracks
Many existing researches are focused on restorative materials. For example, viscosity and mechanical strength of epoxy materials are explored for repairing concrete cracks under low-temperature construction in winter (Dana et al., 2021). Given that the quality monitoring of crack fullness and solidification is important (Bykov et al., 2017), high efficiency of injection processes is pursued by strict observation of the process norms, particularly about the qualitative fullness of crack cavity and full solidification of injected materials. However, no unified test method is available both at home and abroad. Thus, quality control in crack repair is difficult to implement (Wang et al., 2015;Zhu et al., 2015).
Traditional methods are based on manual repair. However, computational and autonomous properties play an increasingly important role in improving the knowledge about the effect of most concrete crack-repair techniques on the mechanical performance of repaired components (Marazani et al., 2017). For self-healing agents in concrete, cement stone samples can deposit a new layer of calcium carbonate minerals on the surface to cover cracks (Boumaaza et al., 2017;Choi et al., 2017). The crack healing potentials of bacteria have been compared with traditional repair techniques through experimental observation (Maedeh et al., 2020). Furthermore, researchers in the process optimization field have invented new technologies and optimized some repair operations. In some researches, a piezoelectric patch was bonded on a beam through an external voltage to affect the closure of concrete cracks (Riccardo et al., 2020). These techniques can also be used to control the failure modes and stress distribution around beam chords (Osman et al., 2017). In a systematic method, Kim et al. (2019) have evaluated the performance of crack repair materials by using PZT-based EMI technology, which can reflect the structural characteristics in evaluating the repair efficiency over time. Ramesh et al. (2021) have deployed non-destructive testing methods to repair and refurbish reinforced concrete structures.
To detect concrete cracks in engineering, personnel may have to enter hazardous environments. Automatic approaches are the only effective way to support human exploring extreme environments. A review of the existing researches on concrete-crack repair techniques finds that they generally pay attention to materials.

Integration of automated recognition and repair for concrete cracks
Several previous studies have attempted to optimize the crack recognition process for concrete values. For example, a computer vision system for a train inspection monorail was proposed and installed in the Large Hadron Collider to gather data from various sensors and capture images by the European Organization for Nuclear Research, only purposed for recording data and reducing personnel intervention (Attard et al., 2018). The recognition process of engineering concrete cracks has been automated to a certain extent based on deep learning methods (Chheng & Likitlersuang, 2018), including Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Transfer Learning (2017b; Cha et al., 2018;Huang et al., 2018;Xue & Li, 2018;Zhang et al., 2017a). Overall, the structural crack repair process is slow, labor intensive, and subjective so far. To overcome these working limitations, automatic repair systems have to be developed (Kovačević et al. 2021). An urgent need is to design a fully automated integration system for inspection and reparation, which shall enable remote operations, without any need for direct human intervention. One selection is to improve the automatic behaviors of robots. It has been shown that the quality of manual operations depends not only on the experience of workers, but also on the level of their fatigue. Therefore, this system to be designed shall ensure the safety and suitability of the control mode. Being semi-autonomous, this system can improve the inspection efficiency and accuracy by automatically identifying concrete cracks. Tele-operation of robots should be considered for the operation process. In addition, a semisupervised computer vision system has been developed in ROBO-SPECT European FP7 project to detect tunnel diseases (Menendez et al., 2018), andHarsh et al. (2020) have used robots and computer vision to detect and quantify defects in dam spillways. As these repair platforms are semi-autonomous and pre-programmed, most of the current crack inspection and repair platforms are focused only on detection. To address this limitation, this study introduces a computer-vision-guided semi-autonomous robotic system, which is dedicated to concrete crack inspection and repair projects that involve decision making by humans.

Methodology
As shown in Fig. 1, four main steps are involved in the computer-vision-guided semi-autonomous concrete crack repair process using robotic arms. The first step includes feature acquisition and trajectory extraction, purposed to recognize concrete cracks. Feature acquisition is performed to determine the length, width, depth, and other measures of cracks through a computer vision process, while trajectory coordinates are calculated via hand-eye calibration. After a knowledge base is created to determine appropriate crack features for the repair method, the overall repair process will be simulated by code programming and software operation. The decision made by humans based on the knowledge base includes the establishment of relevant standards and specification Fig. 1 Architecture of the crack repair system databases, e.g., a crack feature database. Once the simulation process is determined, the crack repair process will be launched, including the design of robotic arms, the execution of repair operation, and the evaluation in the next step. Finally, the semi-autonomous concrete-crack repair process is tested and verified under laboratory conditions based on computer vision. Each aspect of the repair process is described in detail in Fig. 1.

Acquisition of concrete crack features using computer vision
The computer-vision-based crack-feature acquisition involves: detecting cracks, determining locations of crack components, and measuring the length, width, and depth of cracks. A project requires a large amount of data, which have to be recorded and organized through various methods. A database is needed to store the data on the majority of concrete crack repairs. In general, managing the raw data involves an independent database, which can be built with an electronic spreadsheet application. Crack features can be divided into eight categories based on the following engineering feature values: component position, crack position, crack material, crack properties, crack width, crack length, crack depth, and crack direction, as shown in Table 1. Various features are detected with different techniques. For instance, some crack features are acquired in infrared, laser, ultrasonic, and various other computer-vision-based imaging methods. With regard to the infrastructure for the crack feature acquisition using the computer vision technique, a general workflow of such acquisition is shown in Fig. 2.

Trajectory extraction and hand-eye calibration
Extraction of the coordinates of cracks involves image pre-processing, denoising, and edge-region processing for crack trajectory extraction. Region points of a crack trajectory are used in thresholding value selection created from edge points on the basis of human decision making. A region of interest (ROI) needs to be set to insulate the background and the crack trajectory region, and the minimum distance from the region to the crack is not less than 1 cm but not more than 2 cm. A filtering method should be used to eliminate the useless information. To transform a gray image into a binary one, this study implements the thresholding technique, with which a picture can be compartmentalized by using the local threshold. The points of the trajectory are depicted from the points of the area outline. In the marginal area, the sub-pixel type of data contour is generated by utilizing the marginal form. The outer margin of the pixels is utilized as contour points, as shown in Fig. 3.

Camera coordinate calculation
Wiping off useless pixels through the filtering technology is crucial to identifying the important data. The thresholding measure, in which an image is divided into multiple local thresholds, is implemented in this study to transform a gray image into a binary one. The measurement can be carried out under any normal indoor light conditions. Camera coordinates are calculated in the pinhole camera model (Eq. 1), as shown in Fig. 4, where K represents camera intrinsics.

Robot coordinate calculation
Parameter A can be calculated with the form, and the calculation can be completed in the Matlab arm calibration toolbox. Parameter B can be solved by using the Matlab camera calibration toolbox and Zhang Zhengyou Camera Calibration Method (Zhang, 2000). The relative positional relationship between a camera and the sixth axis of the manipulator must be changed by three positions, as shown (1) x c y c z c in Fig. 5. T x is the hand-eye relationship and . 0 T 6 is the kinematics positive value.
The relationship between the polar coordinates of the manipulator and the coordinate system of the target object (calibration board) T G is a fixed value. T 6 represents the transformation relationship between the coordinate system of the 6 degrees of freedom manipulator and the end (forward kinematics) of the manipulator. T X represents the coordinate transformation relationship between the end of the robotic arm and the camera. T C represents the transformation of the camera to the coordinate system of the target object.
(2)  Equation (4) can also be expressed as A · X = X · B , where A = T −1 6 2 · T 6 1 represents the relative relationship between the two position postures, and B = T C 2 · T −1 C 1 represents the relative relationship between the two position pose cameras. Parameters A and B can be obtained through the forward kinematics relationship of the manipulator and the external parameter matrix of the camera.
After the T x value is obtained, the motion coordinates of the manipulator can be calculated. In Eqs. (5) and (6), T x indicates the hand-eye relationship and . 0 T 6 indicates the kinematics positive solution of the manipulator.

Design of knowledge base and simulation of crack repair
A knowledge base is designed for the crack repair method driven by meta-knowledge. This inference process will screen data from the crack feature, repair standard, and robot code databases and contribute to the data warehouse system, as presented in Table 2. Furthermore, the expression of the knowledge base is constantly adjusted on the basis of the robot execution characteristics, so as to accurately select the alternatives containing the tools, materials, standards, and related information.
The process of requirement analysis, database design, data coding, item importing, data verification and screening, back-up, and exporting for analysis is illustrated in Fig. 6. With the RobotStudio design module, a virtual simulation environment is designed, where a robot arm and concrete cracks can be displayed. First, the robot or robotic platform to repair the concrete cracks can be deployed. Second, the execution process can be simulated to detect the movement conflict by using a collision detection module and code compilation. The simulation enables the robot to hold back human beings from the detriment in uncomplicated testing scenarios and improve the efficiency and feasibility of the repair process.

Design of robotic arm for crack repair
A semi-autonomous concrete-crack repair robot is implemented to support repairing a wide range of cracks. A human intelligence-based approach is used to design a convertible terminal tool, which is identified as ideal, because it refers to most of the repair methods for concrete cracks, including the operations, tools, and materials used in the repair process, as shown in Table 3. This study also investigates a robotic arm control method based on Asea Brown Boveri Ltd. (ABB) robot language for applications with related execution instructions.   (2) consists of the length and depth, components, location, material, property, width, and direction of cracks, defined as Enumeration Type (6). The number is in Int Type (1). Each data picture is listed in the table using Blob Type (1). Before the image processing, the pixel resolution and image capture should be done. After preprocessing and enhancement of grayscale images and others images, they can capture the pictures of cracks, as shown in Fig. 7. The threshold method of segmentation is used after smoothening the images' spatial filtering. And the length and width of cracks are also calculated to evaluate the parameters of images of cracks by analogy.
The data are collated in Excel to perform detection and high-performance transformation, and then converted to a record for the crack feature database. Table 4 shows the attribute table structure of the crack features (containing 200 entries of data). The crack feature data for every interface are transformed into the database, as shown in Fig. 8.

Knowledge base of the repair methods
The database framework of the knowledge base is established, as presented in Table 2. The repair methods' data requirements can be divided into 10 types and 8 aspects.
The characteristics of the repair methods are installed in 6 major interfaces. Text Type (7) is composed of technology number, name, range, material, tool, process, and reference. Steps and robotic execution code are defined as Blob Type (2). The number is Int Type (1). Table 5 shows the attribute table structure of the crack repair methods. The repair technology data for each port are converted to the established database. The Excel data table converted for the database is shown in Fig. 9, which now consists of  six types of process data, which will continue to expand in the future.

Simulation system and execution device
Given the expanding ability of robots to take semiautonomous concrete crack repair, it is imperative that mechanisms are put in place to guarantee the safety of their behavior and process simulation. Moreover, semiautonomous robots should be safer; arguably, they should also be explicitly executable. By using the RobotStudio design module, a virtual simulation environment is designed, where a robot arm and concrete cracks can be displayed. First, the robot or robotic platform for repairing concrete cracks can be arranged. Next, the execution process can be simulated to detect the movement conflict by using a collision detection module and code compilation. It is demonstrated that the simulation should enable the robot to prevent human from being harmed in simple test scenarios and make the repair process more efficient and feasible. The semi-autonomous robot arm provides software for offline and online programming of robots. It implements a methodology to create a BIM model of an existing physical robot, which is described by taking the example of 6-axis robotic manipulator (ABB IRB 6700-235). Later, the crack trajectory parameters extracted with computer vision are compiled to execute the code. With the locus coordinate parameters, action simulation is exported. The simulation is developed with RobotStudio, which can connect to Visual Studio to execute the robot motion and collision detection. The application is finally integrated with the robotic manipulator, as shown in Fig. 10. Although most multifunction tools of the semi-autonomous robots available now have a circular flange plate,  this study extends the flange plate from different repair tools for repairing concrete cracks, as shown in Fig. 11. The repair tools presented are mainly composed of sealing, grabbing, blowing, and seaming devices, which use grout nipples and different sealants; and their circular flange plate and switch can be actively controlled through the pneumatic soft bending actuators embedded along the edges. Tools are converted according to the requirements of corresponding steps. The multifunctional and convertible repair device model is validated and simulated experimentally in RobotStudio, finding that the error rate is within 6% of the surface for a number of actuation levels.

Repair process and discussion
The semi-autonomous robotic platform is implemented in a laboratory, which includes industrial robot integrating multifunctional tools, workbench, and repairing components, as shown in Fig. 12. In the course of experiment, each step has its code to command the robot, including:   . 9 Interface of knowledge base system for the repair methods  (Fig. 12a), grout nipple grabbing (Fig. 12b), grout nipple pasting (Fig. 12c), cleaning and injection (Fig. 12d), plug grabbing (Fig. 12e), and plug installation and curing (Fig. 12f ). Functional fixture for grabbing the grout nipple and plug is driven by a MHL2-40D cylinder. Both the motion path and speed of each operation of the robot are controlled through the control procedures, so as to make the pose and the grabbing speed of the construction adjustable. In addition, the coordinates of these moving points that have been calculated before will be compared with those acquired from the process simulation of the concrete crack repair process. The data are utilized when the teaching apparatus is deployed to control the accuracy of the process. And all the operations are implemented in the ABB programming language, as elaborated below. It is important to control the accuracy during the construction process, because the accuracy can even affect the quality of the component repairing results.
The motion execution encoding of Fig. 13 is explained in Fig. 14. The robot's operation and execution procedures are allocated to each step of the repair method. During the process, the robot allocation is decided to build a new workstation, which is divided into many possible tasks in the former decision. In the proposed approach, the original task time is evaluated to a greater value. Once the new best task and waiting time are achieved, all the individuals in the solution will be re-decoded using their fitness values. Then, the corresponding repair operations are implemented by the industrial robot, as shown in Fig. 14.
All the procedures of the crack repair process are debugged with RobotStudio software. The main program of the repair procedure and simulation environment is illustrated in the ABB programming language, as shown in Fig. 15. At the beginning of the procedure, the code script InitAll is defined to initialize all parameters and empty storage spaces. CheckHomePos is presented to find appropriate positions of the concrete crack photo pose, while triggering do08 and di00 to send the photo commands and perform images for analysis. Robot allocation is implemented to blow up and clean the cracks, stick the nozzle, install the plug, and apply the sealant. All the operations correspond to a di signal, as indicated in Fig. 16. If di0x (x = 1, 2, 3, 4, 5) = 1, then relevant operations are triggered and executed.
Eight points are selected in the trajectory of crack extraction based on computer vision. Each point of the cracks is measured by the actual coordinates of the robot operation, and the data set so obtained is used sequentially in calculation. All the points of simulation and real measurements are listed in Table 6. As presented in the experiment, the accuracy reaches the last two decimal points. In this task, Cosine Similarity analysis is implemented to estimate the similarity between the two sets of data, which are calculated in Formula (7). The results are  presented through the Average Cosine Similarity. As can be seen, its value is very close to 1, showing that the two sets of data are highly correlated with each other. Therefore, it is concluded that the crack trajectory recognition based on computer vision is similar to that of crack repair, proving high recognition accuracy.
Network planning is conducted to plan and control projects and to identify the best solution by looking for key jobs and key chains. The activity on the edge network of the restoration project is drawn, as shown in Fig. 17, where the key path is marked in red. The time-consuming factors of the entire project are indicated for the nodes in the critical path. All the time values are calculated in the simulation software. However, some of the steps take an excessive amount of time. Therefore, the simulation can be started with the nodes in the critical path, so as to reduce the time consumed. For example, the algorithm can be optimized to identify the cracks with higher efficiency and calculate the relevant coordinates. In addition, the algorithm can be greatly improved to reduce project execution time by boosting the technology and process execution steps. (7)

Conclusions
This study explores a computer-vision-guided semiautonomous concrete-crack repair method by using robotic arms. The extraction of the coordinates of cracks proves to be an efficient way to acquire the trajectory of cracks. Furthermore, after the feature extraction for  concrete cracks, a crack feature database and a repair method knowledge base are applied in determining the repair process based on human intervention. The method proposed in this study can be optimized to save project execution time. One of the major contributions of this study is that it can make the system fully autonomous. Simulations and experiments demonstrate that the proposed method can improve the crack detection for concrete structures and enhance the scheme decision-making and construction automation in the repair process.
For future improvement, this system shall be maintainable and checkable, while minimizing or even eliminating human intervention or supervision. Further research is ongoing to design an improved system, so as to perform accurate and cost-effective inspection, maintenance, and evaluation of civil structures that can restored and repaired in a safer, less costly manner.