Keywords

1 Introduction

Interaction technology between human and robot has been continuously developed. Among the applications for surveillance, outdoor unmanned security robots have attracted attention in recent years. The robots have to perform autonomous navigation and human interaction in various outdoor environments. First, autonomous driving has been carried out steadily, and many researchers are conducting it. First, autonomous driving has been continuously studied by many researchers. Especially, the navigation techniques for the outdoor environment have been performed mostly for good weather, and it is difficult to cope with most weather conditions. This is because effective feature extraction is difficult in complex weather environments. In order to overcome this problem, robust features are extracted by utilizing all of the multi-modal sensor data available to the robot. Then, there is a need for a method of locating the most useful features using prior knowledge of the current weather, location, and topography.

On the other hand, the security robot is required to interact with humans to perform missions to track or guide interested humans. For this purpose, it is necessary to consider both the control method of measuring and maintaining the distance from the human, the method of driving safely by recognizing the speed and the humility. Therefore, a safe control method is required for rollover that does not overturn while measuring distance.

In this paper, we propose a robust localization method for outdoor environment and a control method for interaction with human. The proposed method can drive outdoors for most weather conditions, and can track and guide humans on the unpaved road or slope way. Experimental results show that robust position estimation is feasible based on knowledge-based features, and a stability model can be defined to enable safe travel with less risk of rollover.

The remainder of paper is organized as follows. In Sect. 2, some related works are summarized. Section 3 describes localization for outdoor autonomous navigation, including multi-layered knowledge based salient map, information filter, and some results. The safety path generation method with ZMP model is described in Sect. 4. Section 5 presents the human robot interaction by following in surveillance application. Finally, we summarize the proposed method.

2 Related Works

Research on the position estimation of robots has been carried out steadily. The Researches based on various sensors estimate a location with a fixed map using vision data [1] or sensor data [2] in indoor environment. However, in order to carry out unmanned security, localization method should be possible in outdoor environment.

There is a method [3] of recognizing the road environment by mixing LiDAR based on GPS, a method [4, 5] using a laser sensor, a method using a single camera [6] or a stereo vision [7] for outdoor localization. These methods determine the location by mixing the global GPS and the local pose estimation using the sensor data. In the same way, there is also research on location recognition using only global and local vision data [8]. However, for practical security, robust driving methods are needed in most weather conditions, including seasonal changes.

In order to overcome the environmental changes, a method of estimating the location using robust features in the image for seasonal changes [9], a method that is useful not only for seasons but also for night and daytime conditions [10, 11], and techniques have been proposed for rain or snow weather conditions [12]. These methods are solving some of the changes in the outdoor environment depending on the feature quantity using a specific image only. Therefore, in this paper, we propose a localization method considering most of season, night/day and weather conditions.

On the other hand, there were various efforts for stable control of robots. However, it seems that most of them are not suitable for roads that have roughness for active steering [13], steering and braking [14, 15], which are mostly controls for flat roads. Therefore, we propose a stable model-based control algorithm by the centrifugal force generated in the turning process in order to prevent the rollover on the road with the slope.

Most of the interaction methods between human and robot were performed based on sensor data in indoor environments [16, 17]. However, in this paper, we propose a human-following interaction method using the above-mentioned location recognition and control method for outdoor unmanned security robot. It performs interactions that follow specific humans for outdoor security applications. Consequently, the proposed interaction method is expected to improve the service of the outdoor unmanned security robot.

3 Localization for Outdoor Autonomous Navigation

3.1 Multi-layered Knowledge Augmented Based Strategy Map

A multi-layered knowledge augmented map database for extracting valid data from multi-modal sensor is shown in Fig. 1. This map is based on a multi-layer including semantic knowledge (semantic data, episodic data, semantic and cloud big data), climate, time, geographical features and driving strategies. The information is used to enhance a dynamic navigation map. Figure 2 shows an example of fusion of valid salient data based strategy map at the current location by query from the multi-layered knowledge augmented DB map. This is a way to represent the most useful data at the current position using a high-level knowledge database. Moreover, each data has a level of robustness and reliability can be measured using it. Therefore, we can choice the salient data which has strong and weak data in the given knowledge data (weather condition, current time, season, temp, RH, etc.).

Fig. 1.
figure 1

Multi-layered knowledge augmented map database configuration.

Full size image
Fig. 2.
figure 2

An example of valid salient data based strategy map.

Full size image

3.2 Reliable Localization Method

Reliable localization estimation is performed by extracting salient pointers and features with the multi-layer prior knowledge-based information filters. This method is estimated by fusion of non-model based Monte Carlo Localizer [18] and model-based EKF Localizer [19] in a reliability pose. Each localizer considers the knowledge-based robust level to determine its uncertainty. The final pose estimation using this is a reliable result, as shown in Fig. 3.

Fig. 3.
figure 3

The overview of reliable localization method.

Full size image

The proposed method employs an information filter for multi-modal sensor data, the salient points and features can be extracted as,

$$ X_{i} = {\text{I}}(x_{i} ,k) + V_{k} $$
(1)

where Xi is observed data, and Vk denotes the noise vectors. The \( {\text{I}}\left( \cdot \right) \) derived from the information state vector Yk

$$ Y_{k} \triangleq {\text{I}}\left( \cdot \right)^{ - 1} . $$
(2)

Therefore, the information filter can be extracted a reliable position from the multi-modal sensor data. Finally, we can estimate the accurate position for security robots.

3.3 Experimental Results for the Proposed Localization Method

In order to show the usefulness of the proposed method, a salient dominant vertical structure was extracted from the heavy rain images. The Fig. 4 shows the results of extracting valid salient features by using information filter. In Fig. 4(a), the image shows a result in the good weather condition. Figure 4(b) is the start image in the heavy rainy day. The third image is a result without filter, as shown in Fig. 4(c). Finally, Fig. 4(d) shows the result image with the proposed information filter (e.g. blurring, lens flare effect repression). Table 1 shows that the salient feature extraction rate improves.

Fig. 4.
figure 4

The results of information filter: (a) sunny day, (b) rainy day, (c) result without filter and (d) result with filter.

Full size image
Table 1. The comparison of salient feature extraction rate for weather conditions.
Full size table

Figure 5 shows the results of applying the proposed method to 3D LiDAR data in a factitious rainy situation. Figure 5(a) and (d) show the start image and final image in a sunny day, and there is no position error. In Fig. 5(b) and (e), the images show that localization error is caused by the raindrops. Finally, the results of our method can reduce the error and obtain a valid location, as shown in Fig. 5(c) and (f). Therefore, the precise location in clear weather loses its accuracy in rainy conditions, but it can be seen that the proposed method is capable of reliable localization by elimination feature error, as shown in Table 2.

Fig. 5.
figure 5

The result of the proposed method: (a) sunny start location, (b) rainy start location without filter, (c) rainy start location with filter, (d) sunny final location, (e) rainy final location, and (f) rainy final location with filter. (white points are localization error)

Full size image
Fig. 6.
figure 6

The robot model on a slope.

Full size image
Table 2. The comparison of 3D LiDAR feature error rate for weather conditions.
Full size table

4 Safety Path Generation for Human Computer Interaction

4.1 Robot Modeling for Overcoming Rollover

The rollover of the mobile robot is determined by the gravitational force in the ramp and the centrifugal force generated in the turning process of the mobile robot [20]. Figure 1 shows a graphical representation of a turn at a ramp and the resulting force. At this time, the case where the robot is rolled over is expressed by Eq. (3),

$$ vw = g\theta + \frac{gd}{2h} $$
(3)

where v is velocity of robot, w is angular velocity of robot, θ presents the angle of slope. This equation represents that the rollover is dependent on v, w, and θ. Moreover, in this equation, only the occurrence of rollover is known, and the risk of rollover is unknown. Therefore, it is necessary to quantify the risk of rollover using ZMP (zero moment point). This model is modified to introduce the ZMP concept, as shown in Fig. 7.

Fig. 7.
figure 7

The robot ZMP model on a slope.

Full size image

The risk of rollover of a robot can be expressed numerically by the concept of ZMP, and if ZMP model exists between the outer and inner wheels of the robot, it can be said to be stable against rollover. Furthermore, this ZMP model was represented, as Eq. (4),

$$ p = h\theta + \frac{vw}{g}h $$
(4)

where p is ZMP, even if it is the same size, which the risk of rollover depends on the robots should be normalized, as Eq. (5),

$$ s = \left| {\frac{2}{d}\left( {h\theta + \frac{vw}{g}h} \right)} \right| $$
(5)

where stable factor s must be between 0 and 1 to be stable, and if it is negative, it can be determined that rollover has been occurred. If the risk of rollover can be determined numerically, a stable path can be created.

The algorithm used in this paper is DWA (dynamic window approach). DWA calculates the cost according to each cost function in the area consisting of robot velocity and angular velocity set, and determines the most suitable robot velocity and angular velocity. This can be expressed by Eq. (6),

$$ G(v,w) = a(\alpha \cdot heading(v,w) + \beta \cdot dist(v,w) + \gamma \cdot vel(v,w) + \delta \cdot rollover(v,w,\theta )). $$
(6)

Figure 8 shows the experimental result through the algorithm applying the rollover model. The robot is turning in a slope, and the result is the same as the graph. In the graph, Rollover cost (sky blue color) indicates the risk of rollover. The results show that the value was 0.4 or higher when the risk of overturning was not taken into consideration, and the risk of overturning was lowered to 0.25 when the risk of overturning was considered.

Fig. 8.
figure 8

The experimental results in slope. (Colour figure online)

Full size image

5 Human-Robot Interaction by Following

In the security mission, the unmanned robot must perform the role of tracking or guiding the human. In particular, a mission to track a specific human is initiated by a tracking command received from an administrator, extracting a human region from an image obtained from a camera mounted on the robot. To do this, we employ a method for extracting human regions from the camera image and an interaction method for performing mission based on tracking method [21].

5.1 Human-Robot Distance Measurement

In order to perform various missions in the outdoor environment, the robot is equipped with a multimodal sensor module. The mission for extracting and tracking human regions in various camera modules conducts by RGBD cameras. When a tracking command is received from the control tower, the robot measures the distance of the tracking area from the depth image. The area from the center (xi, yi) of the region to the width and height of the region is divided as shown in Fig. 9(a) (green rectangle), and the depth of the region is accumulated in units of 1 m, and then the final distance D is calculated, as shown in Fig. 9(b) and Eq. (7).

$$ {\text{D = }}\frac{1}{N}\sum\nolimits_{i = 1}^{N} {d_{i} } $$
(7)
Fig. 9.
figure 9

An example of tracking: (a) distance calculation region, (b) histogram by distance. (Colour figure online)

Full size image

5.2 Human-Following Interaction

The interaction of tracking and following the robot human was performed based on the following scenarios. First, when the abnormal situation (extraction of the crowd density based on the threshold) occurs in the fixed agent, the robot receives the abnormality confirmation command from the control tower. Second, the robot moves to the fixed agent where the abnormal situation occurs, identifies the abnormal situation, and transmits the image to the control tower by distinguishing the person who is falling from the person standing. Next, the robot receives the specific human tracking command, and then starts tracking the specified human, as shown in Fig. 10. Finally, the robot carries out the below sequence to perform the human following interactions.

Fig. 10.
figure 10

Human following mission execution flow

Full size image

Human Following Interaction Sequence

  1. 1.

    Calculate the straight line distance between the robot and the center coordinates of the human from (xi, yi) and D.

  2. 2.

    Calculate the relative angle between the robot and the human.

  3. 3.

    Tracking Flag is executed when straight line distance is over 1.5 m.

  4. 4.

    Left/Right Flag is executed when relative angle is more than 15°.

  5. 5.

    When the Tracking Flag is executed, a linear velocity tv* is generated in proportion to the straight line distance to the object.

    *tv = (tv_gain)*(z_distance) The tv_gain is experimentally determined to be 0.2.

  6. 6.

    When the Left/Right Flag is executed, the rotation speed rv** is generated in proportion to the relative angle with respect to the object.

    **rv = (rv_gain)*(d_angle) The rv_gain is experimentally determined to be 0.35.

Figure 11 shows a result image of human following interaction.

Fig. 11.
figure 11

Human following interaction results: (a) control tower command, (b) drone view, (c) side view, (d) robot view.

Full size image

6 Conclusion and Future Work

Interactions between robots and humans are very important in outdoor unmanned security missions. Among the interactions, the task of following the human is one of the most active that the robot can perform on the control tower orders. In this paper, we propose a localization method robust to outdoor weather environment and a control method for safe driving in rough load for human following interaction. The results of the localization showed the robustness to the weather, and the control method proved to be safe path generation. Human robot interaction based on these methods showed usefulness to perform a given task, and it was found that robot could actively respond. Therefore, it is expected that the proposed methods can be applied to the security robot more practical.

In the future, we will add a snow removal filter to secure application to most robots by using autonomous navigation based on semantic information and driving techniques for various locomotion. Moreover, we are also developing a human following interaction method for collaborative tracking of multiple robots.