1 Introduction

According to the World Health Organization’s world report on children injury edited by Pedan et al. (2008), the leading cause of child death is unintentional injury. In Japan, the Emergency Transportation Report prepared by the Tokyo Fire Department (2018) points out that over ten thousand children of 0–5 years of age are subject to emergency transport every year. This number has not changed greatly for several years, which indicates that current child accident prevention measures are insufficient. In addition, fall accidents are the most frequent type of accident among children. As such, it may be necessary to take countermeasures to prevent fall accidents.

Various approaches have been taken for child injury prevention. As effective methods for preventing accidents involving children, the World Health Organization has recommended six measures: legislation and regulations and their enforcement, product modification, environmental modification, supportive home visits, promotion of safety devices, and education and teaching of skills (Pedan et al. 2008). A great deal of research has been devoted to the investigation of accident prevention through human supervision, e.g., as dealt with by Schnitzer et al. (2015) and Morrongiello et al. (2016). However, among these recommendations, environmental modification, which is often referred to as a passive strategy, is highlighted as one of the most effective countermeasures in home injury prevention.

In the present study, we attempt to establish a method to adapt environmental modification approach to fall accident prevention. In order to prevent child fall accidents, we herein consider how fall accidents occur. Child fall accidents are thought to occur according to the following process: a baby climbs to a height through a product (e.g., a chair, a desk, or a bookshelf) and then falls from a high location by losing balance. If we can predict the climbing behavior that leads to a fall accident, the location of the risk of a fall accident can be determined. The parents can be notified of the risk in advance, and the daily living space can be improved so that the child cannot climb to dangerous high locations. Figure 1 shows an overview of a new approach to accident prevention at which this research is aiming. As shown in the first scenario in this figure, we use simulation results to help create a safer environment when people try to improve their daily living space. In another scenario, real-time monitoring climbing behavior based on simulation could be useful in the complementary supervision of children. In the field of elderly safety, several systems for fall detection has been proposed as a new application of Internet of Things (IoT), e.g., as proposed by Makhlouf et al. (2019) and Gracewell and Pavalarajan (2019). If daily environment gets to have cameras or other IoT device, real-time monitoring scenario is becoming feasible.

Fig. 1
figure 1

Overview of the accident prevention approach enabled by this research

Full size image

However, predicting how children will interact with products in everyday life situations remains very difficult, even for advanced digital human models, i.e., behavior simulation systems, since children significantly change their behaviors depending on their complex muscular and cognitive performance. In addition, the environment around children, including the layout of consumer products in their homes, constantly changes. An effective means to simulate children’s interaction with products in everyday life situations is to perform simulations using measured data obtained from behavioral observations.

On the other hand, deep learning has made great progress in recognizing human postures from images. DeepPose developed by Toshev and Szegedy (2014) and OpenPose developed by Cao et al. (2017) are typical examples. Using these technologies, it has become possible to measure a child’s natural posture without imposing a burden on the child, such as attaching markers or sensors. Thus, recent advances in artificial intelligence, image processing technology, IoT, and other sensing technologies have allowed realization of this data-driven approach. The present paper proposes a new method for predicting human behavior by formulating behavior prediction as a path planning problem. To this end, we use a path planning algorithm known as a rapidly exploring random tree (RRT) in the field of robots (cf. Berenson et al. 2009; Lavalle 2006). In the present paper, we refer to the proposed AI/robotics-based environmental modification support approach as an environmental smartification approach.

The concrete purpose of the present study is to develop a data-driven simulation that can predict children’s climbing behavior in a daily living environment by integrating behavioral recognition technology (OpenPose) and the RRT path planning algorithm. In the present paper, we first report the development of two kinds of systems for acquiring the three-dimensional (3D) coordinates of the posture of a child from images; one is images from RGB-D and the other is images from two cameras. In addition, we describe an experiment in which the climbing posture data of a child are acquired using this system. Finally, we report a method for describing posture data in a configuration space and a simulation method using the path planning algorithm on the configuration space using measured posture data.

2 Three-dimensional posture detection method for children and collection of climbing posture data

In order to develop a data-driven simulation, we first developed a database of children’s climbing behavior data. This database was developed in three steps. First, we estimated the posture of a child and obtained this information as data. We then performed an experiment with children to determine how they interacted with a climbing apparatus. Finally, we generated a configuration space using the posture data obtained from experiments; the configuration space is used for climbing behavior simulation later. This section describes these steps.

2.1 Development of a three-dimensional (3D) posture data detection system

We developed a system that collects 3D posture data from color images and depth images using an RGB-D camera (Microsoft’s Kinect). We subsequently used the pose recognition software (OpenPose) developed by Cao et al. (2017) to acquire posture data, because the accuracy of the Kinect’s posture detection algorithm decreases when a person touches an object. OpenPose allows us to extract the two-dimensional position of 25 parts of the body, such as the hands, feet, shoulders, and knees, from single images. Using the depth data from the Kinect and posture data detected by OpenPose, the developed system obtains 3D coordinate data of a detected person’s posture (Fig. 2). The system processing procedure is presented below.

  1. (1)

    Map a color image on the corresponding depth image. Acquire a 512 × 424 pixel color image.

  2. (2)

    Estimate the posture with OpenPose for the image acquired in (1), and acquire the two-dimensional coordinates of the human posture in the image.

  3. (3)

    Convert each posture coordinate value into 3D coordinates by searching for the corresponding points of the depth image and the two-dimensional coordinates obtained in (2).

Fig. 2
figure 2

System overview for obtaining 3D posture data by Kinect and OpenPose

Full size image

As described in the next subsection, we found the above system had occlusion problems stemming from a single view of RGB-D camera. Therefore, we developed another type of system that collects 3D posture data from color images of two cameras to improve the occlusion problems that the above system has. Using the direct linear transformation (DLT) method (cf. Abdel-Aziz et al. 2015) for posture data detected by OpenPose obtained from two cameras, the developed system obtains 3D coordinate data of a detected person’s posture (Fig. 3). The system processing procedure is presented below.

  1. (1)

    Multiple points with known coordinates are measured with two cameras for calibration.

  2. (2)

    Measurement is started after a light emitting device emits light for synchronization.

  3. (3)

    After the measurement is completed, the two-dimensional coordinate values of the calibration points viewed from the two cameras are obtained.

  4. (4)

    OpenPose is applied to the video measured in the synchronized state according to the light emission in (2), and the two-dimensional coordinate value of each joint are obtained.

  5. (5)

    Using the coordinate values obtained in (3), the two-dimensional coordinate values of each joint obtained in (4) are converted into three-dimensional coordinate values by the DLT method.

Fig. 3
figure 3

Example of 3D posture data obtained by two cameras, OpenPose and DLT

Full size image

2.2 Experiment for collecting posture data

In order to construct a database of children’s climbing behavior using the proposed system, we conducted an observational study on how children interacted with a climbing apparatus (see Fig. 4). Fourteen children aged 20–58 months participated in the study (two 1-year-old, six 2-year-old, one 3-year-old, and five 4-year-old). The experiment used a construction-type jungle gym was performed by Oono et al. (2017) in the living laboratory constructed by Nishida et al. (2017). We thought that it would be easy for children to climb the jungle gym, which we assumed would occur frequently. We observed climbing behavior for various shapes. A total of 184 data points were extracted using the system using RGB-D camera described in Sect. 2.1, and these posture data were compared with the color images to check whether the data were correct. It was confirmed that 141 posture data points corresponding to 76% of the total were valid. The rest 43 data points were excluded because the occlusion problem occurred; an object between the camera and the child prevented extraction of the correct depth value. The experiment was designed to be interrupted or canceled by the research participants at any time, even during the experiment, and was reviewed and approved by the ethics review board of the institution to which the author belongs.

Fig. 4
figure 4

Observation of behavior during climbing a jungle gym

Full size image

A new experiment was performed to increase the number of data and the diversity of objects. The new experiment was conducted using the improved measurement system that utilized the two cameras to improve the occlusion problem. Twenty-one children aged 2–5 years participated in the study (six 2-year- old, seven 3-year-old, seven 4-year-old, and one 5-year-old). The experiment used block playground equipment of various heights (30, 40, 50, and 70 cm) and 110 cm shelf and fence as shown in Fig. 5. This equipment was created with reference to the shape and dimensions of furniture often found in the daily living space such as a stairway, a bookshelf, and a fence in balcony. In this experiment, 3D posture data was obtained by the measurement method using the DLT method described in Sect. 2.1. The experiment was designed to be interrupted or canceled by the research participants at any time, even during the experiment, and was reviewed and approved by the ethics review board of the institution to which the author belongs.

Fig. 5
figure 5

Observation of behavior during climbing the playground equipment created considering the shape and dimensions of furniture often found in the daily living space

Full size image

2.3 Configuration space generated from posture data

The top side (a) part of Fig. 6 shows the configuration space generated using the data obtained from the experiment at the jungle gym described in the first half of Sect. 2.2. Berenson et al. (2009) applied the concept of the configuration space into a path planning task of a robot. In this paper, we utilized the configuration space to describe the possible variation of climbing posture. For Fig. 6, the configuration was visualized using a low-dimensional compression method, t-distributed stochastic neighbor embedding (t-SNE) developed by Van der Maaten and Hinton (2008), because the original posture data are high-dimensional data. We found that the configuration space changes with the age of the child. The top side (a) part of Fig. 6 shows that the distribution of posture data existing on configuration space differs depending on age. The bottom side (b) and (c) parts of Fig. 6 show respectively normalized posture data and color images when the child engages in climbing behavior. The bottom side of Fig. 6 shows the relation between the data sampled in the configuration space and the data in the corresponding posture images. By searching the configuration space, we can determine which posture a target-age child can take in a data-driven manner without a generative human model for considering the center of gravity, physical muscle power, and the cognitive aspects of the child.

Fig. 6
figure 6

Top side: Visualization of the configuration space of normalized posture data. Bottom side: Relation between the configuration space and climbing posture images

Full size image

3 Data-driven climbing simulation

As mentioned in the introduction, in order to prevent child accidents, it is necessary to design products and the environment while predicting the possible actions of the child. Accidents may occur not only with a single product but also in a situation in which multiple products and surrounding environments are combined, such as a sofa and a chair placed by a window and a desk. Therefore, there is a need for a simulation system for predicting the behavior of children in a daily living spaces. In the present study, we developed an algorithm for predicting climbing behavior using climbing behavior data by applying a motion planning method called the RRT (cf. Berenson et al. 2009; Lavalle 2006), which can search for a path at high speed in a high-dimensional space and a configuration space that is often used in robotics. The details of the algorithm are described in the next subsection.

3.1 Algorithm for data-driven climbing simulation

As described above, two functions are required for the simulation system in a daily living space. The first function was implemented to predict a child’s climbing behavior from actual measurement data. The simulation algorithm is shown below.

figure a

Algorithm 1 performs a simple iteration of extending T by adding vertices that extend in the direction of each new point randomly selected by RandomConfig. This situation is shown on the left-hand side of Fig. 7. The NearestNeighbor function (see Algorithm 1 line 4) works to find the nearest vertex to qrand in T. The CollisionFree function checks for a collision by stepping along the interval between arguments. Algorithm 2 searches the configuration space while taking into account the posture in the previous step to determine which posture can be taken at qnext. This situation is shown on the right-hand side of Fig. 7. The SearchConfigurationSpace function (see Algorithm 2 line 2) works by searching for posture data of a child of the age decided by the argument of postureold in the configuration space described in Sect. 2.3 and returning them as a list. However, the currently implemented algorithm does not consider the distance between posture data in the configuration space when searching for posture data in the configuration space. In order to judge climbability in Algorithm 2 line 6, we use the following conditions: the posture in which one foot is on a horizontal plane or one hand is on a horizontal plane and one foot is on a vertical plane or a horizontal plane.

Fig. 7
figure 7

Left-hand side: Illustration of Algorithm 1. Right-hand side: Illustration of Algorithm 2

Full size image

3.2 Data-driven climbing simulation in simple environment

First we demonstrating the developed algorithm using a virtually created environment. We set a simple stepped environment in a 3D space and simulated whether the stepped environment could be climbed to the top. For the simulation, we used the data obtained from the jungle gym experiment described in the first half of 2.2. The details of the set environment are shown in top side of Fig. 8. The first and second steps were set to 34 cm, and one side of the jungle gym was used in the experiment of Sect. 2.2. Environments 1 and 2 were created to have different levels after the third step, and simulations were performed for each environment, as shown in bottom side of Fig. 8. Climbing to the top of Environment 1 was simulated through the third step. The top step is approximately 2 m from the floor, and if there is a window at the top of the environment, there is a risk of an extremely severe fall accident. Although the author could not climb to the top of Environment 2, the author was able to simulate climbing to the second step. In this example, since the height of climbing for Environment 2 is lower than that in the simulation of Environment 1, Environment 2 is judged to be safer than Environment 1.

Fig. 8
figure 8

Top side: Two environments used in the simulation. Bottom side: Simple simulation system of predicting climbing behaviors

Full size image

3.3 Evaluation of data-driven climbing simulation

Next, in order to evaluate the validity of the prediction system, we compare the developed climbing behavior prediction system with a statistical model related to climbing behavior. Conventionally, any versatile simulation system for climbing behavior have not been developed. However, as for hexahedron, a statistical model for evaluating the probability of climbing occurrence was created by Nomori et al. (2009). So we compare the developed system with the statistical model in order to partially evaluate the validity of the developed prediction system. The conventional statistical model allows us to calculate the probability of climbing behavior from the height and width of the hexahedron and the age of the child. Table 1 shows the probability of occurrence of the climbing behavior in the computational model and the success or failure of the climbing behavior in the developed system. In both cases, the child was two years old. The table indicates that the results of the developed system has similar tendency comparing with those of the statistical model; the developed system judged success of climbing if the height was larger than 50 cm and the conventional statistical model predicted low probability less than 0.4.

Table 1 Comparison between the computational model and the developed system
Full size table

Therefore, it is considered that the developed system has given reasonable results for the purpose of supporting safe environment design. Unlike the conventional statistical model, our system can not only predict the climbing occurrence but also visualize climbing behavior described in the next subsection. So it has the possibility that parents, interior decorator, children’s nurse can know which products children can climb in advance to make environment safer.

3.4 Data-driven climbing visualization in complex environment

In this subsection, we describe a system of climbing simulation for 3D shape data of a daily living space. We developed a simulation system using Unity for 3D graphics. The algorithm described in Sect. 3.1 is used for climbing simulation. We created a 3D model of the living laboratory constructed by Nishida et al. (2017) using photogrammetry, which generates a 3D model of an object from multiple photos (cf. Abdel-Aziz et al. 2015). Figure 9 shows the living lab and 3D model considered herein. Figure 10 shows an example of the simulation results for climbing a low table. Thus the developed system can visualize how a child climb a target object in daily living space using a large amount of posture data.

Fig. 9
figure 9

Left-hand side: Living lab. Right-hand side: 3D model of living lab

Full size image
Fig. 10
figure 10

Example of visualization of climbing process from the floor to a low table

Full size image

4 Conclusion

In the present study, as a new approach to home safety, in particular, childhood fall prevention, we developed a data-driven method for preventing child fall accidents in a daily space. The system consists of a children's climbing behavior database created by RGB-D cameras (Microsoft’s Kinect) and a posture recognition system (OpenPose) and DLT method. The simulation system predicts which objects a child can climb based on the climbing behavior data and a motion planning method using the RRT algorithm. We evaluated the fundamental functions of the developed system by conducting a climbing simulation using simple environmental data and complex real living space data.

In the future, based on the fundamental functions of the system, we plan to install the developed system in real living spaces, such as ordinary homes and kindergartens, in order to accumulate more 3D posture data in living spaces in which a variety of objects are placed.