Keywords

1 Introduction

The architecture, engineering and construction (AEC) industry is one of the least digitized industries [12] and only slowly embraces paradigms, such as Building Information Modeling (BIM). BIM emphasizes the processes and technologies to create a digital model, that represents the physical and functional characteristics of the project. In its present form, BIM targets mainly the building, but as Choi et al. [6] point out, the work space and the building environment is an important resource for managing a construction site, as well. The introduction of environmental information requires integration of BIM and Geographic Information Systems (GIS), which however brings new challenges, as GIS and BIM follow different paradigms [1].

Moreover, such systems are limited by their static representation of a building and its environment [13], as they lack any automatic information flow from the construction site to the GIS+BIM model. This synchronization between real assets and their digital model is a key feature of a Digital Twin (DT). The concept of DTs describes a virtual representation of a physical object, and the corresponding flow of data between these two parts [10]. It is a major part of the Industry 4.0 roadmap, already embraced by other industries such as manufacturing and production, whereas its development in AEC is still in infant stages, as Deng et al. [7] state. The majority of current research is exploring the integration of BIM models and sensors, but their focus lies on building related topics during the operation phase, such as indoor hazards monitoring, thus again leaving out the building environment during the construction phase.

This paper presents a methodology for establishing and operating a DT of the environment of construction sites. Inspired by GIS applications, it uses geodata to initialize the DT and employs mobile construction machines to create information flow from the environment to the DT. This methodology is then tested on a local test site.

2 State of the Art

The majority of current research does not directly consider the environment of construction sites, but touches on it in efforts to automate site monitoring. In their literature review, Boje et al. [2] gather several examples, such as the use of drones or laser-scanning to capture and save changes in construction status to a BIM system. Although there is progress in automating the integration of captured data into the chosen building model, the act of operating the drone or preparing and capturing laser scans is manual. Moreover, all examples focused mainly on the constructed object and provided only visual evolution.

Xu et al. [15] employed already existing construction site models to solve multi-objective dynamic construction site layout problems, but considered only facility or process related features, such as safety and environmental hazards posed by a facility. Song et al. [14] developed an automated tool to calculate optimized equipment travel paths using the site layout within BIMs, however they assumed a 2D flat surface and square obstacles without consideration of elevation or ground properties.

One exception is the work by Cheung and Lin [5], which assessed the level of hazardous gases around the construction site using a Wireless Sensor Network. Although this idea represents dynamic updates of the environmental state, only a single attribute is monitored without integration into a more extensive model of the environment. Similarly, Arroyo et al. [1] examined the use of geological shallow subsurface data for construction and design applications, which is only a limited subset of properties of the construction environment.

This analysis indicates that the area of DTs for environmental modeling of construction sites has not yet been extensively explored, as the current focus lies mainly on the constructed building, without taking the rest of the site into consideration. Additionally, there are hardly any automatic approaches for updating the state of the environmental model, which is required by the DT paradigm.

3 Methodology

This section will provide a closer look into the steps of the described process in Fig. 3 and their implementation in the preliminary study currently being carried out on a local test

Available Geodata

Geodata (or geographic data) is data that can be referenced by a location relative to the Earth. Two most common types of geodata are rasterized and vectorized data. The former saves values like height, e.g. in digital elevation models (DEM) (Fig. 2), or color, e.g. in digital orthophotos (DOP) (Fig. 1), on a regular grid. By georeferencing the ‘origin’ and defining the resolution of the grid, every cell can be easily accessed by its geographic position. The latter type explicitly defines the geometry of objects by describing them with geometrical primitives. Every vertex is then assigned geographic coordinates. This work suggests using following geodata, depending on its quality and availability: Digital Surface Models (DSM), Digital Elevation Models (DEM), Digital Orthophotos (DOP), topographical maps, geological maps, city maps and building models. Such data is usually accessible via public databases, such as ‘OpenGeodata.NRW’, the official geodatabase of the government of North Rhine-Westphalia, that was mainly used in this work. The ‘Open Geospatial Consortium’, an authority on geospacial information, offers free and open standards for interaction with such databases via its ‘OGC Web Services’. Geodata can come in different formats, so for consistency and simplified interaction with the future Digital Twin, this work suggests transforming and combining rasterized data into a multilayered GeoTIFF file, an official OGC standard that offers functionalities for georeferenced raster data. A similar tactic is applied to vectorized data to transform them into the CityJSON format.

Fig. 1
figure 1

Digital Orthophoto and building model of the test site area. Building outlines are colored in pink and can be seen in the top right corner

Full size image
Fig. 2
figure 2

Digital Elevation Model of the test site area

Full size image

The described data offers a starting point for the DT. However public geodata sometimes lacks quality, especially its temporal resolution can be in the range of several years and the data is thus quickly outdated. Figures 1 and 2 illustrate this nicely, as they were taken before creation of the construction site and therefore show only an empty field. This issue calls for more frequent updates, ideally in real time, to create a true DT of the environment.

Live Data Acquisition

Mobile construction vehicles are a promising platform for gathering live data about the construction site environment. They frequently traverse and directly interact with it by e.g. excavating earth or transporting material. Moreover, modern machines are already equipped with different sensors. Most sensors are typically used for condition monitoring of the vehicle [9] thus measuring internal data and focusing on the machine, our case, however, focuses on the environment.

This situation calls for two solutions: inference of external, environmental data from available internal sensors and equipping the construction vehicles with additional sensors. This work implemented both strategies. A wheel loader, a typical mobile construction machine for moving material using a front mounted bucket, was equipped with following sensors: Inertial Measurement Unit (IMU), Global Positioning System (GPS) tracker, RGB camera, Light Detection and Ranging (LiDAR) sensors, pressure sensors of the hydraulic cylinders, stroke transducers of the hydraulic cylinders, wheel encoders, measuring the rotation of all four wheels. All measurements are indexed by their unique UNIX timestamps. Depending on the accuracy of the GPS sensor, it can be necessary to correct the measured positions. This can be achieved by fusing GPS and IMU readings as described for example in [3]. After this optional preprocessing step, a set of measurements, each with a unique time and clear geographical position, is ready for further analysis.

Fig. 3
figure 3

Overview of the proposed methodology for modeling the environment of construction sites via a Digital Twin (DT). First a DT must be initialized by using available geodata from public databases. Next, live data from the construction site is gathered by construction vehicles equipped with a variety of sensors. This data is then processed to update the DT with new information, such as changes in the surface model or transported material. The updated environment information can be further used to loop back into analysis of future data as well as be exported back to public databases. Finally, the DT assists the user through suitable interfaces in different tasks such as monitoring or planning and thus ‘acting’ on the real environment, closing the loop between Real and Digital Twin

Full size image

Data Analysis

An effective data analysis strategy is key to extract information from the previously gathered data. As construction progresses, the site experiences changes in material placement, object placement, soil conditions and the general surface model and thus these changes are of special interest for an up-to-date DT.

Surface Model The use of IMU and LiDAR sensors enables Simultaneous Location And Mapping (SLAM) based approaches. Such algorithms try to localize the robot (or machine) within its surroundings, while building a map of those surroundings at the same time. Exactly such 3D map of the environment allows for updates of the surface model. The generated point clouds can then be rasterized by placing them on a regular grid and taking the combination of height values of all points within a cell. This work applied the average height within every cell. Additionally, the differences in resolution of the point cloud and the grid can lead to some cells being empty, as points get sparser with distance to the LiDAR or the rays are simply obstructed by the roughness of the terrain. A straight-forward solution is to use interpolation to estimate missing cell values.

Terrain Condition Besides elevation, the type of terrain is important information about the environment of a construction site. One possible approach is done by Kurup et al. [11], who propose a support-vector-machine based algorithm to classify different terrains with features from camera images and IMU readings. Differences in vehicle speed and the rotational speed of its wheels indicates some form of slip. This information can again be combined with corresponding GPS readings to detect areas with difficult to traverse ground. Soil compaction is another terrain property that can be inferred from the movement and position of the vehicle around the construction site as the wheels exert pressure on the ground. All extracted information about the properties of the terrain is then added to the corresponding layer of the grid describing the environment (Fig. 3).

Material Transport Of course, the main changes in the environment are caused by excavation and transport of material performed by the machinery. Stroke transducers and pressure sensors yield the actuator position of e.g. the bucket of a wheel loader. Since the kinematics of the vehicle and its current geographical position are known, the location of removed or placed material can be observed. Volume and mass of the transported material can be then estimated from its physical properties and technical specifications of the used machine, e.g. its bucket volume. Detected material, e.g. a heap of topsoil, can then be described as geographic objects in vector format (see Sect. 3), containing its coordinates and attributes.

Fig. 4
figure 4

Example of a GeoJSON object

Full size image

Object Localization Finally, all objects that are not terrain or earth material also have to be considered, such as placement of fences or building material. One strategy is to use RGB cameras and LiDARs to detect and localize objects relative to the vehicle [4, 8]. This is, however, only tried out in context of autonomous driving and more datasets with objects from construction sites are needed. Localized objects should be saved in vector format. At the end of this data analysis step, environment measurements from construction vehicles are transformed into information in vector and raster form and ready to be incorporated into the DT.

Structure of Digital Twin

As discussed in Sect. 3, geodata is usually handled with two different formats, vector and raster data. The proposed structure embraces this distinction and splits the DT in two parts: A multilayered GeoTIFF file and a database for vector objects.

The GeoTIFF format is suitable for storing the raster part of the current environmental state due to its native integration of geospatial location, reference coordinate system and possibility to include multiple layers into one file. Since every layer shares the same grid structure, this format enforces consistency between different layers in regard to resolution and geographic location. The GeoTIFF file could contain the following selection of layers: DSM, DEM, red, green and blue channels of color data, terrain type map, soil compactness map, geological map. The second part of the environmental model stores all distinct geographical objects in a JSON document database. We propose using the GeoJSON format. Figure 4 shows an example for such an object. It consists of a set of user defined properties and primitive geometries referenced by geographic coordinates. In this case, it is a ‘fence’-type object in from of a line with additional information about its height. Together with the previously mentioned CityJSON objects (see Sect. 3), the DT needs only one database to manage all vector data, allowing for efficient and consistent access, searching and editing.

With this structure, the information about the current state of the environmental model can then be used to support future updates of the DT by combining them with new measurements. This information can also be exported back into public geodatabases improving and updating their data.

Interfaces and Applications

A complete and consistent DT acts as the single source of truth for the state of the construction site. Through a selection of suitable interfaces and representations, such as 3D models, maps or dashboards and statistics, the site operator is supported in their monitoring, controlling and planning tasks. This constitutes an indirect connection from the Digital Twin back to the Real Twin, as its state directly influences the actions of the user, who in turn changes the state of the real construction site through their managerial decisions. Possible applications can be optimization of route planning based on elevation and ground properties or editing the site layout after changes in the construction plan.

4 Experiments

The presented methodology was implemented in a preliminary study, using a wheel loader and local test site. During initialization, all available data sources depicted the site as an empty and flat area. The machine then performed several test drives around the area and carried out typical construction tasks. It was equipped with sensors described in Sect. 3 to emulate one update pass of the digital twin.

4.1 Surface Model

Figure 5 demonstrates the information gain from LiDAR recordings. The point cloud records the area in front of the machine. For better visibility, points are colored based on their height. At some point in time, a depression in the ground could be clearly detected ahead of the vehicle, indicated by the blue region, as well as some tall vegetation in the form of the ragged yellow-orange structures on the right. Neither of those were present in the initial surface model. The recorded point cloud was then rasterized, linearly interpolating empty pixels (Fig. 6) and used to update the corresponding area of the global DSM, as the current geographical position of the vehicle is known.

Fig. 5
figure 5

Recorded point cloud during wheel loader operation. Points are colored based on their height. The white symbol indicates the position of the viewer. A sink can be clearly observed ahead of the vehicle, indicated by the blue region. On the right-hand side, the rough and yellow-orange patches indicate vegetation

Full size image
Fig. 6
figure 6

Elevation map generated from the point cloud in Fig. 5 with 10 cm resolution. Empty grid cells were linearly interpolated

Full size image

4.2 Slip and Soil Compaction

Figure 7 shows a qualitative map of slip in the area traversed during operation of the machine. The measure of slip was defined as the difference in rotation frequency between wheels on mostly straight movement segments. Darker regions show areas with more slip. Figure 8 shows a quantitative map of soil compaction, which was caused by the pressure of the wheels of the vehicle on the ground. Additionally, crossing the same spot multiple times and the added weight of transported material were also considered in the estimation. Darker regions show higher level of compaction. The dark patch on the right side of the figure shows an area where the wheel loader piled up and subsequently removed a heap of earth. This new environmental information isn’t normally available and offers new possibilities for e.g. future route planning and is added to the corresponding layers of the DT.

Fig. 7
figure 7

Qualitative map of slip

Full size image
Fig. 8
figure 8

Qualitative map of created soil compaction

Full size image

5 Discussion

The experiments demonstrated potential for employing a DT in context of construction site environments. Each test drive can be seen as a single update pass from the real environment to the DT. In a real use case, these updates should happen continuously and near real-time during the whole construction process. The implementation of an automatic system for updating the DT needs to be developed and tested in further studies.

Constant updates provide the possibility to monitor the evolution of the construction site, as the DT can store its past states, which can be a powerful controlling tool. However, it raises ethical questions about surveillance of construction site personnel, like machine operators, since their behavior is indirectly recorded through their machine and thus also stored inside the DT. Another issue can arise in using the full potential of the DT and procure its latest geodata to update databases of public bodies. It is important to manage the distinction of confidential business data and data cleared for public use before their export.

The methodology describes only the essential structure of an environmental DT. Based on the use case, the proposed contents of the DT could be extended or further transformed to fit a certain application. Additional information, like the target layout of the site or safety thresholds on slip values, will enable detection of deviations in the current layout or provide insights for path planning, respectively.

6 Conclusion

We proposed a methodology for modeling the environment of construction sites via a Digital Twin (DT). After the DT is initialized with available geodata, it needs to be coupled with information about events happening right now at the construction site. Using construction vehicles equipped with different sensors, raw data about the environment can then be gathered. Next, this data is fused and analyzed to derive the desired information, such as an elevation model or soil condition. This finally updates the previously initialized DT. Every new set of measurements can then be combined with current information from the DT to enhance future updates. The cycle is finally closed with a range of applications, such as monitoring and controlling. The DT can ‘act’ on the real environment through the management and planning decisions of the user, supported by suitable human-machine interfaces. One update loop of this methodology was then implemented on a test site using a wheel loader. Different information, such as changes in surface model, regions with slip and soil compaction, have been derived from sensor measurements during vehicle operation and added into the existing environmental model of the test site.

Future research will include the implementation and evaluation of a fully connected infrastructure to automate the measurement, analysis and update steps. In combination with that, other sensors, such as radar or stereocameras, should be tested for deployment on construction machinery. In the long term, the extension of the environmental DTs to incorporate DTs of constructed objects and the construction machines themselves will offer new topics for investigation.