Insights into automation of construction process using parallel-kinematic manipulators

– This paper discusses challenges, experiences and lessons learned so far while transforming a masonry build system based mostly on manual labour into a robot automated build system. Our motivation for selection of this masonry process is to try out how robot automation could impact the architects in their design work by providing a tool to directly manipulate wall expression down to individual brick level. Such manipulation is often much too costly for manual labour today. Moreover, masonry is a challenging application to automate. Understanding the manual processes involved and transforming them into automation equivalents faces several challenges; among them handling and distribution of the different materials involved, selection of tooling, sensing for handling of variation and digital tooling for the programming of the process. A novel parallel-kinematic manipulator (PKM) with computerized numerical control (CNC) is used as target for experiments, because the performance properties in stiffness, workspace and accuracy will allow us to extend work into further construction processes involving heavy and dirty manual labour.


Introduction
Attempts at machines to perform automatic masonry have been tried from time to time. Even patents for bricklaying machines have already been announced in 1875 [1]. Despite this, bricklaying machines are not in common use today. There is one commercial machine available, the SAM100, offering automation of bricklaying for large straight building facades. Another commercial machine, Hadrian X, is usually also mentioned but it uses a build system with much larger bricks. A recent online article called "Where are the robotic bricklayers" [2] suggests several reasons why automation for masonry is not widely spread today. Among others, mortar is highlighted as a difficult material to handle whereby it is difficult to produce clean mortar joints, which also mirrors our experiences.
In general, autonomous machines and robots are not common in the construction industry. Several articles investigate reasons why: [3] lists lack of interoperability, design for human installation procedures, lack of tolerance management, power and communications as hampering factors. [4] and [5] list high initial investment and risk for subcontractors, immature technology, unproven effectiveness, lack of experts, low R&D budgets, among others. But there are indications that automation is needed in construction for continued growth [6]. Our interpretation is that digital, technical and regulatory infrastructure is lacking to lessen the effort of introducing autonomous construction machinery in the construction value chain. In the project, which this article is part of, we therefore work towards a model to bring business, technology and infrastructure together for bringing commercial application of autonomous robots and machinery closer to reality [7]. We also started up the Center for Construction Robotics [8] in 2019 as a forum between actors to meet and a test site for experiments on robot automation of construction processes.
The paper focuses on mapping the explained masonry process (2) into robot equivalents, including adapting a new developed PKM for construction processes (3) as well as performing and discussing experiments (4).

Brick masonry fundamentals
The fundament on which our experimental setup is built on, from a masonry point of view, are bricks, mortar, tools and process performance.
Bricks which are used in Sweden have different properties regarding dimensions, structure, surface, weight and colour. Most utilized in the Swedish construction business are standard sized bricks of the following dimensions: • 250 x 120 x 62 [mm] (Swedish brick) [10] • 228 x 108 x 54 [mm] (Danish brick) [10] • 240 x 115 x 71 [mm] (German brick) [11] Bricks, independent of their size are available as vertical coring brick, horizontal coring brick, solid brick, pre-wall solid brick and pre-wall vertical coring brick [12]. Most available colours are yellowish, brownish, reddish and blackish. The surface itself is rough, sometimes sandy, corny or dusty and the weight of solid bricks is about 2000 kg/m^3 [10] [11], whereby single solid bricks of the sizes mentioned in the list above weigh between 1.3 kg and 2.0 kg.
Though there exist many different types of bricks with all their dissimilarities used in the construction industry, they have one thing in common: Deviations in their dimensions. We experienced up to +/-2.5 mm in width and height along their surfaces (see Figure 1). The brick type we focus on in our experimental setup is a red Danish brick with 228 x 108 x 54 [mm]. Next to the described deviations the brick contains the following characteristics: Red Danish bricks having an upper and a lower side and a front and a back. To achieve the best possible wall impression, it is important to arrange the bricks in the same orientation along a wall to build. Moreover, they have a solid body and have fine dust on their surfaces. These parameters are very important for the automatization since they are influencing the tool design for handling the bricks as well as the choice of sensory. A solid body brick can for instance be handled with vacuum technology whereas a hollow brick needs clamping technology to be handled. The deployed vision technology (Realsense D435) features point cloud acquisition. In particular the point cloud measurements are matched towards brick geometry for brick detection and localization. The camera is fairly robust in the utilized pick situations. The algorithm uses thresholding in depth to differentiate between brick surface and background.

Manual brick masonry process
The manual masonry process for building a straight wall with bricks includes different action steps, logistics and use of special tools. To identify parts of the process appropriate for automatization we analyzed this process in detail in a workshop with an expert masoner. The process is divided into three main steps including preparation, performance and post-processing. Sub-steps for the preparation including blending the mortar, building a frame for building a leveled wall, putting a horizontal chord to define a specific height for each layer of mortar and bricks, brick-and tool supply. Sub-steps for post-processing includes grading vertical and horizontal joints, possible wall plastering and disassembly of framework. For building a layer of bricks (process performance see Figure 2) for a straight wall, we have set the horizontal stressed chord to a specific height. Thereby we defined a continuous height through all wall layers and achieved a regular build wall. For the application of bricks, we have put mortar with a defined dispersion on the prebuild brick layer. The mortar was applied with a tool especially designed for the needed mortar dispersion (Figure 3 (A)) for the used sizes of bricks. Thereafter the brick was applied with a specific application strategy. This strategy contains tilting the brick behind the wall along its longitudinal and lateral axis ( Figure 3 (B & C)), pushing it in this configuration on top of the mortar from the back of the wall, while tilting both axes back to "zero". Thereby the current placed brick gets aligned along the former placed bricks (  Through this application technology a vertical joint between two side by side bricks as well as horizontal joints between bricks of two sequenced layers are generated. Furthermore, this application strategy offers us to control the flow of excess mortar whereby the joints become as clean as possible. To increase the walls stability vertical joints between bricks of two sequenced layers needs to be displaced. For the displacement we used normal, half sized bricks, which we generated by breaking them with a special hammer (see Figure 4). In general bricks are used both as facing bricks and bearing brick structure, i.e., a complete wall. The different bricks are in each of these uses laid in different pattern. Both related to visual and physical reasons. In practice also the length of a wall decides how different individual bricks must be cut and placed. Different thickness of vertical and horizontal joints is used and several different recipes for mortar are available.

Wall to build
For the brick experiments presented in this paper, we designed a curved masonry wall ( Figure 5). The base curve of each layer of bricks is a planar sinusoidal curve, where the amplitude decreases with the height of the layer above ground. This shape offers the possibility to demonstrate the ability of changing brick placement.

Figure 5: Curved masonry wall
While having a robot being able to directly interpret the wall's design, we can precisely control the exact position and orientation of individual bricks with our robotic masonry system, which is, manually performed, very challenging. The bricks to set are individually rotated out of the tangent direction of the base curves. This individual rotation is proportional to the curvature of the base curve at the location of the current brick, see Figure 6. Thereby we create a wall with differently sized angles between side-by-side placed bricks, which causes differently sized vertical joints. For creating the 3D model of this wall and for calculating the exact position and orientation of each brick, the algorithmic modelling extension Grasshopper to the 3D modelling software Rhinoceros 3D was used, cf. [13]. Algorithmic modelling tools are essential for robotic masonry since manual specification of the orientation and position of each brick would be impossible in larger projects, e.g., in a complete brick facade of a building.
The 3D model can be used for visualising the finished appearance of the wall, but more important, the Longitudinal axis Lateral axis A B C D algorithmic model also exports the orientation and position of each brick to the robotic system to construct the steering code for the brick-laying robot, which is further described in 4.2.

Parallel-kinematic machines
In contrast to a standard arm manipulator where the robot links are arranged in a serial chain from the base of the robot to the tool, parallel-kinematic manipulators typically consist of a robot structure where several parallel links are attached to a common tool plate. The so-called closed kinematic loops together lock degrees of freedom for the position and orientation of the tool. The PKM's fundamentally different design allows for important properties like e.g., less moving mass and significant higher stiffness which may offer important benefits compared to standard industrial manipulators with respect to acceleration, positioning accuracy, structural rigidity with respect to process forces and e.g., footprint/workspace and complementary broadens the applicability and use of robots [14] [15]. The PKM configuration used within the project is based on a gantry frame and a novel wrist construction. It will be explained and its benefits within construction robotics for masonry operations and efficient use in Architecture, Engineering and Construction (AEC) applications, will be highlighted and further discussed in section 3.1.

PKM for brick masonry
In previous work "Parallel-kinematic construction robot for AEC industry" [16] our work-in-progress to adapt a PKM structure to automate a selected masonry process was presented. In the meanwhile, we have set up the PKM in the laboratory and equipped it with necessary hardware and software items to perform experiments.
The PKM is an eight-link parallel-kinematic manipulator that provides 5-axes continuous motion. There are six links that in three pairs connect the three carriages on the 4 m linear guides with the so-called support platform that positions the base for the robot wrist mechanism. Each of these six links have a fixed length of 2 m. By controlling carts on the three linear guides, the robot can perform translatory movement with the support platforms keeping a very stiff orientation. To provide stiff and precise tool orientation in two directions (tilting the tool, while keeping the third orientation stiff), two telescopic links are mounted between the upper and lower carts respectively. Together with a cardanic joint between the support platform and the tool platform, this results in controlled rotational motions around x-and yaxis. This type of machine provides a large singularity free workspace, high rigidity and precision, as also described in [17]. For PKM including its workspace see  To fit with laboratory conditions the PKM is presently mounted on a horizontal support structure. Though it is also possible to mount it in a setup for working from top for example. Figure 8 shows the PKM connected to a support structure in a brick masonry process. To adapt the PKM for the described masonry process, we designed a tool ( Figure 9) containing two motors allowing rotation around z-and y-axis. Moreover, the brick masonry tool consists of adapter plates and motors that transmit the movement generated by the motors into rotary tool motions via cross-roller bearings. The use of an L-shaped part allows the tool to point down in zero configuration. Furthermore, we decided to use a vacuum gripper consisting of three vacuum cups with 55 mm diameter each, equipped with filters inside to handle the dusty bricks and provided with foam to be able to create vacuum for gripping the rough brick surface. Flow regulation and control of vacuum gripper is realized with Avac injector MFE-300H-AS-1.
With a connected 10 m long and 8 mm diameter air hose and 6 bar pressure, we achieved 60 % flow. For easy and fast tool changing option RSP TC60-8 tool changer and RSP TA60-8 tool changing adapter are included in rotational y-axis.
For the mortar application process, we equipped the tool with an extruder mounted on the L-shaped part mentioned before and connected the extruder to a circular nozzle at its end (Figure 10 left). On the other end a connection for a hose is prepared which connects the mortar pump with the extruder.

Experiments
For performing experiments with the developed parallel-kinematic manipulator, we built the experimental setup described in 4.1., generated the brick data needed for the parallel-kinematic manipulators programming and focused on the challenges to solve (section 4.3). Furthermore, we performed experiments described in 4.4 to evaluate our proposed solution. Moreover, we list and discuss our results (section 4.5), including successful realization as well as obstacles which had been occurred and possible solution options, which offer the base for conclusion and future work (section 5) including Technology Readiness Level enhancement of the parallel-kinematic manipulator.

Experimental setup
Our experimental setup (see Figure 10) located in the Swedish National Center for Construction Robotics [8] contains the PKM equipped with the following peripherals: Described vacuum gripper tool to handle bricks, a palette with bricks placed on stacks for picking application, a double sized palette equipped with a flake board for placing application, a controller, a workstation, acrylic glass walls for safety during robot execution, process peripherals connected to tool for mortar application, computer and Intel RealSense Depth camera for vision integration.

Wall data generation & transformation
For creating the steering code for the bricklaying PKM, data on each stone is exported directly from the algorithmic 3D model of the wall (2.2). In this project, this data is exported in the form of an automatically created Microsoft Excel spreadsheet file. Apart from the orientation and position of each brick, this file contains a column specifying the layer, and consecutive number on this layer from left to right, of the current brick. It also specifies if the current brick is a full-sized or half-sized brick. Half-sized bricks are used in the current design for having straight vertical ends of each module of the wall.
To feed the parallel-kinematic machine, which is programmed in G-code based on the data exported as spreadsheet file, a semi-automatic tool chain is used to transform from wall description to executable G-code.

Challenges
Challenges we are focusing on within our experimental performance contain implementation of a stable process for brick handling and mortar application by building the wall described in 2.2. The processes subchallenges can be divided into: Pick bricks, move bricks, place bricks and apply mortar.
Though we focus in our application on one specific brick type, bricks are having, as experienced, deviations up to +/-2.5 mm in width and height, while having a dusty and rough surface. Our gripper decision is based on the fact, that it would be most convenient to pick directly from a palette. On the palette the bricks are placed with no space in between each other and their upper or lower side points outwards the palette. For this reason, we decided to design a tool containing a vacuum gripper to be able to easy separate the bricks from the palette. For performing pick experiments with the chosen gripper in combination with the bricks to handle, we started to place bricks in stacks on a palette. To pick a brick with the vacuum gripper we need to be close enough to the brick to get it connected to the gripper. In case we drive too close the very sensible foam (grippability in this solution depends on the foam being able to create enough suction force) placed at the end of the vacuum cups releases its connection to the cup which impairs the flow we need to create the vacuum, by what we are not able to create enough vacuum to grip the brick. In addition, to offer a continuous good flow we have a filter implemented in the vacuum cups, which we clean by blowing of several times before we pick a brick and after we have placed a brick. A blocked filter also decreases air flow, which decreases the needed vacuum.
Moreover, the deviations of the bricks are causing bending in the stacks and displacement of bricks (see Figure 11 (upper middle)). A calibration procedure matches the pick position in camera space with the corresponding position in robot space. Measurements during operation calculate the relative change in camera space to the calibrated position. The calculated relative change is then applied to the pick position in robot space.
Regarding cycle times and safety, we define speed and acceleration as fast as possible to not disconnect the brick from the gripper during acceleration and braking / emergency stop. We accelerate the bricks while connected to the robots' end-effector with 3.87 m/s 2 and moved them with a constant speed of 1 m/s.
In terms of placing, we investigate placing strategies for dry stacking and for stacking with mortar between each layer as well as general design limits of bricklaying. Since all bricks have deviations, the deviations get bigger with an increasing number of layers during dry stacking performance. For this reason, we decided to place the bricks a few millimeters over the last applied layer and let it drop on top. For placing bricks on a mortar layer, we had to figure out, if the dead load of the bricks is enough to get a good connection to the mortar or if we need to apply a defined pressure for placing the bricks on the mortar.
Due to process requirements and experiences from former experiments we decided to mix mortar glue according to the provided formula. By this we got the needed viscosity to get the mortar glue pumped through the hose as well as applied on the bricks. Furthermore, it is mandatory for a good process flow to apply the right amount of material with the right speed and the right consistency to achieve proper results. The aim is to apply pumpable mortar, which we are still investigating.
Finally, design limits, including wall instability, will be caused by external factors like 10 mm joint height between brick layers, dry behavior of mortar and wall design itself as well as by internal process concerning masonry robotics hard-and software.

Experiment performance
Our experimental performance is divided into two main parts. First, we investigate dry stacking of the wall described in 2.2. Second, we focus on stacking the wall with different mortar application strategies. The process for dry stacking bricks with the adapted PKM for masonry processes includes driving to a defined position over the stacked bricks, taking a picture of the next brick to pick and calculate the displacement regarding the reference brick defined in the process preparations. Afterwards, the PKM drives to the pick position and picks a brick. Thereafter the brick is moved to the placing position and placed with the defined tilt around the y-axis ( Figure 11). The wall we build with this handling technology is shown in Figure 12. For stacking with mortar, we investigate different mortar application strategies with the objective to identify a strategy e.g. without spill and without exposed mortar glue. Therefor we first focus on mortar application on top of a built layer by generating two paths, see Figure 13. Path A goes through the start point of the first stone, the midpoints of all stones and the endpoint of the last stone with a constant speed. Path B goes through all start-, mid-, and endpoints of every stone contained in the layer and is speeded up over the joints.
Since we investigate Path A as the best, we use this as the base to apply another layer of bricks on top, which is shown in Figure 14.

Results
Results we achieved by performing experiments with regards to the identified challenges are also divided into pick bricks, move bricks, place bricks and apply mortar.
The tool decision including two more rotational axes offer us to handle the bricks in the orientations and positions we need. Furthermore, with the chosen vacuum technology turned it out that the dustiness of the stones caused trouble, because it gets sucked into the air hoses and could, in long term, destroy the injector used for generating the vacuum. A proper solution could be to use another filter unit in front of the injector to offer a longterm use of this part. Furthermore, the vacuum cups turn out very sensible in case we drive too close to the brick while at the same time a good vacuum could just be created when we were close enough to the bricks. Implementing a force-torque sensor into the tool to identify the ideal distance between bricks and the vacuum gripper by control of pressure between these two objects would produce relief. Furthermore, the foam attached to the vacuum cups is very sensible. Investigating a Schmalz FQE-xb-120x60 with foam will be of interest.
Like mentioned the bricks to handle have individualized sides. The identification of the different sides requires the use of cognitive sensory integration into the process. Considering that the bricks need to be placed in a defined orientation and position to offer the best wall impression, a stationary vision system in combination with a further manipulator could be integrated into the setup.
The accuracy for picking the red Danish bricks could be enhanced demonstrably by use of the implemented vision system. The high variety of existing brick types opens the need to enhance the existing vision system for a more flexible future application.
Moving the bricks with the needed acceleration and speed worked very well with our configuration. In dependency on how close human and robot will work together in a demonstrated cooperation an additional gripping solution could be used when the bricks are moved by the robot to avoid disconnection of the brick during movement, which means to avoid possible injuries by a flying stone to a human for example. Cycle times which we achieved within our setup for continuous brick handling are 145 bricks / hour, with time divided between pick (7 s / brick), place (4 s / brick) and travel between these positions (14 s / brick). Cycle times for manual brick handling are 300 bricks / hour, excluding breaks.
A possible force-torque sensor would also be very useful to enhance placing of the bricks. For dry stacking we would be able to handle the deviations in height by an appropriate control. For stacking bricks on a layer of mortar the force-torque sensor could help to push the brick with a predefined force into the mortar up to a predefined position and orientation, which adapts the stressed chord from the manual equivalent.
The application of mortar has different constraints. For applying mortar on a layer of bricks with a continuous flow and a constant dispersion we need, besides a specific material consistency, also a defined distance of 2 -4 [mm] between the nozzle exit and the bricks. For path adjustment in height visual depth cognition could be used.
For keeping the needed mortar consistency during experiments, we blended it with a hand blender, in a barrel connected to the pump, frequently. With regards to automation, a solution with continuous blended mortar in a mortar blender and an additional mortar supply mechanism at the end effector in combination with an on purpose-built nozzle could enhance the process quality. Since a lesson learned is that the mortar material needs to be adapted to the robotic process, we are now starting up collaboration with concrete suppliers partly based on our experiences from this work Our decision to apply mortar along path A ( Figure 13) is based on the fact, that this path overlaps almost with the layer of bricks it is applied on as well as with the layer of bricks which is applied on the mortar (see Figure 14). Apply mortar along Path B ensures less overlapping with the layer of bricks it is applied on, as well as with the layer of bricks which is applied on the mortar path. For enhanced overlapping of mortar and bricks a mathematic model could be used to calculate the best performance mortar path.
Even if the curved wall design used in this project is more stable than a totally straight counterpart, improvements on the stability can, of course, be made. Stability would, for example, be greatly improved by making the wall two bricks deep and placing bricks in the wall which connects the two layers. This is a traditional way of creating a load-bearing masonry wall. The individual rotation of each brick is also introducing instabilities in the wall, both during construction and in the completed wall and while designing the wall we must be convinced that the forces applied on each brick will not push it out of place. It must also be checked that the down-facing side of each brick rests to at least 50 % of its area on bricks in the layer below. This is essential for the adhesion of the current brick to the bricks below to guarantee stability.
Wall assembly and transportation of prefabricated walls are issues, which are currently performed manually. Improvements remain, at this stage, open issues.

Conclusion & future work
Within this paper we have shown our recent work-inprogress in terms of testing, for the brick masonry process adapted parallel-kinematic machine, to Technology Readiness Level 3 -4. Since we have validated our predefined assumptions through dry stacking of bricks as well as through stacking bricks with mortar our investigations are highlighting the potential of parallel-kinematic manipulator's use in construction robotics. This lays the foundation for former explorations including enhancement of future process performance and parallel-kinematic behaviour.
Future work contains implementation of improvement opportunities into the current setup. This includes content, discussed in 4.5 and will mainly focus on further sensory integration for brick handling (pick, place, move) as well as enhancements of mortar application tool-design and mortar application strategies. Furthermore, we will focus on digital chain improvement to generate executable G-code directly out of the used CAD environment. Moreover, we will implement safety and interaction issues for enabling the PKM to be used for prefabrication processes close to construction site with the aim to bring the PKM up to a higher Technology Readiness Level. Our idea to use the PKM on the construction site itself, is to mount the arm system on site onto specific, application oriented support structures.

Acknowledgement
The authors gratefully acknowledge the support received from VINNOVA/"Innovativ Agile Construction for globally improved sustainability" (ACon 4.0 #2019-04750) as well as the support received from Robert Larsson (Cementa AB), Petra Jenning (Fojab), Cognibotics AB, Miklos Molnar (LTH) and Fasadgruppen Group AB. These projects provided the opportunity to investigate the use of parallel-kinematic manipulators in construction applications and to develop a parallel-kinematic manipulator for specific construction tasks.