Robotic assembly of threaded fasteners in a non-structured environment

Over the past two decades, a major part of the manufacturing and assembly market has been driven by the increasing demand for customised products. This has created the need for smaller batch sizes, shorter production times, lower costs, and the flexibility to produce families of products—or to assemble different parts—with the same sets of equipment. Consequently, manufacturing companies have deployed various automation systems and production strategies to improve their resource efficiency and move towards right-first-time production. Threaded fastening operations are widely used in assembly and are typically time-consuming and costly. In high-volume production, fastening operations are commonly automated using jigs, fixtures, and semi-automated tools. However, in low-volume, high-value manufacturing, fastening operations are carried out manually by skilled workers. The existing approaches are found to be less flexible and robust for performing assembly in a less structured industrial environment. This motivated the development of a flexible solution, which does not require fixtures and is adaptable to variation in part locations and lighting conditions. As a part of this research, a novel 3D threaded hole detection and a fast bolt detection algorithms are proposed and reported in this article, which offer substantial enhancement to the accuracy, repeatability, and the speed of the processes in comparison with the existing methods. Hence, the proposed method is more suitable for industrial applications. The development of an automated bolt fastening demonstrator is also described in this article to test and validate the proposed identification algorithms on complex components located in 3D space.


Introduction
Industries are going through a transition from mass production to mass customization in order to satisfy different customer needs. The competitive environment has made industries to desire highly flexible and adaptable systems that are able to cope with never ending product changes [1]. Currently, flexibility in most industrial robotic applications is achieved through automatic change of end-effectors, efficient programming of robots, and the use of component-specific fixtures [2]. But for a system to be considered flexible, it should be able to deal with subtle variations in geometry and positioning of parts or sub-assembled components without fixtures and robot re-programming. In addition to these variations, the system should also be capable of assembling different or same families of products with minimal modifications. This will save the cost and the time required to develop component-specific fixtures for low-volume batch productions.
Threaded fastener assembly accounts for over a quarter of all assemblies carried out in industries [3]. Statistics show that out of the six million parts used in a particular aircraft assembly, half of them are fasteners [4]. This demonstrates the volume and variety of assembly processes in aerospace industry related to fasteners such as bolts and threaded holes. However, while fastening operations are typically simple for manual workers to perform, they could be very complex processes to automate. Many of the fastener assemblies at present are carried out by manual workers with tightening tools or by repetitive automation with complicated fixtures and dedicated part feeders. Manual fastening provides flexibility in assembling different products and adaptability to variation in product geometry and positioning at the expense of reliability (e.g. consistency in applying tightening torque). It is also a time-consuming and labour-intensive process. On the other hand, repetitive automation is reliable and faster, but lacks flexibility and adaptability. This emphasises a need for an intelligent non-structured automated solution which incorporates the flexibility and adaptability of a manual worker as much as possible, and also remains reliable and fast. Such a solution requires high accuracy in identifying randomly located fasteners and threaded holes in a defined workspace. It also needs to have an ability to manipulate the fastener and control the torque applied during fastening. In addition to that, the performance (e.g. cycle time) and the cost of the solution should also be considered.
The work reported in this article was aimed at developing an automated assembly solution for performing assembly of different size fasteners such as bolts, into their corresponding size threaded holes located in a random position within a defined workspace. The experiments were carried out in a laboratory environment on a test rig, which emulates real assembly processes of components with complex geometries, such as those typically found in aerospace industry. However, the reported work in this article focuses on the research exercise carried out as part of a larger research project. Further industrial work is required in order to deploy the solution in an industrial context and to guarantee the robustness and efficiency of the proposed solution in a real industrial application.

Review of the existing solutions
The potential of intelligent and flexible assembly concepts have been realised for a long time and several research works have been carried out in this area. These concepts stated the use of multiple robots and optimization of assembly cell arrangements with standardised, modular, and flexible assembly models. Tsarouchi investigated the use of a dual-arm robot for increasing flexibility in an assembly process [5]. The developments of assembly concept such as fully flexible assembly system (F-FAS), reported by Finetto [6], and hybrid flexible assembly systems (H-FAS), proposed by Rosati [7], deliver effective solution for improving small batch productions requiring frequent reconfigurations. However, as stated by Herakovic [8], modern industrial assembly processes need advanced robot-based object detection, recognition, and grasping techniques to assemble randomly positioned objects in unstructured environments. The recognition and localisation of objects located in random locations require 2D or 3D visual perception of the objects depending on the application. A significant amount of research has been carried out in implementing 2D vision system for flexible automated assembly in industries. Fei [9] developed a 2D automated assembly method for jigsaw puzzle assembly and also demonstrated its potential industrial application. This method can only assemble same shape objects placed at random positions and orientations. In 2010, Pitipong [10] introduced an automated robot screw fastening method. This method uses 2D vision system to track fastener alignment before and during the fastening process in order to prevent cross threading. However, the position values obtained are significantly affected by the surrounding environmental conditions and also by the surface finish of the objects.
The combined use of force, torque, and vision sensors for tight-tolerance applications has also been investigated. Chen [11,12], developed a strategy using 2D vision and forcetorque sensors for performing high-precision insertion of a piston into a valve body hole for the automotive transmission subassembly. The insertion follows a spiral pattern and the input from the force-torque sensor is used to determine the precise location of the hole for insertion. Lopez [13] developed an intelligent strategy to accomplish robotic fixtureless assembly using a 2D vision system and a neural network-based learning system. Vision systems and force-torque sensor were used to correct and learn error patterns occurring during various automated processes such as panel forming by Ilangovan [14] and automated polishing by Kalt [15]. Jayaweera [16,17] described a combined metrological and mathematical method to automate aircraft fuselage skin panel assembly. All the above reviewed methods using 2D vision are limited in degrees of freedom and therefore, they are suitable to specific applications.
The use of 3D vision systems to overcome the limitations of 2D systems has also been investigated. The robotic fixtureless assembly (RFA) reported by Bone [18] and the fixtureless assembly method developed by Fleischer [19] describe a vision-guided work cell for assembling automotive components without fixtures. The 3D positions of the components for alignment are identified by tracking the unique markings placed on the components. But the marking has to be visible to the vision system always and the physical location of marking on the component needs to be pre-determined, which is not an ideal solution for an unstructured assembly.
Bin picking solution based on a modified RANSAC approach [20] and a fast object registration method [21] have been developed. The former is more robust, but less accurate and it is also time-consuming. The latter is fast but only accurate to within ± 2 mm. The robotic bin picking solutions [22] are accurate and fast, but they use feature-based object detection method which requires the feature to be visible all the time to the vision system. Jong [23] developed a bin picking method for automotive sub-assembly automation, which identifies and localises objects by fitting primitives such as planes on the surface of the object. This method has a very high process runtime and also has large errors in estimating the object pose. The errors are more than 10 mm in translation and 1.5 in rotation. Also, they are suitable for components that can be decomposed into geometric primitives.
Several advanced methods for object recognition have been developed and used in industrial applications. 3D object recognition methods [24][25][26][27][28] can be broadly categorised into global and local methods based on the type of feature descriptors used for the recognition. The global object recognition methods suggested by Wahl [28] and Khoshelham [29] are more suited to detect primitive shapes such as planes, spheres, and cylinders from the input 3D data. These methods are slow and also require segmentation of overlapping objects. On the other hand, the local object recognition methods proposed by Chen [27], Aldoma [26], and Xu [25] use point to point correspondence between the reference model and the input 3D data to identify and estimate the pose of the objects. Local methods are prone to noise and occlusion, which affect the object detection and therefore less reliable for industrial applications.
The "model global and match local method" proposed by Drost [24] uses oriented point pair features for creating a global model descriptor and matches the model with the scene locally using a voting system. This method groups similar features of the model and compares with the scene to find the best match without coarse information, and therefore works well in occluded and noisy data. Thus, this method with an enhanced prior knowledge is used in this research, for detecting the fasteners located in clutter form. Despite the good level of accuracy and reliable object detection, this method is slow and hence unsuitable for dynamic manufacturing and assembly systems. In order to enhance the performance of this method, a modified CAD model is used for generating global model descriptors, thereby reducing the computation time. This method is referred to as 'surface matching technique', as it involves grouping similar surface point pairs for matching the reference with input 3D data obtained from the vision sensor. Work reported by Ogun [30] uses the surface matching technique to identify 3D objects in clutter to perform automated flexible assembly.
One of the major challenges in this research, identified based on actual industrial need, is to identify hole features located at multiple planes with chamfered edges and visible internal threaded edges. In addition to that, the hole feature geometry and its location in 3D space need to be determined in order to select the correct bolt sizes. There are very few 3D hole feature detection methods developed at present. The hole detection method developed by Wang [31] finds solid holes by detecting planes and their boundary relationship, which cannot be used in chamfered non-solid holes. The method to detect holes in point set surfaces [32] utilises the angle between the points for detecting edges, but the process is very slow (i.e. nearly 2 min for 400 K points). The feature detection methods based on RANSAC [33] and Hough transform [34] are computationally expensive and are more effective only in detecting planes. Therefore, as part of this research, a novel hole feature detection algorithm was developed to extract, identify, and localise the chamfered and threaded holes located in random positions. Based on the results obtained from experiments, the hole feature detection method proposed in this research is highly accurate and fast in detecting the chamfered threaded hole features in 3D.

Proposed solution
The work reported in this paper is focused on the assembly of metallic bolts into the corresponding size threaded holes, located on a target component in a semi-unstructured environment. Similar to methods used by other researchers, laboratory test scenarios were used to replicate industrial case studies, in which the required precision for identifying the sizes of the bolts and hole features is 1 mm, as the minimum difference between two standard metric bolts and holes used in this research. Also, the translational and rotational accuracies of recognition should be less than 1 mm and 1°respectively, to comply with the specifications required for industrial applications. Figure 1 illustrates the generic steps developed in automated non-structured assembly process.
A number of 3D imaging sensors were evaluated under different lighting conditions and surface finish, similar to the conditions occur in an industrial environment, in order to select a suitable imaging device for this research application. To improve performance, prior knowledge such as CAD models are used for determining bolts in clutter and a novel hole feature detection algorithm was developed to identify threaded holes. As illustrated by Fig. 2, a laboratory rig was designed and integrated to a robotic arm and a series of sensors for monitoring and control mechanisms. This is to complete the fastening process, while monitoring for the required force/ torque and the potential component misalignments such as cross threading that is likely to occur in manual operations.

Object and feature detection
The two major challenges involved in this research include identifying bolts located in clutter and threaded hole features with chamfer on the target component (i.e. fastening plate shown in Fig. 2) located in random 3D space. The bolts can be different sizes and may be placed one above the other as shown in Fig. 3. This scenario is defined to examine the ability of the proposed method in identifying bolts in an unknown 3D environment.
The threaded holes are located on the fastening plate at various complex angles. The target component with threaded holes will have chamfered edges, which overshadow the inner edge of the holes as highlighted in Fig. 4. The chamfer features are one of the reasons for low accuracy of the existing identification methods.
This section describes the novel algorithm developed for identifying chamfered threaded holes on the fastening plate and also the object detection technique used for identifying bolts located in clutter.

Hole detection-novel feature detection technique
The proposed feature detection technique starts with scanning of a section of the fastening plate. The information about the scanning area is extracted from the CAD data to improve the speed of the process. The output in the form of point cloud is stored as a 3D model, referred to as the 'hole scene'. This data represent the fastening hole with a chamfer and an internal thread section on the component, located at a random position and orientation. The hole scene data is resampled to make the points equidistant for further processing. The next step in the process is an innovative method to remove the points that represent the threaded section of the hole. This is to eliminate the negative impact of these points on the edge feature  detection process. This process is carried out by finding a surface normal (perpendicular axis to the surface of object) of all the points using the 'moving least square' approximation method. After finding the surface normal, it is then required to find a difference in angle between every point surface normal and the normal of its local neighbours. The point pairs, which exceed the set angle limit, are considered to be a threaded section and can be removed.
The proposed feature detection algorithm is used to identify the points that represent edge features in the hole scene. The resulting edge points may represent any sharp features such as the outer and the inner edge points of the hole, the planar edge points, and the noise points (i.e. unwanted points or discrepancies) present in the hole scene. The noise points in the data affect the classification of the edge features. So the isolated noise, which is not close to the edge features, can be removed by identifying a point or small group of points which have fewer neighbours within the set distance limit. The entire hole feature detection process is illustrated by Fig. 5.
After removing the isolated noise points, the remaining points are grouped into connected regions, only if the points are at close distance. This segmented group of points is used in categorising the features based on their sizes. The segmented edge feature points are analysed for outliers by identifying the points that have a larger distance than the average maximum distance between the points in the group. After removing outliers, the features which satisfy the circle equation are considered as a hole edge. Thus, the resulting circular edge features represents the total number of holes present in the hole scene. For each circular feature, the average of the largest distance between each point with other points is calculated to obtain the diameter. The centre point (x, y, and z) of each circle feature in 3D is obtained by fitting a smallest cuboid to   the circle and calculating its centre point. In order to determine the orientation in 3D space, a plane is fitted to the points and the orientation along all three axes (θ x , θ y , and θ z ) is calculated. The centre point and orientation along all the three axes together provide the final pose of the hole feature.

Edge detection by difference in gradient method
To improve the accuracy in line with the requirement for industrial applications, a new edge detection method was developed. This method involves finding the gradient differences between three consecutive points in the point cloud. If the difference is greater than a set threshold, the points are then subject to a series of conditions (i.e. Eqs. (3) to (5)) to be considered as an edge point. The conditions used include the different possible arrangements of points in a 3D flat and angled surface, and their relation with their immediate neighbouring points. The selection of points for processing is traversed along y-axis (direction of scanning) to cover the entire point cloud data. The algorithm eliminates most of the plane points at the initial gradient difference stage and only the points which have greater difference are subject to the remaining conditions to identify an edge point. Selecting the key points for determining the edge makes this algorithm considerably faster and significantly more accurate than the existing methods.
The gradient difference between any three consecutive points is given by Eq. (1), where '∇' denotes gradient, 'dz' denotes difference in 'z' points, and 'i' denotes index of a point. If the gradient difference between three points is greater than the set threshold (Eq. 2), then the points are subject to a series of conditions (Eqs. (3)- (5)). If a point satisfies any one of the conditions illustrated below, then the point is considered as a potential edge point.

Bolt detection algorithm
Bolts may be located in clutter at random location within the work area this is to experiment geometrical complexity in a near real case study. To identify bolts in clutter, an enhanced CAD-based surface matching technique is developed. The bolt detection process includes a pre-processing step, which provides prior knowledge to the system and a post-processing step for detecting bolts from the work area.

Pre-processing-preparing reference surface model
The pre-processing step is performed to prepare a CAD model of a bolt as a reference model as shown by Fig. 6. Preparing reference surface model for matching is then used to identify the bolts from the bolt scene. The CAD model which is in the form of triangulated surfaces defined by unit normal with the vertices of triangles is sampled at a distance of 0.5 mm, to convert the model into equally spaced points for processing. After sampling, a gripping region is identified from the CAD model. The gripping region is a section of the bolt which is used to grasp the bolt using the two-finger gripper designed for this experiment. A centre point of this region in 3D is the final position which needs to be sent to the robot to pick up the bolt. Thus, the coordinates representing the CAD model in its initial stage is then transformed to this final position. In order to reduce the computation time for the surface matching process and to concurrently maintain important features for efficient matching, the model is reduced to its top section (head section). This modification reduces the computation time by about 65%. To create the reference surface model for matching, the point to point relations such as distance between the points, angle made by the points normal to the line connecting them, and the orientation between their normal are mapped and stored.

Post-processing-surface matching technique
The post-processing stage involves obtaining the point cloud data of the work area, which contains bolts in clutter, referred to as the 'bolt scene'. In order to obtain an equidistant point model, the bolt scene is then resampled with the same sampling distance that has been used to create the reference surface model. The surface model created during the preprocessing step is used as a reference to search for the bolt inside the sampled bolt scene, using the surface-based matching technique, as depicted in Fig. 7. The basic principle of the matching technique is to use a set of key points selected from the reference surface model and try to match the model within the bolt. The result of this matching is a set of 3D poses of the surface model in the bolt scene coordinate system (i.e. scanner coordinate system) and a matching score which depends upon approximate number of key points from reference model matching the surface of the bolt scene. The final bolt

Non-structured assembly experiments
To investigate the potential enhancement made by the proposed identifications algorithms, an experimental demonstrator was developed to emulate fastening process in a near real industrial scenario. In the following sections, the proposed fastener demonstrator is explained and the results from the experiments using the developed algorithms are reported.

Experiment scenario
The experiment scenario proposed for 3D fastener assembly involves assembling different size bolts into the corresponding threaded hole in a semi-structured environment. The target fastening component is a plate mounted on a tilt table located at a random position. The plate has different size of threaded holes at multiple planes and the plate can be pitched at any angle unknown to the system using the tilt table and the top rotary plate, as shown in Fig. 8. This scenario represents an environment that is not structured for automated assembly, which is the case in many application domains focused in this research. In order to perform the assembly task, the bolts and the threaded holes have to be identified first from the work area. Then, the size of the bolt should be determined to facilitate fastening of the right size bolt into the corresponding size hole on the plate with appropriate torque as shown in Fig. 9.

Test rig setup
The test rig setup includes a Motoman SDA20 dual-arm robot, a Bosch Rexroth bolt tightening tool, a Micro-Epsilon scanControl 2900-50 laser profile scanner (resolution and accuracy of 100 and 190 μm respectively), a Schunk two-finger gripper, and a fastening hole plate mounted on a tilting table, as shown in Fig. 10. The computing power was provided with an Intel Core i7 processor with 3.40-GHz speed.
The scope of this research was initially restricted to hexagon head fasteners size ranging from M5 to M14 (grade 8.8 and coarse thread). The tightening device was equipped with a continuous and controllable rotating drive mechanism which is used to fasten the bolts with appropriate torque. The fastening plate is designed as the target component with different size threaded holes ranging from M5 to M14 at different angled planes (i. e. 15°, 30°, 45°, and 60°).
An extensive evaluation was carried out to select the most suitable 3D imaging technology that could be used for assembly in such semi-structured environment. The evaluation includes determining the accuracy of a known object under different lighting conditions and surface finish. Three different technologies were evaluated, including depth sensor with IR coded light projector, stereo camera with pattern projector, and laser profile scanner. The detail of the evaluation process carried out can be found in the related document disseminated by the author [35].

Assembly process
The automated non-structured assembly process involves acquisition of the 3D point cloud data of the work area, object identification and pose estimation, and performing torquecontrolled fastening. Figure 11 illustrates the process flow involved in the automated assembly of fasteners.

Calibration process
The consistency in the geometrical coordination amongst the devices used in the developed rig was of a paramount importance. Therefore, a calibration method was developed as part of this research to facilitate maintaining a singular reference coordination for the system.
The bolt pose obtained from the bolt detection algorithm and the hole pose obtained from the hole detection algorithm both are in the laser scanner coordinate system. The poses Fig. 13 Calibration plate with styli placed at different step heights Fig. 12 Coordinates of robot arms, laser scanner, and calibration plate need to be transformed to the robot coordinate system for the robot to manipulate the objects. This requires an external reference that is common to both robot tools and the laser scanner as shown in Fig. 12.
A reference object shown in Fig. 13 was developed as a common reference to be used for calibration. It is a multistep square plate with equidistant holes for mounting styli pointers with 1-mm-diameter ruby sphere at the tip. The multistep design facilitates calibration of the system at different heights (i.e. for defined 3D space). There are 15 styli at different heights ranging from 5 to 10 mm on the reference plate. The distance relation between the stylus mounted at the centre and the remaining styli was measured using a coordinate measuring machine (CMM). This is to obtain the precise distance relation between the stylus in x, y, and z in 3D space.
The 'H' denotes the transformation matrix and it can be represented by a 4 × 4 matrix (Eq. 7) where 'r' is rotational elements, 'T' translational elements, and 'P' point in space.
Equating each row of the matrix will give the following three equations (Eqs. 8-10) From the above equations, for each coordinate values of the 15 styli noted from the laser scanner and the robot, a total of 45 simultaneous equations can be derived. These equations are solved to obtain the rotational and translational elements of the transformation matrix required to transform the scanner pose into a robot R1 tool pose. The above mentioned steps are repeated for determining the transformation matrix required to transform the pose with respect to the scanner and the robot R2's tool pose.
Thus, the bolt pose obtained from the bolt detection algorithm is transformed into the robot R1 user coordinate for picking up the bolt by using Eq. 11 Similarly, the hole pose is transformed into the robot R2 tool coordinate for carrying out the fastening process. The transformation is performed using Eq. 12  Clearly, the accuracy of the calibrated transformation matrix depends upon the accuracy of the laser scanner and the robot.

Experiments and evaluations
Two series of laboratory experiments were carried out to evaluate the precision and repeatability of the proposed fastening processes in view of the industrial application requirements. The first experiment set was targeted at determining the geometrical precision of the proposed method, and the second set was focused on the success rate and the processing time of the actual fastening operations.
To examine the geometrical precision, i.e. translational and rotational accuracies, 4 batches of 20 assembly tests were carried out on the rig. For each batch, the fastening plate was pitched at different angles from 0°to 30°with a random rotational angle around its centre shaft. A series of metrology equipment was then used on the rig to measure the position of the threaded holes on the plate relative to the system's reference point within a ± 10-μm accuracy range. This process was independent to rig's components and was repeated for each batch size. The information obtained were used as a benchmark to evaluate the accuracy of the coordination points calculated by the developed algorithms. The experiments were considered successful, if the geometrical errors were within the defined application requirements, i.e. up to 1 mm for bolt sizes, and less than 1°rotational errors. The tests proved the system capability to obtain averages of 0.53 mm and 0.67°errors.
Following this experiments, the actual fastening process of the bolts was examined. Three batches of 20 tests were carried out on the fastening of M10 bolts to the corresponding threaded holes on the random located plate. The plate was randomly rotated for each batch of the tests and the force and torque of the fastening process (not the final tightening torque) were observed. This was to identify the potential forceful assembly or cross threading due to the geometrical errors. In this series of experiments, a test was considered successful if (a) the correct size bolt was picked up and the correct threaded hole was  identified, (b) the fastening process was completed without creating excessive force and torque, and (c) the identification process was completed within an acceptable time period, in accordance with the time available in an industrial application scenario. In total of over 60 tests, 98% of the tests were considered successful. The experiments are further explained in the following sections.

Evaluating the precision of the algorithms
The developed hole feature detection algorithm was used to determine the top edge of the chamfered threaded holes located on the fastening plate. Four holes of two different sizes (M10 and M14) were drilled onto the fastening plate on a single geometrical plane. The size of the holes and the distance between them were measured using a precision metrology device (SmartScope Flash 200 [36]) to 10-μm accuracy. Figure 14 shows the threaded holes, their sizes, and the distances between them.
The section of the fastening plate, where the holes were located, was scanned to get the 3D coordination data of the area. This data was then processed using the hole feature detection algorithm to identify the threaded holes, their diameters, and the pose information in a 3D space. The diameter and the distance between holes, measured using the algorithm, were compared with the reference measurement obtained using metrology equipment. These tests provided the translational accuracy of the algorithm in 3D space (i.e. x, y, and z accuracy). To obtain the rotational accuracy, a planar region (shown in the inner square on Fig. 14) located near the holes was selected and a 3D plane was fitted to the planar region data.
The pose of the plane representing the planar region was used as the reference to analyse the rotational elements of the holes locations in 3D space (i.e. θ x , θ y , and θ z accuracy). The evaluation process was carried out by pitching up the tilt table at different orientation (0°to 30°) along its centre. Tables 1 and 2 provide the hole diameter accuracy and the translation accuracy measurements obtained from the detection algorithms. The average error in hole diameter measurement on the different angles was 0.58 mm with 0.53-mm average translation error, which are within the targeted range considered suitable for industrial applications.
Furthermore, the rotational accuracy was evaluated through a series of similar tests and an average error of 0.67°was obtained, as illustrated by Table 3. In addition, the repeatability of the system was calculated at ± 26 μm based on the rotational and translational results gathered from a sample of 80 experiments.

Evaluating the performance of the algorithms
To evaluate the capability of the complete fastening process, three different trials were carried out with 20 tests for each trial batch. The bolt detection algorithm was able to identify M10 bolt from the group of different size bolts (M10, M12, and M14) placed in a clutter. The 3D pose of the bolt gripping section, obtained as a result of the matching process, was accurate as the robot with the gripping fingers was able to pick up the bolt successfully each time. To evaluate the bolt detection algorithm, three different clutters of bolts (M10, M12, and M14) were randomly selected. The bolt detection method (using surface matching) was used to find the M10 bolt on each clutter form 20 times. The results of the trials are shown in Table 4 and Fig. 15.
The average geometrical repeatability of the algorithm in determining the position of the bolt was ± 0.09 mm in translation and ± 0.16°in rotation. These results illustrate that the bolt detection method is accurate and highly repeatable.
Twenty assembly trials were carried out by placing the bolts one above other at the same time, positioning the fastening plate in different random orientations using a tilt table. The  average identification time for the threaded holes and the bolts were recorded as 0.9 and 1.1 s, consecutively. In total, 98% of the identification processes were successful. Figure 16 illustrates the point cloud processing times in further detail for the successful identification processes.
The errors reported in the results are cumulative errors, which include error in the robot manipulation, the laser scanner data acquisition, the determination of tool point (TCP) for the robot arm, the 3D calibration process, and the algorithms developed for object and feature identification.

Conclusion
Assembly of threaded fasteners is a significant part of manufacturing processes. In low-volume high-value manufacturing, fasteners are assembled manually to provide flexibility and to eliminate the need for precision jigs and fixtures. A review of the existing automated assembly technologies showed that the current research and commercial methods are not sufficiently fast and accurate for industrial applications related to the threaded holes with chamfers. In this paper, the authors present the development of new identification algorithms tested on an automated system for assembly of threaded fasteners in a semi-structured environment.
A novel edge detection algorithm for threaded holes and a new bolt detection method were introduced in this article. A demonstrator rig was presented, comprising a robot, a tightening tool, and a laser scanner. The bolts and mating treaded holes were randomly located in 3D space within a confined area. A series of laboratory experiments were carried out. The results of over 160 tests showed that the proposed detection methods can identify bolts in clutter with 0.58-mm accuracy in diameter and manipulate the bolts with 0.53-mm translational accuracy and 0.67°of rotational precision. This is considerably more accurate than existing identification methods (reported in literature as 10 mm and 1.5°) and is reasonably within the range required for