Low cost three-dimensional virtual model construction for remanufacturing industry

Abstract

Remanufactured products can save up to 80% of production and energy costs whilst generating lower CO₂ emissions. The key success factors for remanufacturing are quality, lead-time and cost. Extensive work within the industry and the detailed analysis of the remanufacturing process has shown that component inspection has significant bearing on overall productivity. Remanufacturing lacks automation because activities are predominantly manual. Automation of remanufacturing process will not only decrease the number of non-remanufacturable components, through decreasing cost and increasing consistency in quality, but also attract industries to design for remanufacture. A digital model of the component is required to automate the disassembly process and move towards industry 4.0 and cyber physical systems. There are several expensive techniques to create a digital model, which are not feasible for the remanufacturing industry. The research paper aims to check feasibility of using Visual Structure for Motion (VFM), a relatively low cost method, to develop a 3D digital model, for automation of the automotive engine (in as received condition) disassembly process using industrial robots. These experiments assess the scientific feasibility of using Videogrammetry to acquire pre-disassembly 3D model of the engine. Multiple 2D images were acquired and processed to find matching common features. The location of the camera was calculated through the matching features, producing a three-dimensional digital representation of the captured volume. A sparse point cloud was initially created and was then converted into a dense 3D point cloud. The 3D point cloud was converted into a meshed model. 2D images were stitched together to create a virtual model of the engine with surface texture and colour. Small features were clearly visible in the 3D model.

Introduction

Remanufactured products can save up to 80% in production and energy costs whilst generating lower CO₂ emissions. Up to 85% of a remanufactured product’s weight can come from used parts thus reducing environmental and recycling impacts. The key remanufacturing success factors are quality, lead-time and cost [18]. Currently, the remanufacturing process is performed manually. It is crucial to apply emerging technologies and digital manufacturing systems to remanufacturing, to exploit its full potential, whilst reducing lead time and cost [5, 6, 8, 16]. In current work, initial experiments are performed to assess the feasibility of Videogrammetry to acquire pre-disassembly three dimensional (3D) model of the engine. The application of this vision based technique will help automate the remanufacturing process. Integration of machine vision will help in metrology, machine learning, robot path planning and thus automate disassembly, quality control and other processes. 3D reconstructed model with colour texture will not only aid machine vision tools to be easily implemented, but also will pave way for the use of machine learning algorithms. Both these methodologies, machine vision and machine learning, will be used for robust identification and calculating location of features, for example, bolts type and location. This information will then be communicated to the robot to perform a particular operation, for example, selection of a particular tool to unscrew the bolt. Accuracy of the process will also be assessed in the second phase of research. 3D model of the engine, acquired using a conventional 3D scanner, will be compared to the 3D reconstructed model using VSFM technique.

This study discusses implementation of digital manufacturing in remanufacturing, whilst highlighting a new non-contact metrology technique. Suitability of several metrology techniques, currently being researched under the umbrella of digital manufacturing are discussed with respect to remanufacturing. VSFM, one of these emerging techniques is then selected because of low cost and efficiency. Future research is planned based on this initial study.

Implementation of digital manufacturing

The most important strategy to deal with the rising threat of climate change is to opt for a low carbon economy which will enable reduction in pollution, emission of greenhouse gases and energy consumption [2, 15]. Recent legislation and engineering accomplishments have reduced automotive emissions. Improved automotive component utilization, specifically the engine, is being achieved through remanufacturing. [19]. It is very important to introduce systems of digital manufacturing into remanufacturing. This work is part of an ongoing research focusing on remanufacture of engines, using robots and digital technology. The implementation of cost effective digital technologies will help increase range of the remanufactured products.

Cloud-based design manufacturing (CBDM) systems incorporate communication and information infrastructure and employs the internet of things (IOT) to merge design and manufacturing related data [11]. The concept of CBDM can be directly used to remanufacture products, in areas such as design for remanufacturing, reverse supply chain management, information management, automation, inspection and metrology. The essence of this system is to present physical objects (e.g. an engine) in digital form, and connect humans with machines [31]. These systems can then be used to automate the remanufacturing process. Metrology has immense importance to automate the process, as it provides data which is used in robotic path planning such as feature size and location. Metrology has been widely used in many areas to capture the dimensional deviations of parts geometries [25].

Metrology

The data generated through 3D-Metrology can be effectively used in remanufacturing. Thanks to the recent advancements of technology, three-dimensional geometric measurement can be done by either interacting with the given component mechanically (contact metrology) or radiometrically (non-contact metrology) [22]. Over the last few decades, non-contact metrology technologies have made a big contribution to precision manufacturing and cost reduction. Cost reduction of remanufactured parts is always a big challenge for remanufacturers of automotive components. There are various kinds of non-contact metrology devices used to manufacture automotive components (e.g. engines), but it is a big challenge to define appropriate metrology technology, techniques and devices, which could be utilized to remanufacture components. There are several non-contact metrology techniques to measure components, such as laser scanning [24], laser tracking [1], structured light techniques [27], 3D reconstruction [7]. Cost, accuracy and speed are important parameters while selecting a metrology system for remanufacturing. Comparisons of these techniques are provided in Table 1. Most of these methods are used extensively in the manufacturing industry. After analysis [Table 1], 3D reconstruction was selected because of the additional information which can be acquired through this method, such as colour and texture [30]. This information can be used in machine learning and robot path planning. Furthermore, this technique is low cost and easy to use.

Table 1 Comparison of non-contact metrology techniques

In general, 3D reconstruction pros are, its comparatively of low cost (£500), has higher inspection range (1000 m) and higher resolution. Cons are, it has relatively lower scanning speed (not overall process speed) (255,045 points per sec) and low information, on how to make scans more accurate. This study lays the basis of figuring out the influential parameters to fill this research gap.

Data acquisition through 3D reconstruction

It is now possible to reconstruct component’s geometry due to the advancements in metrology data acquisition and digitization techniques [12]. 3D reconstruction is widely used for large scale buildings and cultural heritage, by means of photogrammetry methods that produce significant improvements in accuracy and scalability. In current study, Multi-view stereo (MVS) system is used to 3D reconstruct the engine geometry. MVS data is generally very detailed ([6, 17, 20, 23]; C. [29]). This technique has not been used in the context of manufacturing or remanufacturing. Appropriately reconstructed components can decrease the cost of remanufacturing by facilitating metrology and robot path-planning. Moreover, remanufacturers often do not have access to component’s original Computer Aided Design (CAD) files and technical drawings, required to remanufacture the components. 3D reconstruction techniques also provide additional data, such as surface texture and colour (C. [30]), which will help to digitize remanufacturing. This is another reason MVS and Structure for Motion (SFM) techniques have been preferred to other non-contact metrology techniques. These techniques were applied to reconstruct the surface geometry and texture of an “as received” automotive engine.

Structure for motion

SFM integrates point matching, feature extraction and existing knowledge of vision systems [4, 23]. Generally, SFM has been used either with limited number of images developed by academic researchers or using costly commercial software such as Autodesk remake [10], Agisoft [13] to combine large numbers of images. Currently, open-source Structure for Motion (SFM) packages such as Visual Structure for Motion (VSFM) (C. [30]) and Open Multiple View Geometry (OpenMVG) [21] are available. It is a low-cost method generally used in multi temporal surveys [13]. These packages offer fast and free processing of several thousand images and produce results comparable to commercial software.

Problem statement

Currently, the disassembly of the engine, for remanufacturing is done manually. Labour and training cost are high to perform the disassembly process. Injury rate due to oil spillage and heavy components is quiet high as well. In the proposed setup, the disassembly process will be performed through an industrial robotic arm. Without digital model, the industrial robotic arm will not be able to identify and locate bolts, nuts and other components to be disassembled. After identifying a component (e.g. bolt), a particular end-effector (tool) will be selected and brought to the specified location (calculated through the 3D model). This low cost 3D digitization technique will enable the machine vision and artificial intelligence components of the system to select a tool and move it in a particular location.

Aim and methodology

The main purpose of this research work is to demonstrate the usability of SFM technique for remanufacturing, by 3D reconstructing an automotive engine. Effects of luminous intensity and number of 2D images was observed during the experiments. In the next step of this research, the authors intend to improve and integrate 3D reconstruction, machine vision and artificial intelligence to robotics for remanufacturing.

In this work, video-based photogrammetry (Videogrammetry) was used to acquire data. Videogrammetry is already applied in industrial manufacturing and robotics [3]. SFM techniques are not fool proof; various parameters such as dynamic background, rapid movement of the camera, shiny surfaces and extreme changes in light can complicate the reconstruction process. Different types of features, background and different amounts of overlap between the pictures affect the reconstruction quality and time. Thus a video was taken with consistent background and lightning. A 3D model was reconstructed through the SFM technique using a sequence of images, without prior knowledge of the camera pose (location, orientation and field of view). Although it uses several modern techniques for detection of key points and dense reconstruction, it also borrows methods developed for classic photogrammetry, such as self-calibrating bundle adjustment [26] to automatically estimate the camera pose. Viewing parameters and 3D structure of a scene was simultaneously solved through SFM while using multiple feature recognition algorithms and computer vision techniques. The output meshed model was saved in Stereo-Lithography format (STL), which can be used for manufacturing and remanufacturing purposes.

Experimental setup

The main aim of experiment was to develop and examine the low cost three-dimensional virtual model construction for remanufacturing industry. Experiments were performed on a used, as received engine of a car, with complicated geometrical features. A smartphone camera of 16 megapixel, 4.3 mm focal length and F1.9 aperture was used for current work. This information increases the quality of the reconstruction. The use of a smartphone demonstrates the low cost characteristic of the reconstruction method. It can be easily accessible by the remanufacturers with no particular setup required. Another advantage is that the user can monitor the acquisition of the video and pictures in real-time, thus improving the data acquisition process. An ultra-high definition video with frame size of 3840 by 2160 pixels was acquired and images were extracted from the video stream (see Fig. 1). The video was taken in four sections (A-B, C-D, E-F, G-H), as shown in Fig. 1b. Each section line was 45 degrees apart from each other. Matlab® was used to acquire variable number of images from the video, which were then processed through VSFM to acquire results.

Fig. 1

a As received automotive engine, b Videogrammtery strategy

Results

VSFM compares every image in a set with every other one to find the best match. Although this technique guarantees to find the best match, the time taken to complete the 3D reconstruction increases exponentially with the number of pictures. Matches were not found with inadequate light on the engine and thus the 3D model was not created for this case. The advantage of using a video footage is that it restricts the comparison within neighbouring images. Thus, reconstruction time is reduced to a linear relationship, which allows for much larger datasets to be used in the reconstruction. This approach provided with necessary overlap between the pictures and did not compromise the performance of the Scale-invariant feature transform (SIFT) algorithm, as VSFM resize images and corrects radial distortion before processing.

The SFM process for the engine was based on the following steps: (1) feature detection, (2) alignment, (3) bundle adjustment and (4) reconstruction. SFM is available under the Berkeley Software Distribution (BSD) License [4]. The set of features defined by SIFT can contain outliers or points not common to both images (depending on the overlap). VSFM quickly and robustly matches images by using the iterative technique RANdom SAmple Consensus (RANSAC) [9], which constructs an eight-point alignment model in linear time. Bundle adjustment refines a visual reconstruction to jointly produce the optimal 3D structure and the viewing parameters. Sparse point-cloud (PC) with camera locations and dense point-cloud are shown in Fig. 2a. After bundle adjustment, VSFM uses Multi-view stereo (MVS) ([14]; C. [28]) to create a dense point cloud from the scene as, shown in Fig. 2b. 15,561 points were created through 101 pictures extracted from the video.

Fig. 2

Point cloud; a Sparse PC of engine with camera location, b Dense PC of engine with colour

The model acquired through this technique can be compared to the 3D scanned model through the STL output and the post-processing steps, using an algorithm available in the Cloudcompare software. This step is planned for the next phase of research. The point-cloud was used to create a polygonal mesh of the geometry using the Meshlab software (Fig. 3). While several post-processing tools are available, Meshlab was selected due to its flexible post-processing and editing capabilities. The point-cloud was loaded in the Meshlab. Then it was cleaned; outlier points were removed from the cloud through the appropriate Meshlab tools. The point-cloud was finally used to create a smooth and textured mesh.

Fig. 3

Virtual models in Meshlab: a Textured Mesh of Engine, b Un-smoothened surface of engine

The smoothened surfaces were rendered and texture was applied to create a 3D virtual model of the as received engine, as respectively shown in Fig. 4. The colours in the virtual model match the real one and small details, such as the small truss and the folded tape can be easily observed. Future work will use this process for the automation of remanufacturing.

Fig. 4

Virtual model of Engine

Discussion

Currently, the disassembly of the engine for remanufacturing is done manually. Labour and training cost are high to perform the disassembly process. Injury rate due to oil spillage and heavy components is quiet high as well. In the proposed setup, the disassembly process will be performed through an industrial robotic arm. Without digital model, the industrial robotic arm will not be able to identify and locate bolts, nuts and other components to be disassembled. Intellectual property right (IPR) issues might affect the relationship of the remanufacturer with the original equipment manufacturer (OEM). The 3D model generated from the VSFM technique is used for disassembly process only, which does not influence IP issues. Similar IP issues will be faced by other digitization techniques if used inappropriately. More research is required in the security of cyber physical system to avoid information leakages.

After identifying a component (e.g. bolt), a particular end-effector (tool) will be selected and brought to the specified location (calculated through the 3D model). This low cost 3D digitization technique will enable the machine vision and artificial intelligence components of the system to select a tool, perform metrology, inspection and robot path planning, to move it in a particular location.

Influential input parameters to 3D reconstruct the digital model were decided on the basis of current research. These parameters are luminous intensity and number of 2D images to create the model. For example, the current system was not able to generate a dense 3D point cloud and 3D model without proper light. Similarly, the reconstruction simulation process crashed with higher number of images. Further experiments were then planned and performed based on this information.

Future work

3D model will also be reconstructed using a machine vision camera and robotic arm as shown in Fig. 5a. The robot arm will be programmed to move around the engine while capturing video. Integration of robots will add capability of repeatable data acquisition. The data will be acquired for three different light conditions i.e. low, medium and high. 3D model will be reconstructed with different number of images in different light conditions to evaluate the optimized position of camera and lighting conditions. The 3D reconstructed models will be compared to the 3D scanned model (Fig. 5b) to check the accuracy and uncertainty of the metrology process.

Fig. 5

Future work; a Robot and machine vision camera to 3D reconstruct engine, b 3D scanning of engine

Currently, a machine learning software is being trained on a testing rig to identify different types of bolts. Research challenge is to identify damaged bolts. Both 3D reconstruction and machine learning codes will help to solve this problem. This information (type of bolt) will then be communicated to the robot to perform a particular operation, for example, selection of a particular tool to unscrew the bolt.

Conclusion

Generally, remanufacturers do not have access to the component’s technical drawings, CAD models, surface features and texture. Conventional optical metrology methods used in the manufacturing industry to acquire this data are expensive, time consuming and require expertise. Experiments were performed to check the feasibility of using Videogrammetry with a low-cost 3D reconstruction method VSFM and determine influential parameters effecting the process. A 3D Virtual model of an automotive engine including geometry, surface texture and colour was developed using a cell-phone camera. Future work will further evaluate these parameters and use this process for metrology, automated disassembly, robotic path planning and inspection purposes in order to enhance remanufacturing productivity.