Keywords

1 Introduction

Maintainability is important to reflect whether product maintenance is convenient, fast and economical [1]. In order to ensure that the product has high availability and low life cycle cost, the product must have good maintainability, so as to reduce the maintenance requirements for manpower, time and resources [2, 3]. Therefore, during the development process of industrial products, sufficient maintainability tests must be carried out to verify and evaluate their maintainability to ensure that they meet the required maintainability requirements.

The traditional method of physical maintainability evaluation relies too much on physical prototype, which is expensive and sometimes impractical [4]. The method of virtual maintainability simulation evaluation using digital prototypes is difficult to accurately evaluate the maintenance force characteristics and maintenance time indicators due to the difficulty of accurate human-machine force interaction. However, virtual and real fusion can present the real world and the virtual world at the same time, providing information extensions for real scenes. In the field of maintenance and assembly, the application of virtual and real fusion has made certain progress. Deshpande designed AR-assisted visual features and interactive modes for support-as-assembly (RTA) furniture [5], and developed an application on Microsoft Hololens™ headsets, which enabled users to quickly conceive the spatial relationship of their components and can support assembly tasks that require high spatial knowledge. And it was tested on the users of RTA furniture for the first time. Vicomtech studies the creation method of AR workspace with interaction and visualization mode as the core, and provides more effective support means for the assembly task of hybrid man-machine production line [6]. It can be considered that the virtual and real fusion maintainability test has good accuracy and economy by reducing the hardware scale, which has a huge application prospect. The key issue here is to integrate the physical equipment and the virtual environment according to the actual positional relationship. The three-dimensional pose of the physical equipment must be accurately identified and then the virtual environment is superimposed. The paper focuses on this research and conducts the application of maintainability evaluation.

2 Overall Solution

In the process of a virtual and real fusion maintainability test, a full set of digital prototypes of the product are usually provided as the basic information for the test. The digital prototypes reflect the relationship between the physical product and the surrounding environment. In order to superimpose the virtual maintenance environment model on the periphery of the physical product object and make it sure that it is a part of the maintenance environment, it is necessary to identify the physical product and make the virtual world fully aligned with the physical world. In this paper, the binocular camera is used to obtain the video stream of the real maintenance scene and the characteristics of the video image are extracted on the basis of calibrating the internal parameters of the camera. The transformation matrix is solved for pose estimation. Then, the virtual scene is registered to the real scene through coordinate transformation to complete the construction of virtual and real fusion maintainability test scene. The overall process is shown in Fig. 1.

Fig. 1.
figure 1

Overall process of maintainability assessment based on virtual and real fusion.

Full size image

3 Key Technology Implementation

The key problem to achieve seamless integration of virtual and real maintenance scene is how to accurately identify physical objects and match them with virtual models. In order to construct a realistic scene of virtual and real fusion maintainability test, the main research is based on the ORB feature extraction method, and ICP matching is carried out with the corresponding equipment model in the digital prototype. On this basis, the qualitative and quantitative maintainability index evaluation method based on virtual reality information fusion is formulated.

3.1 Image Feature Extraction of Maintainability Test Object Based on ORB

At present, many local features such as SIFT, SURF, ORB, BRISK, FREAK, etc. are widely used in the fields of image matching and object recognition [7]. Since the object of the maintainability test process is usually a mechanical product, its surface sometimes lacks rich texture features. Considering the stability and rapidity based on feature point extraction and matching, the ORB local feature is selected here. ORB local features use FAST as the feature point detector, and use the improved BRIEF as the feature descriptor, and use the BF pattern matching algorithm for feature descriptor matching.

FAST feature points are not directional, and the directional parameters are determined by obtaining the center of gravity of the feature point neighborhood. The neighborhood moment is:

$$ m_{pq} = \sum\limits_{x,y} {x^{p} y^{q} I(x,y)} $$
(1)

where \(I(x,y)\) is the gray value at point \((x,y)\), \(x,y \in [ - r,r]\), \(r\) is the radius of the circle, \(p\) and \(q\) are non-negative integers, when \(p\) is 1 and \(q\) is 0, the value \(I_{x}\) of \(I\) in the \(x\) direction can be obtained, when \(p\) is 0 and \(q\) are 1, the value \(I_{y}\) of \(I\) in the \(y\) direction can be obtained, and the \(C\) coordinate of the image center of gravity can be obtained as:

$$ C = \left( {\frac{{m_{10} }}{{m_{00} }},\frac{{m_{01} }}{{m_{00} }}} \right) $$
(2)

The angle between the feature point and the center of gravity is defined as the direction of the FAST feature point:

$$ \theta = \arctan (\frac{{m_{01} }}{{m_{10} }}) = \arctan (\frac{{\sum\limits_{x,y} {yI(x,y)} }}{{\sum\limits_{x,y} {xI(x,y)} }}) $$
(3)

ORB extracts the BRIEF descriptor according to the direction parameters obtained in the above formula. However, due to environmental factors and the introduction of noise, the direction of feature points will change, and the correlation of random pixel block pairs will be relatively large, thereby reducing the discrimination of the descriptor. ORB adopts a greedy algorithm to find random pixel block pairs with low correlation. Generally, 256 pixel block pairs with the lowest correlation are selected to form a 256-bit feature descriptor. Note two descriptors:

$$ K_{1} = x_{0} x_{1} \cdots x_{255} ,K_{2} = y_{0} y_{1} \cdots y_{255} $$

3.2 Matching of Physical Equipment Characteristics and Virtual Environment Registration

The ORB feature set is extracted from the real maintainability test object and the virtual maintenance environment model, and the corresponding feature descriptors \(K_{1} ,K_{2}\) are obtained. The similarity between two ORB feature descriptors is characterized by the sum of the exclusive ORB Hamming distances:

$$ D(K_{1} ,K_{2} ) = \sum\limits_{i = 0}^{255} {x_{i} \oplus y_{i} } $$
(4)

The smaller the \(D(K_{1} ,K_{2} )\), the higher the similarity, and the greater the probability that the two describe the same feature. Conversely, the lower the similarity, the more likely they are not describing the same feature.

Use BF matcher to get all possible matching feature pairs, assuming that the minimum Hamming distance of feature pairs is MIN_DIST. In order to select the best matching pair and improve the operating efficiency, an appropriate threshold is selected and the matching pair smaller than the threshold is selected for the next camera pose estimation. The threshold value cannot be too small, which will affect the final effect, and it is necessary to select the best threshold value through experiments on the image frame.

Given the point \(k_{1i}\) in \(K_{1}\), find the point \(k_{2i}\) with the shortest Euclidean distance of \(k_{1i}\) from \(K_{2}\), and take \(k_{1i}\) and \(k_{2i}\) as the corresponding points to obtain the transformation matrix. Through continuous iteration, the following formula is minimized and the iteration is terminated, and finally the most Optimal transformation matrix is obtained to make them coincide.

$$ f(R,T) = \frac{1}{n}\sum\limits_{i = 1}^{n} {\left\| {k_{1i} - (Rk_{2i} + T)} \right\|^{2} } $$
(5)

In the formula, \(R\) indicates the rotary transform matrix, \(T\) indicates the translation transform matrix.

The essence of the ICP algorithm is to calculate the transformation matrix between the feature sets, minimize the registration error between the two through rotation and translation and then achieve the best registration effect. Assuming two feature point sets \(K_{1} = \left\{ {k_{1i} \in R^{3} ,i = 1,2, \cdots ,n} \right\}\) and \(K_{2} = \left\{ {k_{2i} \in R^{3} ,i = 1,2, \cdots ,n} \right\}\), the registration process using the ICP algorithm is introduced below:

  1. (1)

    Sample set \(K_{1}\), \(K_{10} \subset K_{1}\), \(K_{10}\) represents a subset of set \(K_{1}\);

  2. (2)

    Search in set \(K_{2}\), find the closest point to each point in \(K_{10}\), and get the initial correspondence between \(K_{1}\) and \(K_{2}\);

  3. (3)

    Remove the wrong corresponding point pairs using algorithms or constraints;

  4. (4)

    Calculate the transformation relationship between the two according to the corresponding relationship in step (2), minimize the value of the objective function and apply the calculated transformation matrix to \(K_{10}\) to obtain the changed new \(K_{10}^{^{\prime}}\);

  5. (5)

    Determine whether the iteration is terminated according to \(d = \frac{1}{n}\sum\limits_{i = 1}^{n} {\left\| {K_{2i} - K_{1i} } \right\|^{2} }\). If \(d\) is greater than the preset threshold, return to step (2) to continue the iteration; if \(d\) is less than the preset threshold or reach the set number of iterations, the iteration stops.

By obtaining the transformation matrix through the above steps, the pose transformation relationship between the physical equipment and the virtual maintainability test environment can be obtained, and then virtual registration can be performed to complete the construction of the virtual and real fusion maintainability test environment.

4 Experimental Verification

Take the auxiliary engine room of a ship as a case to carry out the test verification to verify the correctness and applicability of the virtual and real fusion maintainability test evaluation method studied in this paper. The auxiliary engine room is powered by a diesel engine,which is composed of a crank connecting rod mechanism, a gas distribution structure, a fuel system, a lubrication system, a cooling system, a starting system,etc. The engine needs to replace consumable parts such as fuel filter and air filter, and the cylinder and starter motor have a certain failure rate. It needs to be well designed for maintenance to ensure rapid maintenance at the crew level.

In the ship cabin environment, the equipment maintenance process has certain complexity, and other equipment around the equipment and peripheral pipelines and cables are easy to cause insufficient accessibility of the maintenance objects and insufficient operating space. Therefore, in the process of maintainability test of the engine, it is necessary to be able to simulate actual cabin maintenance scenes and maintenance space, and fully consider the impact of various operational obstacles on maintainability, so as to obtain more accurate maintainability test results.

Since the establishment of a 1:1 full-physical maintainability test condition is very costly and has a long cycle, the virtual and real fusion maintainability test evaluation method studied in this paper is adopted, and a small part of the physical equipment and a large number of virtual environments are used to realistically simulate a complete test scene. During the test, the available test conditions include the YN92 physical diesel engine and the complete digital model of the auxiliary engine compartment, as shown in Fig. 2 and Fig. 3. Next, take the repairing and replacing the starting motor as an example to verify.

Fig. 2.
figure 2

YN92 diesel engine.

Full size image
Fig. 3.
figure 3

Virtual maintenance scene of ship auxiliary engine cabin.

Full size image

4.1 Verification of the Establishment Method of Virtual and Real Fusion Test Scene

In order to build a realistic virtual and real fusion maintenance scene, it is necessary to consider the impact of multiple factors on the registration accuracy of the virtual environment. The feature extraction method is an important factor affecting the registration accuracy.

Firstly, the feature extraction and recognition of diesel engine are carried out. Different feature extraction methods have different feature extraction results. The feature extraction of the same object (diesel engine) is performed using SIFT, SURF, and ORB methods respectively, and the comparison of the diesel engine feature extraction results of the three methods is shown in Fig. 4.

Fig. 4.
figure 4

The comparison of the diesel engine feature extraction results of the three methods.

Full size image

The data results of the three methods for feature extraction are shown in Table 1.

Table 1. Experimental results of different feature extraction methods.
Full size table

Through experimental analysis and comparison, the feature points detected by SIFT, SURF and ORB are 502, 454 and 1023 respectively under the same experimental conditions. The feature points matched by SIFT, SURF and ORB are 112, 168 and 136 respectively. It can be found that although the number of feature points matched by the three methods is roughly the same, the time required for ORB matching is significantly shorter and the operation efficiency is obviously higher.

Inject the above two algorithms into AR glasses, obtain the three-dimensional visual information of the physical equipment through the binocular lens of the glasses, and then perform feature extraction and match them with the virtual model one by one. The resulting virtual and real fusion ship cabin repair scene is shown in Fig. 5.

Fig. 5.
figure 5

The obtained virtual and real fusion ship engine maintenance scene.

Full size image

4.2 Maintainability Test Operation and Result Analysis

Next, according to the established virtual and real fusion maintainability test scene of YN92 physical diesel engine, the maintainability operation test of the replacement of the starting motor is carried out. The tester wears AR glasses to carry out maintainability test operation and obtain basic test data.

A total of 5 groups of tests are carried out, and each group of tests is carried out in three scenes of real environment, virtual and real fusion and without surrounding environment respectively, and the comparison of maintenance operations in three scenes is shown in Fig. 6.

Fig. 6.
figure 6

The comparison of maintenance operations in three scenes

Full size image

In the virtual and real integration maintenance test, the maintenance personnel can feel the existence of the surrounding cabin equipment through vision. During maintenance, in order to avoid collisions with the virtual cabin equipment, the bending angle of the arm will be smaller and the movement range will not be large. The posture of the maintenance personnel should be adjusted accordingly to be closer to the real maintenance situation, so the maintainability evaluation error is smaller.

5 Conclusion

This paper proposes a method of constructing a maintainability test scene based on the fusion of virtual and reality for maintainability evaluation. The ORB feature of the equipment is extracted based on binocular vision, and then the ICP method is used for feature matching and recognition according to the feature extraction results, and the virtual environment is registered to complete the construction of virtual reality fusion maintainability test scene. Experiments show that the use of orb features can effectively extract equipment features, with high speed and high precision. The ICP method can be used to realize the registration of the physical object and the virtual environment, thereby completing the registration of the virtual environment. The maintainability test is carried out and evaluated in the built virtual and real fusion test scene. The results show that the surrounding virtual environment has a certain impact on the maintenance process, and the maintainability verification is closer to the maintenance process in the real maintenance environment.

The virtual and real fusion maintainability test method studied in this paper provides a novel and efficient method for simulating the real maintenance performance of the products under complex maintenance conditions. It can carry out the main test operations on the real object and simulate the spatial characteristics at low cost, so as to make the index evaluation of visibility, accessibility and maintenance time more accurate.