Efficient integration process of production data into Augmented Reality based maintenance of machine tools

With the increasing requirements on machine tools such as higher complexity, individuality and digitization, their maintenance becomes also more complex, which increases time consumption and demands special knowledge. The maintenance engineer can be supported in this process using the Augmented Reality (AR) technology, mobile devices and suitable production data. This requires the integration and adaptation of the maintenance documentation as well as further production systems and their data such as the CAD system, product data management (PDM) system, manufacturing execution system (MES) and CNC. For this purpose, a method is developed that allows an efficient integration process of various production data into an AR supported maintenance documentation. This includes concepts for the integration process of different production systems and data into an AR maintenance system as well as the design of the AR maintenance system architecture. Basis is the analysis of the required production and AR systems, data and processes. The aim of the method is to reduce the high expenditure of the overall planning process and the necessary expertise of the maintenance planner of the AR technology. The developed method is verified with different maintenance scenarios, production systems and data as well as machine tools and AR devices.


Motivation
The high complexity and individuality of machine tools and their maintenance lead to more difficult tasks for the maintenance engineer and require specialist knowledge about the particular machine tool. The result is a higher potential for mistakes of the maintenance engineer, increasing maintenance time, production downtime and consequently increasing production costs [1].
By digitization in production and deployment of new technologies such as Augmented Reality (AR) and mobile devices, the maintenance engineer and processes can be supported [2]. AR offers new ways for human-machine interactions by superimposing the real environment with virtual information in the right location in the user's field of view [3]. With suitable AR mobile devices and a digital maintenance documentation, the maintenance engineer can use interactive step-by-step instructions to reduce the complexity of the process and sources of error can be shown directly on the machine tool based on current production data to improve the maintenance process [2].

Problem of the production data integration process
The basis for the maintenance work is the maintenance documentation [4], containing a detailed description of the machine tool, user instructions for manual tasks and pursuing documents, photos and graphics. The maintenance documentation is often delivered on paper that can be timeconsuming, out-of-date, not action-oriented and unfavorably structured for the maintenance engineer [5]. The created digital form in a document has not the appropriate data structure for using in an interactive AR application. Additionally, data from CAD systems and product data management (PDM) systems are required or can expand the AR documentation 1 3 but has to be integrated with an efficient process. Furthermore, data from diagnosis, condition monitoring and further production systems, e.g. a manufacturing execution system (MES) as well as CNC data, can also support the maintenance engineer by detecting defects during the maintenance. Therefore, interfaces and visualization elements have to be available to avoid much programming effort. On the one hand, AR has the potential to improve the maintenance process by supporting the engineer [2], on the other hand the integration process of various production data and the creation of a new digital AR maintenance documentation has a high expenditure with many different data sources and data formats and needs a high level of AR expertise that cannot be covered by the employees. Therefore, an efficient integration process is required that includes various AR components (e.g. camera image processing, tracking of the environment, visualization, interaction) and production systems to support the maintenance planner.

State of the art of Augmented Reality for maintenance of machine tools
In recent years, several explorations were carried out to support the maintenance engineer with AR based instructions using 3D models for manual working steps, e.g. a gearbox maintenance for milling machines [6]. A product disassembly process for maintenance without expert intervention is developed in [7]. For this purpose, a disassembly sequence table is generated based on the product ontology and linked with virtual information. Moreover, AR systems were created that establish a connection to the CNC to use online data [8,9]. This allows the visualization of all CNC values for monitoring, error detection, process control, simulations and movement of 3D models. For the integration of AR required data various approaches were explored [10,11]. Besides the video recording of real workflows, programming and manual addition of 3D models and instruction texts, the usage of an authoring application is a common concept. This allows the user to append and manage the maintenance data in a special application that creates the AR based data structure. In [10] an AR system for assembly instructions is described that integrates a special process planning application to create the assembly sequence and include CAD data. Furthermore, the AR assembly system described in [11] additionally imports PDM data to the authoring process.
As a basis for maintenance AR applications, general AR programming libraries and frameworks provide the opportunity to use several AR components. However, they need high-level expertise in AR, enhancements and programming for the maintenance requirements [12]. Existing AR systems for maintenance, which are using authoring applications, require additional software in the planning process. Furthermore, they cover only special elements, cannot map the required complexity or require a high level of expenditure for the integration process. Neither sufficient automated processing of the data, nor specialized efficient workflows for the production exist. Moreover, it is not sufficient to integrate existing maintenance documentations and comprehensive data sources. Figure 1 illustrates a typical current integration process of AR based maintenance data. In this case, a special authoring system lies in the center of the process where the maintenance documentation and all additional data is entered and managed. The necessary 3D model from the CAD system has to be prepared separately in an application for digital content creation (DCC). This process is characterized by a high manual expenditure and new software systems, data formats and workflows that have to be used by the maintenance planner.
The authors' preliminary work describes the development of the AR framework ARViewer, which is specialized for production usage [13]. It includes various AR components and interfaces for different use cases and is designed for including external systems such as CNCs [12] and PDMs [14] as well as different devices such as tablet computers and head-mounted displays (HMDs) [15]. It provides the basis for realizing a new method for the efficient integration of production data into the AR based maintenance of machine tools.

Aim of the work
The aim is the design of a new method for the efficient integration of production data into the AR based maintenance of machine tools by developing a new creation process for the AR based documentation including different production systems as well as a concept for the overall AR maintenance system architecture. With this method the high expenditure of the AR planning process and the necessary AR expertise of the maintenance planner should be reduced. Figure 2 illustrates the aim for the new integration process of AR based maintenance data. The center of this process comprises software systems that are already in use. This   includes CAD, documentation and PDM systems that are used to create and compose the AR based maintenance data. Existing production data and available AR components can be applied for different maintenance use cases. In addition, CNC and MES should be integrated for online data support. The target process is characterized by a high level of automation, a continuous data model and an industry-oriented workflow.

Approach of the work
To achieve the aim of the work, the state of the art of current AR concepts for the maintenance of machine tools was researched. The next step is the analysis of the required AR based maintenance process and data as well as the creation process for the AR based documentation as a basis for the design of the new method. Subsequently, the new method for the efficient integration process of production data into the AR system is developed and a design of the overall AR maintenance system architecture is created. Finally, the developed method is verified with a sample implementation for different maintenance scenarios and AR devices.

Definition of the AR based maintenance process
For developing the method of an efficient integration process of production data, the AR based maintenance process as a basis has to be defined and is shown in Fig. 3. First step is the identification of the specific machine tool by tracking an attached label with the AR device. Using the tracked number, the connection to all required external systems is established. Now all maintenance tasks are available for the maintenance engineer. By selecting the desired task, all data is visualized step by step and the maintenance engineer executes the task. The data includes the prepared maintenance documentation for AR with superimposed 3D models as well as live data of connected production systems. This is repeated until the maintenance of the machine tool is completed. Finally, the results are stored in the system for subsequent evaluation. Different AR devices, e.g. tablet computers and HMDs, with different options for visualization and interaction should be supported for the maintenance process. Table 1 summarizes the data required for the AR system and the maintenance engineer for the specified AR based maintenance process shown in Fig. 3. The method should consider various AR devices, maintenance scenarios and production systems. Therefore, different properties of AR systems, interfaces to online data sources from external systems and the maintenance documentation with manual tasks have to be included in the integration concept. The goal is to integrate and use all data in one single AR system to cover various maintenance scenarios and to minimize the integration expenditure for the planning process. The basis for the concept is the increasing level of digitization in production today and the available production systems and data.

Requirements to the creation process of the AR based documentation
The requirements to the AR based maintenance process and data that are defined in the last sections have to be considered to the creation process of the AR based maintenance documentation. That implies that many different existing data types and sources have to be integrated to create an efficient and automated process without much manual effort.
That includes the usage of 3D models of the CAD system without a manual converting process, the usage of the existing documentation system without creating the instructions twice for the AR system as well as the usage of interfaces to further external systems such as PDM, MES and CNC to extend the AR based documentation with enhanced information. Additionally, the requirements to the AR technology shown in Table 1 have to be considered without extensive expertise of AR or programming of the maintenance planner.

Integration process of production data into the AR system
The industrial process of maintenance planning has to be adapted to the AR technology. The goal is a continuous process of data linkage from the first planning steps up to the real maintenance of the machine tool. This leads to a continuous data model and continuous engineering to minimize the expenditure for development and service. By using the AR technology, the maintenance data is visualized in the right location on the machine tool. For this purpose, a 3D model of the machine tool is necessary and the existing CAD system and CAD model is used as a starting point of the method.
The following fundamental properties of the method are defined to achieve the goal of high efficiency of the overall integration process of production data: (1) Usage of existing source systems All maintenance data is created with software systems that are already available in the production and the user is familiar with, e.g. CAD, PDM and documentation systems. An additional authoring application is not used. Therefore, the current maintenance planning process is adapted by using these systems in the entire process. (2) Usage of existing production data All the production data already available in the production systems is used to avoid the creation expenditure of additional AR required data, e.g. CAD models, maintenance documentation and PDM data. Furthermore, MES and CNC are directly connected via defined interfaces for the enhancement with online data. (3) Creation of a user-oriented workflow A fast and short process is realized by using existing production systems and data as well as templates for new information (e.g. maintenance labels in the CAD system) and automation of particular steps. (4) Automation of manual processes To reduce the integration expenditure, time-consuming manual processes are automated, e.g. the necessary polygon reduction and format conversion of CAD data and the adaptation of the AR application and user interface to the current maintenance scenario. (5) Creation of uniform interfaces The application is usable with different AR devices, maintenance scenarios, data sources and machine tools. By providing various interfaces, a specialized customization by the maintenance planner is avoided. (6) Providing various AR components To cover different use cases, simplify the integration complexity and avoid keen knowledge about AR and programming, various developed AR components are used and configured.
For realizing the presented six fundamental properties and the goals shown in Sect. 3 and Fig. 2, Fig. 4 illustrates the sequence of the new method for the efficient integration process of the production data into the AR based maintenance. Initial point is the CAD model of the machine tool whose components have to be restructured for highlighting or moving special components depending on the maintenance use case. Furthermore, special visual labels have to be appended for highlighting the maintenance tasks and AR tracking purposes. An automated polygon reduction and format conversion is processed after the export of the 3D model to eliminate manual work.  The second step is creating the maintenance documentation with the technical documentation system. With these systems the information can be entered by the maintenance planner with graphical elements in a user-friendly way but are stored in a structured form commonly in an xml [16] based format. The data structure is divided into the main maintenance tasks that consist of many single steps that have detailed information attached such as descriptions of the manual work, materials and tools. Additionally, the external data e.g. 3D models, PDM or CNC data sets are attached by their names. With these technical documentation systems and the structured data format, the documentation can be used for different output mediums such as digital documents, paper documents and AR data. Additionally, pursuing documents, photos and graphics can be included for detailed information.
All created and required maintenance data is stored and managed in a systematical structure in the PDM system. Additional PDM data as well as configurations for AR components, MES and CNCs are also stored in the PDM system and linked with the maintenance documentation.
For the maintenance process, the PDM system deploys all required data via network to the AR device. This device processes an automatic adaptation of the user interface based on the current maintenance use case and deposited device properties (see Table 1) by calculating graphic elements and activating interaction methods. For verifying the results, it is possible to obtain a preview of the AR application during the planning process.
With the described new method, the high expenditure with many manual steps, different data sources and data formats is reduced by using existing systems, data and automated processes. With the user-oriented workflow, provided interfaces to external systems and AR templates the complexity of the process and the necessary AR expertise is reduced.

Design of the AR maintenance system architecture
For realizing the new method defined in Sect. 5.1 and Fig. 4, a concept for the AR maintenance system architecture is designed (see Fig. 5). It consists of five main AR application components. The data management component integrates the existing maintenance data from the production systems. This includes the created maintenance documentation from the PDM system as well as online data from MES and CNCs. In the presented concept, the AR application is directly connected to the source systems of the maintenance documentation and integrated into the production infrastructurewithout using an additional authoring system. The PDM system is used as the central system for the prepared maintenance instructions. In the PDM system, all created maintenance data from the CAD system, documentation system and pursuing documents is stored, managed and provided via network. Online data from the machine tool and production is used directly via defined interfaces to the CNC and MES, e.g. for diagnosis and condition monitoring. The interfaces are designed with standard technologies and can connect different types of systems to reduce the expenditure of customization.
The presented process and concept can be implemented with different software systems, so that they are customizable for different requirements and companies. Examples are described in the verification Sect. 6 and details about individual components of the concept are available in [14]. With the presented concept of the AR maintenance system architecture and its interfaces, various data sources and data formats can be integrated without a high expenditure and changes of the AR application.

Aim and approach of the verification
The developed method for an efficient integration process of production data into an AR maintenance system and the  Fig. 5 Concept of the AR maintenance system architecture 1 3 design of the AR maintenance system architecture are verified with several maintenance scenarios of real industrial environments to demonstrate the feasibility, viability and advantages of the method. Therefore, the developed AR application, different machine tools and AR devices are used. Basically, the particular components of the method can be implemented with different systems depending on the company's requirements. For the verification, each component is exemplary implemented and tested for the specific maintenance scenario. Starting with the CAD model the described maintenance planning process is executed and the result is visualized on the example machine tools. Finally, an analysis of the executed planning process and conclusions about the created method are discussed.

Sample implementation of the developed method
The developed method was verified on different maintenance scenarios including a CNC turning and milling center [17] and a wire drawing machine [18]. Extracts of their paperbased maintenance documentations and CAD models were used as basis for the AR based documentation. An example use case is the maintenance of the slideway wipers of the CNC turning and milling center including the steps disassembly, inspection, cleaning and reassembly. The task requires carefully cleaning of all relevant parts in the machine, without omitting any working steps. Target AR devices, to assist the maintenance engineer, are industrial tablet computers [19] and HMDs [20]. The implemented components of the AR application shown in Fig. 5 are based on the AR framework ARViewer (see Sect. 2 and [12]). The method was applied as shown in Fig. 4, considering the defined six fundamental properties (see Sect. 5.1). The structure in Fig. 6 illustrates the implemented setup with the included systems. The maintenance planning is realized with the defined standard production applications at the office workspace (a) not requiring an additional authoring system. The created maintenance documentation is stored and managed on a server with the PDM system (b). On the shop floor, the maintenance engineer accesses the data from the PDM server and the online production systems (CNC, MES) using the AR maintenance application on a tablet computer (c).
For the exemplary CNC turning and milling center, the 3D model was designed with the CAD application Autodesk Inventor [21]. To move the machine axes and to show 3D model components in correct position in the AR application, assemblies had to be restructured in the model hierarchy. The corresponding assemblies themselves were used as markings in the AR view. Markers for tracking were added to the machine as predefined assemblies with a defined name. The open file format Wavefront obj [22] was used for exchange with the AR application. It offers various options for the resolution, whereby a small number of polygons is sufficient for the accuracy of the AR visualization.
In the next step, all other required data was arranged, such as photos of real machine components (e.g. jpg file format) and additional documents with detailed descriptions and tables (e.g. pdf file format). In order to gain access to the data of the MES and CNC during the maintenance, a configuration file was created in the xml file format [23] with entries for the connection data and required data records of the machine (only required once per machine). Next, the individual maintenance steps were created with the provided data. For this purpose, the technical documentation system XMLmind XML Editor [24] was used as an example, whereby the data is stored in a structured xml based form for the system requirements but is created in a user-friendly form with a clear graphical user interface. The used standard DocBook [25] specifies, checks and stores specifications for the structure in an xml file [23].
The created maintenance data and described systems are integrated in a structured form in the PDM system Siemens Teamcenter [26]. With the existing API (application programming interface) of the PDM system and its client-server architecture, the current maintenance data can be accessed from any device and location via network, which is used by the AR maintenance application. The papers [12,13] and [14] describe more details about the usage, interfaces and implementation for the source production systems (CAD, PDM, documentation system, CNC, MES).
At the machine tool, the maintenance engineer receives the created step-by-step instructions including 3D models and online data for the maintenance tasks, so that all information is available on a single device without a separate handling of external systems. Thereby, the online systems are used to support the maintenance engineer with the latest machine information and to detect problems on the machine  Fig. 6 Implementation setup of the new efficient integration process of production data with a desktop standard production applications, b maintenance data management server, c AR maintenance application on mobile AR devices tool. The user interface is adjusted automatically with different visualization elements based on the current working step data. For the different AR devices, e.g. with different display sizes and interaction devices, separate approaches for visualization and user interaction were implemented to create the best benefit.

Evaluation of the sample implementation
The application of the new developed method in the various maintenance scenarios demonstrates their applicability for the maintenance of machine tools. The goal of the efficient process of integrating production data into AR based maintenance applications (see Fig. 2) is achieved by realizing the defined properties of the method described in Sect. 5.1: (1) By using the existing source systems (CAD, PDM, documentation system) the maintenance planner works with well-known software applications, needs no induction to new systems and uses its knowledge and experience. (2) By using the existing production data, less data has to be created, has not to be manually transferred to a new system and fast updates are executed since no intermediate system is required. (3) A user-oriented and short workflow is created by the integration of available production systems and data, the automation of processes and the usage templates for maintenance instructions. (4) The automation of the process is applied by the polygon reduction and format conversion of the 3D models, the adaption of the AR user interface and by connecting and using available production systems and online data. Thus, for the creation process no programming is required. (5) By implementing uniform interfaces, the AR application is used with different maintenance cases, data sources and machine tools.
(6) By providing various AR components, the application is used with different AR devices appropriate to the current use case, e.g. with different visualization and interaction components for tablet computers and HMDs.
A summarized comparison of the typical previous and the new developed creation process is shown in Table 2. This shows that the effort in the planning process, the required expertise of the maintenance planner and the overall complexity is significantly reduced.
Especially the use of the existing standard documentation system and the integration of external systems for generating the AR based documentation as well as the automatic update, if changes of the documentation are necessary, can reduce the additionally time for the AR based documentation by half, because all texts, pictures and the data structure have to be created only once. The 3D model components have to be integrated additionally, but the effort is reduced by automated processes. For complex machine tools with text-based documentations over a hundred pages, the savings can be multiple days. In a company with many different machine tools, the expenses can be reduced accordingly.

Conclusion
The presented work describes a new developed method for an efficient creation and integration process of the maintenance documentation and production data into AR based maintenance applications for machine tools, thus reducing the high expenditure of the process and the necessary expertise of the maintenance planner for the AR technology. As basis for the development of the new method, an analysis of the AR based maintenance process and required data as well as user and technology requirements was processed. The result is an improved planning and integration process for required production data and systems that is user-oriented and partly automated. New maintenance instructions are created in a structured form and in an action-oriented Several manual export and import processes of the CAD model Additional DCC application for polygon reduction and format conversion Additional authoring system and process for creating the AR based documentation and extend the 3D model Several manual export and import processes of the AR based documentation data Manual process to add existing data from external production systems Manual update of the maintenance documentation data Automatic integration of the CAD model into the PDM system Automatic process for polygon reduction and format conversion Existing documentation system for generating the AR based documentation and CAD system for extending the 3D model Automatic integration and deployment of the AR based documentation data with the PDM system Integration of existing data from external production systems (online) via interfaces Automatic update of the maintenance documentation data manner with existing standard applications. Furthermore, existing production data can be included through uniform interfaces. Updates of the maintenance data are processed fast because no additional software and manual working steps are required. Besides the proposed creation and integration process, a design of an AR based maintenance software architecture with several AR components was created to realize the verification of the method. The implementation and verification of the new method in various maintenance scenarios demonstrates their applicability for different AR devices and machine tools.