Concepts for creating augmented reality based technical documentations for the maintenance of machine tools

The maintenance process of machine tools becomes more complex and difficult due to its increasing complexity and individuality. Augmented reality (AR) can support the maintenance engineer by visualizing the process with individual working steps and further information in a user-friendly manner. In order to an efficient deployment, a short and automated creation process of the AR based documentation is required including the integration of existing systems and data such as the technical documentation, CAD models and PDM data. This avoids a complete rewriting and manual creation of the AR based documentation in addition to the standard paper-based version. Therefore, various concepts for creating AR based technical documentations for the maintenance of machine tools are presented and verified including an analysis of the required documentation data, systems and processes.


Motivation
The augmented reality (AR) technology can be used to support the maintenance engineer in the maintenance process of machine tools [1]. With the deployment of a supporting AR application, the increasing complexity of the machine tools and their maintenance can be handled better and errors can be reduced. Furthermore, the AR application provides the required specialist knowledge about a particular machine tool and assists the worker on the individual machine tool. With the use of AR, the maintenance process is visualized through individual working steps and further information by superimposing the real environment with virtual information in the right location in the user's field of view [2]. This improves the human-machine interaction with a user-friendly visualization of all required data.
An important area of the deployment of AR systems is the content creation of the specific use case. For the maintenance B Philipp Klimant philipp.klimant@mb.tu-chemnitz.de 1 Institute for Machine Tools and Production Processes, Chemnitz University of Technology, Reichenhainer Straße 70, 09126 Chemnitz, Germany of machine tools, a technical documentation is required [3]. In addition, a digital documentation offers the possibility to include existing systems and data of the company. The creation process can lead to a high effort and must be designed efficient for a use in a company. Therefore, concepts for creating AR based technical documentations in different use cases are needed.

Problem of the creation process
A machine tool requires a technical documentation that is typically created as a digital office document and printed on paper [4]. This existing form cannot be used directly for the AR application because of its data format and structure. For current AR maintenance applications, the documentation is typically created twice with a special application for AR that leads to a high creation effort. Furthermore, updates of the documentation have to be made twice for each use case.
The AR application needs a special data structure containing two main aspects. On the one hand it represents the machines' technical documentation in single steps and on the other hand it defines the special AR data including e.g. tracking information [5]. Additionally, existing data in the company's infrastructure should be referenced to extend the documentation possibilities and to reduce the creation effort. This requires the connection of available systems such as CAD or PDM (product data management) for an efficient integration of the data.

State of the art of creating AR content in production environments
For production and maintenance use cases, several approaches for content creation were explored. The project "AR-Handbook" [6] used videos and artificial intelligence (AI) to create digital manuals with step-by-step instructions. Therefore, manual tasks were recorded with a camera. Afterwards, the videos were split automatically in single sequences and recognized in the following executed tasks. The support for the user is provided by a semi-transparent video overlay. The advantage is the simple and quick handling of the system. The disadvantage is the limitation to videos, no further information is shown. In [7] a special authoring software is used to create assembly instructions for an AR based support of the worker by arranging 3D models, texts and multimedia data. The sequence of the assembly is imported from an assembly planning program. The processed data is stored in a database. The position and orientation of the 3D models in the real environment is shown in the editor using previously recorded images. In [8] an AR system for assembly is created using middleware to connect external systems. By using a special authoring software, individual information for each working step is created. This method has the advantage of creating individual information in a user-friendly way, but needs a high effort to create an additional instruction besides the standard documentation.
The connection to external data systems is an important component to reduce the effort of the creation process and to extend the documentation with additional data. In [9] an AR based maintenance system is presented that uses data from a PDM system. With an authoring application, maintenance files for the AR usage are generated based on a formal description of various maintenance activities. Also in [10], PDM and CAD data is used with an authoring application for the maintenance planning. Some information for the worker is arranged manually and some information is arranged automatically to reduce the effort and extend the instructions.
In summary, Fig. 1 illustrates a typical current creation process for the AR maintenance documentations. A common way is the use of a special authoring system to create the AR documentation with the required AR structure. The maintenance documentation text and steps have to be typed manually in the authoring system by the maintenance planner. Additional information is included by importing and combining external data manually, e.g. pictures, documents, 3D models and product data. This data is needed to extend the textual maintenance documentation and to exploit the  Fig. 1 Current creation process for AR maintenance documentations with a special authoring system with high manual effort possibilities of AR. Afterwards, the created AR documentation is exported from the authoring system on a PC and imported in the AR maintenance application on a mobile AR device to use it on-site the machine tool. This process is characterized by a high effort with many manual steps by the maintenance planner. The additional authoring system has to be managed in the company and the documentation has to be created and updated a second time besides a standard paper-based version.
In the authors' preliminary work, the AR framework "ARViewer" was developed, which is designed for production usage and which includes various AR components (e.g. for tracking) and interfaces for different use cases (e.g. data connections) [5,11]. Also the usage of PDM data was explored to extend the AR support for the worker in the maintenance process [12]. The AR framework is used to visualize the results of the developed creation concepts.

Aim of the work
The aim is the usage of an existing technical documentation system for the maintenance of machine tools to create the AR documentation in contrast to the special AR authoring system described in chapter 2 and Fig. 1. The advantages are that the maintenance documentation has to be created only once for different required formats, e.g. paper-based versions or websites, and updates can be generated fast for all formats. Moreover, the required external data, e.g. pictures, documents, 3D models and product data, is linked between the technical documentation system and its source systems, so that they get automatically updated on external changes. With the usage of an existing technical documentation system, the maintenance planner can use a system he is familiar with and no further system needs to be maintained in the company. Figure 2 shows the new proposed creation concept for AR maintenance documentations. The requirements of maintenance documentations and the current documentation systems and processes in companies are different. Therefore, various documentation concepts with different documentation systems are examined in this paper. To ensure an efficient deployment and reduce the creation effort, existing systems and data, such as pictures, documents, CAD models and PDM data, are integrated.

Approach of the work
To reveal the problem of the creation process and develop the new concepts for AR documentations for the maintenance of machine tools, the state of the art of creating AR content in production environments was examined. It shows that typical current creation processes are using special authoring systems to create the AR documentation with the required AR structure which leads to a high manual effort for the maintenance planner.
Next, an analysis of the required maintenance data, creation processes and systems is performed to create the basis of the new documentation concepts that should reduce the high manual effort. Especially the CAD application is examined because it shall be the starting point for the documentation process since the 3D model of the machine tool is the basis of the AR application to determine the location of the information in the environment and to overlay the information over the real machine tool on the basis of the tracking.
Afterwards, the main concepts for the documentation systems and processes are developed. For different requirements in different companies, three documentation methods are examined: technical documentation system, spreadsheet application and PDM system. All three systems provide a graphical user interface for the maintenance planner and save the created data in a structured form that is applicable for AR maintenance applications.
Finally, a verification of the developed concepts is performed with sample implementations of their components.

Required maintenance data
The technical documentation contains all information for the worker to perform the maintenance. This includes all required tasks, e.g. assembly instructions, as well as general hints for the worker, e.g. safety warnings. Task instructions have to be created as procedure descriptions in text form and can be extended with pictures, data sheets and checklists. In a digital form the maintenance documentation can be extended with additional information such as videos, 3D models and other data sources, e.g. diagnosis and condition monitoring systems. The data can illustrate the tasks, help to find errors and make the overall maintenance process more efficient.
Furthermore, with a digital maintenance application the results of a maintenance process can be recorded and evaluated. This can include occurred problems, disorders and damages on the machine tool. Besides text-based descriptions, pictures and videos can describe the current state. This improves the quality management.
To use all the described data in an AR application, it has to be created in a structured form. This makes it possible to map different use cases in a uniform AR application. The user interface for creating the data is described in the following chapter of the creation concepts. Figure 3 shows the developed data structure for a maintenance task. A task consists of a unique number, name, description, note and interval to save the period of time in which the task has to be executed. Each task includes different steps that the worker has to  Fig. 4 General creation process for the maintenance documentation perform. A single step has a unique number, name, description, note as well as materials and tools to show required items. In addition, different external data can be assigned to a step. The 3D model is required for the correct position and orientation of the information in the AR view and used to highlight parts of the machine tool. The PDM data set integrates current production information. The MES and CNC data set can connect live data on the machine tool e.g. for diagnosis purposes. Also, available documents, pictures and videos of the machine and process can extend the maintenance documentation.
At the end of a maintenance task, the result is saved with a description and date for the quality management. Pictures and videos that are recorded directly with the AR application can be appended. With the defined data structure, each required information and the overall process using single steps is mapped. For different use cases in different companies an adaptation of the data structure with additional data fields is possible without interfere with the basic concept.

General creation process
Based on the required maintenance data, Fig. 4 shows a general creation process for the maintenance documentation. The approach is the creation from general down to the detailed information. First, for a new maintenance scenario a new task with its data, such as name and description, is created. Next, the single steps with their data are created. For each step external data fields are linked, such as the 3D model, external systems (PDM, MES, CNC) and media files (pictures, videos, documents). Afterwards, more steps and tasks can be added. At the end of the creation process, the created data is stored in the developed data structure and in a standard file format.
A basic concept is that all the existing data is reused and linked to the maintenance documentation. In that way, the creation effort is reduced and the data can be updated easily. Furthermore, for a user-friendly creation process, templates for each task are provided. The templates represent the developed data structure with an appropriate user interface. The specific process, file format and user interface depends on the used systems in the company. Therefore, different creation concepts are developed in the following chapter.

Required systems for the creation process
To reach the goal to create the AR documentation for the maintenance of machine tools directly without an additional authoring system, three different systems were worked out and presented in detail in the following chapter. Different concepts are required because of the different preconditions with existing systems in various production environments. Figure 5 shows an overview of the presented systems. The first investigated maintenance documentation system is the technical documentation system that is especially developed for creating technical documents. Problems are the integration of external data and the usage for AR applications.
The second system is the spreadsheet application that is not prepared especially for documentations but provides a structured user interface to store the data in a useful way. It shall be investigated because not every company is using a special technical documentation system but office software. The problem of using this is typically the integration of external data.
The third system is the PDM system that is often used in production environments. Advantages are the graphical user interface, the structured storage and the directly linkage of data. But the system has to be prepared for special documentation requirements and AR applications.
For the integration of external data, the three areas CAD application, PDM system as well as pictures and existing documents shall be included. This additional information is required for the AR use case or can extend the standard documentation.

CAD application as starting point
3D models are required in AR applications to determine the position and orientation of information in the environment and to overlay the information over the real machines on the basis of the tracking. In addition, the 3D models are used for visualization, e.g. to highlight components of a machine that are not visible or that are relevant for maintenance. For that reason, the existing CAD data of a machine should be used as a source for the 3D models. The CAD application can be used as a starting point for the AR based maintenance documentation because the 3D models are already existing and the users are familiar with the software. Furthermore, no other software has to be managed in the company. Figure 6 shows the information created with the CAD application for the AR maintenance documentation. Assemblies can be added or rearranged for the maintenance requirements. Also marker objects for the tracking are added and readout in the AR application. For the implementations in this paper, the CAD application Autodesk Inventor [13] is used.
The existing CAD data of a machine is initially in the data format of the CAD application used for design. These proprietary formats are only supported by the respective CAD application and cannot be used directly in the AR application. This can be done using exchange formats, e.g. jt, step, iges, obj, fbx or mesh, that are read by many different applications and libraries, but not all original data are retained. They are interpreted differently by different systems. Various software can be used for converting and reading the 3D model in the AR application, e.g. Assimp [14] or PiXYZ [15].
Due to the mostly very large amounts of data of the CAD models and the real-time requirements of the AR application on mobile devices, a polygon reduction of the 3D model is advisable of the machine or of assemblies. The reduction usually has to be made to less than a tenth and should be carried out automatically in order to avoid manual processing and the use of an additional application. Therefore, additionally algorithms can be added to the workflow, e.g. with VCGlib [16].

Technical documentation system
Technical documentation systems are specially developed for the creation of documents in the technical area and are Fig. 7 User interface of a technical documentation system also used for the maintenance documentation of machines. In contrast to office documents, the individual parts of information are given certain properties in order to define a special structure and to make them available for various use cases. Furthermore, they provide additional functions, e.g. versioning of the documentation and the collaboration of different users on documents. Figure 7 shows an example of a graphical user interface (GUI) of the documentation system XMLmind XML Editor [17]. By entering text and adding images, a document is created via a user-friendly interface. An explicit structure is made in order to assign all entered information to a special meaning. Formatting can also be specified. By defining the document structure, a required format can be ensured and saved. For the maintenance documentation the data structure and creation process defined in Sects. 4.1 and 4.2 is used. The data and GUI structure is based on Fig. 3.
The possibility of defining certain properties and using them in the document enables it to integrate additional data. These cannot be displayed in the documentation software itself, but can be linked for later use. For example, a machine component number can be entered in a field for CAD, whereby the corresponding 3D model of the component can be loaded and displayed in the AR application.
The data formats of technical documentation systems are mostly based on xml [18]. This format saves the information in a well-structured form and can therefore also be processed in other applications. In addition, standards such as DocBook [19] or DITA (Darwin Information Typing Architecture) [20] are used to specify the data structure and to validate the entered information.
Due to the used data structures and formats, the information created with the technical documentation system can be processed in the AR application. The documentation system also offers the option of generating documentations for different media, such as text documents and websites. The common basis means that changes can be quickly adopted in all formats.

Spreadsheet application
Spreadsheet applications offer a user-friendly and widely used way of entering and storing information in a structured manner. Maintenance tasks can also be documented and structured with this application type especially if no additional software shall be used in the company. In order to evaluate the created documents in other applications, defined structures must be considered. Figure 8 shows an example of the graphical user interface of the spreadsheet application Microsoft Excel [21] for entering the maintenance tasks. A specific meaning is defined for each column, e.g. the number in column one and the name of the task in column two. The maintenance tasks are entered in the individual lines. A uniform structure can be specified by means of a corresponding template using the data structure defined in Sect. 4.1.
External data can only be integrated into the documentation by specifying names that are replaced by the actual data in the maintenance AR application. For example, by specifying a component number, the 3D model exported from the CAD application can be assigned and visualized.
The created documents can be saved in various formats for further processing. One option is the office format xls [22] or xlsx [23]. The data can also be saved in structured exchange formats such as xml [18]. An exchange via a text file (txt, csv) is also possible.
The AR application reads the stored documents and displays the specified information accordingly. This requires that the structure of the created documents match the data structure defined in the AR application. Additionally, the spreadsheet application can also save the data in a format for direct use (e.g. pdf [24]) or the data can be imported into a word processing program in order to format and include it into a more extensive documentation.

PDM system
In a PDM system, all required product, process and organizational data of a production is stored, managed and provided during the entire product life cycle [25]. In this way the Fig. 9 User interface of a PDM system for the maintenance documentation complexity and variety of products and processes can be controlled. With the adaptable graphical user interface and data structures, maintenance documentations can be created. The advantage is the integration of existing data that can be linked directly in the PDM system. In addition, automatic versioning of the documentation and collaboration between several people are possible. Figure 9 shows an example of the graphical user interface of the PDM system Siemens Teamcenter [26] for entering the maintenance documentation. To realize this, corresponding input masks and data structures must be created in the PDM system in order to implement the defined data structure of Sect. 4.1. The maintenance documentation is structured and stored directly in the PDM system together with all other available data, which enables a direct link between them.
For the integration of further data and systems, corresponding interfaces are available as standard in the PDM system. In this way, e.g. 3D models from CAD applications are added and versioned. Required 3D models in the maintenance documentation can be selected, linked and visualized using attributes. Overall, this creates a consistent data model, which means that updates can be carried out quickly. Existing product, process and organizational data, e.g. part lists, documents, drawings and spare parts information, can be integrated into the maintenance documentation.
The maintenance documentation is stored internally in the underlying database in the PDM system on the basis of the created data structure. For use in other applications, either direct access via an interface of the PDM system can be implemented or an xml file [18] can be exported.
Due to the client-server architecture and the interfaces (API-application programming interface) provided by the PDM system, the AR application can access and visualize the maintenance documentation on-site the machine. Linked data can also be downloaded and displayed. With the direct connection of the systems, a quick update is possible if some information changes. In addition, the PDM system offers the option of generating office documents, which means that they can be reused for other output media.

Setup and sample implementation
The creation concepts for the AR based technical documentations for the maintenance of machine tools were verified with a maintenance scenario of a real industrial environment. With the implementation, the feasibility and viability of the concepts shall be demonstrated. For the verification, the AR application "ARViewer" [11] with different AR devices were used. Based on an existing paper-based documentation, a new AR based documentation was created with the three presented concepts. Therefore, the CAD application was used as a starting point and the 3D models were included in the documentations. The result was visualized with the AR application on the example machine tools.
For the sample implementation of the developed creation concepts for the AR based technical documentations, the Niles-Simmons CNC turn and mill center N20 MC with a Siemens Sinumerik 840D sl control system was used (see Fig. 10). Detailed machine specifications are available in [27]. Due to the complex structure of the machine, it is suitable to demonstrate the support of complex maintenance steps with AR. On the basis of the existing paper-based maintenance documentation and CAD model, the three presented concepts of Sect. 5 were realized with the process of Sect. 4.2. The basis for saving the AR maintenance documentation is always the same data model from Sect. 4.1, which means that no adjustments need to be made to the AR application for different creation systems. Besides the CAD model, text descriptions, pictures, documents and PDM data were integrated as input values into the AR maintenance documentation (see Fig. 3). As mobile AR devices, industrial tablet computers [28] and HMDs [29] were used to visualize the resulting AR maintenance documentation. Figure 10 shows the resulting AR application on a tablet computer with the created maintenance documentation. The individual maintenance steps with the associated information for the respective maintenance case are visualized stepby-step in an information field in order to guide the user through the maintenance process. Using the AR technology, the linked 3D model is used to superimpose the relevant components on the real machine and an information field is displayed in the correct position in order to highlight the location of the maintenance work for the user. This visualization can also be used during the maintenance planning process. The AR visualization can be previewed by displaying a photo of the real machine or just the 3D model instead of the live camera image.

Results of the sample implementation
The implementation of the three developed creation concepts in a maintenance scenario of a real industrial environment demonstrates the applicability for the AR based maintenance of machine tools. The AR maintenance documentation can be created directly in a production application without an additional authoring system. With this method, the creation effort can be reduced significantly because the documentation has to be created only once for different media and updates can be made faster. The three different concepts are useful because of the different requirements and conditions in different companies.
With the defined data structure that represents the technical documentation in single steps, all required information can be created and saved in all of the three production applications and visualized in the AR application. Existing data in the company's infrastructure, such as CAD and PDM, can be included in different ways to extend the documentation and to reduce the creation effort. The CAD system provides the required 3D model in a user-friendly way without the use of an additional software.

Conclusion
With the presented concepts for creating AR based technical documentations for the maintenance of machine tools, the process is realized without a special authoring software. Therefore, the three documentation methods: technical documentation system, spreadsheet application and PDM system were examined. As basis, the required documentation data, systems and processes are analyzed and the three developed concepts are verified with an AR application and a machine tool. The implementation and verification of the method shows that a short creation process and efficient deployment of the documentation is realized because it avoids a complete rewriting and manual creation of the AR based documentation in addition to the standard paper-based version and updates can be done fast. Furthermore, existing systems and data of a company such as technical documentations, CAD models and PDM data are integrated for an efficient overall process as well as to extend the documentation and to use the advantages of an AR application. Moreover, live data of MES or CNC on-site of the machine tool can be integrated with the same concept to extend the documentation [5]. With the usage of an existing technical documentation system, the maintenance planner can use a system he is familiar with and no further system needs to be maintained in the company. As future work, adjustments of the documentation should be possible in the AR application on-site of the machine tool during the maintenance process that are also stored in the source systems. This offers a simple way for the user to extend the documentation and to make required changes if the working environment has changed or faults were occurred.