RetrofittAR: Supporting Hardware-Centered Expertise Sharing in Manufacturing Settings through Augmented Reality

Abstract

Since almost the onset of computer-supported cooperative work (CSCW), the community has been concerned with how expertise sharing can be supported in different settings. Here, the complex handling of machines based on experience and knowledge is increasingly becoming a challenge. In our study, we investigated expertise sharing in a medium-sized manufacturing company in an effort to support the fostering of hardware-based expertise sharing by using augmented reality (AR) to ‘retrofit’ machines. We, therefore, conducted a preliminary empirical study to understand how expertise is shared in practice and what current support is available. Based on the findings, we derived design challenges and implications for the design of AR systems in manufacturing settings. The main challenges, we found, had to do with existing socio-technical infrastructure and the contextual nature of expertise. We implemented a HoloLens application called RetrofittAR that supports learning on the production machine during actual use. We evaluated the system during the company’s actual production process. The results show which data types are necessary to support expertise sharing and how our design supports the retrofitting of old machines. We contribute to the current state of research in two ways. First, we present the knowledge-intensive practice of operating older production machines through novel AR interfaces. Second, we outline how retrofitting measures with new visualisation technologies can support knowledge-intensive production processes.

1 Introduction

Computer-supported cooperative work (CSCW) research has, from the beginning, focused on how knowledge and expertise are constituted in their respective practices and how they can be mediated through the use of technology. Indeed, CSCW has already contributed significantly to alternatives to the ‘standard’ knowledge management approach (see e.g. Nonaka 1994) in and through a focus on knowledge as embodied in action. This ‘practice paradigm’ is by now prevalent in the CSCW community, and assumes the need for a detailed understanding of workplace practices prior to the development of technical artefacts to support expertise sharing (e.g. Schmidt 2012). While early approaches around knowledge sharing focused mainly on artefact-centred repositories that encompass information production, storage, retrieval and reuse, more recent approaches focus on the ways that situated and socially contextualised knowledge is shared among knowledgeable actors and how it could be best supported through information technology, as ‘expertise sharing’ (Ackerman et al., 2013). These expertise-sharing approaches, often referred to as second-generation knowledge management, establish a perspective that embeds knowledge exchange as mutual learning in and through work practice. This relates to older discussions regarding the distinction between ‘knowing how’ and ‘knowing that’ (Ryle 1945). One classic example is Answer Garden. This tool supports the development of an organisational memory by helping people to find answers to existing questions (Ackerman 1998) or by locating experts within an organisation (Reichling 2008). Recent research has demonstrated that new technologies have the potential to support expertise sharing in some radical ways (Hoffmann et al., 2019), suggesting that AR and sensor technology, for instance, could result in a ‘third generation of knowledge and expertise sharing research’ (de Carvalho et al., 2018). Such a move is quite distinct form orthodox views of knowlegde management insofar as they position themselves in relation to the ‘embodied’ rather than the ‘tacit’, and in doing so allow for a conceptualisation of knowledge as practice-based, contextually driven and evolving in realtime.

The discussions around expertise sharing become particularly relevant given the current upheaval in manufacturing industries. Many production facilities are presently being transformed by so-called cyber-physical systems, which intertwine physical and software production facilities that operate on different spatial and temporal scales. These systems interact with each other in ways that change contextually. Nevertheless, the supposition that technological innovation of this kind will necessarily transform all manufacturing environments is unwarranted. The manufacturing which takes place in many small- and medium-sized companies is usually based on specialised machines that have been in place for many decades. These machines were designed and built explicitly for the production of specific and individual products. Such machines cannot simply be replaced (both from an economic and organisational point of view) by modern cyber-physical systems.

The ‘SME problem’, so to speak, is compounded by the diminishing availability of special machine operators in these SMEs, who are gradually ageing and retiring and thus creating what was once termed an ‘organisational knowledge’ problem. ‘Know how’, put simply, is leaving the company. The experiential and embodied knowledges of long-serving employees often play an important role in production quality. In the context of Industry 4.0, then, we argue that the transfer the accumulated experiential knowledge to young professionals through improved contextualisation with the help of new technologies is a relevant and significant challenge for SMEs (Ludwig et al., 2016). In this paper, we report on a case study of a medium-sized company that is facing this precise situation. On the one hand, the company’s machinery consists of old machines of proven reliability. On the other hand, the company must repeatedly replace its retiring employees – without losing the expertise acquired over decades. Specifically, we examined how legacy machines can be ‘retrofitted’ using modern lightweight augmented reality (AR) technology to prevent this loss of expertise and to support new employees in learning how to operate these specialised machines. With this paper we take up developments in the CSCW literature regarding expertise sharing by developing new mechanisms for storing machine settings as parameters. Visualization techniques associated with AR allow us to partially explicate knowledge in the form of parameters and to display it in situ in the working environment using AR technology. Hoffmann et al. (2019) show that the processes described can be supported by AR, as the knowledge transfer model is essentially realised by the technical possibilities of AR. The goal in using this technology, then, is to initiate an expertise sharing process for complex production processes. Based on the findings of our detailed pre-study of the practices of workers in operating and configuring special machines, we designed, fully implemented and evaluated the AR-based HoloLens application, RetrofittAR. RetrofittAR enables relatively inexperienced machine operators to operate machines through an innovative interface. Our study clearly outlines how AR technology is beneficial for an expertise-sharing process on the shop floor in a real production environment, even when old machines without numerical controls are being used. The novelty of the paper is to present the results of a practical study with an AR retrofitting application to support expertise sharing. This paper shows, how AR technology can support the operator to remain at the centre of a production process and to make the essential decisions in a production process. In addition to current research, our work demonstrates the role of AR in supporting dynamic, real-time, operations.

2 Related work

Our research combines three research discourses. The first focuses on human-centred manufacturing and the importance of expertise sharing. The second encompasses the appropriation of sociable technologies, and the third deals with novel interfaces for manufacturing settings in particular. Central search terms in the course of our literature search were: ‘human centered CIM’, ‘organisational memory’, ‘augmented reality’ AND ‘expertise sharing’ AND ‘hardware’, ‘augmented reality’ AND ‘retrofit’ AND ‘industry’, ‘human in the loop’ AND ‘industry’.

2.1 Human-centred manufacturing

Various industrial revolutions over time have seen the development of new and different specialised markets. Because customers increasingly demanded individualised and customised products, the dynamics of production, were transformed. For this reason, new forms of manufacturing, work design and innovation became necessary to cope with highly dynamic markets. In this constantly evolving field, computer integrated manufacturing (CIM) has been challenged by alternative approaches around human-centred manufacturing systems. These human-centred forms of organising manufacturing processes, which are substantially based on human skills, experience and competences, are often called ‘anthropocentric production systems’ or ‘human-centred CIM’ (Badham 1991; Brödner 2007). There is an extensive literature which discusses the characteristics of human-centred manufacturing and the challenges associated with it (Corbett 1990; Peruzzini and Pellicciari, 2017). As Grandi et al. (2018) point out, this requires a focus on “performances, aesthetics, reliability, usability, accessibility and visibility issues, costs, and many other aspects.” (p.702) and hence, a transdisciplinary approach is necessary (Grandi et al., 2020). A more limited literature has focused on knowledge sharing in the manufacturing context (Li et al., 2019) who reference the urgent need for “a holistic framework for identifying and accommodating individuals’ needs and expectations of relevant data, information and knowledge” (p.380).

The radical change of management strategy from Tayloristic models to a human-centred, expertise-based approach that promotes continuous competence is ongoing and, represents the beginning of a new phase of industrial development. In contrast to previous manufacturing revolutions, which generated significant productivity growth, the new industrial revolution arguably generates new innovation dynamics dependent on the sharing of knowledge and expertise. The exchange of expertise presents itself as a complex social interaction process (Brödner 2007), whereby developing an organisational memory (OM) is likely to be an essential driver of a company’s success. The concept of OM reveals how organisations use and maintain knowledge in various forms (Bannon and Kuutti, 1996). However, OM is neither simply stored nor unproblematically accessed (Ackerman 1998; Ackerman and Halverson, 1999; Randall et al., 1996). Knowledge and expertise are not subject to rigid structures but rather constitute living and changing content in practice, which in turn carries special implications for possible support technologies (Bannon and Kuutti, 1996).

2.2 Hardware-based expertise sharing

With regard to supporting expertise sharing within manufacturing settings, a number of standard procedures exist, including for instance the use of plans and shift protocols (e.g., Carstensen et al., 1999) or coordinating meetings (e.g. Schmidt 1991). However, when focusing on the practice of manufacturing itself, these mechanisms usually have fallen short because a large number of contextual circumstances are neglected in respect of actual machine operation (Lewkowicz and Liron, 2019). These contextual factors, such as the detailed nature of the materials, the actual condition of the machine (Bowers et al., 1995) or even the current room temperature, can play a decisive role in manufacturing.

Ludwig et al. (2017) focused primarily on these contextual and environmental factors to design a hardware-based approach for collaboratively dealing with this complexity in practice. Using Turkles’ (2002) original concept of sociable technologies, they show how Internet of Things technologies can offer operators new possibilities for communicating, documenting and sharing hardware usage practices. Sociable technologies offer three dimensions for understanding a machine’s functionality and sharing its practical uses: (1) the internal dimension, which provides the user with information on the inner workings of a machine, its current status and the functioning of the complex component structure; (2) the spatial-material dimension, which provides detailed information about the machine’s location and environment (e.g. room temperature and brightness), so that possible relationships with the internal context can be derived; and (3) the task- and process-related context, which provides information on the position in a production chain or purpose and objective of machine use (Ludwig et al., 2014).

When interacting with specialised machinery, the operator is the central unit (Cimini et al., 2020). With the advent of human-centred manufacturing (see Chapter 2.1), new technologies are usually introduced in manufacturing settings with the aim of supporting users rather than replacing them. Although physical demands are changing due to the use of new technologies, cognitive abilities are also potentially increased by assistance systems (Romero et al., 2016). This interaction between operator, technology and organisation (Dregger et al., 2016) leads to an augmented knowledge base, in which human abilities and technical possibilities complement each other (Engelbart 1988). Specifically, the ability to interpret data and derive decisions through automated collected data paves the way for extensive expertise sharing (Romero et al., 2016).

2.3 Supporting hardware-based expertise through augmented reality

Historically, hardware has often only poorly provided information about machine behavior, especially in the case of machines from the first to third industrial revolutions. Written manuals, checklists (paper-based and digital), status updates and so on, provide information but only in a static manner. While modern machines with various sensors provide for much more sophisticated monitoring procedures, traditional interfaces are significantly more restricted in their ability to provide real-time visualizations (Ludwig et al., 2019). In the context of human-centric manufacturing, it is apparent that effective operations need exactly this kind of reactive, dynamic, visualization and AR is considered to have the potential to provide interfaces to display real-time feedback during the operation of machinery. Such interfaces can also be considered as second-level functionalities which not only allow the user to interact with the machine but also explain and support the interaction thus, inter alia speeding up accurate decision-making (Jasche and Ludwig, 2020; Ludwig et al., 2017; Stevens et al., 2009). Sensing the machines’ behaviour and interconnectivity are prerequisites for these interfaces (Al-Maeeni et al., 2020).

Guerreiro et al. (2018) have demonstrated how retrofitting old machines (adding components not included with the original machine) can meet the requirements of modern manufacturing. The technology used in the retrofitting process is aimed not simply at automating the machines but also ensuring that operators receive better feedback from the process. Here, appropriate visualisation of process data is an essential interface between the machine and the human being. The individual design of the retrofit application must also consider the specific user requirements (Ramakers et al., 2016). In the case of retrofitting measures, smart glasses are increasingly used as a visualisation tool, which permits rapid data interpretation and immediate decision-making (Al-Maeeni et al., 2020; Guerreiro et al., 2018). In addition, the spatial recognition capabilities of an AR device can be used to recognise machines, suggest suitable components for a retrofit and visualise them directly. In addition, subsequent actions such as the purchase or sale of such components as well as instructions for assembly or disassembly can be supported with the AR device (Mourtzis et al., 2020).

Research in the area of smart glasses has shown that workers in industrial settings can readily be supported in assembly tasks (Baird and Barfield, 1999; Bhattacharya and Winer, 2019; Dey et al., 2018). Research on systems which support workers in performing assembly tasks by displaying the necessary information at the relevant times, goes back almost 30 years (Caudell and Mizell, 1992). In comparison to other instructional media, such as written paper or videos, AR and its multimodal form of feedback provide speed advantages in the assembly task’s execution (Baird and Barfield, 1999; Fiorentino et al., 2014; Henderson and Feiner, 2011; Jetter et al., 2018). In addition, AR-based instructions allow for a higher level of detail and thus further reduce cognitive load (Bresciani and Eppler, 2009; Paas et al., 2003; Tang et al., 2003). When combined with cyber-physical systems, smart glasses may support the exchange of knowledge and experience between experienced and inexperienced production workers (de Carvalho et al., 2018; Hoffmann et al., 2019). However, most AR studies in industrial contexts focus on more static situations like assembly tasks or maintenance and there is little research on using AR as support during the actual operation of a machine.

Augmented Reality clearly requires authoring processes, scripts or instructions in order to reduce training and other overheads (Ramirez et al., 2013; Roberto et al., 2016; Seichter et al., 2008). These authored instructions, in the main, aim to support reoccurring and static situations like assembly or maintenance tasks. Further work focuses on automatic algorithmic authoring techniques which are based on existing data such as product information or prerecorded videos of a workflow (Fernández del Amo et al., 2018; Petersen et al., 2013). Techniques which are based in particular on video recorded workflows are useful because the data can not only be used to automatically create instructions, but can also be used to analyze the users’ actions and to provide feedback on accuracy. In unforeseen and dynamic situations without prerecorded instructions smart glasses remain useful. Most smart glasses can transmit the field of view of the support seeking person to a remote expert over the internet. The expert can now see the situation and can provide support via speech or by placing virtual objects like arrows in the local environment (Ludwig et al., 2021).

3 Research approach and application field

Our review of the literature suggests that there are no current examples of AR technology being used to enhance knowledge sharing while operating and visualising setting parameters on (older) special machines. There is a clear need, we argue, in settings where the operation of machinery requires specialised knowledge, and where the passing on of knowledge and expertise cannot be guaranteed, for dynamically available information to be provided in support of manufacturing practice in situ. Visualisation techniques hold evident potential. We addressed this specific, but important, research gap in CSCW by examining how AR technology might support operators’ expertise sharing in adjusting and handling machines continuously during manufacturing. We introduced, for the first time, the possibility of using AR to retrofitted legacy machines by designing new interfaces for expertise sharing. The novelty of the paper therefore, and as indicated, consists of the results of the evaluation of an AR retrofitting application in a real production environment. By doing so, we aim to answer the research question: How might an AR application be designed to retrofit legacy machines to foster expertise sharing for knowledge-intensive production processes?

To answer our research question, we examined manufacturing processes in a small German company. Although the increased use of numerical controls for production machines, in particular, can result in high manufacturing precision at fast working speeds (Kumar et al., 2018; Schlegel et al., 2018), specialised machinery that was constructed for specific products decades ago is not equipped with such technological accessories. The machine park of the company we investigated consists of several specialised machines, which are operated by experienced and long-standing employees, and whose expertise has accumulated over decades. However, this type of manufacturing process comes with a risk. If an experienced employee retires, the company must hire a new employee who can handle the machine. Because such employees are difficult to find, the company struggles with prolonged idle time, which can quickly problematise its ability to survive in today’s market.

The company we investigated produces tank heads for apparatus and plant construction and specialises in the metal spinning production process. The process is a single-stage, tool-bound forming process that offers high productivity due to short cycle times. The company owns several specialised spinning machines. Some of the machines are equipped with computerized numerical controls, and others are without such controls. The machines produce heads with different geometries and various materials. The basic machine structure is shown in Figure 1.

Figure 1.

Metal spinning machine.

Figure 2 shows the machine’s production process set-up: (1) a rotating tool provides the final contour of the head, (2) the head is pressed against the tool by a press roll (3) with a certain pressure, (4) which is applied to the press roll along the x-axis and thus produces plastic deformation. In addition, the pressure roller moves at a certain speed v along the z-axis to the edge of the head. The x- and z-axis and their parameters p and v are individually controlled by the machine operator for each head by a rotary control.

Figure 2.

Machine set-up, (1) tool, (2) head, (3) press roll, (4) x-axis with pressure p, (5) z-axis with speed v.

The combination of the tool shape, the raw material in the form of a circular blank and the parameter settings for the x- and z-axis result in the final product of a head. Thus, the metal spinning process comprises several variables that must be assessed and analysed without the support of a numerical control.

For more insight into the company’s structures and practices, we conducted a design case study, which typically consists of three phases (Wulf et al., 2011). The first phase entailed an empirical analysis of the existing practices and usage tools. The second phase comprised analysing results for an iterative and participatory generation of design ideas and solutions. In the third phase, the appropriation of the technological artefact in a real setting, its potential redesign and the artefact’s potential future impact were documented.

4 Empirical study

4.1 Methodology

To examine current practice, we organised and documented eight workshops, observed machine set-up and spinning processes through eye tracking and conducted four semi-structured interviews. The workshops were held together with various representatives in different organisational positions (see Table 1; W1–8). A total of seven representatives from production management, work preparation, machine set-up and technical management participated in the workshops. The workshops were documented through notes made by the researchers and jointly prepared working papers that were discussed and improved in the ensuing workshop. In concrete terms, the contents discussed in the workshops were recorded in note form. After the workshop, the contents were formulated and recorded with sketches (examples are Figures 2 and 4). These contents were verified at the beginning of the subsequent workshop with the core team or distributed to the workshop participants and then improved in a joint session. In workshop 7, a scheme for data collection was jointly elaborated and converted into an Excel document. Subsequently, the documents produced were progressively filled with data by the company (see Figure 6 sheet of paper). Overall, a summary document—a kind of manual—was written, which was improved in several iterations and finally approved by the participants and the management as a complete project documentation. This collection resulted in a 250-page document with all details of the spinning process during the one-year cooperation recorded.

Table 1 Overview of workshops during the pre-study and design phases.

Besides a general briefing, we wanted to observe and quantify the machine operator’s activities and perspective during the process (E1–2). With the help of eye tracking, fixations (points of view that are looked at closely) and saccades (rapid eye movements between fixations) were recorded and analysed. We equipped an experienced operator with the Tobii Pro Eye Tracking Glasses 2. To ensure accurate tracking, the machine operator went through a short calibration phase. Afterwards, the operator started the spinning process. Because eye tracking can be operated wirelessly, the glasses did not restrict the machine operators’ work, providing us with real-time and detailed insights. Thus, we could observe how the employees interacted with the machine during the process without disturbing it. Practical limitations meant that only two recordings were made with one individual who had the capacity to carry out the complex processes and simultaneously record and explain his rationale with the glasses. Nevertheless, eye tracking allowed us to collect data that the machine operator might not otherwise have been able to directly communicate. To evaluate the collected data, we created visuals in a form typical for eye-tracking data: (1) the gaze plot was used to show the sequence and duration of fixations during the process and (2) the heat map also made it possible to identify which areas of the machine and the machine interface were viewed particularly frequently and at what time. The operator was also asked to comment on their actions.

After the workshops and eye-tracking recording, we conducted semi-structured interviews. The interviews were recorded, transcribed and analysed and lasted on average for 45 min. The interviews were based on an interview guide with 14 questions for the Master Craftsman and Logistics Specialist. The five central questions were as follows:

• What makes the forming process difficult to learn?

• How did you learn to operate the machine? What training levels did you go through?

• What makes a successful forming process? How does a good result look like?

• What do you use as a guide when you operate the machine, for example increasing or decreasing the pressure? What role does your experience play in this?

• Are there differences between the various heads in terms of the forming process and the degree of difficulty?

The interview guide for the Production Manager and Technical Manager comprises 12 questions. The five central questions were as follows:

• What do you consider to be relevant data for operating the machine?

• What specifications/content would you like to make/teach?

• What are the disruptive factors (Why is process-safe production a challenge)?

• Please describe the connection between school-based and in-company training and the acquisition of process knowledge!

• Everything is allowed and possible: What would you do/provide to make process-safe production possible?

Based on the empirical data, we derived some design challenges (see Chapter 5), which led to our concept for supporting hardware-based expertise sharing through AR (see Chapter 6). We implemented our concept as the HoloLens prototype RetrofittAR, the performance of which was later evaluated.

4.2 Results of the empirical study

As previously mentioned, the company is in competition with several other companies and has to maintain productivity levels while managing costs. Thus, the company tries to differentiate itself through a special product portfolio. Additionally, it offers low prices, which presuppose a high degree of economic efficiency on the production side and the use of highly specialised machinery. Both aspects require that there is a huge amount of expert knowledge deployed in the specific production practices. We examine these aspects in detail below.

4.2.1 Practical expertise sharing – more important than theoretical knowledge transfer

The operators play a decisive role in successful production, as they must be able to operate the specialised machines reliably (W1, P1). The challenge is that this reliability is based on implicit knowledge and experience gained over a period of time. Therefore, the production process’s success relies almost entirely on individual operators. There is a large risk, as one interviewee suggested: ‘At the moment, everyone who retires with us takes all the knowledge they have acquired in 20–25 years on the machines with them. Because it’s not recorded or written down anywhere’ (I1).

Relying solely on school-based training is problematic because content adapted to specific company production processes is not available. The school-based part of the training, in particular, did not fit the company’s requirements because its specific machinery was not included in school-based provision (W1, P2).

When we send the young people to basic training, filing is part of the curriculum. This does not add any value to our forming process. Rather, the trainees must learn what physical properties and forces are involved in the forming process. (I1)

The company, therefore, tried to tackle this challenge through a long-term, dual training strategy. In addition to the theoretical content, which is taught during school-based training, the company fostered the transfer of practical content. However, it turned out that the significant level of error generated during the training process was unacceptable. Put simply, no mistakes are allowed.

The forming process is simply based on learning-by-doing. Gaining experience, gaining experience. And this experience is gained over time. And when you’re faced with productivity and time pressure, those are two factors that are difficult. (I4)

Under these conditions, it is of particular importance that the machine and its components is adequately explained, as this is the only way to ensure actual learning-by-doing within the actual production environment. I3 outlined:

First of all, it is important to know the machine and its characteristics. What you produce with the help of the machine. So that the new employee understands this. Then the control panel itself is important. What do the buttons mean? Where do I have to adjust what? But this static content can easily be transferred. (I3)

I1 highlighted the importance of the expertise gained onsite during the operation of a machine:

Every article has a certain diversity. They may only be nuances, but you have to recognise the nuances in the production process on the manually controlled machines and react immediately. A cause–effect relationship must be established. Now this and that happens on the part, and I have to turn the wheel. Another part may have different adjustments. (I1)

As a result, a knowledge gap was identified which had to do with inadequate theoretical training content and the highly contextualised manufacturing processes. The company, therefore, established a training phase for new employees in which they would be accompanied by an experienced foreman. The experienced employees withdrew from their day-to-day business, thus creating the capacity for in-house training activities. However, this in-practice teaching had some shortcomings:

From my perspective, the learning environment seems to be a big problem. We’re training people on the equipment in a very loud and dirty environment. You have practically no learning workshop or closed learning area. You are in the production hall, in the dirt, right next to the noisy plant. And we don’t manufacture any training parts either; these are all parts that are sold. (I1)

To better plan and track staff training, the qualification levels of in-house training within W4 were jointly recorded and documented, as no corresponding documents were currently available: ‘We also don’t have any documents or tools that could be used to do this’ (I2). Thus, the training’s success was highly dependent on the people involved. Furthermore, in addition to the technical expertise, the ability to train new personnel was of decisive importance:

It is not up to every employee to train the employees on the machine. One can do it; the other can’t. One can do it well; the other can do it badly. One person may misunderstand or take it the wrong way. It must be someone who passes it on sensibly. Not just anyone can do it. (I3)

4.2.2 Determining deviations during the manufacturing process is highly contextualised

During the production of tank heads, deviations between the planned products and the actual products sometimes occur. To determine the cause of these deviations, it was necessary for the machine operator to compare the head’s current state with the target specifications. The head’s target geometric specification is determined by drawings and standards. These standards included the head’s written dimensions which instruct the operator which characteristics must be measured while doing the quality check (W5, P4). These pre-specified and written characteristics were labelled in documentation as hard quality characteristics. In addition, the surfaces and overall visual impressions were constantly checked by the operator using so-called soft quality characteristics (W5, P5). The operator either detected deviations after production by checking the head’s circumference or by determining the head’s behaviour on the machine at an early stage during the production process through observation (W5, P6) (see also Figure 3): ‘I determine the quality by closely observing the head on the machine. I pay attention to the gap between tool and head and to the fluttering movements of the head edge’ (E2, P5).

Figure 3.

Operator observes the spinning process in detail; left: head flutters; middle: the gap between tool and head; right: crimping of the head.

The results of our eye-tracking study clearly showed that the operator observes the behaviour of the head on the machine in a highly focused manner at each stage: ‘I must always keep my eyes on the head. This is the best way to detect deviations’ (E1, P5). These observations allowed the operator to intervene during the production process if needed. Thus, they constantly compared the actual situation with a target specification and derived the control deviation. When a head was completed, the circumference was also measured. If the measurement was outside the tolerance, the head might need to be reworked or scrapped (W3, P5).

4.2.3 The real expertise can be seen in the adjustment of the parameters

The operator typically uses the deviations described in the previous chapter to define the settings or manipulate input parameters. Our workshops revealed that observing the production process required considerable in situ expertise. Moreover, the parameter adjustments required extensive expertise acquired over many years and, above all, a constant check of adjusted parameters and desired effects in practice. We conducted a precise elaboration of the detailed settings requiring manipulation during the seventh workshop. First, the head was divided into different areas. However, these areas depended to a significant degree on the size and geometry of the head. We, therefore, developed a uniform definition of the areas together with the operators. Figure 3 shows that at least three phases occurred during a head’s production. In the early production phase, the operator focused on the fluttering of the head. In the further course of the process, the operator focused their attention on the gap between the head and the tool, as well as the head’s outer edge.

From the beginning, the first step, that is pure experience. That means that when I have a 3 mm sheet in front of me, I start with a pressure of 30, for example. That is pure experience. As soon as I start pressing, I have to watch the head very closely. That means I have to see how the sheet is behaving and increase the pressure according to the observed behaviour of the sheet. (I4)

I2 expanded on this statement by pointing out several important parameters: ‘From my point of view, there are three parameters to consider: pressure, speed and rotational speed. The pressure and speed vary along the radius of the head, and the rotational speed is always constant’.

The interviewees told us that how the pressure is adjusted is of particular importance: ‘The pressure must be increased slightly. Caution is required’ (E1, P5) or as P9 explained: ‘The pressure must be carefully increased in several stages’. We, therefore, developed the head plan shown in Figure 4.

Figure 4.

Variation of the pressure profile along the radial direction of the head.

To enable the machine operators to adjust the machine, using the machine’s three main control variables: (a) the pressure of the x-axis, (b) the speed of the z-axis and (c) the rotational speed of the tool. For example, the following instructions resulted from observation and comparison with the setpoint. ‘If the head flutters too much, I'll have to relieve the pressure’ (E1, P5). As the process progressed, it became more important to increase the pressure so that ‘there is no gap between the head and the tool’ (E1, P5). Both settings referred to the pressure of the x-axis (blue line in Figure 4). As the degree of deformation increased along the radius of the head, the pressure of the x-axis needed to be increased. This increase needed to be entered manually by the operator, via a rotary control on the control panel beside the machine (W7, P5).

As we previously mentioned, the exact course of the blue curve in Figure 4 was different for each article to be produced. Therefore, the settings were recorded relative to the following characteristics (W7):

• Geometry and surface of the head

• Material of the semi-finished product

• Production machine

Any combination of features created new and very specific pressure profiles of the head and required that the machine operator know the settings for all feature combinations. In this case, the operator needed to have about 70 pressure profiles at hand. The former machine operator I4 described the relationships:

The bulge from the head is practically the pressure on the roll. The surface of the head is affected by the speed. The parameters are directly related to the machine and the raw material. Also, the speed and pressure are directly related. (I4)

This complexity, consisting of the product-specific dependence between radial position and pressure was further increased by disturbance variables, such as wear and tear of the tools and the characteristics of the raw material. The results of the empirical study indicated that these pressure adjustments had to be changed by up to 30% due to disturbance variables (W7, P5). As I1 explained, ‘On the one hand, the raw material … we cannot guarantee that one component is really like another. On the other hand, the machine, where wear and tear cause changes in the condition of the tools’ (I1).

5 Summary of the empirical results

Our empirical study showed that the machine operator plays a crucial role in current manufacturing processes when handling specialised machines. Their expertise and training have a significant influence on the success or failure of the production process. The absence of an automated control system made this particularly relevant. Technically speaking, the machine operator took the place of this controller by performing sensory tasks, determining control deviations and defining manipulable variables. Inexperienced machine operators usually do not have such expertise. However, current strategies like the intensive training of inexperienced machine operators, as we have shown, fall short due to inappropriate theoretical training content and the highly contextualised manufacturing processes.

6 Design challenges

Based on the empirical findings, we derived challenges and design implications that guided the later implementation of the HoloLens application RetrofittAR. The existing ways of operating the machines posed several challenges in relation to technical support for the practices of machine operators. On the one hand, the existing machinery was indispensable for economical production but, on the other hand, the machines did not convey relevant real-time data to the operator during the production process. When coupled with insufficient machine operator training, it became apparent that inexperienced operators required the machine’s operations to be made visible during its actual operation. As the company’s production requires specialised machines, the instructions had to be adapted individually to each machine. Due to the lack of qualified machine operators, the instructions needed to be easy to understand and easy to follow during the production process. The instructions also had to contain the essential single-setting parameters and be designed to support machine operators’ practices. Furthermore, the instructions needed, as far as possible, to be integrated into the existing machine infrastructure, so that a retrofitting of the existing machinery would be possible.

As chapters 4.2.2 and 4.2.3 show, the skills of machine operators played an essential role in production. They observed the quality of the output during the production process and directly initiated appropriate measures by adapting the setting parameters. Thus, placing all instructions directly in the operator’s field of view was deemed important. Here, the required input parameter (see Figure 2) must be visualised and coupled with machine data. Specifically, a fusion of input parameters with the machine data (e.g., speed or pressure) allow a deeper understanding of how to operate the machine appropriately. By visualising these values, the machine operator no longer needs to constantly look at the machine control panel to see the actual setting of the two parameters mentioned and is thus able to maintain the necessary head quality observation.

Because the parameters set vary along the head’s radius, it is necessary to display them according to the head’s geometry. For this purpose, a connection between parameter visualisation and the actual position of the machine axes is needed. Therefore, the target geometry of the head must be placed directly on the machine itself, including the parameter specifications. This visualisation enables a continuous comparison of the real machine data with the visualised specifications that can be carried out by the operator. However, for older machines, this was not a simple matter, as the positions of the machine axes and further machine information were not provided digitally.

Because the empirical study showed that several heads were produced on a single machine, we determined a visualisation of each head and the parameter specifications that must be adapted in real time. This is accompanied by the adjustment of the given parameters using the disturbance variables. As our results in chapter 4.2.3 show, the influence of the disturbance variables depended on the degree of wear and the characteristics of the raw material.

Table 2 summarises the design implications based on the empirical findings and links to the implementation presented in chapter 7. We derived all design implications directly from the findings of the workshops, interviews and eye-tracking sessions presented in chapter 4.2.

Table 2 Summary of empirical findings, design implications and implementation.

Figure 5.

General concept of the AR-interface: (1) Virtual segmentation of the head along the radius, (2) Default parameters for the spinning process, (3) Live data of the machine.

Figure 6.

Data structure of RetrofittAR.

Table 3 Data structure.

Figure 7.

Main Menu: (1) Article selection, (2) Dimension of the head according to the selected article and calculated with the virtual head generator, (3) Machine selection, (4) Adjusting the disturbance parameters; Virtual representation of live data: (5) Live data speed, (6) Live data pressure, (7) Live signal.

7 Concept

To tackle the aforementioned design challenges, we designed an AR concept that enabled the machine operator to observe the head’s behaviour during the production process and, if needed, to make appropriate adjustments. Our empirical work, along with the eye-tracking, showed the importance of the continued monitoring of the machine by operators. Thus, we focused on using a head-mounted display to visualise the relevant information directly in the operator’s field of view. At the same time, this allowed the operator’s hands to be free to perform the necessary process steps.

We created a visual representation of our concept, shown in Figure 5. The segments of the radius were mapped to the actual physical head so that the operator could see the current radius range, including the parameters for pressure and speed. These virtual objects were always aligned with the position and angle of the operator’s view. We also displayed the default values for pressure and speed. Coupled with the virtual segmentation of the head, we highlighted the place where the operator must manually put in the values when the physical press roll of the machine touched the virtual segmentation. In addition, there was a live visualisation of the actual machine parameters. This visualisation always displayed the current speed and pressure. Our concept not only gave the machine a new way of representing the configurations but also made the monitoring of head behaviour easier.

8 Implementation

We implemented the concept as a Microsoft HoloLens application called RetrofittAR. The aim was to provide the machine operator with the relevant information about the machine, the article and setting options. RetrofittAR was implemented with Unity. We used the Mixed Reality Toolkit (MRTK) because it already offers basic assets for lower-level tasks, e.g. for user interfaces. We mainly used the MRTK version 2 and extended it with some legacy assets of the MRTK version 1/Holotoolkit assets.

As the data within our concept was closely related to the nature of the physical and spatial environment, we used the image and object tracking Software Development Kit (SDK) Vuforia 8.3.8. Vuforia allowed us to scan image targets with the HoloLens camera to precisely position our AR images. The image targets were positioned directly on a flat and somewhat protected place on the machine body. Due to the poor lighting conditions in the production hall, tracking was very occasionally unreliable and deviations of up to 5 cm could occur. Therefore, a handle was implemented that allows manual correction of the positioning of the AR images. The user is now able to compare the virtual representation of the head with the real head and correct accordingly with the handle.

The requirements of our application meant we had to integrate different relevant data types and sources with RetrofittAR. Figure 6 shows the different data types and structures of our application. To implement the data about articles, the team first had to create databases with parameters for speed and pressure. We therefore created corresponding Excel sheets for each article. Within each Excel sheet, the operator listed the pressure levels, pressure profiles, speed and rpm for each article. Our findings had indicated that personnel-intensive training phases were largely unproductive due to the production environment and the low level of prior knowledge (see Table 2). Hence, an asynchronous knowledge transfer was initiated. For this purpose, the experienced machine operator had to fill Excel sheets with his preferred parameter settings and these were then implemented as scriptable objects in the Unity development environment (see Figure 6) and visualised as default parameters in the ToolTips (see Figure 8). The geometries of the heads were usually standardised. We, therefore, stored the basic dimensions and the exact shape of the head geometry for each article according to the construction guidelines. Thus, we were able to create each head geometry virtually on the HoloLens. The geometries could be quickly determined using the stored formula, even for new head dimensions.

Figure 8.

Torispherical Head UI: (1) Rendered part of the virtual head, (2) Indicators for segmenting the head along the radius, (3) ToolTip containing predefined parameter for pressure, (4) ToolTip containing predefined parameter for speed, (5) ToolTip containing the current parameters for pressure.

For the machine’s live data visualisation, we connected the machine’s rotary control, signal lights and pressure indicator to a Raspberry Pi, which in turn provided the data via a WebSocket connection. The number of verticies of the virtual head and the amount of data from machine to HoloLens via the WebSocket connection were designed so that the performance did not fall below 40 fps.

The following data structure emerged (see Table 3), which was necessary for the productive use of our application.

The freely manoeuvrable AR image with the main menu allows the user to select articles from a multi-page list with the known article numbers (see Figure 7). After selecting an article, its geometric dimensions are displayed. If this article can also be produced on another machine and parameters for the settings have also been stored, the user can then select an alternative machine and the corresponding parameters for this machine are displayed. In addition, the wear of the roll and batch fluctuations of the material can be set with two sliders. According to the settings of the disturbance variables, the default parameters on the ToolTips adapt. The freely manoeuvrable Live Data UI showed live data of the currently selected machine (see Figure 7). In addition to the pressure display of the ToolTips, the current pressure value was also displayed on the Live Data UI by means of a circular scale which was designed according to the scale on the machine control panel. The same scale was implemented for the speed display. Furthermore, a single machine signal was transmitted and displayed, which indicated when a certain position on a machine axis had been reached.

The Torispherical Head UI was placed as an overlay over one of the respective real machines and tools in three dimensions with Vuforia and the MultiVuMark package. Based on the drawings and standards supplied, the head was calculated and finally rendered on the HoloLens. A small section of the head was displayed and rendered from one side only so that the operator had a clear view of the head (see Figure 8). In addition to the head, indicators were placed on relevant spots of the currently displayed ‘TorisphericalHead’ shape, segmenting the head along the radius. As shown in Figure 8, the relevant spots were at 0,2r; 0,3r; 0,6r and 0,9r.

The indicators were connected to the ToolTips by a curved line. The ToolTips contained the PressureProfileCollection and showed the predefined parameters for pressure and speed at the relevant spot. Next to the predefined parameters, the current pressure value transmitted by the machine over a WebSocket connection was displayed to the right (see Figure 8). These virtual objects were billboarded so that the position of the object was always aligned with the position and angle of the machine operator’s view.

9 Evaluation

9.1 Methodology

To evaluate RetrofittAR, we conducted an evaluation study with 10 machine operators aged between 25 and 60 (average age of 35) (see Table 4). None of the participants had previous experience with AR technology. Our evaluation consisted of a task-based in situ evaluation of our prototype in a real production environment. We followed with an interview with each participant.

Table 4 Evaluation participants.

During the in-situ evaluation, we first explained the HoloLens and its interaction modalities. Afterwards, all participants independently went through the HoloLens’s built-in calibration app. Calibration was necessary so that the HoloLens fit individually into the participant’s viewport. We then instructed the participants to launch our app and select the corresponding production article from a list. Although we had previously stored the data of 60 production articles within the application, the evaluation was performed only using four of them. We asked the participants to place all freely placeable AR images according to their preferences. Subsequently, the production process started, and the participants were advised to produce up to ten similar heads with the help of the application.

Our evaluation was split into two modes. In the first mode, an accompanied evaluation was conducted. In this case, an inexperienced machine operator wore the HoloLens and articulated the setting values that were visualised by RetrofittAR to an experienced machine operator. The experienced machine operator set the values exactly as specified and only intervened if a collision was imminent. This procedure was implemented to ensure work safety, as the inexperienced machine operators were operating the machine for the first time. In the second mode, a completely independent evaluation was carried out. The machine operator performed the evaluation without external assistance (see Figure 9).

Figure 9.

Accompanied evaluation (left) and unaccompanied evaluation (right) of the HoloLens application.

During our evaluation, we were particularly interested in the use of RetrofittAR in the real production environment and the difficulties that arose. The philosophy behind the evaluation was derived from Twidale et al.’s (1994) ‘situated evaluation’. Therefore, qualitative methods were applied which allowed us to draw conclusions from the use of the technology in the real production environment. Experts and novices were asked to describe their views on the usefulness of the application. The participants were instructed to ‘think aloud’ (Nielsen 1993). Due to the noise level in production, statements could not be recorded, and only notes of the comments made by the operators were taken. Immediately after the in situ evaluation, we conducted a semi-structured interview with the participants. Our questions focused on the following topics:

• Information content of the visualisation.

• Interaction between real machine and virtual content.

• Placement of the information.

• Interaction possibilities and usability of the application.

• Integration of the glasses into everyday work.

• The extent to which the AR images represent knowledge-relevant data for the experienced employees.

• How the integration of the machine values into the virtual interface was perceived.

9.2 Results of the evaluation

9.2.1 Display and placement of AR images’ decisive influence on the timely processing of the illustrated parameters

The participants emphasised that the AR images had to be moved and placed freely when needed (T2, T3). This functionality allowed them to keep the relevant values in view, even if the operator moved. At the same time, the participants were also able to observe the physical machine, which is one of the main advantages of AR:

You can observe everything on the machine and in the environment. These are not VR [Virtual reality] glasses, where you are simply isolated from the real environment. So, you still have everything in view and can place your windows as you need them and always have everything in view. (T2)

The placement of the Live Data UI was especially important, as these values had to be continuously observed:

I put the pressure display directly over the tool and then I could always see how the pressure was. You have all the lines on the tool of the machine and next to them boxes with the values that show how much pressure you have to set. (T2)

However, because the AR images of the live values could be moved freely, the operator had the choice of where to display the values, which challenged inexperienced operators. Thus, they expressed a preference for the pressure values of the machine, at a minimum, to be placed directly next to the reference values: ‘The pressure values of the machine should actually be right next to the target values. So, I can compare both values directly and do not have to look at another place’ (T6).

The virtual line’s location marking on the tool (see Figure 8) was found to be accurate and appropriate, and the machine operator could see exactly where they needed to adjust the pressure and speed values according to the target values: ‘So the lines definitely match the real tool. I followed the lines and adjusted the parameters accordingly’ (T3).

Nevertheless, during the evaluation, it became apparent that with the small head diameters, the lines for the virtual representation of the head (virtual lines) were very close together at the beginning of the spinning process. The operator had to react quickly by increasing the pressure values at a precise time. As a result, the operator found it difficult to map the virtual target values with the real position of the machine in a timely fashion, losing the overview.

I wanted to set exactly what the glasses were supposed to do, but then I couldn’t do it anymore. The lines were too close together because the diameter of the head is very small. Then I continued producing the head without this information. (T8)

Thus, simply displaying additional and more detailed information was problematic, as the operators found it difficult to process too much information. ‘So, the more information, the better. However, the more the information, the more difficult it is to process in practice’ (T6).

To simplify the interface, one operator suggested visualising only selective information: ‘For me, there were too many markings constantly visible. If the marks would gradually appear and then disappear again depending on the position of the pressure roller, only the acutely important marks would be visible’ (T7).

9.2.2 Using AR as an upgrade or supplement highly depends on the expertise

The experienced machine operators had difficulty growing accustomed to the new interface, as they found some information to be redundant and were distracted from the actual observation of the head’s behaviour on the machine:

I concentrated on the glasses and also perceived the information. But for me, it was a strange feeling. I don’t know how to explain it: I actually know what I have to do because I have been doing it for a long time. It was difficult for me to concentrate on the glasses and ignore how I had done it before. (T1)

T6 described the experience as follows: ‘It is decisive how I increase the pressure. If I increase the pressure, I must immediately pay attention to the reaction and behaviour of the head. Therefore, I will not rely completely on the glasses’. T6 also indicated that the AR glasses are only a support for the operator and their value lies mainly in ensuring that the human is provided with the necessary information for setting the control parameters. For this reason, some operators thought RetrofittAR should only be regarded as a supplement, depending on experience. T8 explained:

I think it’s perfect for training people. You don’t have to constantly look at the machine panel. All the values are collected on the interface, and at the same time, you have the head in view. Especially for the beginning, it is really good. For the people who have been working on it for a longer time, for them, it is not upgrading but supplementing. For people who are doing it for the first time or for people who are just getting started, it is definitely an upgrade. Because you save yourself the trouble of drawing lines on the desk or making other markings you can really have a view of the tool or the head and learn what you need to pay attention to. Instead of looking away from the machine every time to look at the panel and check what pressure is set – you have it right in front of you. You have all the information you need right in front of you. And that’s definitely an upgrade, especially for training. (T8)

Based on the existing data, T8 also pointed out that the interface should be seen as a supplement because not all external influences can be represented in the target values and therefore the visual control of the head’s behaviour still had to be the main focus:

If the batch of the metal melt is changed, the entire material behaviour during the spinning process changes as well. Then the pressure is sometimes too high or sometimes too low. Both are problematic. So, the eye has to control itself. (T6)

However, when focusing on inexperienced operators, T9 provided an important insight by stressing how sometimes one could gain a deeper understanding of the process:

Well, I’ve been working here for several years now, and from time to time I asked my colleague: What are you doing there now? What are you actually looking at here? The interface on the head made me realise which details he pays attention to and how the process works. (T9)

RetrofittAR allowed the machine operator to perceive an almost undiminished environment and thus maintain communication with colleagues: ‘And you can also talk to your colleague when something is going on somewhere’ (T4). There are situations where glasses are not necessary, and the machine operator does not need the data provided by the interface. ‘When changing the material, the glasses can be flipped up so that you can see more’ (T5).

Fifty years span the machine and the introduction of smart glass. The operators initially had problems thinking away from pure hardware and towards a data-oriented view. Once this perspective was adopted, operators recognised the potential of AR as an additional interface for the machine that supports the production process:

Well, my opinion is that glasses and machines don’t really fit together. The machine is from the ‘70s, the glasses from 2020, but from a content point of view, it makes sense to wear the glasses when operating the machine. (T1)

9.2.3 Enabling expertise sharing directly at the physical machine

To support on-the-job learning and build up knowledge about the machine, it was important that the operators had a constant view of the machine’s current values and ‘behaviour’:

You have the head and the machine in focus all the time, and you don’t have to look at the machine console again and again. And that way you can start learning. In other words: To combine the virtual of the glasses with the real haptic of the machine. So that in the end, you get to be able to produce the head – I would say – by feeling. (T8)

Although RetrofittAR could make learning on the machine easier from the operator’s point of view, there was a risk that inexperienced operators would rely too heavily on the application, thus preventing the actual building of expertise. T9 suggested:

The only risk I see is that people might try to learn by heart and not react to the behaviour of the head. That the operators don’t take the glasses as a support, but that they now say: I take the glasses now and I know what it is written, and I don’t have to do anything else. Then a new batch of material comes in, and then the values are again not 100% correct. You just can’t rely on it 100%, but you rather have to take it as a guideline. (T9)

The lack of experience becomes evident when deviations or problems occur. Gaining this complex process knowledge is part of the efforts of inexperienced machine operators: ‘You want to know what you are doing. You don’t just want to press buttons; you also want to know when, how and why’ (T3). T3 described the exact content that he received through the AR visualisation:

For me, it is definitely a help because I am inexperienced. And I think it makes sense to get support as someone who is inexperienced in using such a machine. And if you have such a help, especially in the beginning, when you first have to get to know the machine and learn when you have to increase pressure and when you have to decrease the pressure again, until when you have to keep a certain pressure and when you fold the head. I find that helpful, absolutely. (T3)

RetrofittAR can even act as a mediator between experienced and inexperienced users. This role emerged during classical training situations in the company (consisting of an expert who explained and implemented the training and a novice who listened and watched). Questions that often arose could be better answered by using AR:

When I watch the experienced one during production, I always think: Why is he turning there now? Why is he spinning fast now? But this way, I understand the process more precisely. If you didn’t have this insight before, with the help of the glasses, you can now understand it more easily. (T10)

Even beyond the initial training phases, for some it made sense to use the glasses, especially when they were not sure how to handle a certain dimension of the head because they had never produced it before.

These glasses are not designed to be worn for 8 hours. You use them for the first 5, 6, 7 or 8 heads. After that you know: Here to this area, I have to turn there, until then I have to turn there, until then I have to turn there. In some cases, the following situation can occur: At 2 o’clock on the night shift there’s no one there, and you have never produced 1250 × 5 head dimension. (T2)

Thus, the glasses can create an autonomous learning environment from the operator’s point of view, which partially allows the novice to learn the basic settings autonomously: ‘The operator is taught the basics in a relatively compact time: pressure, speed, feed: ‘And after that, he can teach himself many things, not everything, but a large part of it using these glasses’ (T8).

9.2.4 Usability issues

Wearing the glasses was not always a comfortable experience, though this varied considerably from operator to operator. One user found that 'the glasses fit well on the head and can be adjusted well' (T10), while others felt 'a feeling of pressure on the forehead and temples' (T5). The interaction options were perceived as basically precise, although 'it takes several minutes to get used to them' (T2). The menu navigation of the prototype was perceived as occasionally cumbersome due to the division of the contents into individual windows:

I had to operate via two different windows, which initially confused me. It would have been better to integrate all content in one window. In the end, however, only a few settings had to be made, which is good in this industrial environment and everyone can manage the operation. (T1)

Due to the limited number of settings (article selection, wear setting and connection setup to the WebSocket), the menu structure was considered easy to use and T1 estimated that all colleagues could cope with the operation of the application.

Due to the difficult lighting conditions and the low illumination of the working area on the machine, the machine operators found the AR images too opaque:

It is not very bright in the production hall. It was really hard to see how the material was touching the tool. (T8)

T8's statement confirms that observing the machine is of paramount importance (see Figure 3). However, by adjusting the brightness of the HoloLens displays and reducing the size of the virtual head, problems could be remedied quickly and observation of the machine could take place without difficulty.

From a usability point of view, the integration of the machine displays into the AR interface was perceived as very positive. T7 states: 'The interface is complete and all information is visible on the machine'. In general, no inconsistencies were found by the operators.

10 Discussion

The results of our evaluation demonstrate how the practices and skills of operators inform their attitudes towards RetrofittAR. More experienced workers had habitual ways of doing things and, to a certain degree, found the application to be unnecessary for their work practices. In particular, with the real-time processing of information and the execution of measures based on the visualised information, an information overload turns out, for them, to be problematic (Hiltz and Turoff, 1985). As s our evaluation shows, experienced operators find the visualised information redundant (Kalyuga et al., 2003), which may lead to an information overload during the process’s execution. The experienced operator simply preferred to rely on their own assessment and visual inspection of the process rather than trusting the given values.

Conversely, the inexperienced machine operator does not have this experience and is, therefore, grateful for the target values, even though these values can only provide reference points. Thus, the AR display is particularly suitable for inexperienced operators because the interface both provides a visual of the target values and allows the observation of the process. RetrofittAR provides inexperienced machine operators with the ability to perform the process independently and thus they can gain important practical experience and expertise in situ. This ability is important, especially against the background of the lack of theoretical training because all subsequent training content can be based on this expertise. Particularly, the semi-automation of expertise sharing in an actual production environment can positively affect expertise sharing processes (Hoffmann et al., 2019). As we point out above, in a highly competitive environment, it is important to avoid interrupting normal production, and further to avoid unnecessary errors in production. Our evaluation demonstrates that the application helps in fulfilling these objectives.

The results of the empirical study showed that users pay considerable attention to the fact that AR content adapts to the actual situation. We, therefore, implemented a dynamic head geometry creation based on standardised design specifications. This implementation means that any dimensions of the heads could be generated through RetrofittAR in real time without having to create models externally in a design software environment. As Bannon and Kuutti (1996) indicate, the visualised contents must adapt to the actual tasks in practice. The results of the empirical study show how very process-specific contents and representations are necessary when providing instructions for use with machines of the kind we were dealing with (see Table 2). Existing software solutions on the market are oriented towards the display of standard geometries such as boxes or arrows and do not offer the possibility of displaying process-specific setting data (Vuforia Expert Capture or Microsoft Remote Assist) or only support remote scenarios (Microsoft Remote Assist). Existing solutions simply do not meet the requirements for context specific information as described in chapter 5. On the other hand, the geometries and data must be made available to the software. Many of the authoring tools require extensive expert knowledge for the creation of the AR content (Nebeling and Speicher, 2018). To mitigate this issue, the virtual head generator (see Figure 6) was implemented, which automatically generates the geometries when a new article is created. The input parameters were packaged in scriptable objects (see Figure 6), which can also be easily created. One limitation is that the data in the current software have to be carried out in the Unity development environment, and although no programming knowledge is necessary for this, Unity must be operated. At this point, however, the process can be further simplified in the future on the basis of the virtual head generator and the scriptable objects by implementing an end user oriented authoring environment.

We argue that AR technologies are especially suitable to foster expertise sharing processes in a fast-changing knowledge environment where real-time production routines need to be maintained. They are suitable, not just because data and knowledge content can be adapted quickly and comprehensibly, but because the user must react to these changes and can build up new expertise more quickly and efficiently. In our opinion, the reason for the support of this AR technology transformation is the fusion of the AR images with the real environment. Additionally, the possibility—as shown in the evaluation—arises that default values and the process’s live reaction to these values can be assessed by providing effective feedback through the glasses (Burke et al., 2006; Shah et al., 2012; Tang et al., 2003), thus facilitating accurate judgement. Due to the necessary spatial mapping capabilities of the technology (Evans et al., 2017), AR leads to a good contextualisation of the visualised content, which can support the establishing of an OM (memorising or storing information and action of recalling and remembering) (Bannon and Kuutti, 1996). This process enables the user to build up their own process knowledge by complementing human abilities and technical possibilities (Engelbart 1988).

While the application stores the exact setting parameters and thus reveals some of the explicit knowledge of the experienced operator, the application also recognises that not all OM can be stored (Ackerman 1998; Randall et al., 1996) by allowing the observation of the processes through the interface. This ensures that the inexperienced machine operator can build up expertise independently. Thus, AR technology provides a good transition between formal knowledge content and the local, specific, knowledges characteristic of expertise (de Carvalho et al., 2018).

Due to the strongly context-dependent visualisation, AR has been found in general to lead to a lower cognitive load among users (Bresciani and Eppler, 2009). In this context, however, it was clear that some measures to reduce the cognitive load were necessary. Specifically, the results suggest that two measures could be effective in this context. Firstly, a reduction of the information to the essential process parameters is needed. Secondly, a representation in which real machine and virtual information merge as effectively as possible in one consolidated interface would help reduce that overhead.

AR seems to be particularly suitable for retrofitting machines because an interface for any machine can be created more or less independently of the hardware (Al-Maeeni et al., 2020). It establishes a direct reference to the machine by placing AR images with centimetre precision at the physical hardware itself. Through standard communication protocols, AR glasses can deliver near real-time information transmission of certain machine signals, which in turn has a positive effect on expertise sharing (de Carvalho et al., 2018; Hoffmann et al., 2019).

The HoloLens assists the actual process with the visualisation of the geometry and parameters and provides a compact new interface. The evaluations show how the machine operators appreciated the ability to concentrate on only one interface. In the course of retrofitting, this means that all optical signals provided by the old, human–machine interface (HMI) must be migrated to the new interface. The separation of an HMI and an AR interface for information visualisation, according to Romero et al. (2016), should be avoided to encourage the increase of cognitive abilities by support systems. Within the AR interface, default and actual values should be placed beside each other in such a way that a direct comparison of the two values is possible. However, the results of the evaluation also show that for the near-real-time processing of the information, it is also important to display the contents selectively according to the process’s progress to avoid operator misinterpretations. In concrete terms, the virtual representation of the set parameters should be built up depending on the position of the press roll.

With regard to retrofitting special machines, it is important to focus on the machine operator as a key decision-maker and to support them by providing individual sensor values. The point here is that the second-by-second skilful work practices that lead to error avoidance and smooth production require AR interfaces that can be built around the changing boundary conditions (e.g., the fluttering of the head depending on the pressure) or can allow visualising of context parameters directly in the form of process values. In addition, the AR images also allow the prominent placement and highlighting of certain real processes which particularly require the operator’s expertise (e.g., marking the head with lines).

As our evaluation shows, the AR interface is a useful tool for assisting in the qualification process of inexperienced operators. The visualised information gives the operator an indication or an idea of the settings with which the head can be produced. Based on this information, the operator can gain experience and ultimately, since there is no strict target parameter specification, determine how much he can and should deviate from the given parameters (Cimini et al., 2020). The displayed default values should only serve as an orientation, and the operator must be allowed to trust their own experience. Therefore, visualisation technology must leave space for the interpretation of the values under constantly changing boundary conditions and context variables. Our application pays due respect to, and is predicated on, the concept of sociable technology according to Ludwig et al., (2017) and the three contexts (internal context, spatial-material context and task- and process-related context) which must be considered.

10.1 Limitations

Our study is, in certain respects, limited. Our application was rolled out on only one machine (a function of the need to keep as many machines working full-time as possible). We cannot say at this time what effects the technologies will have on other machines. However, from a technological point of view, further machines can be prepared for use with RetrofittAR by using ubiquitous sensor technologies. Another limitation is that the AR application is specific to the production process considered in our study. Generalising statements can only be made to a certain extent, and the results must be validated in similar settings but based on a different production process. Unfortunately, only one operator with sufficient expert knowledge was available for the eye-tracking study. Therefore, limited general conclusions can be drawn from these results, but the findings were, even so, very important for the concept and implementation (see Table 2). The results of the evaluation also confirm the findings of the eye-tracking study. The evaluation particularly emphasises that the interface still allows the practice of observing the head on the machine and even intensifies the practice by the fact that the operator finds the complete interface on the machine and therefore in his direct field of vision. Because only one process was considered, only some the complexities of the expertise-sharing process can be examined in detail. In addition, the implementation was only carried out for the HoloLens. Statements about other AR glasses and their suitability for use in such a scenario can therefore not be made with any great confidence. However, it is to be expected that due to the strong spatial reference of the displayed AR images (for example, the indicators for segmenting the head with the corresponding tool tips), AR glasses with no spatial mapping capabilities are unlikely to offer any added value for this use case. Nevertheless, we successfully demonstrated that, for the process considered in this paper, the AR application provided a common information space (Schmidt and Bannon, 1992) which effectively facilitated ‘learning by doing’. The paper shows how configurable knowledge content can potentially be visualised utilising AR in a wider range of industrial and other contexts, thus promoting competence development. For a long-term study of expertise sharing support, an appropriation study in the company and accompanying quantitative evaluations to measure the effectiveness of systemic support are necessary.

11 Conclusion

The literature uses the term 'anthropocentric production systems' or 'human-centred CIM' to describe a human-oriented view of computer-assisted production processes. Such a perspective gives the essential decision-making function to the human being, largely due to the existence of various existing uncertainty factors. This is why the term 'competence-based manufacturing' is often used in this context (Brödner 2007). In CSCW, this competence has a direct relationship to problems associated with organizational memory (OM) and the sharing of expertise and knowledge in particular (Bannon and Kuutti, 1996) and remains a vibrant issue. New technologies are said to be able to remedy this to a certain extent and to increase cognitive abilities through assistance systems (Romero et al., 2016). In this context, novel AR technologies stand out, as they can visualise highly contextualised instructions and support embedded in CPS competence building (Hoffmann et al., 2019). This paper shows how AR technology can support the operator to remain at the centre of a production process and to make the essential decisions in a production process by supporting the operator in competence development (Cimini et al., 2020). Figure 10 shows how the prototype we present allows the operator to act as a controller in a control loop and thus remain at the centre of the production process. For this purpose, the image of a control loop is used: The target variables are defined by drawings and standards. However, deviations (fluttering head, gap, circumference) occur during production, which the operator must compensate for with the help of parameters (pressure, speed and rotation speed). RetrofittAR supports this process with the three main evaluation results shown in the figure. The spinning process carried out on the basis of the parameters is disturbed by the disturbance variables (tool abrasion and material properties), so that the head must be repeatedly checked by a quality check. AR technology is also suitable here because it allows the operator to carry out the quality check by observing the head (see Figure 10) while at the same time viewing the default parameters.

Figure 10

Human machine control loop supported by RetrofittAR.

The point is, as we have demonstrated, that in the real world, the risks (and costs) associated with radical change are too much for SMEs to bear. Older machinery continues to be reliable and functions well for the purposes required of it. The problems encountered are much more to do with continuity of knowledge between ‘old hands’ and inexperienced operators.

For the concrete answer to the research questions, the results of the study show that operators need to manage different data types, all in real time.

• (1) Geometric data represents sections of the head’s final geometry to be produced and exactly positioned on the machine. A special feature is that the geometry is dynamically recalculated on the HoloLens depending on the article to be produced based on nominated parametric specifications so that no rigid computer-aided design models need to be stored. The AR display on the machine allows the machine operator to compare the actual head with the visualised geometry and compare the position of the real pressure roller with the values specified along the head’s radius.

• (2) Default values for pressure and speed settings are displayed along the radius of the head. Based on the experience of one machine operator, values for all heads are recorded in tabular form and displayed onto the corresponding radius sections of the real head with the help of AR glasses. These are default values which allow the successful production of a part. The defaults can be dynamically adjusted depending on the wear and tear of the tools and the raw material quality.

• (3) Visualisation of the live data of the machine directly next to the default values allows a continuous target or actual comparison between the values. Thus, all the data required for the production of a head is located on the new AR interface, and the machine operator can both concentrate on the interface and observe the head on the machine directly.

Furthermore the qualitative results of the evaluations are such that we can make the following claims about the design of AR retrofitting interfaces to support expertise sharing processes.

• AR is well suited for retrofitting old machines. Retrofitting is especially effective if operators typically focus on certain areas of a machine since they can be clearly highlighted during a running production process. It is also relevant to situations where a wider field of view covering the whole of the work piece to see progress in detail during production. The interfaces can be aligned in such a way to provide a clear view of certain areas and position essential virtual content in the direct field of view.

• AR contextualises the content with the wider environment, which has a positive impact on expertise sharing processes. The contextualisation between real machine and virtual content allows inexperienced machine operators to understand previously unknown and difficult-to-explain relationships between the process and the process parameters. The results of the evaluation show how process knowledge can be gained more quickly in the course of expertise sharing on the shop floor in a real production environment – especially for old machines without numerical controls. Furthermore, the technology allows the establishment of an expertise-sharing environment in running production.

• Ensuring the operator’s ability to control processes requires the interface to contain all information necessary for operation and to avoid media discontinuity. Additionally, it must also gradually build up content according to the process status and thus selectively visualise the content.

Our study clearly outlines the role that humans and their expertise play within complex production processes. The AR technologies seem to support the developing expertise of human operators where electronic controllers are not feasible (for reasons of cost etc.). Applications like RetrofittAR allow users to actively manage processes with requisite skill, even if they have little previous experience in operating the machine.

Our design case study contributes to current CSCW discourses on expertise sharing and the role of humans in manufacturing settings and provides technological interventions that illustrate how continuity of skill and knowledge can be supported in environments where neither formal training nor generic applications are adequate. Such environments, we suggest, are common.

Future studies might further investigate the appropriation of the technology, focusing on long term adaptation to the technology, its impact on workplace practice and on knowledge transfer. Subsequently, it is planned that the application will be rolled out to other use cases and machines to better assess the transferability of our findings.