1 Introduction

Industry 4.0 has been proposed as a new phase in the industrial revolution, integrating digital and physical worlds through technology into the industrial procedures [53, 67]. Despite the promising benefits of using smart sensors, embedded systems, cyber-physical systems and Internet-of-Things (IoT) in manufacturing [2, 20, 27, 48, 63, 70], human operators remain an essential part of any industrial processes [27, 46]. Smart factories are essential in the stock market [69] and almost doubled their market size in the last 5 yearsFootnote 1. Regardless, in logistics procedures, human intervention still remains the most important asset to accomplish the intended goals [30]. One example is “kit assembly.” Usually, this task uses a mixed-model assembly, i.e., on the same line, several distinct kits, composed from different components must be assembled. During the kit preparation, the operators have to identify the necessary components and their corresponding location, so that they can be picked up correctly [23].

This information is mostly conveyed through traditional printed paper manuals, requiring that operators read the list of components composing the kit and then search for their location in the line [23].

However, paper manuals can be mentally and physically demanding to some operators after long hours of labor, resulting in mistakes and decreasing efficiency. Furthermore, the learning and training phases for novice operators are not straightforward, requiring a long period to prepare them for the shop floor activities. One possible solution to improve the kit assembly operations is the use of Augmented Reality (AR), considered one of the nine pillars of Industry 4.0 to support operators with real-time information for faster decision-making, while improving work processes [15, 24, 50, 54, 56, 62]. This technology can integrate virtual information in the operators workspace [35, 42], helping them in assembly tasks [18, 43, 49], provide context-aware assistance [5], data visualization and interaction (acting as a Human-Machine Interface (HMI)) [16, 40], indoor localization [60], maintenance applications [8, 18, 61], quality control [4, 65], material management [16, 51] or remote collaboration [7, 39, 66], by presenting additional layers of digital information on top of real-world environments [3, 28, 33, 37, 38, 57]. Prior studies identify certain benefits of applying AR for technological industrialization, like increased work safety, effective learning and training, as well as more task effectiveness [10, 12, 31], as well as improved Human-Robot Interaction (HRI) [1, 13, 19, 34].

However, most AR solutions are limited to a given space, which means operators must have all necessary materials within a limited range, otherwise, the tracking capabilities of existing tools will fail, affecting operators experience, and in turn, task performance.

To overcome this limitation, the concept of Pervasive AR offers experiences that can be visualized continuously as users move through the environment. Such experiences are also characterized by being aware of and responsive to the user’s context and pose [21, 36, 47].

In fact, this concept has the potential to evolve AR solutions to multi-purpose experiences, providing easier access and better perception of information, changing how users interact with virtual content and their surroundings.

Pervasive AR and Mixed Reality (MR), depending on the definition, can differ; however, they are both part of the “Virtuality Continuum” [41]. MR has no single definition and can be considered different things as its understanding is based on one’s context [59, 68]. One possible definition is the combination of AR and Virtual Reality (VR) parts that interact with each other but are not necessarily tightly integrated [52, 68]. Quint et al. [55] define it as the combination of real and virtual world information. On the other hand, Holz et al. [25] and Nebeling et al. [45] refer MR to the real environment that allows for shared interaction with virtual experiences. Hereupon, Pervasive AR extends the original AR concept in time and space based on the surrounding context [21], which can be inserted in the scope of MR.

Based on the constraints of traditional paper manuals and the research opportunities to enhance kit assembly tasks, in this work, a Pervasive AR tool for production assistance was developed.

This tool was designed with partners from the industry sector using a Human-Centered Design (HCD) methodology.

The proposed solution can be used in Head-Mounted Displays (HMDs) and Handheld Devices (HHDs), enabling to configure new training experiences on the shop floor, or edit existing ones to incorporate new steps. The goal is to provide training experiences that are continuous in space and time, without the need of visual markers (e.g., QR Code, Aruco). By doing so, it is possible to support training sessions, not only of novel operators, but also of more experienced operators, when new kits are introduced. Besides, this work also has the objective of evaluating the developed tool and comparing it with the traditional paper manual method. Two user studies were conducted on the shop floor at different moments in the development process. Individuals having distinct expertise were considered to improve the range of the data collected and create a solution that is generic enough to be used by operators from different areas of a factory, if such a need ever exists.

Besides this section, the paper is structured as follows: Section 2 describes related work on AR in industrial scenarios. Section 3 details the logistics scenario considered. From here, requirements are presented, leading to the proposal of the AR prototype. Finally, an initial user study on the shop floor to gather first impressions is described and its results discussed. Section 4 reports a second user study on the shop floor to compare different methods. The results obtained are reported and discussed in Section 5. Section 6, draws concluding remarks and presents ideas for future work.

2 Augmented Reality for logistics in industrial scenarios

Logistics is a primordial part of industrial systems, although it is one of the least explored AR research areas, as illustrated by Fig. 1. While several logistic procedures, such as storage and stock removal are fully automated, there are still some processes where automation is not implemented, being highly dependent on human intervention [30, 71].

Order picking is one of the logistic procedures where most of the work is performed by humans, requiring flexibility and motor skills. Even processes, such as assembly tasks, require order picking to fetch the components constituting the product to be assembled. There is a wide range of research in order picking, as it is a complex operation with high economical relevance, having great potential for process optimization [71]. Hereupon, besides the traditional printed paper manual method, other forms of supporting the task can be applied, such as pick-by-voice systems [6] consisting in communicating with the warehouse management system via a voice recognition system; pick-by-light strategies [6], guiding operators by lights or digital displays; Head-Up Displays (HUD) [22] where the picker can use a small display to see the next component to pick; or by using AR tools [23, 29], overlaying digital picking information aligned with the real-world environment.

Fig. 1
figure 1

Distribution of the different fields of industrial application. In red, the logistics field, corresponding to 12%. Adapted from [15]

Full size image

Due to its wide range of applications, AR is considered one of the key technologies for assisting human operators in such scenarios [29]. The literature contains some works proposing AR systems to assist order picking tasks. Blattgerste et al. and Büttner et al. [9, 11] presented AR systems for assembly tasks, implemented for HMDs and projection hardware, correspondingly. Part of their system assists the component picking with AR; however, the corresponding picking task does not require the user to move. [17] uses in-situ projection for assisting industrial assembly tasks. Their study showed the learning success for untrained workers and the decreasing performance for expert operators. Reif and Günthner [58] compared an AR marked-based order picking system with the traditional paper manual system. The AR system proved its potential, being 4% faster than the traditional paper manual method and producing seven times less errors. Murauer et al. [44] conducted a laboratory study for comparing an AR method displayed in HoloLens with traditional paper manual methods (in both participants native language and unknown foreign language). Participants were faster using the AR system, followed by using the textual instructions in the native language and the textual instructions in the foreign language. The AR system led to the same amount of errors over the paper manual. It was also found that their AR system leads to significantly less cognitive effort.

All things considered, it seems that AR research has not focused in the usage of continuous experiences, i.e., markerless AR solutions to support complex order picking tasks in real industry environments, which imply human motion through the environment. , instead of the traditional tasks, implying having operators seated and facing all the necessary materials to accomplish the picking/assembly tasks.

In fact, solutions that explore continuous experiences exist, being most of the research effort devoted to serious games or museums exhibitions [26, 33, 64].

Furthermore, there is also no record of a comparison between different methods for creating Pervasive AR experiences in industrial scenarios. Nevertheless, in order for the field to mature and contribute to a higher level of knowledge of the research community, the authors argue that it is paramount to consider more complex use-cases as is the case of logistics contexts. In particular, the authors are referring to training operators in industrial scenarios, where the authors strongly believe this technology can have an important impact by displaying instructions on what to do throughout the shop floor.

All things considered, it seems that complex logistics tasks have been strongly addressed by Industry 4.0. There are several works applying AR for such tasks, but other technological systems were considered [6, 22]. These works consist of marker-based or space limited solutions [17, 58] or markerless solutions related to tasks that do not require human motion through the environment [9, 11, 17]. Also, most researchers address the real industrial task in realistic laboratory procedures [9, 11, 23, 44]. Certainly solutions that explore continuous experiences in real-life scenarios exist, being most of the research effort devoted to serious games or museum exhibitions [26, 33, 64]. Table 1 summarizes the related work of the AR usage in industrial logistics and the applicability of Pervasive AR for other use-cases.

Table 1 Overall description of the related work in industrial logistics and continuous AR experiences: Application area, functionality and method
Full size table

Hereupon, the novelty of the authors’ research is the usage of continuous AR experiences, i.e., markerless solutions to support complex order picking tasks in real industry environments, which imply human motion through the environment.

Furthermore, there is also no record of a comparison between different methods for creating Pervasive AR experiences in industrial scenarios.

Nevertheless, in order for the field to mature and contribute to a higher level of knowledge of the research community, the authors argue that it is paramount to consider more complex use-cases as is the case of logistics contexts. In particular, industrial operators’ training, where the authors strongly believe this technology can have an important impact by displaying instructions on what to do throughout the shop floor.

3 Methods and materials

This section describes a human-centered approach to identify the needs and challenges of human operators during logistics tasks in shop floor scenarios (see Fig. 2).

The authors focused on a picking use-case, benefiting from an ongoing collaboration with partners from the industry sector.

A set of requirements were identified and analyzed.

These were used in the conceptualization and development of a Pervasive AR tool.

Afterwards, the authors discuss the first impressions derived from an initial user study in the authors’ partner’s factory.

Finally, the authors describe a set of improvements conducted before a larger study, described in Section 4.

Fig. 2
figure 2

Methodology adopted: a) Human-Centered Design approach to identify the needs and challenges of human operators during logistics tasks on shop floor scenarios; b) identification of a set of relevant requirements; c) design and development of a Pervasive AR based prototype for HHD and HMD; d) interactive evaluation of the prototype developed using real-life picking tasks

Full size image

3.1 Shop floor scenario

To help guide the authors’ research, a picking line from an industrial factory was considered. This picking line is acknowledged, by the authors’ industrial partners, as one of the production lines that leads to most the errors and mistakes. It is composed of seven large shelves, arranged in an “L” shape (see Fig. 3), a computer and printers close to the middle.

Each shelf contains several component boxes identified with a label. The labelling system has the format “shelf_number-row_number-column_number” (e.g., E04-02-03) and is arranged from the bottom-right box to the top-left box.

The type of component relative to the label position remains static during large periods (6 months or larger); however, if the production changes significantly, some components can be swapped with others or even removed.

This shop floor scenario is located at an extremely noisy sector. Nearby, machinery, equipment and vehicles operate all day long.

Fig. 3
figure 3

Overview of the real-life logistics scenario. Examples of shelves of the industrial shop floor composed of distinct boxes with various distinct components

Full size image

Per eight hours shift, three operators, working in parallel, perform kit assembly tasks.

The kit content varies according to the needs of the final product which the kit is associated with. These kits are composed of predefined components (documentation, screws, pipes, buttons, caps, batteries, etc.) to be merged with the final product before being packed for worldwide distribution.

Currently, this task is still supported by traditional printed paper manuals, containing the list of material composing each kit. Each component has the following information:

  • Component: The reference of the desired item;

  • Denomination: Component common name;

  • Quantity: Number of item to include in the kit;

  • Picking location: Position of the component (e.g., E02-01-06).

The kit assembly requires the location of components, so these can be collected by hand, and temporally placed into a cart with 16 sections for later packaging (Fig. 4).

According to operators, the picking process is performed paying attention to the picking location and the quantity of components. Although other information exists, most is actually ignored, being non-relevant to the picking process, which can actually cause confusion and distractions.

In order to save up time, operators often do not follow the kit components list in a sequential order. To elaborate, when they spot that some components are close to each other, they pick these consecutively. After picking all the kit components, the packing process is finalized and the kit is sent to the subsequent assembly line.

It is apparent that operators perform the task walking along an “L shape” corridor, also designated as “Supermarket,” using their mental abilities and task expertise to draw the shorter path for the kit components picking location. Furthermore, they have to memorize which components were already picked while assembling the kit. This behavior often leads to mistakes, such as skipping components in the list, resulting in incomplete kits, which in turn, delays the production of other factory lines and can result in client complaints. In addition, new operators require a long training period, needing to spend a significant amount of time to reach the task performance of more experienced operators.

Fig. 4
figure 4

Example of the industry cart used during the picking procedures by the participants. The goal was to allow them to pick and place the components during the selected tasks

Full size image

3.2 Requirements elicitation

To better understand the operators’ needs and challenges, a HCD methodology was considered. The authors coordinated various on-site and remote meetings with the authors’ industrial partners, as well as brainstorming sessions and visits to the shop floor. These meetings involved data engineers, process development engineers, line operators and line managers. This leads to the elicitation of a set of requirements.

To address the described scenario and tasks, it is necessary to select the kit that needs to be assembled, read its list of components and inform about their location. It is also relevant to consider that operators must move continuously along the production line to perform the picking at the corresponding locations.

Another important topic is the need to facilitate the picking process, by making this activity simpler and faster to conduct. Hence, decrease the mental and physical effort from these operators, allowing them to walk less, with the same or higher productivity.

Ideally, solutions for these tasks must be applied throughout the entire shift, or even work-day.

Besides, the authors considered important to validate the picking process in order to reduce errors raised from grabbing components from the wrong boxes or forgetting to pick others.

In this context, it is also important to consider different hardware alternatives at this moment, being able to have a solution that can run in different platforms, to better comprehend which is more adequate to support the requirements for long-term usage.

Table 2 describes the requirements outlined above.

Table 2 Requirements identified during the meetings to support the described picking task
Full size table

3.3 Pervasive AR tool for picking tasks in industrial scenarios

To fulfill the previous requirements, a Pervasive AR tool was designed and developed.

Two distinct hardware alternatives were considered: HMD and HHD. Thus, having options with different financial costs, enabling to compare their advantages and disadvantages and evaluate which is more beneficial for the selected industrial use-case.

Both methods share a similar architecture, being divided into two modules: configuration and visualization (Fig. 5).

Fig. 5
figure 5

Architecture overview. The Pervasive AR tool supports two display technologies: HMD and HHD. The configuration of the virtual information over the real-word is performed once and stored. Later, it is possible to visualize the information in an AR setting, thus supporting operators’ tasks

Full size image

The configuration module consists in the placement of virtual content over the real-world, so this information can be stored for later access. This procedure is mandatory, however required only once.

Since the real-world context occasionally changes, these changes have to be updated in the application too. Hereupon, the configuration module supports minor changes to the virtual content, not requiring to repeat the configuration process for the components that remained static.

This module maps the points of interest of the real-world as a whole and allows to add virtual content, in the form of cubes over the real-word boxes, arranged on the shelves. These cubes can be translated, rotated, scaled and copied, providing the required geometric transformations to perform the entire AR configuration efficiently and adaptable to various layouts and box sizes.

The box placement does not require high accuracy, depending on the real-world box size, the virtual box can be displaced from 10 to 30 cm; however, the tool offers higher accuracy when necessary.

Then, from a list containing the existing components on the shop floor, each cube is associated to the corresponding component (Fig. 6; watch sample videos for the HMDFootnote 2 and HHDFootnote 3 version).

After having the virtual content correctly placed and labelled, it is stored, in relation to the real-world mapping, so it can be used by the visualization module.

Fig. 6
figure 6

Configuration module. Virtual cubes can be placed in the desired poses. The components are associated with the cubes. At the end, the AR environment is stored

Full size image

The visualization module uses the stored information to automatically display the AR content in the correct pose over the real environment.

This module compares the stored real-world mapping with the current camera viewpoint. When a match occurs, the associated AR content is set to its configured pose.

Selecting the kit to be assembled, the virtual content associated to its components is acknowledged as such. Then, the AR information appears above the locations to perform the picking. The AR content is presented in a floating balloon form with the relevant information about the component (Fig. 7).

Two methods for presenting AR content were created:

  • Sequential: The AR balloons are presented one by one in the sequential order of the kit list. When a component is picked, the next one automatically appears (watch sample videos for the HMDFootnote 4 and HHDFootnote 5 versions);

  • Non-sequential: Every AR balloon, i.e., component from the kit list is presented. The picking is performed according to the operators’ preference (watch sample videos for the HMDFootnote 6 and HHDFootnote 7 versions).

To validate the picking, when a component is collected, the corresponding balloon changes its background color from white to blue, indicating the last picked location. When a new component is picked, the last blue balloon is hidden.

This mechanism ensures that distracted operators can recall from which box the picking was being done or allow them to keep knowing the respective box, even if they put their hands over the box by fault.

At any time, the operator can access how many and which components compose the kit, as well as, which were already picked and which were not.

Fig. 7
figure 7

Visualization module. The blue information balloon represents the last picked component. The white information balloon represents the next component to be picked

Full size image

As for the development, and although the two methods are very similar, different technologies were considered in combination with the Unity game engine based on C# scripts. For the HMD, the Mixed Reality ToolkitFootnote 8 (MRTK) was considered with its local world anchors feature, allowing local persistence. Regarding the HHD, it uses ARCoreFootnote 9, taking advantage from its cloud anchors feature, which requires internet connection to store the real-world information in an external API.

While configuring the AR environment using HMD, the cubes are translated, rotated and scaled applying mid-air hand gestures, that can be used for close (by intersecting the operators hand with the 3D box) and far (by using the rays casted by the hands as pointers) interaction. The following gestures were consideredFootnote 10: air-tap, i.e., detects operators index finger to interact with existing buttons; one-handed mid-air pinch, i.e., grab and rotate 3D objects; and two-handed mid-air pinch, i.e., spreading both hands to change the scale of the object. As for the HHD, this can be performed applying the drag, turn to rotate and pinch/spread gestures in the device screen [14]. Furthermore, the HMD menu is accessed through the left hand facing upwards, while in the HHD is shown/hidden in the screen side panels (Fig. 6). Those menus also include a scene selector, allowing the configuration of multiple environments. This can be useful when considering multiple assembly lines, each one with its own set of components to be picked.

Regarding visualization, different validation mechanisms were implemented for the picking tasks. For the HMD, the validation is acknowledged when a human hand enters the virtual balloon area, i.e., when grabbing components from boxes. Differently, the HHD validation is not automated and requires operators to press the virtual balloon on the device screen, before or after picking the component.

3.4 First impressions on the shop floor

To understand the viability of using the initial version of the Pervasive AR tool, two visits to the shop floor were conducted. In total, more than 12 hours were spent in the industrial scenario.

These visits consisted in a preliminary user study. The goal was to evaluate the acceptance towards the use of the proposed technology in real-world context, as well as, gather first impressions regarding usability, interaction and tracking, allowing to establish the next steps of the authors’ research.

Participants were asked to conduct the picking process of a selected kit using the HMD and the HHD versions of the Pervasive AR tool (Fig. 8). To avoid learning effects, the order in which the tests were conducted was balanced.

Concerning the hardware used, the authors considered the Microsoft HoloLens 2 for the HMD method. In turn, the Asus Zenfone AR or the Samsung Galaxy A52 were attached to a smartphone hand grip for the HHD method, testing two devices with different sensors.

Participants were introduced to the goals of the study, as well as the experimental setup and design. After giving their informed consent, they were able to perform the tasks. Next, the picking task was completed, while being observed by a researcher who assisted them if necessary, and registered their actions and difficulties.

In addition, the route taken by three participants (1 experienced, 2 inexperienced) while performing the task was registered by hand over the architectural plan of the assembly line (see Fig. 4 — right). For these three participants, it was also requested to perform the picking task using the traditional paper manual.

In the end, subjective data was collected during informal interviews with the participants.

Participants’ opinion towards the different versions was registered by the researchers that accompanied the study. Plus, demographic data, and previous experience with AR was also considered.

The data collection process was conducted under the guidelines of the 1964 Helsinki Declaration.

Twenty-six (26) individuals (10 female, 16 male) from distinct departments of the factory participated in the study (e.g., operators, line managers, logistics, ergonomics, maintenance, process engineering, engineering manager, production manager).

From these, 40% of participants had previous experience with AR, and 25% were familiar with the picking task.

Fig. 8
figure 8

Participant performing the picking task using the Pervasive AR tool to visualize the kit components location. Left: Using a HMD. Right: Using a HHD

Full size image

Overall, participants considered the Pervasive AR tool robust and accurate to be applied in a real industrial setup. It was also deemed easy to use, as well as having great potential to support operator’s tasks, specially for inexperienced operators.

Another advantage mentioned, is the fact that operators do not need to be familiar with the line layout, the components localization or even the labelling system, which can facilitate situations in which the layout is changed for efficiency purposes or to expand its production.

Additionally, operators do not need to memorize which components were already picked, thus decreasing the cognitive effort they endure daily.

As for the methods considered, an adaptation period was considered necessary for both variants, although, the interaction quickly becomes familiar to the users, stating that with time, task performance would easily improve.

Regarding the HMD method, using the Microsoft Holo-Lens 2, it was arguably the preferred option, with participants reporting that it was more stable when presenting the AR content in relation to the desired locations of the production line.

Furthermore, the hands-free approach was appraised as extremely useful to visualize additional information while conducting the picking tasks. In particular, when compared to the HHD alternative, which requires operators to hold such devices at all times, unless some sort of support is used, which was mentioned by various participants.

The need to test this method for a longer period of time was also mentioned, in order to verify if it is comfortable/ergonomic for long-term use, while also understanding how to handle the battery limitations. The possibility of using power-banks was also brought up.

Another relevant insight uncovered during the task execution of an operator, is associated with the fact that the operator decided to use a pair of gloves, in particular, black and shiny ones, which caused the HMD hand detection algorithm to struggle with recognizing the operators hands. This was very important, given that until that moment, such characteristics were not considered.

Concerning the HHD method, the Samsung Galaxy A52 showed a considerable decrease in its motion tracking technology at the industrial environment. In turn, the Asus Zenfone AR smartphone did not suffer from such a high performance impact when used in this scenario.

Yet, this method appears to reduce operator’s understanding of their surroundings when compared to the HMD alternative. It was also considered relevant having an automatic process that validates that the correct component was picked, instead of requiring human input at all times, which may become overwhelming after a certain period of time.

It was also stated that sequential picking would only be interesting for training new users and that non-sequential order is deemed the better option, given that it can provide a more detailed overview of the environment, while also giving more freedom to the operators.

As expected, participants suggested various improvements, deemed relevant to refine the AR tool and enhance operators performance in the future.

Most of these were considered and included in a revised version of the proposed tool. To elaborate, the following was considered: (1) visualize additional information of the component that needs to be picked besides its location (e.g., reference, denomination, quantity); (2) integrate different colors for the AR content, given that the colors initially selected would sometimes be confused with the environment; (3) display the percentage of components gathered during task resolution to improve awareness and motivation;

4) keep the augmented content in the display for a larger period of time after a given component is picked to avoid confusions and providing a clear overview of the task status;

5) include ergonomic warnings and notifications, e.g., suggest adjusting the HMD headset prior to its use to improve operators well being later on; 5) incorporate guidance on the next component when a sequential order is used, and when it is not in sight of the device field of view, thus avoid wasting time looking around. This was integrated through the use of directional arrows (Fig. 9). While in the HMD it appears on the field of view sides and just points top, down, left and right, in the HHD, it is fixed in front of the device camera viewpoint and rotates with 3-degrees of freedom.

Fig. 9
figure 9

Examples of an improvement conducted. Directional arrow used to guide operators to the next component to be picked. Left: HMD; Right: HHD

Full size image

Finally, Fig. 10 presents a set of Spaghetti diagrams with the paths travelled by the three participants, allowing their comparison while using the three methods considered (HMD, HHD and paper manual), and to verify which of them requires participants to walk less during the picking procedure [32]. It is observable that the routes walked by the inexperience participants (Fig. 10b and c) while using the AR applications are remarkably more efficient than using the traditional paper manual. On the other hand, the experienced participant (Fig. 10a) adopts a more efficient route using the paper manual, which is more familiar. However, the routes travelled using the HMD and the HHD were nearly as efficient. All in all, these results lead us to infer that Pervasive AR tool can contribute to less physical effort and higher efficiency when used by novice operators, which are learning or training the picking task.

Fig. 10
figure 10

Spaghetti diagrams representing three participants routes for the three methods considered: paper (blue), HMD (green) and HHD (red)

Full size image

4 Shop floor user study

In Section 3, the potential of employing Pervasive AR on a shop floor environment was assessed. The feedback obtained was used to improve the AR tool. This section reports a second user study, conducted to compare how different conditions affected operators performance during distinct picking tasks.

4.1 Experimental setup

Three distinct methods were considered: C3 — HHD using the smartphone Asus Zenfone AR attached to a hand grip.

The necessary AR content, corresponding to the components that needed to be picked for each kit was configured before the user study. Plus, during the study, a cart was used, so that participants could pick and place the components while moving through the environment.

4.2 Experimental design

The null hypothesis (H0) was considered, i.e., all experimental conditions are equally usable and acceptable to support the picking tasks.

The independent variables were the visualization method used during the picking tasks by the participants, with three levels corresponding to the experimental conditions: C1 — paper manual; C2 — HMD; C3 — HHD.

Plus, the methods for presenting the AR content: Sequential and Non-sequential. The dependent variables were participants’ opinion regarding the conditions used and the average time per component picking.

As secondary variables, participants’ demographic data, previous experience with AR and with the industrial task were considered.

A within-subjects experimental design was used, meaning that all participants used every condition.

4.3 Task

Regarding the tasks, three real-life industrial kits were considered, taking advantage of an ongoing research project with partners from the industry sector: small kit, frequent kit, uncommon kit.

The goal was to pick a list of components based on the kits selected.

In addition, a preparation kit was also considered, composed with three components not picked in the previous mentioned kits. This was used to introduce participants to the AR tool.

During the task execution, the components were placed in a cart, which could be carried around the line or not, leaving this decision up to the participants.

4.4 Measurements

Two types of data were collected. Task performance, which consists in the average time required to pick a component, recorded in seconds by a manual stopwatch (condition C1) or by the device (condition C2 and C3).

Participants’ opinion was gathered through a post-task questionnaire, including the following: demographic information (age, gender, factory department and role), experience with AR and on the assembly line; the post-task questionnaire collected the following dimensions: D1 — Level of confusion or distraction about the content used; D2 — Level of physical effort; D3 — Level of mental effort; D4 — Level of satisfaction.

Dimensions data was collected using a Likert-type scale: 1 — Low; 7 — High.

Furthermore, Participants’ difficulties, comments and observations, as well as their opinion on general questions were requested: Q1 — Condition order of preference; Q2 — Willingness to use AR for this task on a daily basis;

4.5 Procedure

Both methods share a similaParticipants were introduced to the goals of the study, as well as the experimental setup and design. After giving their informed consent, they were introduced to the tasks and the conditions considered.

Participants started by filling a demographic questionnaire.

Then, an adaptation time was provided, in which participants used the conditions to conduct the picking task with the preparation kit.

Next, the selected tasks were completed with all conditions, while observed by a researcher who assisted them if necessary, and registered any relevant event.

To minimize bias, i.e., learning effects, the order in which every condition was presented across the group of participants was counterbalanced.

At the end, participants answered a post-task questionnaire associated with their preferences towards the conditions used.

Last, a small interview was conducted.

The data collection process was conducted under the guidelines of the 1964 Helsinki Declaration.

Moreover, two groups were created, splitting participants in half. The first half performed the picking tasks using the sequential method.

The second group picked the required components in a non-sequential order of their choosing.

4.6 Participants

In total, 10 participants were recruited (1 female, 9 male), whose ages ranged from 22 to 39 years old (M=26.6, SD=4.9).

These had distinct professions at the factory, e.g., data engineer, automation engineer, industrial engineer, maintenance engineer, mechanical engineer, line manager, line worker (did not answer the post-experiment questionnaire) and manufacturing digitalization engineer. From these, 60% of participants had previous experience with AR and beyond the expert line worker and two participants with little experience, the participants had no experience with this production line.

5 Results and discussion

Next, the results from the user study are described and discussed. All participants were able to assemble the designated kits using the three conditions. On average, each session lasted for 40 min (about 30 min to complete the task).

Figure 11 and Table 3 present the average picking time per component in each condition considering the picking order: sequential and non-sequential. The chart illustrates that the time efficiency increases using condition C2 (HMD).

To elaborate, this condition was 70% faster than the traditional paper condition using the sequential picking order (11s and 36s, correspondingly) and 54% more efficient applying the non-sequential (13s and 28s, correspondingly).

Besides, using condition C3 (HHD) was more efficient compared with the paper manual in both methods (25s and 36s applying the sequential picking order and 17s and 28s using the non-sequential, respectively). However, not as much as condition C2 (HMD) (25s and 11s applying the sequential picking order and 17s and 13s using the non-sequential, respectively).

Hence, a decrease in time can be observed when using the non-sequential picking order compared to the sequential method with condition C3 (HHD) (25s and 17s, correspondingly) and condition C1 (paper) (36s and 28s, correspondingly). This does not apply to condition C2 (HMD) (11s and 13s, correspondingly) since the directional arrow feature was not available in this condition during the non-sequential method.

Looking to condition C3 (HHD) efficiency increase, the authors believe that condition C2 (HMD) would be even more time efficient applying the non-sequential method with this feature included.

Table 3 Average picking time per component using sequential and non-sequential order in three conditions: C1 — Paper; C2 — HMD; C3 — HHD
Full size table
Fig. 11
figure 11

Average picking time per component using sequential and non-sequential order using three conditions: C1 — Paper; C2 — HMD; C3 — HHD

Full size image

Analyzing the dimensions addressed in the questionnaire, rated by the participants based on a Likert-type scale: 1 — Low; 7 — High (Fig. 12 and Table 4), the following was reported. Regarding the level of confusing and distraction about the presented information (D1), condition C2 (HMD) rated lower (median=2, sum=18), then condition C3 (HHD) (median=2, sum=26) and condition C1 (paper) (median=4.5, sum=43). As the data is on an ordinal scale and each user performed the three conditions (matched sample), the equality of medians was tested with the Friedman test (ANOVA nonparametric test), which rejected the null hypothesis (equality of distributions/medians, p-value=0.006). Furthermore, multiple paired comparisons, with Bonferroni correction, considered C3 not different from C1 (p-value=0.110) and not different from C2 (p-value=0.395), but established C2 as significant different (preferred) to C1 (p-value = 0.005). With respect to the level of physical effort (D2), condition C2 (HMD) was rated lower (median=2, sum=16), followed by condition C1 (paper) (median=3, sum=28) and condition C3 (HHD) (median=3, sum=35) respectively. The Friedman test rejected the null hypothesis (p-value=0.013), indicating differences among conditions. Multiple paired comparisons, with Bonferroni correction, considered C1 not different from C2 (p-value=0.270) and not different from C3 (p-value=0.270) but established C2 as significant different (preferred) to C3 (p-value=0.011). As for the level of mental effort (D3), condition C2 (HMD) was rated lower (median=2, sum=21), then condition C3 (HHD) (median=3, sum=30) and condition C1 (paper) being rated higher (median=4.5, sum=45). The Friedman test rejected the null hypothesis (p-value=0.019), indicating differences among conditions. Multiple paired comparisons, with Bonferroni correction, considered C3 not different from C2 (p-value=0.471) and not different from C1 (p-value=0.140) but established C2 as significant different (preferred) to C1 (p-value=0.011). Concerning the level of satisfaction (D4), condition C2 (HMD) was rated higher (median=6.5, sum=64), followed by condition C3 (HHD) (median=5, sum=50) and condition C1 (paper) (median=2.5, sum=28). The Friedman test rejected the null hypothesis (p-value<0.001), indicating differences among conditions. Multiple paired comparisons, with Bonferroni correction, considered C3 not different from C2 (p-value=0.050) and not different from C1 (p-value=0.219) but established C2 as significant different (preferred) to C1 (p-value=0.001). Condition C2 (HMD) was preferred in all dimensions to condition C1 (paper). On the other hand, there was no statistical difference between condition C3 (HHD) and condition C1 (paper), as well as, between condition C3 (HHD) and condition C1 (paper).

Table 4 Overview of the participants post-task questionnaire (median and sum) regarding the dimensions and questions for each condition (C1 — Paper manual, C2 — HMD, C3 — HHD): D1 — Level of confusing and distraction about the presented information; D2 — Level of physical effort; D3 — Level of mental effort; D4 — Level of satisfaction; Q1 — Condition order of preference; Q4 — Level of AR helpfulness in learning/completing the task
Full size table
Fig. 12
figure 12

Overview of the results concerning the dimensions: D1 — level of confusing and distraction about the presented information; D2 — level of physical effort; D3 — level of mental effort; D4 — level of satisfaction. Conditions: C1 — Paper; C2 — HMD; C3 — HHD. Data collected using a Likert-type scale: 1 — Low; 7 — High

Full size image

When questioned about their preferences (Q1), condition C2 (HMD) (median=7, sum=66) was selected as the preferred alternative, followed by condition C3 (HHD) (median=5, sum=48) and lastly, condition C1 (paper) (median=2.5, sum=24).

The Friedman test rejected the null hypothesis (p-value<0.000). Furthermore, multiple paired comparisons, with Bonferroni correction, considered C1 not different from C3 (p-value=0.219) but established C2 as significant different (preferred) to C1 (p-value=0.000) and to C3 (p-value=0.024). In general, participants would be willing to use the AR tool (Q2) (median = 6, sum=56). However, the usage time ranged from “while training” and “for a short time, one or two hours, as it would be uncomfortable for a longer time” to “the entire shift.” Most participants disliked the smartphone holder (Q3) while using condition C3 (HHD) because it occupies one of the hands. Despite requiring additional testing, other holders were suggested: “smartphone holder fixed to the cart,” “wristband smartphone holder” or “smartphone holder attached to a vest.”

Both AR conditions were seen as identically helpful in learning (C2-HMD: median=7, sum=66; C3-HHD: median=6, sum=62) assessed with the Wilcoxon matched pairs nonparametric test (p-value=0.178) and in completing the task (C2-HMD: median=7, sum=63; C3-HHD: median=6.5, sum=63) assessed with the Wilcoxon matched pairs nonparametric test (p-value=1.000), result of the analysis of the users’ answers to question Q4.

Participants preference regarding condition C2, as suggested by their comments, is associated with the fact that, when using the HMD, it was easier to maintain a good visual-motor coordination (P4 — “As the smartphone field of view is smaller, sometimes it was harder to find the virtual information.”). Having a hands-free setting facilitates accomplishing the components picking tasks (P4 — “Holding the smartphone with one hand and perform the operations with the other hand is not practical.”). Also, they stated that the stability of the virtual content in this condition was better, which is beneficial for situation understanding and awareness (P6 — “With the smartphone I was afraid to lose the tracking of the virtual world, what did not happen when using the HMD.”).

Also important, both conditions using AR have been considered to become more intuitive and easy to use over time, which in turn may be reflected in lower cognitive effort (P2 — “[The AR tool is] very intuitive, easy to interact and facilitates the component picking.”).

A Cluster Analysis of the answers of the four dimensions in the three conditions revealed: (i) high satisfaction with C2 (HMD) and C3 (HHD) and low satisfaction with C1 (paper); (ii) low perceived level of confusion, mental and physical effort concerning C2 (HMD) and high level of confusion about C1 (paper); (iii) moderate and equivalent level of mental and physical effort regarding C1 (paper) and C3 (HHD).

Additionally, the use of Correspondence Analysis allowed to note some user profiles, namely: (i) some resistance to the change (from paper to AR) of older participants, with more daily experience of using paper in the task and who considered moderate the mental effort of using AR; (ii) a straightforward acceptance of AR, by users of manufacturing digitalization and industrial engineering, who penalize the use of paper in the dimension of confusion, highlight the very low physical and mental effort of AR and express, also, a high level of satisfaction for this new paradigm; (iii) the automation, mechanical and maintenance engineering users’ attention to ergonomics issues, namely the physical effort in C3 condition (HHD), although they also express a high satisfaction for using AR.

6 Conclusions and future work

Nowadays, the value Augmented Reality (AR) can generate for several industrial applications such as assembly, maintenance, quality control, logistics and others is undeniable. Regardless, research is still needed for AR solutions to be fully integrated in the daily activities of human operators.

In this paper, the authors proposed a Pervasive AR tool for supporting logistics operators during order picking tasks on industrial shop floors. A Human-Centered Design (HCD) methodology was used with partners from the industry sector to identify operators’ difficulties, challenges, and define requirements. Following, two methods were developed (Head-Mounted Display (HMD) and Handheld Device (HHD)), allowing to configure and visualize AR content in the industrial environment. All in all, virtual content can be visualized, allowing operators to know how to conduct picking operations, i.e., having step-by-step instructions, including text, images and 3D models on how to assemble a given kit.

The proposed methods were evaluated during picking tasks on the shop floor in two phases: first, to have an understanding of first impressions from twenty-six participants, including individuals with different occupation and expertise. This feedback was used to improve the prototypes before the second study, in which, ten participants that had never conducted (except one expertise operator and two participants with little experience) the selected tasks used three different conditions: C1 — paper; C2 — HMD; C3 — HHD. The goal was to verify which conditions were more adequate and could contribute with higher productivity for the task.

Results emphasize the potential of Pervasive AR for operators picking activities on the shop floor environment, in particular for training operators not familiar with the tasks and to help in finishing the tasks. Although both conditions seem as valid options to support operators, condition C2 was preferred by all participants, being considered more useful and efficient on the shop floor scenarios, mostly due to its hand-free setting, as well as higher time efficiency, less cognitive and physical effort and providing a higher level of satisfaction. On the other hand, although condition C3 was faster than C1, it was not considered by the participants as presenting significant advantages in the other dimensions compared to C1.

The study also revealed that careful attention is required when designing the experimental protocol to ensure that the assembly line is not disrupted during labor. Additionally, it is important to ensure the safety of participants when using the developed tool, as their level of awareness may be affected, especially if they are not familiar with the devices used. The access to a considerable number of participants on the shop floor during the conducted evaluations was difficult to achieve and time consuming. Besides, some workers were hesitant to cooperate as they seemed concerned that the technology could replace their jobs in the future, or that could be used to monitor their performance.

As next steps, the authors intend to study upgraded validation mechanisms, as well as, to conduct longitudinal studies on the shop floor environment. Although some insights were already obtained, longer studies and more complex tasks are an opportunity to better comprehend how AR solutions may affect operators health, motivation, and productivity on a daily basis.

Furthermore, explore co-located collaborative use-cases, requiring various operators to work together on shared goals, while taking advantage of the benefits of Pervasive AR in Industrial scenarios.