Abstract
Augmented Reality (AR) is a pillar of the transition to Industry 4.0 and smart manufacturing. It can facilitate training, maintenance, assembly, quality control, remote collaboration and other tasks. AR has the potential to revolutionize the way information is accessed, used and exchanged, extending user’s perception and improving their performance. This work proposes a Pervasive AR tool, created with partners from the industry sector, to support the training of logistics operators on industrial shop floors. A Human-Centered Design (HCD) methodology was used to identify operators difficulties, challenges, and define requirements. After initial meetings with stakeholders, two distinct methods were considered to configure and visualize AR content on the shop floor: Head-Mounted Display (HMD) and Handheld Device (HHD). A first (preliminary) user study with 26 participants was conducted to collect qualitative data regarding the use of AR in logistics, from individuals with different levels of expertise. The feedback obtained was used to improve the proposed AR application. A second user study was realized, in which 10 participants used different conditions to fulfill distinct logistics tasks: C1 — paper; C2 — HMD; C3 — HHD. Results emphasize the potential of Pervasive AR in the operators’ workspace, in particular for training of operators not familiar with the tasks. Condition C2 was preferred by all participants and considered more useful and efficient in supporting the operators activities on the shop floor.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Industry 4.0 has been proposed as a new phase in the industrial revolution, integrating digital and physical worlds through technology into the industrial procedures [53, 67]. Despite the promising benefits of using smart sensors, embedded systems, cyber-physical systems and Internet-of-Things (IoT) in manufacturing [2, 20, 27, 48, 63, 70], human operators remain an essential part of any industrial processes [27, 46]. Smart factories are essential in the stock market [69] and almost doubled their market size in the last 5 yearsFootnote 1. Regardless, in logistics procedures, human intervention still remains the most important asset to accomplish the intended goals [30]. One example is “kit assembly.” Usually, this task uses a mixed-model assembly, i.e., on the same line, several distinct kits, composed from different components must be assembled. During the kit preparation, the operators have to identify the necessary components and their corresponding location, so that they can be picked up correctly [23].
This information is mostly conveyed through traditional printed paper manuals, requiring that operators read the list of components composing the kit and then search for their location in the line [23].
However, paper manuals can be mentally and physically demanding to some operators after long hours of labor, resulting in mistakes and decreasing efficiency. Furthermore, the learning and training phases for novice operators are not straightforward, requiring a long period to prepare them for the shop floor activities. One possible solution to improve the kit assembly operations is the use of Augmented Reality (AR), considered one of the nine pillars of Industry 4.0 to support operators with real-time information for faster decision-making, while improving work processes [15, 24, 50, 54, 56, 62]. This technology can integrate virtual information in the operators workspace [35, 42], helping them in assembly tasks [18, 43, 49], provide context-aware assistance [5], data visualization and interaction (acting as a Human-Machine Interface (HMI)) [16, 40], indoor localization [60], maintenance applications [8, 18, 61], quality control [4, 65], material management [16, 51] or remote collaboration [7, 39, 66], by presenting additional layers of digital information on top of real-world environments [3, 28, 33, 37, 38, 57]. Prior studies identify certain benefits of applying AR for technological industrialization, like increased work safety, effective learning and training, as well as more task effectiveness [10, 12, 31], as well as improved Human-Robot Interaction (HRI) [1, 13, 19, 34].
However, most AR solutions are limited to a given space, which means operators must have all necessary materials within a limited range, otherwise, the tracking capabilities of existing tools will fail, affecting operators experience, and in turn, task performance.
To overcome this limitation, the concept of Pervasive AR offers experiences that can be visualized continuously as users move through the environment. Such experiences are also characterized by being aware of and responsive to the user’s context and pose [21, 36, 47].
In fact, this concept has the potential to evolve AR solutions to multi-purpose experiences, providing easier access and better perception of information, changing how users interact with virtual content and their surroundings.
Pervasive AR and Mixed Reality (MR), depending on the definition, can differ; however, they are both part of the “Virtuality Continuum” [41]. MR has no single definition and can be considered different things as its understanding is based on one’s context [59, 68]. One possible definition is the combination of AR and Virtual Reality (VR) parts that interact with each other but are not necessarily tightly integrated [52, 68]. Quint et al. [55] define it as the combination of real and virtual world information. On the other hand, Holz et al. [25] and Nebeling et al. [45] refer MR to the real environment that allows for shared interaction with virtual experiences. Hereupon, Pervasive AR extends the original AR concept in time and space based on the surrounding context [21], which can be inserted in the scope of MR.
Based on the constraints of traditional paper manuals and the research opportunities to enhance kit assembly tasks, in this work, a Pervasive AR tool for production assistance was developed.
This tool was designed with partners from the industry sector using a Human-Centered Design (HCD) methodology.
The proposed solution can be used in Head-Mounted Displays (HMDs) and Handheld Devices (HHDs), enabling to configure new training experiences on the shop floor, or edit existing ones to incorporate new steps. The goal is to provide training experiences that are continuous in space and time, without the need of visual markers (e.g., QR Code, Aruco). By doing so, it is possible to support training sessions, not only of novel operators, but also of more experienced operators, when new kits are introduced. Besides, this work also has the objective of evaluating the developed tool and comparing it with the traditional paper manual method. Two user studies were conducted on the shop floor at different moments in the development process. Individuals having distinct expertise were considered to improve the range of the data collected and create a solution that is generic enough to be used by operators from different areas of a factory, if such a need ever exists.
Besides this section, the paper is structured as follows: Section 2 describes related work on AR in industrial scenarios. Section 3 details the logistics scenario considered. From here, requirements are presented, leading to the proposal of the AR prototype. Finally, an initial user study on the shop floor to gather first impressions is described and its results discussed. Section 4 reports a second user study on the shop floor to compare different methods. The results obtained are reported and discussed in Section 5. Section 6, draws concluding remarks and presents ideas for future work.
2 Augmented Reality for logistics in industrial scenarios
Logistics is a primordial part of industrial systems, although it is one of the least explored AR research areas, as illustrated by Fig. 1. While several logistic procedures, such as storage and stock removal are fully automated, there are still some processes where automation is not implemented, being highly dependent on human intervention [30, 71].
Order picking is one of the logistic procedures where most of the work is performed by humans, requiring flexibility and motor skills. Even processes, such as assembly tasks, require order picking to fetch the components constituting the product to be assembled. There is a wide range of research in order picking, as it is a complex operation with high economical relevance, having great potential for process optimization [71]. Hereupon, besides the traditional printed paper manual method, other forms of supporting the task can be applied, such as pick-by-voice systems [6] consisting in communicating with the warehouse management system via a voice recognition system; pick-by-light strategies [6], guiding operators by lights or digital displays; Head-Up Displays (HUD) [22] where the picker can use a small display to see the next component to pick; or by using AR tools [23, 29], overlaying digital picking information aligned with the real-world environment.
Distribution of the different fields of industrial application. In red, the logistics field, corresponding to 12%. Adapted from [15]
Due to its wide range of applications, AR is considered one of the key technologies for assisting human operators in such scenarios [29]. The literature contains some works proposing AR systems to assist order picking tasks. Blattgerste et al. and Büttner et al. [9, 11] presented AR systems for assembly tasks, implemented for HMDs and projection hardware, correspondingly. Part of their system assists the component picking with AR; however, the corresponding picking task does not require the user to move. [17] uses in-situ projection for assisting industrial assembly tasks. Their study showed the learning success for untrained workers and the decreasing performance for expert operators. Reif and Günthner [58] compared an AR marked-based order picking system with the traditional paper manual system. The AR system proved its potential, being 4% faster than the traditional paper manual method and producing seven times less errors. Murauer et al. [44] conducted a laboratory study for comparing an AR method displayed in HoloLens with traditional paper manual methods (in both participants native language and unknown foreign language). Participants were faster using the AR system, followed by using the textual instructions in the native language and the textual instructions in the foreign language. The AR system led to the same amount of errors over the paper manual. It was also found that their AR system leads to significantly less cognitive effort.
All things considered, it seems that AR research has not focused in the usage of continuous experiences, i.e., markerless AR solutions to support complex order picking tasks in real industry environments, which imply human motion through the environment. , instead of the traditional tasks, implying having operators seated and facing all the necessary materials to accomplish the picking/assembly tasks.
In fact, solutions that explore continuous experiences exist, being most of the research effort devoted to serious games or museums exhibitions [26, 33, 64].
Furthermore, there is also no record of a comparison between different methods for creating Pervasive AR experiences in industrial scenarios. Nevertheless, in order for the field to mature and contribute to a higher level of knowledge of the research community, the authors argue that it is paramount to consider more complex use-cases as is the case of logistics contexts. In particular, the authors are referring to training operators in industrial scenarios, where the authors strongly believe this technology can have an important impact by displaying instructions on what to do throughout the shop floor.
All things considered, it seems that complex logistics tasks have been strongly addressed by Industry 4.0. There are several works applying AR for such tasks, but other technological systems were considered [6, 22]. These works consist of marker-based or space limited solutions [17, 58] or markerless solutions related to tasks that do not require human motion through the environment [9, 11, 17]. Also, most researchers address the real industrial task in realistic laboratory procedures [9, 11, 23, 44]. Certainly solutions that explore continuous experiences in real-life scenarios exist, being most of the research effort devoted to serious games or museum exhibitions [26, 33, 64]. Table 1 summarizes the related work of the AR usage in industrial logistics and the applicability of Pervasive AR for other use-cases.
Hereupon, the novelty of the authors’ research is the usage of continuous AR experiences, i.e., markerless solutions to support complex order picking tasks in real industry environments, which imply human motion through the environment.
Furthermore, there is also no record of a comparison between different methods for creating Pervasive AR experiences in industrial scenarios.
Nevertheless, in order for the field to mature and contribute to a higher level of knowledge of the research community, the authors argue that it is paramount to consider more complex use-cases as is the case of logistics contexts. In particular, industrial operators’ training, where the authors strongly believe this technology can have an important impact by displaying instructions on what to do throughout the shop floor.
3 Methods and materials
This section describes a human-centered approach to identify the needs and challenges of human operators during logistics tasks in shop floor scenarios (see Fig. 2).
The authors focused on a picking use-case, benefiting from an ongoing collaboration with partners from the industry sector.
A set of requirements were identified and analyzed.
These were used in the conceptualization and development of a Pervasive AR tool.
Afterwards, the authors discuss the first impressions derived from an initial user study in the authors’ partner’s factory.
Finally, the authors describe a set of improvements conducted before a larger study, described in Section 4.
Methodology adopted: a) Human-Centered Design approach to identify the needs and challenges of human operators during logistics tasks on shop floor scenarios; b) identification of a set of relevant requirements; c) design and development of a Pervasive AR based prototype for HHD and HMD; d) interactive evaluation of the prototype developed using real-life picking tasks
3.1 Shop floor scenario
To help guide the authors’ research, a picking line from an industrial factory was considered. This picking line is acknowledged, by the authors’ industrial partners, as one of the production lines that leads to most the errors and mistakes. It is composed of seven large shelves, arranged in an “L” shape (see Fig. 3), a computer and printers close to the middle.
Each shelf contains several component boxes identified with a label. The labelling system has the format “shelf_number-row_number-column_number” (e.g., E04-02-03) and is arranged from the bottom-right box to the top-left box.
The type of component relative to the label position remains static during large periods (6 months or larger); however, if the production changes significantly, some components can be swapped with others or even removed.
This shop floor scenario is located at an extremely noisy sector. Nearby, machinery, equipment and vehicles operate all day long.
Per eight hours shift, three operators, working in parallel, perform kit assembly tasks.
The kit content varies according to the needs of the final product which the kit is associated with. These kits are composed of predefined components (documentation, screws, pipes, buttons, caps, batteries, etc.) to be merged with the final product before being packed for worldwide distribution.
Currently, this task is still supported by traditional printed paper manuals, containing the list of material composing each kit. Each component has the following information:
-
Component: The reference of the desired item;
-
Denomination: Component common name;
-
Quantity: Number of item to include in the kit;
-
Picking location: Position of the component (e.g., E02-01-06).
The kit assembly requires the location of components, so these can be collected by hand, and temporally placed into a cart with 16 sections for later packaging (Fig. 4).
According to operators, the picking process is performed paying attention to the picking location and the quantity of components. Although other information exists, most is actually ignored, being non-relevant to the picking process, which can actually cause confusion and distractions.
In order to save up time, operators often do not follow the kit components list in a sequential order. To elaborate, when they spot that some components are close to each other, they pick these consecutively. After picking all the kit components, the packing process is finalized and the kit is sent to the subsequent assembly line.
It is apparent that operators perform the task walking along an “L shape” corridor, also designated as “Supermarket,” using their mental abilities and task expertise to draw the shorter path for the kit components picking location. Furthermore, they have to memorize which components were already picked while assembling the kit. This behavior often leads to mistakes, such as skipping components in the list, resulting in incomplete kits, which in turn, delays the production of other factory lines and can result in client complaints. In addition, new operators require a long training period, needing to spend a significant amount of time to reach the task performance of more experienced operators.
3.2 Requirements elicitation
To better understand the operators’ needs and challenges, a HCD methodology was considered. The authors coordinated various on-site and remote meetings with the authors’ industrial partners, as well as brainstorming sessions and visits to the shop floor. These meetings involved data engineers, process development engineers, line operators and line managers. This leads to the elicitation of a set of requirements.
To address the described scenario and tasks, it is necessary to select the kit that needs to be assembled, read its list of components and inform about their location. It is also relevant to consider that operators must move continuously along the production line to perform the picking at the corresponding locations.
Another important topic is the need to facilitate the picking process, by making this activity simpler and faster to conduct. Hence, decrease the mental and physical effort from these operators, allowing them to walk less, with the same or higher productivity.
Ideally, solutions for these tasks must be applied throughout the entire shift, or even work-day.
Besides, the authors considered important to validate the picking process in order to reduce errors raised from grabbing components from the wrong boxes or forgetting to pick others.
In this context, it is also important to consider different hardware alternatives at this moment, being able to have a solution that can run in different platforms, to better comprehend which is more adequate to support the requirements for long-term usage.
Table 2 describes the requirements outlined above.
3.3 Pervasive AR tool for picking tasks in industrial scenarios
To fulfill the previous requirements, a Pervasive AR tool was designed and developed.
Two distinct hardware alternatives were considered: HMD and HHD. Thus, having options with different financial costs, enabling to compare their advantages and disadvantages and evaluate which is more beneficial for the selected industrial use-case.
Both methods share a similar architecture, being divided into two modules: configuration and visualization (Fig. 5).
The configuration module consists in the placement of virtual content over the real-world, so this information can be stored for later access. This procedure is mandatory, however required only once.
Since the real-world context occasionally changes, these changes have to be updated in the application too. Hereupon, the configuration module supports minor changes to the virtual content, not requiring to repeat the configuration process for the components that remained static.
This module maps the points of interest of the real-world as a whole and allows to add virtual content, in the form of cubes over the real-word boxes, arranged on the shelves. These cubes can be translated, rotated, scaled and copied, providing the required geometric transformations to perform the entire AR configuration efficiently and adaptable to various layouts and box sizes.
The box placement does not require high accuracy, depending on the real-world box size, the virtual box can be displaced from 10 to 30 cm; however, the tool offers higher accuracy when necessary.
Then, from a list containing the existing components on the shop floor, each cube is associated to the corresponding component (Fig. 6; watch sample videos for the HMDFootnote 2 and HHDFootnote 3 version).
After having the virtual content correctly placed and labelled, it is stored, in relation to the real-world mapping, so it can be used by the visualization module.
The visualization module uses the stored information to automatically display the AR content in the correct pose over the real environment.
This module compares the stored real-world mapping with the current camera viewpoint. When a match occurs, the associated AR content is set to its configured pose.
Selecting the kit to be assembled, the virtual content associated to its components is acknowledged as such. Then, the AR information appears above the locations to perform the picking. The AR content is presented in a floating balloon form with the relevant information about the component (Fig. 7).
Two methods for presenting AR content were created:
-
Sequential: The AR balloons are presented one by one in the sequential order of the kit list. When a component is picked, the next one automatically appears (watch sample videos for the HMDFootnote 4 and HHDFootnote 5 versions);
-
Non-sequential: Every AR balloon, i.e., component from the kit list is presented. The picking is performed according to the operators’ preference (watch sample videos for the HMDFootnote 6 and HHDFootnote 7 versions).
To validate the picking, when a component is collected, the corresponding balloon changes its background color from white to blue, indicating the last picked location. When a new component is picked, the last blue balloon is hidden.
This mechanism ensures that distracted operators can recall from which box the picking was being done or allow them to keep knowing the respective box, even if they put their hands over the box by fault.
At any time, the operator can access how many and which components compose the kit, as well as, which were already picked and which were not.
As for the development, and although the two methods are very similar, different technologies were considered in combination with the Unity game engine based on C# scripts. For the HMD, the Mixed Reality ToolkitFootnote 8 (MRTK) was considered with its local world anchors feature, allowing local persistence. Regarding the HHD, it uses ARCoreFootnote 9, taking advantage from its cloud anchors feature, which requires internet connection to store the real-world information in an external API.
While configuring the AR environment using HMD, the cubes are translated, rotated and scaled applying mid-air hand gestures, that can be used for close (by intersecting the operators hand with the 3D box) and far (by using the rays casted by the hands as pointers) interaction. The following gestures were consideredFootnote 10: air-tap, i.e., detects operators index finger to interact with existing buttons; one-handed mid-air pinch, i.e., grab and rotate 3D objects; and two-handed mid-air pinch, i.e., spreading both hands to change the scale of the object. As for the HHD, this can be performed applying the drag, turn to rotate and pinch/spread gestures in the device screen [14]. Furthermore, the HMD menu is accessed through the left hand facing upwards, while in the HHD is shown/hidden in the screen side panels (Fig. 6). Those menus also include a scene selector, allowing the configuration of multiple environments. This can be useful when considering multiple assembly lines, each one with its own set of components to be picked.
Regarding visualization, different validation mechanisms were implemented for the picking tasks. For the HMD, the validation is acknowledged when a human hand enters the virtual balloon area, i.e., when grabbing components from boxes. Differently, the HHD validation is not automated and requires operators to press the virtual balloon on the device screen, before or after picking the component.
3.4 First impressions on the shop floor
To understand the viability of using the initial version of the Pervasive AR tool, two visits to the shop floor were conducted. In total, more than 12 hours were spent in the industrial scenario.
These visits consisted in a preliminary user study. The goal was to evaluate the acceptance towards the use of the proposed technology in real-world context, as well as, gather first impressions regarding usability, interaction and tracking, allowing to establish the next steps of the authors’ research.
Participants were asked to conduct the picking process of a selected kit using the HMD and the HHD versions of the Pervasive AR tool (Fig. 8). To avoid learning effects, the order in which the tests were conducted was balanced.
Concerning the hardware used, the authors considered the Microsoft HoloLens 2 for the HMD method. In turn, the Asus Zenfone AR or the Samsung Galaxy A52 were attached to a smartphone hand grip for the HHD method, testing two devices with different sensors.
Participants were introduced to the goals of the study, as well as the experimental setup and design. After giving their informed consent, they were able to perform the tasks. Next, the picking task was completed, while being observed by a researcher who assisted them if necessary, and registered their actions and difficulties.
In addition, the route taken by three participants (1 experienced, 2 inexperienced) while performing the task was registered by hand over the architectural plan of the assembly line (see Fig. 4 — right). For these three participants, it was also requested to perform the picking task using the traditional paper manual.
In the end, subjective data was collected during informal interviews with the participants.
Participants’ opinion towards the different versions was registered by the researchers that accompanied the study. Plus, demographic data, and previous experience with AR was also considered.
The data collection process was conducted under the guidelines of the 1964 Helsinki Declaration.
Twenty-six (26) individuals (10 female, 16 male) from distinct departments of the factory participated in the study (e.g., operators, line managers, logistics, ergonomics, maintenance, process engineering, engineering manager, production manager).
From these, 40% of participants had previous experience with AR, and 25% were familiar with the picking task.
Overall, participants considered the Pervasive AR tool robust and accurate to be applied in a real industrial setup. It was also deemed easy to use, as well as having great potential to support operator’s tasks, specially for inexperienced operators.
Another advantage mentioned, is the fact that operators do not need to be familiar with the line layout, the components localization or even the labelling system, which can facilitate situations in which the layout is changed for efficiency purposes or to expand its production.
Additionally, operators do not need to memorize which components were already picked, thus decreasing the cognitive effort they endure daily.
As for the methods considered, an adaptation period was considered necessary for both variants, although, the interaction quickly becomes familiar to the users, stating that with time, task performance would easily improve.
Regarding the HMD method, using the Microsoft Holo-Lens 2, it was arguably the preferred option, with participants reporting that it was more stable when presenting the AR content in relation to the desired locations of the production line.
Furthermore, the hands-free approach was appraised as extremely useful to visualize additional information while conducting the picking tasks. In particular, when compared to the HHD alternative, which requires operators to hold such devices at all times, unless some sort of support is used, which was mentioned by various participants.
The need to test this method for a longer period of time was also mentioned, in order to verify if it is comfortable/ergonomic for long-term use, while also understanding how to handle the battery limitations. The possibility of using power-banks was also brought up.
Another relevant insight uncovered during the task execution of an operator, is associated with the fact that the operator decided to use a pair of gloves, in particular, black and shiny ones, which caused the HMD hand detection algorithm to struggle with recognizing the operators hands. This was very important, given that until that moment, such characteristics were not considered.
Concerning the HHD method, the Samsung Galaxy A52 showed a considerable decrease in its motion tracking technology at the industrial environment. In turn, the Asus Zenfone AR smartphone did not suffer from such a high performance impact when used in this scenario.
Yet, this method appears to reduce operator’s understanding of their surroundings when compared to the HMD alternative. It was also considered relevant having an automatic process that validates that the correct component was picked, instead of requiring human input at all times, which may become overwhelming after a certain period of time.
It was also stated that sequential picking would only be interesting for training new users and that non-sequential order is deemed the better option, given that it can provide a more detailed overview of the environment, while also giving more freedom to the operators.
As expected, participants suggested various improvements, deemed relevant to refine the AR tool and enhance operators performance in the future.
Most of these were considered and included in a revised version of the proposed tool. To elaborate, the following was considered: (1) visualize additional information of the component that needs to be picked besides its location (e.g., reference, denomination, quantity); (2) integrate different colors for the AR content, given that the colors initially selected would sometimes be confused with the environment; (3) display the percentage of components gathered during task resolution to improve awareness and motivation;
4) keep the augmented content in the display for a larger period of time after a given component is picked to avoid confusions and providing a clear overview of the task status;
5) include ergonomic warnings and notifications, e.g., suggest adjusting the HMD headset prior to its use to improve operators well being later on; 5) incorporate guidance on the next component when a sequential order is used, and when it is not in sight of the device field of view, thus avoid wasting time looking around. This was integrated through the use of directional arrows (Fig. 9). While in the HMD it appears on the field of view sides and just points top, down, left and right, in the HHD, it is fixed in front of the device camera viewpoint and rotates with 3-degrees of freedom.
Finally, Fig. 10 presents a set of Spaghetti diagrams with the paths travelled by the three participants, allowing their comparison while using the three methods considered (HMD, HHD and paper manual), and to verify which of them requires participants to walk less during the picking procedure [32]. It is observable that the routes walked by the inexperience participants (Fig. 10b and c) while using the AR applications are remarkably more efficient than using the traditional paper manual. On the other hand, the experienced participant (Fig. 10a) adopts a more efficient route using the paper manual, which is more familiar. However, the routes travelled using the HMD and the HHD were nearly as efficient. All in all, these results lead us to infer that Pervasive AR tool can contribute to less physical effort and higher efficiency when used by novice operators, which are learning or training the picking task.
4 Shop floor user study
In Section 3, the potential of employing Pervasive AR on a shop floor environment was assessed. The feedback obtained was used to improve the AR tool. This section reports a second user study, conducted to compare how different conditions affected operators performance during distinct picking tasks.
4.1 Experimental setup
Three distinct methods were considered: C3 — HHD using the smartphone Asus Zenfone AR attached to a hand grip.
The necessary AR content, corresponding to the components that needed to be picked for each kit was configured before the user study. Plus, during the study, a cart was used, so that participants could pick and place the components while moving through the environment.
4.2 Experimental design
The null hypothesis (H0) was considered, i.e., all experimental conditions are equally usable and acceptable to support the picking tasks.
The independent variables were the visualization method used during the picking tasks by the participants, with three levels corresponding to the experimental conditions: C1 — paper manual; C2 — HMD; C3 — HHD.
Plus, the methods for presenting the AR content: Sequential and Non-sequential. The dependent variables were participants’ opinion regarding the conditions used and the average time per component picking.
As secondary variables, participants’ demographic data, previous experience with AR and with the industrial task were considered.
A within-subjects experimental design was used, meaning that all participants used every condition.
4.3 Task
Regarding the tasks, three real-life industrial kits were considered, taking advantage of an ongoing research project with partners from the industry sector: small kit, frequent kit, uncommon kit.
The goal was to pick a list of components based on the kits selected.
In addition, a preparation kit was also considered, composed with three components not picked in the previous mentioned kits. This was used to introduce participants to the AR tool.
During the task execution, the components were placed in a cart, which could be carried around the line or not, leaving this decision up to the participants.
4.4 Measurements
Two types of data were collected. Task performance, which consists in the average time required to pick a component, recorded in seconds by a manual stopwatch (condition C1) or by the device (condition C2 and C3).
Participants’ opinion was gathered through a post-task questionnaire, including the following: demographic information (age, gender, factory department and role), experience with AR and on the assembly line; the post-task questionnaire collected the following dimensions: D1 — Level of confusion or distraction about the content used; D2 — Level of physical effort; D3 — Level of mental effort; D4 — Level of satisfaction.
Dimensions data was collected using a Likert-type scale: 1 — Low; 7 — High.
Furthermore, Participants’ difficulties, comments and observations, as well as their opinion on general questions were requested: Q1 — Condition order of preference; Q2 — Willingness to use AR for this task on a daily basis;
4.5 Procedure
Both methods share a similaParticipants were introduced to the goals of the study, as well as the experimental setup and design. After giving their informed consent, they were introduced to the tasks and the conditions considered.
Participants started by filling a demographic questionnaire.
Then, an adaptation time was provided, in which participants used the conditions to conduct the picking task with the preparation kit.
Next, the selected tasks were completed with all conditions, while observed by a researcher who assisted them if necessary, and registered any relevant event.
To minimize bias, i.e., learning effects, the order in which every condition was presented across the group of participants was counterbalanced.
At the end, participants answered a post-task questionnaire associated with their preferences towards the conditions used.
Last, a small interview was conducted.
The data collection process was conducted under the guidelines of the 1964 Helsinki Declaration.
Moreover, two groups were created, splitting participants in half. The first half performed the picking tasks using the sequential method.
The second group picked the required components in a non-sequential order of their choosing.
4.6 Participants
In total, 10 participants were recruited (1 female, 9 male), whose ages ranged from 22 to 39 years old (M=26.6, SD=4.9).
These had distinct professions at the factory, e.g., data engineer, automation engineer, industrial engineer, maintenance engineer, mechanical engineer, line manager, line worker (did not answer the post-experiment questionnaire) and manufacturing digitalization engineer. From these, 60% of participants had previous experience with AR and beyond the expert line worker and two participants with little experience, the participants had no experience with this production line.
5 Results and discussion
Next, the results from the user study are described and discussed. All participants were able to assemble the designated kits using the three conditions. On average, each session lasted for 40 min (about 30 min to complete the task).
Figure 11 and Table 3 present the average picking time per component in each condition considering the picking order: sequential and non-sequential. The chart illustrates that the time efficiency increases using condition C2 (HMD).
To elaborate, this condition was 70% faster than the traditional paper condition using the sequential picking order (11s and 36s, correspondingly) and 54% more efficient applying the non-sequential (13s and 28s, correspondingly).
Besides, using condition C3 (HHD) was more efficient compared with the paper manual in both methods (25s and 36s applying the sequential picking order and 17s and 28s using the non-sequential, respectively). However, not as much as condition C2 (HMD) (25s and 11s applying the sequential picking order and 17s and 13s using the non-sequential, respectively).
Hence, a decrease in time can be observed when using the non-sequential picking order compared to the sequential method with condition C3 (HHD) (25s and 17s, correspondingly) and condition C1 (paper) (36s and 28s, correspondingly). This does not apply to condition C2 (HMD) (11s and 13s, correspondingly) since the directional arrow feature was not available in this condition during the non-sequential method.
Looking to condition C3 (HHD) efficiency increase, the authors believe that condition C2 (HMD) would be even more time efficient applying the non-sequential method with this feature included.
Analyzing the dimensions addressed in the questionnaire, rated by the participants based on a Likert-type scale: 1 — Low; 7 — High (Fig. 12 and Table 4), the following was reported. Regarding the level of confusing and distraction about the presented information (D1), condition C2 (HMD) rated lower (median=2, sum=18), then condition C3 (HHD) (median=2, sum=26) and condition C1 (paper) (median=4.5, sum=43). As the data is on an ordinal scale and each user performed the three conditions (matched sample), the equality of medians was tested with the Friedman test (ANOVA nonparametric test), which rejected the null hypothesis (equality of distributions/medians, p-value=0.006). Furthermore, multiple paired comparisons, with Bonferroni correction, considered C3 not different from C1 (p-value=0.110) and not different from C2 (p-value=0.395), but established C2 as significant different (preferred) to C1 (p-value = 0.005). With respect to the level of physical effort (D2), condition C2 (HMD) was rated lower (median=2, sum=16), followed by condition C1 (paper) (median=3, sum=28) and condition C3 (HHD) (median=3, sum=35) respectively. The Friedman test rejected the null hypothesis (p-value=0.013), indicating differences among conditions. Multiple paired comparisons, with Bonferroni correction, considered C1 not different from C2 (p-value=0.270) and not different from C3 (p-value=0.270) but established C2 as significant different (preferred) to C3 (p-value=0.011). As for the level of mental effort (D3), condition C2 (HMD) was rated lower (median=2, sum=21), then condition C3 (HHD) (median=3, sum=30) and condition C1 (paper) being rated higher (median=4.5, sum=45). The Friedman test rejected the null hypothesis (p-value=0.019), indicating differences among conditions. Multiple paired comparisons, with Bonferroni correction, considered C3 not different from C2 (p-value=0.471) and not different from C1 (p-value=0.140) but established C2 as significant different (preferred) to C1 (p-value=0.011). Concerning the level of satisfaction (D4), condition C2 (HMD) was rated higher (median=6.5, sum=64), followed by condition C3 (HHD) (median=5, sum=50) and condition C1 (paper) (median=2.5, sum=28). The Friedman test rejected the null hypothesis (p-value<0.001), indicating differences among conditions. Multiple paired comparisons, with Bonferroni correction, considered C3 not different from C2 (p-value=0.050) and not different from C1 (p-value=0.219) but established C2 as significant different (preferred) to C1 (p-value=0.001). Condition C2 (HMD) was preferred in all dimensions to condition C1 (paper). On the other hand, there was no statistical difference between condition C3 (HHD) and condition C1 (paper), as well as, between condition C3 (HHD) and condition C1 (paper).
Overview of the results concerning the dimensions: D1 — level of confusing and distraction about the presented information; D2 — level of physical effort; D3 — level of mental effort; D4 — level of satisfaction. Conditions: C1 — Paper; C2 — HMD; C3 — HHD. Data collected using a Likert-type scale: 1 — Low; 7 — High
When questioned about their preferences (Q1), condition C2 (HMD) (median=7, sum=66) was selected as the preferred alternative, followed by condition C3 (HHD) (median=5, sum=48) and lastly, condition C1 (paper) (median=2.5, sum=24).
The Friedman test rejected the null hypothesis (p-value<0.000). Furthermore, multiple paired comparisons, with Bonferroni correction, considered C1 not different from C3 (p-value=0.219) but established C2 as significant different (preferred) to C1 (p-value=0.000) and to C3 (p-value=0.024). In general, participants would be willing to use the AR tool (Q2) (median = 6, sum=56). However, the usage time ranged from “while training” and “for a short time, one or two hours, as it would be uncomfortable for a longer time” to “the entire shift.” Most participants disliked the smartphone holder (Q3) while using condition C3 (HHD) because it occupies one of the hands. Despite requiring additional testing, other holders were suggested: “smartphone holder fixed to the cart,” “wristband smartphone holder” or “smartphone holder attached to a vest.”
Both AR conditions were seen as identically helpful in learning (C2-HMD: median=7, sum=66; C3-HHD: median=6, sum=62) assessed with the Wilcoxon matched pairs nonparametric test (p-value=0.178) and in completing the task (C2-HMD: median=7, sum=63; C3-HHD: median=6.5, sum=63) assessed with the Wilcoxon matched pairs nonparametric test (p-value=1.000), result of the analysis of the users’ answers to question Q4.
Participants preference regarding condition C2, as suggested by their comments, is associated with the fact that, when using the HMD, it was easier to maintain a good visual-motor coordination (P4 — “As the smartphone field of view is smaller, sometimes it was harder to find the virtual information.”). Having a hands-free setting facilitates accomplishing the components picking tasks (P4 — “Holding the smartphone with one hand and perform the operations with the other hand is not practical.”). Also, they stated that the stability of the virtual content in this condition was better, which is beneficial for situation understanding and awareness (P6 — “With the smartphone I was afraid to lose the tracking of the virtual world, what did not happen when using the HMD.”).
Also important, both conditions using AR have been considered to become more intuitive and easy to use over time, which in turn may be reflected in lower cognitive effort (P2 — “[The AR tool is] very intuitive, easy to interact and facilitates the component picking.”).
A Cluster Analysis of the answers of the four dimensions in the three conditions revealed: (i) high satisfaction with C2 (HMD) and C3 (HHD) and low satisfaction with C1 (paper); (ii) low perceived level of confusion, mental and physical effort concerning C2 (HMD) and high level of confusion about C1 (paper); (iii) moderate and equivalent level of mental and physical effort regarding C1 (paper) and C3 (HHD).
Additionally, the use of Correspondence Analysis allowed to note some user profiles, namely: (i) some resistance to the change (from paper to AR) of older participants, with more daily experience of using paper in the task and who considered moderate the mental effort of using AR; (ii) a straightforward acceptance of AR, by users of manufacturing digitalization and industrial engineering, who penalize the use of paper in the dimension of confusion, highlight the very low physical and mental effort of AR and express, also, a high level of satisfaction for this new paradigm; (iii) the automation, mechanical and maintenance engineering users’ attention to ergonomics issues, namely the physical effort in C3 condition (HHD), although they also express a high satisfaction for using AR.
6 Conclusions and future work
Nowadays, the value Augmented Reality (AR) can generate for several industrial applications such as assembly, maintenance, quality control, logistics and others is undeniable. Regardless, research is still needed for AR solutions to be fully integrated in the daily activities of human operators.
In this paper, the authors proposed a Pervasive AR tool for supporting logistics operators during order picking tasks on industrial shop floors. A Human-Centered Design (HCD) methodology was used with partners from the industry sector to identify operators’ difficulties, challenges, and define requirements. Following, two methods were developed (Head-Mounted Display (HMD) and Handheld Device (HHD)), allowing to configure and visualize AR content in the industrial environment. All in all, virtual content can be visualized, allowing operators to know how to conduct picking operations, i.e., having step-by-step instructions, including text, images and 3D models on how to assemble a given kit.
The proposed methods were evaluated during picking tasks on the shop floor in two phases: first, to have an understanding of first impressions from twenty-six participants, including individuals with different occupation and expertise. This feedback was used to improve the prototypes before the second study, in which, ten participants that had never conducted (except one expertise operator and two participants with little experience) the selected tasks used three different conditions: C1 — paper; C2 — HMD; C3 — HHD. The goal was to verify which conditions were more adequate and could contribute with higher productivity for the task.
Results emphasize the potential of Pervasive AR for operators picking activities on the shop floor environment, in particular for training operators not familiar with the tasks and to help in finishing the tasks. Although both conditions seem as valid options to support operators, condition C2 was preferred by all participants, being considered more useful and efficient on the shop floor scenarios, mostly due to its hand-free setting, as well as higher time efficiency, less cognitive and physical effort and providing a higher level of satisfaction. On the other hand, although condition C3 was faster than C1, it was not considered by the participants as presenting significant advantages in the other dimensions compared to C1.
The study also revealed that careful attention is required when designing the experimental protocol to ensure that the assembly line is not disrupted during labor. Additionally, it is important to ensure the safety of participants when using the developed tool, as their level of awareness may be affected, especially if they are not familiar with the devices used. The access to a considerable number of participants on the shop floor during the conducted evaluations was difficult to achieve and time consuming. Besides, some workers were hesitant to cooperate as they seemed concerned that the technology could replace their jobs in the future, or that could be used to monitor their performance.
As next steps, the authors intend to study upgraded validation mechanisms, as well as, to conduct longitudinal studies on the shop floor environment. Although some insights were already obtained, longer studies and more complex tasks are an opportunity to better comprehend how AR solutions may affect operators health, motivation, and productivity on a daily basis.
Furthermore, explore co-located collaborative use-cases, requiring various operators to work together on shared goals, while taking advantage of the benefits of Pervasive AR in Industrial scenarios.
Data Availability
The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request
Notes
reports.valuates.com/market-reports/360I-Auto-4S58/industry-4-0 (Accessed: September 2022)
https://www.youtube.com/watch?v=9M7SEWPmno0 (Accessed: September 2022)
https://youtu.be/cLSlQ86ekmo (Accessed: September 2022)
https://www.youtube.com/watch?v=gInR-8IJ8p0 (Accessed: September 2022)
https://youtu.be/TvYGmNoi4eo (Accessed: September 2022)
https://www.youtube.com/watch?v=XJQFl8o28hc (Accessed: September 2022)
https://youtu.be/mINhAxZrJ9M (Accessed: September 2022)
docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/?view=mrtkunity-2022-05 (Accessed: September 2022)
developers.google.com/ar (Accessed: September 2022)
docs.microsoft.com/en-us/dynamics365/mixed-reality/guides/authoring-gestures-hl2 (Accessed: September 2022)
References
Aivaliotis S, Lotsaris K, Gkournelos C, Fourtakas N, Koukas S, Kousi N, Makris S (2022) An augmented reality software suite enabling seamless human robot interaction. Int J Comput Integ Manufac: 1–27
Al-Maeeni SSH, Kuhnhen C, Engel B, Schiller M (2020) Smart retrofitting of machine tools in the context of industry 4.0. Procedia CIRP 88:369–374
Alves J, Marques B, Ferreira C, Dias P, Santos BS (2021) Comparing augmented reality visualization methods for assembly procedures. Virtual Reality 26(4):1–14
Alves JB, Marques B, Dias P, Santos BS (2021) Using augmented reality for industrial quality assurance: a shop floor user study. Int J Adv Manufact Technol 115(1):105–116
Aromaa S, Väätänen A, Aaltonen I, Goriachev V, Helin K, Karjalainen J (2020) Awareness of the real-world environment when using augmented reality head-mounted display. Appl Ergonomics 88
Battini D, Calzavara M, Persona A, Sgarbossa F (2015) A comparative analysis of different paperless picking systems. Indust Manag Data Syst 115(3):483–503
Belen RAJ, Nguyen H, Filonik D, Favero DD, Bednarz T (2019) A systematic review of the current state of collaborative Mixed Reality technologies: 20132018. AIMS Electr Elect Eng 3:181
Besbes B, Collette SN, Tamaazousti M, Bourgeois S, Gay-Bellile V (2012) An interactive augmented reality system: a prototype for industrial maintenance training applications. In: IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp 269–270
Blattgerste J, Renner P, Strenge B, Pfeiffer T (2018) In-situ instructions exceed side-by-side instructions in augmented reality assisted assembly. In: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, p 133140
Bottani E, Vignali G (2019) Augmented reality technology in the manufacturing industry: a review of the last decade. IISE Trans 51(3):284–310
Büttner S, Prilla M, Röcker C (2020) Augmented Reality Training for Industrial Assembly Work - Are Projection-Based AR Assistive Systems an Appropriate Tool for Assembly Training?, p 112
Chin KY, Wang CS (2021) Effects of augmented reality technology in a mobile touring system on university students learning performance and interest. Australasian J Edu Technol Assoc Comput Mach 37(1):2742
Dimitropoulos N, Togias T, Zacharaki N, Michalos G, Makris S (2021) Seamless human–robot collaborative assembly using artificial intelligence and wearable devices. Appl Sci 11(12)
Dong Z, Zhang J, Bai X, Clark A, Lindeman RW, He W, Piumsomboon T (2022) Touch-move-release: studies of surface and motion gestures for mobile augmented reality. Frontiers Virtual Reality:90
Egger J, Masood T (2020) Augmented reality in support of intelligent manufacturing a systematic literature review. Comput Indust Eng 140:106195
Fraga-Lamas P, FernáNdez-CaramáS TM, Blanco-Novoa Ó, Vilar-Montesinos MA (2018) A review on industrial augmented reality systems for the industry 4.0 shipyard. IEEE Access 6:13358-13375
Funk M, Bächler A, Bächler L, Kosch T, Heidenreich T, Schmidt A (2017) Working with augmented reality? a long-term analysis of in-situ instructions at the assembly workplace. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, p 222229
Gavish N, Gutiérrez T, Webel S, Rodríguez J, Peveri M, Bockholt U, Tecchia F (2015) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23(6):778–798
Gkournelos C, Karagiannis P, Kousi N, Michalos G, Koukas S, Makris S (2018) Application of wearable devices for supporting operators in human-robot cooperative assembly tasks. Procedia CIRP 76:177–182
Gomes P, Magaia N, Neves N (2020) Industrial and artificial internet of things with augmented reality. Converg Artif Intell Int Things:323–346
Grubert J, Zollmann S (2017) Towards pervasive augmented reality: context awareness in augmented reality. IEEE Trans Visual Comput Graph 23(6):1706–1724
Guo A, Wu X, Shen Z, Starner T, Baumann H, Gilliland S (2015) Order picking with head-up displays. Computer 48(6):16–24
Hanson R, Falkenström W, Miettinen M (2017) Augmented reality as a means of conveying picking information in kit preparation for mixed-model assembly. Comput Industr Eng 113:570–575
Henderson S, Feiner S (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Visual Comput Graph 17(10):1355–1368
Holz T, Campbell AG, O’Hare GM, Stafford JW, Martin A, Dragone M (2011) Miramixed reality agents. Int J Human-Comput Studies 69(4):251–268
Hou W (2019) Augmented reality museum visiting application based on the microsoft HoloLens. J Phys: Conf Series 1237(5):052018
Jwo JS, Lin CS, Lee CH (2021) Smart technology-driven aspects for human-in-the-loop smart manufacturing. Int J Adv Manufact Technol 114(5):1741–1752
Kim K, Billinghurst M, Bruder G, Duh HB (2018) Revisiting trends in augmented reality research: a review of the 2nd decade of ISMAR (2008–2017). IEEE Trans Visual Comput Graph, TVCG 24(11):2947–2962
Klinker K, Berkemeier L, Zobel B, Wüller H, Huck-Fries V, Wiesche M, Remmers H, Thomas O, Krcmar H (2018) Structure for innovations: a use case taxonomy for smart glasses in service processes. In: Multikonferenz Wirtschaftsinformatik Lüneburg, Deutschland
Kota L (2018) Artificial intelligence in logistics: applications and algorithms. Adv Logistic Syst-Theory Pract 12(1):47–60
Li X, Yi W, Chi HL, Wang X, Chan AP (2018) A critical review of virtual and augmented reality (vr/ar) applications in construction safety. Automation Construct 86:150–162
Lopes R, Teixeira L, Ferreira C (2019) Lean thinking across the company: successful cases in the manufacturing industry. Lean manufacturing: Implementation, opportunities and challenges:1–31
Maio R, Alves J, Marques B, Dias P, Lau N (2022) Exploring an augmented reality serious game for motorized wheelchair control. In: Proceedings of the 2022 international conference on advanced visual interfaces
Makris S, Karagiannis P, Koukas S, Matthaiakis AS (2016) Augmented reality system for operator support in human-robot collaborative assembly. CIRP Annal 65(1):61–64
Marino E, Barbieri L, Colacino B, Fleri AK, Bruno F (2021) An augmented reality inspection tool to support workers in industry 4.0 environments. Comput Indust 127:103412
Marques B, Carvalho R, Alves J, Dias P, Santos BS (2019) Pervasive augmented reality for indoor uninterrupted experiences: a user study. In: UbiComp/ISWC’19, p 141144
Marques B, Alves J, Neves M, Maio R, Justo I, Santos A, Rainho R, Ferreira C, Dias P, Santos BS (2020) Interaction with virtual content using augmented reality: a user study in assembly procedures. Proc ACM Human-Comput Int 4(196):1–17
Marques B, Silva SS, Alves J, Araujo T, Dias P, Santos BS (2021) A conceptual model and taxonomy for collaborative augmented reality. IEEE Trans Visual Comput Graph:1–18
Marques B, Silva S, Alves J, Rocha A, Dias P, Santos BS (2022) Remote collaboration in maintenance contexts using augmented reality: insights from a participatory process. Int J Int Design Manufact, IJIDeM:1–20
Martins NC, Marques B, Alves J, Araújo T, Dias P, Santos BS (2021) Augmented reality situated visualization in decision-making. Multimed Tools Appl 81:1–24
Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst 77(12):1321–1329
Mourtzis D, Zogopoulos V, Katagis I, Lagios P (2018) Augmented reality based visualization of cam instructions towards industry 4.0 paradigm: a cnc bending machine case study. Procedia CIRP 70:368–373
Mourtzis D, Zogopoulos V, Xanthi F (2019) Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling. The Int J Adv Manufact Technol 105(9):3899–3910
Murauer N, Müller F, Günther S, Schön D, Pflanz N, Funk M (2018) An analysis of language impact on augmented reality order picking training. In: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, New York, PETRA ’18
Nebeling M, Speicher M, Wang X, Rajaram S, Hall BD, Xie Z, Raistrick AR, Aebersold M, Happ EG, Wang J, et al. (2020) Mrat: the mixed reality analytics toolkit. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp 1–12
Neumann WP, Winkelhaus S, Grosse EH, Glock CH (2021) Industry 4.0 and the human factor a systems framework and analysis methodology for successful development. Int J Prod Econ 233:107992
Neves M, Marques B, Madeira T, Dias P, Santos BS (2022) Using 3d reconstruction to create pervasive augmented reality experiences: a comparison. In: 2022 IEEE Conference on virtual reality and 3D user interfaces abstracts and workshops (VRW), pp 712–713
Okano M (2017) Iot and industry 4.0: The industrial new revolution. In: CMIS-17 - international conference on management and information systems, vol 17
Paelke V (2014) Augmented reality in the smart factory: Supporting workers in an industry 4.0. environment. In: Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA), pp 1–4
Pierdicca R, Frontoni E, Pollini R, Trani M, Verdini L (2017) The use of augmented reality glasses for the application in industry 4.0. In: De Paolis LT, Bourdot P, Mongelli A (eds) Augmented reality, virtual reality, and computer graphics. Springer International Publishing, Cham, pp 389–401
Pierdicca R, Frontoni E, Pollini R, Trani M, Verdini L (2017b) The use of augmented reality glasses for the application in industry 4.0. In: De Paolis LT, Bourdot P, Mongelli A (eds) Augmented reality, virtual reality, and computer graphics, pp 389–401
Piumsomboon T, Dey A, Ens B, Lee G, Billinghurst M (2019) The effects of sharing awareness cues in collaborative mixed reality. Front Robot AI 6:5
Plakas G, Ponis S, Agalianos K, Aretoulaki E, Gayialis S (2020) Augmented reality in manufacturing and logistics: Lessons learnt from a real-life industrial application. Procedia Manufact 51:1629–1635
Quandt M, Knoke B, Gorldt C, Freitag M, Thoben KD (2018) General requirements for industrial augmented reality applications. Procedia CIRP 72:1130–1135
Quint F, Sebastian K, Gorecky D (2015) A mixed-reality learning environment. Procedia Comput Sci 75:43–48
Rani NA, Aziz FA, Hairuddin AA, Ahmad SA, Hemdi AR (2020) Augmented reality: capabilities and challenges in machining industry aligned with industry 4.0. Adv Mater Process Technol 0(0):1–9
Rashid Z, Meliá-SeguíJ Pous R, Peig E (2017) Using augmented reality and internet of things to improve accessibility of people with motor disabilities in the context of smart cities. Future Gen Comput Syst 76:248–261
Reif R, Günthner WA (2009) Pick-by-vision: augmented reality supported order picking. Visual Comput 25(5):461–467
Rokhsaritalemi S, Sadeghi-Niaraki A, Choi SM (2020) A review on mixed reality: current trends, challenges and prospects. Applied Sciences 10(2):636
Saeedi S, Bodin B, Wagstaff H, Nisbet A, Nardi L, Mawer J, Melot N, Palomar O, Vespa E, Spink T, Gorgovan C, Webb A, Clarkson J, Tomusk E, Debrunner T, Kaszyk K, Gonzalez-De-Aledo P, Rodchenko A, Riley G, Kotselidis C, Franke B, O’Boyle MF, Davison AJ, Kelly PHJ, Lujàn M, Furber S (2018) Navigating the landscape for real-time localization and mapping for robotics and virtual and augmented reality. Proc IEEE 106(11):2020–2039
Sahu CK, Young C, Rai R (2021) Artificial intelligence (ai) in augmented reality (ar)-assisted manufacturing applications: a review. Int J Product Res 59(16):4903–4959
Santi GM, Ceruti A, Liverani A, Osti F (2021) Augmented reality in industry 4.0 and future innovation programs. Technologies 9(2)
Santos LMALD, da Costa MB, Kothe JV, Benitez GB, Schaefer JL, Baierle IC, Nara EOB (2021) Industry 4.0 collaborative networks for industrial performance. J Manufact Technol Manag 32(2):245–265
Scargill T, Chen Y, Eom S, Dunn J, Gorlatova M (2022) Environmental, user, and social context-aware augmented reality for supporting personal development and change. 2022 IEEE conference on virtual reality and 3d user interfaces abstracts and workshops (VRW). IEEE, Christchurch, New Zealand, pp 155–162
Segovia D, Mendoza M, Mendoza E, González E (2015) Augmented reality as a tool for production and quality monitoring. Procedia Comput Sci 75:291–300
Sereno M, Wang X, Besançon L, Mcguffin MJ, Isenberg T (2020) Collaborative work in augmented reality: a survey. IEEE Trans Visual Comput Graph pp 1–20
Sony M, Naik S (2020) Industry 4.0 integration with socio-technical systems theory: a systematic review and proposed theoretical model. Technol Soc 61:101248
Speicher M, Hall BD, Nebeling M (2019) What is mixed reality? In: Proceedings of the 2019 CHI conference on human factors in computing systems, association for computing machinery, New York, CHI ’19, p 115
Tang Y, Chau KY, Li W, Wan T (2020) Forecasting economic recession through share price in the logistics industry with artificial intelligence (ai). Computation 8(3)
Tang YM, Ho GTS, Lau YY, Tsui SY (2022) Integrated smart warehouse and manufacturing management with demand forecasting in small-scale cyclical industries. Machines 10(6)
Wang W, Wang F, Song W, Su S (2020) Application of augmented reality (ar) technologies in inhouse logistics. E3S Web Conf 145:02018
Acknowledgements
To everyone involved in the user studies, and discussion groups, thanks for your time and expertise. The context of the user study was conducted under the Augmented Humanity project, thanks to a co-promotion between Bosch Termotecnologia S.A. and the University of Aveiro.
Funding
Open access funding provided by FCT | FCCN (b-on). This research was developed in the scope of the Augmented Humanity project (POCI-01-0247-FEDER-046103 and LISBOA-01-0247-FEDER-046103), financed by ERDF through POCI. It was also supported by IEETA - Institute of Electronics and Informatics Engineering of Aveiro, funded by National Funds through FCT, in the context of the project (UIDB/00127/2020).
Author information
Authors and Affiliations
Contributions
Each author has substantially contributed to conducting the underlying research and drafting of the manuscript.
Corresponding author
Ethics declarations
Ethical approval
The authors confirm that this manuscript is original and has not been published, nor is it currently under consideration for publication elsewhere. All procedures performed were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Consent to participate
Informed consent was obtained from all participants included in the study.
Consent for publication
Not applicable
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Maio, R., Santos, A., Marques, B. et al. Pervasive Augmented Reality to support logistics operators in industrial scenarios: a shop floor user study on kit assembly. Int J Adv Manuf Technol 127, 1631–1649 (2023). https://doi.org/10.1007/s00170-023-11289-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00170-023-11289-1