1 Motivation and preparatory work

At present, manufacturing companies are confronted with a multitude of challenges [1]. Increasing product variety and smaller batch sizes have to be managed, while quality, time and costs have to be competitive in international comparison. Particularly in assembly, the need for greater flexibility increases the pressure on employees to remain productive and competitive. This applies especially to high-wage countries. Overall this leads to a more complex work environment for the employees. More variants and components lead to more decisions to be made by the executing employee. For this reason, better support with information to assist decision making is needed [2]. In the context of these developments, it is urgently necessary to offer the affected employees better access to information, e.g. through digital assistance systems.

Companies have high expectations regarding the introduction of such systems, but are hindered by the high implementation expenses in terms of time, costs and organizational or process adjustments [3]. Furthermore, there is great uncertainty regarding the benefits of single assistance technologies [3].

Thus, research on the effect of digital assistance has so far concentrated mainly on the comparison of different communication media and thus addresses the question of which hardware implementation of digital assistance is most promising. Funk et al. [4] and Blattgerste et al. [5], for example, show that digital assistance does not have a beneficial effect on assembly time, but primarily has a positive effect on quality. In contrast, Uva et al. [6], Hou et al. [7] and Aehnelt [8] show that digital assistance has a positive effect on both assembly time and assembly quality compared to paper-based work instructions.

The question of the influence of qualification and experience of employees on the effect of digital assistance is addressed by Jeske et al. [9] and Watson et al. [10], who examine in particular training processes and the appropriate media. On the one hand, they conclude that informatically rich and less abstracted assembly information is well suited for training processes. On the other hand, the learning curves determined show that these effects converge with increasing practice. Experienced and qualified assembly employees do not benefit as much from digital assistance as less qualified employees.

Although it is possible to deduce influences on the effect of digital assistance from these studies, two further aspects need to be considered with regard to the state of research: On the one hand, the studies mentioned above consider simple digital implementations for the transmission of assembly information. More extensive forms of digital assistance, which are, for example, already implemented in learning factories are not used (see e.g. [11,12,13]). However, in particular less qualified assembly workers could benefit from these. On the other hand, the studies do not vary the work task. This is remarkable, since the complexity of work task is mentioned in the practice-oriented literature as the main motivation for the use of digital assistance (see [14, 15]).

This results in two major fields of action. On the one hand, by quantifying the effects of digital assistance systems, their acceptance is to be promoted and in a further step, a frame of reference is required that serves companies as a support to introduce the appropriate digital assistance systems, depending on their individual situation and challenges. In order to start closing this research gap, Keller, Bayer, Bausch and Metternich [16] have developed an investigation model based on a systematic literature research, which describes the effects of digital assistance on production processes. In addition, various test scenarios were defined and a procedure was developed to verify the investigation model. This preparatory work serves as a basis for the following elaborations and will be presented in more detail in chapter 2. On the other hand, it should be noted that with the qualification of employees and the complexity of the work task, two major influencing variables on the effect of digital assistance have so far been analyzed little or insufficiently. This is the starting point of this article, which presents an empirical study that aims to quantify these effects.

In this paper the study carried out and the corresponding results for the benefit evaluation of digital assistance are presented. This study will test the hypothesis that complexity and competence have an influence on the positive effect of digital assistance. As a result, we expect to find measurable effects of digital assistance, which differ depending on the test scenario. The purpose is to identify the scenario with the greatest effect. This scenario will be selected to investigate the functional relationships between digital assistance and the observed effects in higher detail.

In this article:

  • an existing research model is adapted so that selected key performance indicators (KPIs) can be measured and the formulated hypotheses can be tested in the laboratory study (Sect. 2)

  • an experimental setup is presented (Sect. 3)

  • the design of the study is described (Sect. 4)

  • the study results are presented and evaluated (Sect. 5)

2 Investigation model and adaptations

To carry out the study, the above-mentioned investigation model and the test scenarios had to be adapted and suitable KPIs had to be derived. The modified model for the study is shown in Fig. 1. The model illustrates the hypothetical interdependencies (red arrows) between the functions of the assistance system, the effects on the employee and the KPIs, which can be aggregated to labor productivity as an overall measure.

Fig. 1
figure 1

Investigation model adapted to the study

Full size image

2.1 Modifications of the model

In a previous publication, eight functions of digital assistance were defined [16]. To validate the fundamental suitability of our approach we use the following five functions of digital assistance in this study: Documentation, Control, Context Identification, Initiation of the Process and Information Output.

Three further functions were not taken into account. The Decision Support, based on data analytics, as a function of an assistance system was not implemented, because at the time the study was conducted, there was not enough data available. Since the focus of the study is on the effect of digital assistance on the individual workplace, the function of Communication was not considered. Contrary to the focus of the study, this function describes the possibility of communicating with knowledge carriers or people in charge and thus is primarily focusing on network effects. Due to the study design, as it is described in Sect. 4, the function Learn and Practice was removed from the model. The design developed for the study does not provide for repetitions of the assembly process, but these are necessary to generate learning curves. Consequently, the effects on the employee and KPIs of the Production Process associated with the three removed functions have also been eliminated.

2.2 Labor productivity as KPI to measure the benefit

In order to measure the effect of digital assistance, a target variable is necessary. This variable must meet two requirements: It must be possible to causally attribute a change in this target variable to the use of digital assistance systems. In addition, it must be possible to measure or determine the components of the variable with reasonable effort for conducting the study and especially for a practicable application in a production process of a company. Furthermore, labor productivity is a common and widely used metric to measure the performance of work systems in production.

For manual assembly as a highly human-centred process, the definition of labor productivity as the ratio of output to labor input [17] is suitable. As a commonly used key figure in industrial production it enables the comparison of introducing digital assistance systems with other improvement measures in production. The operationalization of labor productivity (hourly output) is based on the Nakajima report on Overall Equipment Effectiveness (OEE) [18]. The factors availability, performance and quality are included in the calculation with one single term each, as it can be seen in Eq. 1. Since the assembly is carried out without repetitions, only time variables were used for calculation:

$$ LP = \frac{{t - t_{d} - t_{s} }}{t} \times \frac{{t_{a} }}{{t - t_{d} - t_{s} }} \times \frac{{t_{w} }}{{t_{w} + t_{r} }}. $$
(1)

In Eq. 1\( t \) represents the total time spend on the assembly line, \( t_{d} \) describes the downtime \( t_{s} \) the setup time, \( t_{w} \) the time used for the work task, \( t_{r} \) the time for reworking and \( t_{a} \) is the target time for the assembly process of the product. This target time was determined for both products using the Methods-Time Measurement (MTM) method, which is a predetermined motion time system and does not contain any time for the handling of information, which means that the determined target time describes the absolute minimum time required for the work task [19]. The first term represents the ratio of available time to total time, the second the ratio of target time to required time (performance) and the third the proportion of working time in which work was carried out without errors (quality).

2.3 Questionnaire meCUE as KPI to measure the user experience

The success of digital assistance systems depends not only on economic considerations but also on the assessment of the employees who work directly with these systems. Because assembly is strongly human-centered, it is important that employees can and want to work effectively with digital assistance systems. For this reason, the experience of use is also recorded in addition to work productivity. This can be measured using the questionnaire meCUE [20]. This questionnaire comprises a total of 33 items in the categories task-related product perception, non-task related product perception, user emotions and consequences of use, which are recorded using a seven-level Likert scale.

2.4 Categorisation of the test scenarios

The categorisation of the test scenarios with regard to employee competence and process complexity is implemented as follows.

To address competency it was distinguished between trained and untrained test persons. In the study carried out, process-experienced test persons (“trained”) received an introduction to the corresponding product before assembly, in which the product variant was assembled and disassembled twice under instruction. This is considered sufficient to gain adequate process knowledge. Untrained test persons have no knowledge of the product before assembly.

Process complexity was addressed by the use of two products, which are available in the process learning factory CiP of the TU Darmstadt: A pneumatic cylinder from Festo and a gear motor from SEW Eurodrive (see Fig. 2). These products, which serve as demonstrators, were used to conduct the study. After a detailed methods-time measurement (MTM), the assembly of the gear motor (Target time for sum of activities = 5.7 min.) can be considered much more complex than the pneumatic cylinder (2.1 min.). This result corresponds to the classification into a low and a medium complex product according to the categorization of a previous publication of the authors [16].

Fig. 2
figure 2

Test scenarios for the conducted study

Full size image

3 Experimental setup in the process learning factory CiP

Both products are assembled on the assembly line shown in Fig. 3 in the process learning factory CiP. This manual assembly line consists of three workstations which are passed through by the test person during the assembly of a product in sequence in a one-piece flow. The assembly line is equipped with pick by light technology, touch screen monitors, controlled assembly tools and other sensor technology. These components are interconnected via a process control system and can be controlled by this system during the assembly process. This setup allows the realization of all five functions mentioned in Sect. 2.1 (see Fig. 1). The functions are partly realized by a combination of individual elements of the technical setup. For example, the control is carried out by data acquisition via the sensor system and a comparison with the target value in the process control system.

Fig. 3
figure 3

Assembly line in the process learning factory CiP

Full size image

4 Study design

The aim of the study is to investigate the effect of digital assistance depending on process complexity and employee competence. For this purpose, we must first test whether digital assistance has a significant positive effect on productivity:

H1: It is assumed that the productivity of the assembly process is greater with digital assistance (\( \mu_{a} \)) than without (\( \mu_{w} \)). The hypotheses can be formulated as follows:

$$ H_{0} : {\varvec{\Delta}} = \mu_{a} - \mu_{w} \le 0\;{\text{and}}\;H_{1} : {\varvec{\Delta}} = \mu_{a} - \mu_{w} > 0. $$
(2)

Once the hypotheses formulated in Eq. 2 have been tested for each scenario, the respective effect size of digital assistance is compared between the scenarios via the Cohens d. On this basis the following assumptions can be verified:

  • H1.1: The effect (size) of digital assistance on labor productivity is larger for the gear motor than for the pneumatic cylinder.

  • H1.2: The effect (size) of digital assistance on labor productivity is larger for untrained test persons than for trained test persons.

It was also tested whether the digital assistance has a significant positive effect on the user experience:

H2: It is assumed that the experience of use is greater with digital assistance (\( \mu_{a} \)) than without (\( \mu_{w} \)). The hypothesis can be formulated accordingly to Eq. 2. Cohens d is used to compare effect sizes between the scenarios for experience of use:

  • H2.1: The effect (size) of digital assistance on experience of use is larger for the gear motor than for the pneumatic cylinder.

  • H2.2: The effect (size) of digital assistance on experience of use is larger for untrained test persons than for trained test persons.

The within-subject design was chosen for the study, which means that all participants were exposed to several conditions. Each test person assembles both products once with and once without digital assistance. Thereby, each participant is trained for one product and untrained for the other. The assembly sequence was systematically varied to counterbalance order effects. One procedure contains three essential components:

  1. 1.

    Set up of the assembly line;

  2. 2.

    Assembly of the product;

  3. 3.

    If a product is assembled incorrectly, the error must be reworked afterwards.

When considering one test scenario separately, the mean values of labor productivity with and without digital assistance are compared by means of a paired t test. The t-test is used to test the statistical significance of a difference in means. The comparison is one-sided.

The comparison between the test scenarios is based on the Cohens d as a measure of the effect size and can be performed based on the t-tests. Compared to statistical significance, the consideration of effect size offers two advantages:

  1. 1.

    The informational value is greater, because in addition to the significance, the practical relevance (relative size of the effect) is also evaluated.

  2. 2.

    The comparability of results of different studies (in our case test scenarios) is simplified.

Cohen [21] provides a frame of reference for the interpretation. Thus d = 0.20 corresponds to a small effect size, d = 0.50 to a medium effect size and d = 0.80 to a large effect size [21].

To determine the required number of participants, the expected effect size must be estimated. A very strong effect is anticipated when the complex product is assembled by an untrained test person. Since the effect is likely to be weaker for the remaining three test scenarios, a medium to large effect is assumed. This leads to the expected effect size of \( \dot{d}_{3} = 0.70 \). Following the comments of Cohen [9] the required number of test persons (n) can be calculated:

$$ n = \frac{{n_{10} }}{{100 \times \left( {\dot{d}_{3} \times \sqrt 2 } \right)^{2} }} + 0.7. $$
(3)

Choosing a conventional significance level of α = 0.05 and a statistical power of β = 0.80, the table value \( n_{10} \) contained in Eq. 3 can be determined and it follows that 30 test persons are required in total. Looking at it from the other perspective, significant results can be expected if the study is performed with at least 30 test persons.

5 Study results and evaluation

Thirty subjects (25 male/5 female) with an average age of 22.9 years participated in the study. Twenty-seven of them were students, most of them in technical subjects. None of them had any previous experience with the assembly line, but some had experience with the assembly of the pneumatic cylinder. These were placed in the group of subjects trained for the cylinder.

Figure 4 shows the mean difference in labor productivity Δ, its improvement in relation to the averaged productivity without digital assistance, the Cohens d (if the effect is statistically significant), the t-value and the p-value for each test scenario. The stars represent the statistical significance with zero stars corresponding to p > 0.05, one star corresponding to 0.01 < p≤0.05 and two stars corresponding to p ≤ 0.01.

Fig. 4
figure 4

Statistical Results of the analysis of labor productivity and user experience

Full size image

Regarding the hypotheses raised in Sect. 4, the analysis of labor productivity reveals two main points:

  • The effect size of digital assistance is greater for the gear motor than for the pneumatic cylinder

  • The effect size of digital assistance is greater for untrained employees than for trained employees

Analysing the setup time separately, it is observed that digital assistance has a significantly negative effect on setup time for the gear motor (t(29) = − 2963, p = 0.003) and for the pneumatic cylinder (t(27) = − 1880, p = 0.036). The reason for this effect may be the simplicity of the setup process in terms of time and content. The digital assistance seems to unnecessarily bind the attention of the operator. Due to this effect, the lead time not including the setup process is examined and the following is observed:

  • The relation between the test scenarios is identical to the previous analysis of labor productivity

  • By removing setup time from the analysis, all observations are reinforced in the dimensions of significance and effect size

Figure 4 shows the mean difference of the meCUE results Δ, its improvement in relation to the averaged experience of use without digital assistance, Cohens d, the t-value and the p-value for each test scenario. The stars represent the statistical significance as mentioned previously. Regarding the hypotheses raised in Sect. 4, the results of experience are consistent with those for work productivity:

  • The effect size of digital assistance is greater for the gear motor than for the pneumatic cylinder

  • The effect size of digital assistance is greater for untrained employees than for trained employees

Thus, the hypotheses raised in Sect. 4 can be clearly confirmed. More generally, this study indicates that the effect of digital assistance increases with greater process complexity and decreases with higher employee competence.

From the study results the following transferable new knowledge can be derived: As shown the effect of digital assistance increases for more complex tasks. Taking the setup process into account, the rising benefit of digital assistance could be observed over three levels of complexity. In the same way, the increase of the positive effect with decreasing employee competence could be verified. These findings assist companies in identifying meaningful areas of application for digital assistance systems. The focus should be on rather complex assembly processes and less competent employees (e.g. working students). This shows, for example, that digital assistance is predestined to support training processes for workplaces with complex assembly tasks, especially if the employees have little knowledge of the (or a comparable) assembly process. Furthermore it could be observed that the experience of use was in every scenario better with digital assistance than without. This leads to the expectation that the acceptance for a (well implemented) digital assistance exists and even increases with higher usage (high complexity, low competence).

6 Summary and outlook

The aim of this study was to investigate the influence of process complexity and employee competence on the effect of digital assistance. Therefore, it was set up an assembly line with several functions of digital assistance in the process learning factory CiP. First, the investigated target value of labor productivity and experience of use and various test scenarios were defined. In order to operationalize the categorization of test scenarios, practice-oriented criteria were introduced. The central result of the investigation is that the effect of digital assistance increases for the more complex product and untrained test persons, supporting [9] and [10]. If absolute values are considered, digital assistance always has a positive effect for the complex product. For simple products (cylinder) and simple activities (set up), it is shown that digital assistance has no significant or a negative effect.

Based on these results, the model shown in Fig. 1 will now be analysed in further studies. For the identified test scenario with the greatest effect size of digital assistance, the individual functions of the assistance system are isolated and systematically varied. In addition, the influence of the individual functions on the subordinate categories of Labor Productivity (Availability, Performance and Quality) is differentiated in following studies. Using multiple regression, the influences of the individual functions are then quantified to create a reference framework for the introduction of digital assistance, individually adapted to existing processes, as mentioned in the introduction.