1 Introduction

The fourth industrial revolution, often referred to as Industry 4.0, is already on its way enabled by advanced digitalization, industrial internet and smart technologies such as Internet of Things (IoT). For manufacturing industry, this revolution is expected to provide shorter development periods, individualization in demand for the customers, flexibility, decentralization and resource efficiency [1, 2]. Industry 4.0 will also change factory floor work, making it significantly more demanding in terms of managing complexity, abstraction and problem solving [3, 4]. Employees are likely to act much more on their own initiative and organize their personal workflow. Flexible work organization models meet the growing need of employees to strike a better balance between work and private life as well as between personal development and continuing professional development [3].

A central part in Industry 4.0 is the human-centricity of the Factories of the Future, described as development towards Operator 4.0 [5]. Operator 4.0 refers to smart and skilled operators of the future, who are assisted by automated systems that provide a sustainable relief of physical and mental stress and let the operators utilize and develop their creative, innovative and improvisational skills, without compromising production objectives [5].

Operator 4.0 factory work will be qualitatively enriched and flexible and will require new qualifications to master the digital technology invading to factories. Future factories should support current workers in learning new skills while tempting new workers who are already familiar with digital solutions. The advanced work environment and work tools are expected to enhance work well-being. Well-being is crucial for keeping qualified and experienced personnel in the company and, additionally, work well-being positively correlates with productivity [6]. Thus, work well-being serves both the employee and the employer.

New Operator 4.0 work tools and related new work practices have the potential to enhance productivity and work well-being but these positive impacts do not materialize automatically. How should the targeted impacts then guide the design process? It requires, at least, that the objective of the design is set accordingly and kept in focus throughout the design process. In the Factory2FitFootnote 1 project, we aim to design Operator 4.0 solutions that directly support the workers themselves and enhance their possibilities to influence their work. The challenge in designing such solutions is to find ways to guide the design so that the main aim of worker wellbeing is affecting constantly and efficiently the design. This cannot be obtained by just emphasizing wellbeing in design as wellbeing is the result, not the means. In the design process, we need to focus on immediate implications of the design decisions from all relevant perspectives such as usability, user experience and safety. The design decisions then have wider impacts on worker well-being, and those impacts should be foreseen in the design. Thus, we need a design and evaluation approach that will focus both on immediate implications of the design and wider impacts on work wellbeing. To face this challenge, we have created a framework to support the design, evaluation and impact assessment activities throughout the project.

In the following, in Sect. 2, we first give an overview of related research. Then in Sect. 3, we introduce the basic structure of our design and evaluation framework. In Sects. 4 and 5, we describe the work well-being approach as well as the design and evaluation approach that we have chosen to the framework. Finally, in Sect. 6, we outline how we plan to use the framework in design, evaluation and impact assessment activities and present examples of using the framework. At the end, we present conclusions and our future work plans.

2 Related Research

In the literature, various frameworks for design and evaluation of work systems have been proposed. Activity theory can be used as a framework for analyzing and redesigning work [7]. Activity theory analyses work, roughly, from the perspectives of instruments, subject, rules, community, division of labor and object of work, and can be used for developing work in a systemic level [7]. A framework for design and evaluation of complex interventions to improve health was created by Campbell et al. [8]. Their framework is focused on addressing the additional problems resulting from the evaluation of complex interventions. The framework used in the study of Sluga et al. [9] provides a conceptual framework for collaborative product design and the related operations in manufacturing work systems. They created a conceptual model for collaboration and an ICT platform, which supports collaboration over the web. None of these models address the worker’s satisfaction or well-being that are of main interest in our project.

Danna and Griffin [10] present an organizing framework that highlights the major elements of health and well-being in the workplace. They propose that the antecedents personality traits, occupational stress and the work setting influence work well-being and the worker’s health in the workplace. Furthermore, the well-being at work determines individual (e.g., psychological consequences) as well as organizational consequences (e.g., productivity). There are about 1,300 articles citing the one of Danna and Griffin, but hardly any of them really use their framework. Some studies test or refer to some relationships that are part of the framework [11,12,13,14,15,16,17,18]. Examples are briefly outlined below.

When assessing the impact of healthy work organization intervention in a retail setting, DeJoy et al. [11] used the conception presented in the study of Danna and Griffin [10]. So an expanded view of organizational effectiveness was assumed, including both business performance and employee health and well-being. Wood and de Menezes [12] state, referring to Danna and Griffin [10], that the policies that increase employee well-being are essential, because, among other things, stress at work extends to general health [12]. Koopmann et al. [13] examined predictors of daily regulatory focus at work and the foci’s impact on employee well-being at work and home, conceptualizing well-being as mood and psychosomatic health complaints, in accordance with the prior work of Danna and Griffin [10]. In the review of current state of research in employee well-being, Ilies et al. [14] refer to Danna and Griffin [10] in stating that happy workers are less often late and show up for more days of work. Nielsen et al. [15] reported having used the definition of well-being of Danna and Griffin [10], so that well-being is the state of employees’ mental, physical, and general health, including also their experiences of satisfaction both at work and outside of it.

Only a few studies, found by the authors, have utilized the framework as such or with minor modifications, like in the present study. Dežmar-Krainz [16] has analyzed and assessed in a theoretical level how socially responsible orientation incorporated in human resource management can contribute to well-being of employees. In line with Danna and Griffin [10], Dežmar-Krainz acknowledges that home and workplace well-being often overlap. Furthermore, the original model [10] was only slightly changed by reformulating the concept of occupational stress (preferring the concept of psychosocial stress).

Nykänen et al. [17] combined the model of Danna and Griffin [10] and the framework of Newsham et al. [18] with the goal to develop a new user-friendly design model for intensive and intermediate care facilities in hospital context. The model by Newsham et al. [18] links the physical environment, through environmental satisfaction, to job satisfaction.

In the framework by Nykänen et al. [17], multitasking, job stress and burnout as well as indoor environmental satisfaction are regarded as mediating factors between antecedents (physical conditions and personality factors) and consequences, in this context well-being indicators (job satisfaction and work engagement). The usage of physical factors is a natural choice when indoor design is focused on, and multitasking reflects an elementary part of nurse’s work. Nykänen et al. [17] were using their framework for evaluation purposes.

Although other frameworks exist besides the one of Danna and Griffin [10], none of them covered the concepts satisfaction and well-being in a similar way. Their framework included the design aspect and the characteristics of work-related aspects and company benefits as needed within our research. Furthermore, various authors have referred to Danna and Griffin’s framework [10]. Thus, we decided to utilize the model of Danna and Griffin [10] as the fundamental basis and we decided to relate also to the model of Nykänen et al. [17]. The framework used in this study is described in the next section.

3 The Design and Evaluation Framework

When designing Operator 4.0 solutions, individual worker’s point of view should be central in the design and evaluation activities. New work tools and related work practices should be designed so that they result in meaningful, motivating and engaging work tasks. Accordingly, the design outcomes should have a positive impact on work well-being and company benefits.

Our aim was to define a framework, which supports the (i) design and evaluation of the new Operator 4.0 tools and practices and the (ii) assessment of their impact on work well-being and company benefits. The design and evaluation framework that we suggest is presented in Fig. 1. The framework utilizes the framework of Danna and Griffin [10] by modifying its content to meet the design and evaluation needs of our project. In that sense, our study is similar with the studies of Dežmar-Krainz [16] and Nykänen et al. [17]. Conceptually, the framework by Nykänen et al. [17] is quite close to our approach. Similar to those approaches, in our framework, work well-being is affected by both non-work and job-related satisfaction. The novelty value of our approach is to use the same, theoretical framework for design, evaluation and impact assessment purposes. This provides a firm basis for practical decision making in designing the solutions to be developed in the project. Danna and Griffin [10] as well as Nykänen et al. [17] differentiate antecedents and consequences from the immediate implications of the intervention, i.e. the introduced new work tool or practice. In our framework, we use the term impacts instead of consequences to stress that design decisions might have impacts on work well-being and company benefits.

Fig. 1.
figure 1

The design and evaluation framework for Operator 4.0 solutions that support work well-being.

Full size image

Antecedents include the work environment, work organization and worker characteristics. They describe the original context of use [19] where the intervention - in the form of new tools and new work practices - is implemented.

Immediate implications cover workers’ experience with their work and work tools. Desired immediate implications can directly guide the design process, and be in the focus of evaluation activities. Well-designed tools that are acceptable and safe to use, that provide high usability as well as positive user experience and that are ethically sound, will most likely have an positive impact on work well-being. We chose these perspectives to study immediate implications.

Impacts include work well-being and company benefits. For investigating the impacts of new Operator 4.0 tools on work well-being, we focus on the positive indicators of well-being: job satisfaction, work engagement and job motivation. Company benefits, based on the new work tool and work practice solutions, origin from the efficiency of the new solutions on work output directly and from the increased well-being at work. Expected company benefits include optimized processes, productivity, and quality. In addition, the developed solutions are expected to affect the company image so that the company becomes as a more desired place to work. Thus, the company benefits are the ones related to the core task or function of the company, often related to economic matters. Company benefits are also the impacts that the companies themselves find interesting and important.

As a whole, the new tools are evaluated during the design phase with a specific set of perspectives, focusing on the usage-centric qualities of the tool (located in the center of Fig. 1, Operator 4.0 Design & Evaluation approach). After the piloting of the new tools is performed, the users (workers in the factory in question) are asked for feedback from the same perspectives as used during the design, as well as from the well-being approach (Work well-being, in the upper right corner of Fig. 1). After the piloting it is also possible to evaluate or foresee the company-level benefits (the lower right corner of Fig. 1).

In this paper, we focus on the framework from two perspectives: (i) the design and evaluation approach and (ii) the approach to assess the impacts of the new tools and practices on work well-being. Together these constitute the part of the framework that supports designing Operator 4.0 solutions that might enhance work well-being. In the following, we will describe in detail these parts of the work well-being framework.

4 Work Well-Being

Work well-being is a concept affected by various phenomena. Thus, it can be a result of a combination of, for example, wellbeing due to an experience of meaningful life during leisure time, as well as finding the objective of work motivating (related to job motivation) and get satisfaction of performing work tasks (related to job satisfaction). The relative importance of each factor is individual so a straightforward formula cannot be created - for instance, for some, interesting work can balance unsatisfactory leisure time whereas others feel better at work only when happy at home. The concept of work well-being is, then, just the well-being as it realizes itself at work, irrespective of its reasons.

Job satisfaction and work engagement are considered to represent major components of work-related well-being [20]. Van Beek, Hu, Schaufeli, Taris, and Schreurs [21] have shown the relevance of motivation for explaining differences between work engagement and negative dimensions of well-being. Thus, job motivation represents an important mediating factor for well-being that also needs to be considered. Our work well-being approach includes these three factors: job satisfaction, work engagement and job motivation, which are described in more details in the following sub sections.

4.1 Job Satisfaction

Work is an important part of life in the western society and job satisfaction a vital factor reflecting work related attitudes and feelings. Job satisfaction, or the lack of it, is shown, for instance, in the willingness to go to work and the pride, or lack of it, when talking about it. Two common definitions of job satisfaction describe job satisfaction as “a pleasurable or positive emotional state resulting from the appraisal of one’s job or job experiences” [22] p. 1300 and “the extent to which people like (satisfaction) or dislike (dissatisfaction) their jobs” [23] p. 2. Based on these definitions, job satisfaction reflects an overall attitude towards work in general, or in relation to different aspects of work, such as, colleagues, salary, the nature of work itself, supervision, or working conditions [23] p. 2. The extent to which work properties meet or exceed the personal expectations of employees determines the level of job satisfaction [22]. Satisfaction with the nature of work, including job challenge, autonomy, variety and scope, predicts best the overall job satisfaction [24].

Because work tools and related new work practices may have strong impact on job satisfaction, the implementation of new solutions for operator 4.0 should consider the individual expectations of the workers and thereby fostering the satisfaction at work. In addition, job satisfaction as a part of our framework, is an important premise for increasing job performance and is closely connected to other concepts such as work engagement.

4.2 Work Engagement

Work engagement is something that the employer does not require, as long as work is well done and employees stay at the company: For the worker him/herself, the level of engagement to work is important, indicating how interesting (s)he truly finds the work.

Maslach and Leiter [25] argue that work engagement refers to energy, involvement and professional efficacy of a worker. Schaufeli et al. [26] p. 74 define work engagement as “a positive, fulfilling, work-related state of mind, which is characterized by vigor, dedication and absorption”. Schaufeli et al. [26] p. 74 characterize an engaged worker as feeling energetic while working, being willing to invest effort in one’s work and persistent when facing difficulties. An engaged worker is also dedicated in one’s work and such feelings as the feeling a sense of significance, enthusiasm, inspiration and pride are included in engagement. According to JD-R (job demands-resources) model of work engagement [27], both job resources and personal resources predict work engagement, having a positive impact on it especially when the job demands are high.

Work engagement has a positive impact on performance at work. When the workers are engaged and perform well, they are able to create and strengthen their resources, which creates a positive spiral. The measurement of work engagement is important to learn as it provides a rather stable reference to other work well-being related phenomena (job satisfaction and job motivation). The enhancement of work engagement, based on the introduction of the new tools, would be a strong signal about the positive impact the tools bring along.

4.3 Job Motivation

Job motivation is rather superficial as regards the relationship with work, compared to the more closely work-related concepts of job satisfaction and work engagement. A person may be motivated to work without finding pleasure in the work tasks, for instance, as long as the status of the work role is good enough or because of the social environment at work.

Porter and Lawler [28] divide job motivation into intrinsic and extrinsic motivation. If workers fulfil their tasks because they derive spontaneous satisfaction from performing the activity itself and are interested in their work, they are intrinsically motivated. Extrinsic motivation, in contrast, describes that workers’ satisfaction comes not from their tasks, but rather from the consequences that go along with the fulfilment of the task, such as verbal or financial rewards. In Herzberg’s motivation-hygiene theory [29], motivators involve factors determined by the job itself such as achievement and recognition. Besides that, hygiene factors are extrinsic to the job, such as interpersonal relations and salary. Hygiene factors prevent negative feelings and dissatisfaction; the existence of motivators determines satisfaction. According to Hackman and Oldham [30], the degree of motivation depends on whether (1) the work is experienced as personally meaningful, valuable and worthwhile, (2) employees perceive themselves as personally accountable and responsible for the work outcomes, and (3) employees are aware how well they perform in their work. High work engagement goes along with high intrinsic and extrinsic motivation [31]. Nie and colleagues [31] argued that intrinsic motivation has a positive influence on well-being and is positively correlated with job satisfaction whereas extrinsic motivation might have positive as well as negative effects.

In summary, extrinsic as well as intrinsic factors are important for fostering motivation at work. Therefore, job motivation within our design and evaluation framework refers to both intrinsic and extrinsic motivational aspects. Negative consequences of new operator 4.0 solutions should be avoided and individual preconditions should be established for positive effects of the work utilizing the new tools.

5 Approaches for Design and Evaluation

As pointed out earlier, we chose five complementary design and evaluation perspectives regarding the immediate implications:

  1. 1.

    Usability was included in the framework as it is traditionally a central viewpoint in human-centered design [19]. Usability studies focus on the appropriateness of the tools for work tasks. Systems usability [32] has extended the view to complex industrial systems where both tools and work practices are studied and developed, thus contributing to smooth work processes.

  2. 2.

    User experience (UX) has also become an essential viewpoint in human-centered design. User experience is widely used as a design driver in the design of consumer products but in industrial contexts, user experience driven design is still rare. UX extends the focus from mere usage of the tools to how the users feel while working, and wider how the new tools shape their image as professionals [33]. User experience driven design has potential especially in designing radically novel concepts.

  3. 3.

    User acceptance studies are widely used to support the uptake of new technical solutions at work places [34]. User acceptance studies aim to explain the reasons for people’s attitudes towards work systems and tools as well as further adoption of the systems. The studies connect usage behavior to both user characteristics and attributes of the work tools.

  4. 4.

    Safety assessment is essential in industrial systems to prevent risking workers’ health. Safety is typically addressed with expert assessment of safety risks and proposing mitigation measures. Safety should be assessed also from the worker point of view and then trust is an important concept.

  5. 5.

    Ethical assessment is important as the adoption of new technologies may raise concerns. For instance, Industry 4.0 solutions are often based on gathering data of the work environment and even the workers, and this may be ethically sensitive.

In the following, we will describe these five perspectives and analyze in more details what contribution each perspective brings to the framework.

5.1 Usability

Usability refers to “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” [19]. The satisfaction with a new tool forms a precondition for its positive impact on work satisfaction.

In the context of complex socio-technical systems, a more holistic system-oriented approach to usability is called for. In the Systems Usability approach [32], understanding the appropriateness of the tools in industrial production processes begins by investigating the purpose and meaning of the tools in human use. In activity theory, one main principle is that of mediation. According to Vygotsky [35], tools and instruments mediate the relationship between the subject and the object of work (what is acted on). Based on the activity theory [7], the tool’s mediating role can be elaborated by making distinction between the different functions of a tool: the instrumental, psychological and communicative functions [32]. The three functions can be scrutinized from three different types of activity, i.e., performance, way of acting and UX (see Fig. 2).

Fig. 2.
figure 2

Systems usability framework [32].

Full size image

In the framework, systems usability brings in the viewpoint of the work practices: new work tools and new work practices need to be developed in parallel. Thus, design and evaluation should not only focus on the interaction with the new tools but actually working with the tools as well as adopting and developing new ways to work.

5.2 User Experience

Good user experience (UX) is the goal of most products and services intended for the consumer market. UX is also receiving increasing attention in the development of industrial systems. According to Hassenzahl [36], UX consists of both pragmatic and hedonic aspects of product use. UX at work is the way a person feels about using a product, service, or system in a work context, and how this shapes the image of oneself as a professional [33]. This definition connects UX to well-being, especially to work engagement.

Kaasinen et al. [37] propose UX goals to guide the design of industrial systems. UX goal setting integrates the viewpoints of different stakeholders, thus committing them to the defined UX goals and emphasizing user experience as a strategic design decision. In our design and evaluation framework, UX goals are defined as suggested by Kaasinen et al. [37], and used as suggested by Roto et al. [38]. High-level UX goals help to create and maintain UX mindset within the design team, thus keeping the focus on how users would feel at work while using the new tools. In the evaluation activities, UX goals can be utilized to study if the users actually have the intended experiences such as feeling of control. UX can also be evaluated more openly, aiming to identify widely the actual experiences.

5.3 User Acceptance

Technology acceptance models aim at studying how individual perceptions affect the intentions to use technology as well as the actual usage. Acceptance models mainly focus on explaining adoption behavior and often include behavioral intention. In the initial Technology Acceptance Model (TAM) [39], the aim is to explain the determinants of user acceptance of a wide range of end-user computing technologies. TAM points out that perceived ease of use and perceived usefulness affect the intention to use, which determines actual usage. TAM was designed to study information systems at work to predict whether the users will actually take a certain system into use in their jobs. The model provides a tool to study the impact of external variables on internal beliefs, attitudes and intentions.

Van der Laan, Heino and De Waard [40] define attitudinal acceptance as “predispositions to respond, or tendencies in terms of “approach/avoidance” or “favorable/unfavorable” (p. 2) and describe two dimensions of “attitudinal” acceptance of technological innovations: satisfaction and usefulness.

Technology acceptance models provide solid frameworks to identify issues that may affect user acceptance of technical solutions in work. The original TAM has been extended to provide further aspects about the drivers and obstacles for adoption of technology. Especially for new information and communication technologies, the UTAUT2 model [34] that includes TAM as well as other theoretical frameworks (e.g., Theory of Planned Behaviour [41]), are used to assess drivers and barriers for acceptance of new technical solutions. In detail, performance expectancy, effort expectancy, social influence, facilitating conditions, hedonic motivation, and habit are relevant factors influencing workers’ intention to use the technology or solution as well as actual usage.

In the design and evaluation framework, user acceptance perspective supports designing tools that the users want to use, rather than tools that they have to use. User acceptance studies are especially important during the early phases of the design to identify factors that affect the acceptance. High acceptance of Operator 4.0 solutions is a precondition for positive impacts on work well-being.

5.4 Safety

Safety is related to physical and mental well-being and thus has a clear role in the framework. Safety design starts with generic safety requirements, which cover many kinds of machines and their typical risks. Safety requirements are an agreement of acceptable risk levels available in the form of various safety standards (e.g., [42]). Generic standards do not cover all safety aspects of all kinds of machinery. This is especially true for new developments. Therefore, risk assessment, which covers also new and unusual risks, needs to be performed by safety experts. In addition, the industrial user organization should perform risk assessment focusing on potential accidents. Risks are described with the severity and the probability of each risk [42]. Risk assessment methods focus on system operations and functions as well as human operation and tasks [43]. The possibility to avoid a hazard is often depending on the skills and awareness of the worker.

Safety risks may lead to hazards and may threat worker’s health. The feeling of safety influences the worker’s well-being. In the framework, we focus on the worker’s point of view. We will study perceived safety, as it indicates the worker’s own feeling of the safety of the work and the work environment.

5.5 Ethics

The adoption of new technologies can reveal ethical challenges. Especially when the new technology is used for measuring person related attributes, such as health in work context in our study, ethical matters become vital, from the sake of ethicality itself and also as it affects user acceptance. This raises the need to include ethics in the framework. Niemelä, Kaasinen and Ikonen [44] propose the Ethics by Design approach based on positive, forward-looking and proactive ethical thinking. Ethical points of view are considered in the early project phases, with the aim of creating a positive, ethical-solution-oriented mindset among project partners. The ethical approach should not just identify current or future problems but also actively design for and be inspired by achieving ethically sustainable solutions [45].

In the design and evaluation framework, we have adopted the Ethics by Design approach [44]. The focus is on creating ethically sound solutions rather than just identifying potential problems. Thus, we focus on ethical guidelines that guide the design and form a basis for evaluation activities.

5.6 Summary

The aim of the design and evaluation framework is to guide design and evaluation activities so that they support creating solutions that have positive impact on work well-being. The approaches of systems usability, user experience (UX), user acceptance, safety and ethics support this goal. Systems usability brings in the viewpoint of focusing not only on the interaction with work tools but also on the new work practices that are developed in parallel to the new work tools. User experience points out focusing on how people feel at work when using the new tools, and how this shapes their image of themselves as professionals. User acceptance focuses on users’ willingness to adopt new tools and practices, and the underlying factors affecting adoption behavior. Safety assessment and ethical assessment are typically performed by experts. The framework emphasizes that they both also have the worker point of view that should be included. Do the users consider the solutions safe; can they trust the solutions? Do the workers judge the solutions as ethically sound? Is there something that they have doubts about?

Even if the five perspectives all bring in a specific, important viewpoint, they also include similar elements. Systems usability includes UX as one element but it does not include the idea of concrete UX goals. Ease of use element of user acceptance is very similar to performance outcome in systems usability. Feeling of safety, trust and ethical concerns are closely related to UX and user acceptance. These similarities should not be a problem in using the framework, as the aim is to integrate the perspectives so at all important aspects will be studied.

6 Utilizing the Framework

The aim of the design and evaluation framework is to support design and evaluation activities throughout the project. Now that we have defined the framework, our aim is to continue utilizing it in practice, analyze the experiences of use, and update the framework accordingly.

In the design activities, systems usability puts the focus of the design to new work practices and how the new tools could best support smooth ways to work. UX covers the design aspect of how people feel at work. Both systems usability and UX support early design phases where different stakeholders of the project build a common vision of the solutions to be defined. Drivers and obstacles for user acceptance should be identified early in the design to find solutions that will be accepted. Safety and ethics viewpoints emphasize the consideration of those aspects early in the design, and from the worker’s point of view. In this way, they bring in a positive, forward-looking viewpoint that helps finding safe and ethically sustainable solutions.

The evaluation perspectives included in the framework are all using different methods. Integrating these methods is challenging but necessary to avoid having too many parallel evaluations that would be difficult to organize in practice. Furthermore, parallel evaluations might result in contradicting results, the implications of which would be difficult to interpret. The chosen perspectives should be integrated in evaluation so that we address all the necessary perspectives. Depending on the evaluation activity, the focus may be on higher-level issues or on details. Using similar questionnaires in all evaluation activities allows following the progress of the development work. These questionnaires can be extended by further items and complemented with interviews for individual evaluations to focus on selected issues. The interviews can be evaluation specific even if the evaluation themes follow the perspectives of the framework.

When Operator 4.0 tools are developed, at least as prototypes, they can be tested in real-life working contexts, where a more realistic impact assessment can be gained. In the impact assessment, measuring worker’s well-being before and after the introduction of new tools and practices allows for direct comparisons. This gives us the opportunity to draw conclusions about the impact of the new work tools and practices on well-being. Impact assessment can be continued later on in actual use. We emphasize that impact assessment can also be part of the early phases of a project. Foreseen impacts, assessed by the workers themselves, other company representatives and external experts give valuable insight to the potential of the new solutions already during the design phase. From the participating companies’ point of view, early feedback on foreseen impacts has been seen especially valuable, as it helps assessing which solutions are most promising regarding business benefits.

When performing evaluation, validated and widely used measures are beneficial as they are solid means producing clear results, easy to compare with the results of other studies using the same measures. However, the set of various perspectives is rather large in our project. Using validated questionnaires would mean hundreds of questions for each worker. Such a load cannot be set to the users. As we feel it to be important to include all the perspectives of the framework, a compact set of questions, with just a few questions for each perspective is needed. Thus, in practice, the questionnaires used and to be used in this project are tailored for this purpose. Of course, validated tests can still be used in some individual evaluations if needed for some specific goal. Furthermore, the questionnaires are complemented with solution specific questions. Using the common questionnaires allows comparing different solutions, while solution specific questions and complementary results of user interviews give detailed user feedback.

In the following, we describe our first experiences of using the framework in design and evaluation activities in the Factory2Fit project.

6.1 Overall Concept Design

The Factory2Fit project started with a series of co-design activities that involved all the project partners. First, factory workers of the project’s pilot factories were interviewed and their work was observed. Then, a series of workshops were conducted in which the results of the factory worker studies were analyzed and discussed among project partners. Industrial partners, representing the pilot companies, presented their expectations related to the new tools and the foreseen impacts. In the workshops, a common vision for the project was gradually set, and the vision was complemented with descriptions of new concepts for work tools and related work practices. The common vision was based on how the future work should feel and based on the vision, two high-level UX goals were defined in the workshops: empowerment and engagement. UX experts analyzed these high-level goals and based on the results of the user studies and the design workshops, they proposed more detailed experience goals (Table 1). The experience goals will guide the design of the Operator 4.0 solutions, as they will be analyzed to identify design implications for each individual concept and tool.

Table 1. User experience goals in Factory2Fit project.
Full size table

To support ethics by design, we defined ethical guidelines (Table 2) based on the ethical guidelines for ambient intelligence by Ikonen, Kaasinen and Niemelä [46]. We chose five out of the six ethical themes (we left out “benefit for society” as it was felt too abstract in our case) and for each of the five ethical themes, we modified the original guideline so that it reflected what the guideline would concretely mean in Operator 4.0 solutions in factory environments. In the design phase, the guidelines form a checklist, with which the design team can assess proposed solutions.

Table 2. Ethical guidelines in Factory2Fit project.
Full size table

Safety issues are typically studied by expert assessments, and many standards give instructions on safety issues. While these safety activities are important, in the framework, we wanted to focus on the workers’ point of view. Based on safety standards [42], we defined safety requirements to guide design (Table 3). In the design process, the proposed solutions and implementations have been checked against these requirements.

Table 3. Safety requirements in Factory2Fit project.
Full size table

By now, systems usability and user acceptance have been utilized in design activities indirectly: they have been included as the evaluation viewpoints in evaluation activities that support human-centered iterative design. In the next sub section, we describe our experiences of the evaluation activities.

6.2 Using the Framework in Individual Design and Evaluation Activities

In the following, we describe three case studies of utilizing the design and evaluation framework in the design and evaluation activities in the Factory2Fit project. We describe how we used the framework and we discuss related benefits and challenges.

Case 1: Evaluation of Four Factory2Fit Concepts

The design and evaluation framework was utilized in the evaluation of four concepts that were developed in the project. The concepts aimed to engage workers for participatory design of factory operations, knowledge sharing and training. The concepts were utilizing augmented and virtual reality (AR/VR) and context-aware social media. Factory workers evaluated the concepts in two workshops. The concepts were presented one by one to the participants as demonstrators and/or videos. After each presentation, the participants filled in a questionnaire that guided them to consider the chosen evaluation themes (user acceptance, UX and safety). In the questionnaire, participants also evaluated foreseen impacts on job satisfaction. After the participants had filled in the questionnaires, we had a follow-up group interview. In the interview, they were asked about the benefits, the challenges and development ideas of the presented concept. This revealed attitudes and expectations to explain the results gathered with the questionnaires. After the concepts were evaluated in this way, the group discussed about the presented entity of concepts and their potential to support their work.

Besides workers’ experience and acceptance of the concepts as well as safety issues and expected impacts on job satisfaction (i.e., aspects of the framework), concepts’ main benefits, challenges and development ideas were formulated by the factory workers. They could point at diverse issues related to usability, safety or ethical issues. In general, workers’ feedback of the concepts was positive.

Our aim is to develop the framework while using it. That is why we analyzed the benefits and challenges of the framework after the case study. As a benefit, the framework guides the data collection in evaluations and makes data comparable with other research partners. This is important in a large research project with several evaluation activities carried out by different partners in different countries. The framework also supports keeping in mind the main aim of the project - to influence work well-being. Even if direct impacts cannot be measured until the solutions are in actual use, focusing on foreseen impacts in the evaluations helps developing the solutions to support this aim. During the design process, we have several different evaluation activities. The framework needs to be utilized adaptively, so that the focus is chosen according to the current design phase. The perspectives of the evaluation should then be chosen accordingly. However, in the user interviews, we should be open for all the perspectives. For instance in this study we did not focus on usability as we were dealing with early concepts. Still the users were commenting about the expected usability challenges with the forthcoming solutions. We also left ethics out, as we considered that the concepts did not include ethically sensitive issues. However, in the interviews the users pointed out foreseen ethical problems. Based on these results, we decided to include in the forthcoming user studies all the perspectives, even if with a smaller set of questions.

Case 2: Evaluation of a Social Media Platform

The design and evaluation framework was applied in the evaluation of a social media platform. The social media platform integrates a social media messaging system with a production environment so that the discussions can be connected to physical elements of the production line or to the status of the line such as a certain error situation. The purpose of this evaluation was to get feedback to the design from workers and other stakeholders. Nine participants (factory workers and other factory stakeholders) participated in the evaluation. The prototype system was first introduced to them and then they were divided into two groups, where they could try out the system guided by the facilitators. When trying out the system, the participants were asked about problems, improvement needs as well as pros and cons related to the system. In the end, the participants filled in a questionnaire. The questionnaire included UX, user acceptance and usability perspectives from the framework. Impacts were assessed with two questions in the questionnaire regarding anticipated changes in knowledge sharing practices. Results show that the participants liked the system and agreed that the usability was on a good level. They suggested many important improvements for the system.

As such, this study could be regarded as a basic usability research. However, in the workshop also impacts and development ideas were discussed. For example, the integration of workers discussions with production data could improve performance and job satisfaction. In addition, the possibility to share knowledge could engage workers more with their work and work community. The gamification feature could motivate some people to share knowledge more actively.

As a benefit, by using the design and evaluation framework it was possible to enhance traditional usability studies towards considering impacts of a system on work well-being and on productivity. The results pointed on the high value of qualitative data gathered via interviews or group discussions as many possible impacts were raised in the discussions but in the questionnaire, we were only asking about impacts on knowledge sharing. Similar to Case 1, this case points on the importance of qualitative methods for complementing questionnaires for assessing the various aspects of the framework in the future.

Case 3: Design and Evaluation of Worker Feedback Dashboard

The design and evaluation framework was utilized in the participatory design of Worker Feedback Dashboard, a web-based solution giving personal data-based feedback to machine operators at a factory (the design process is described in [47]). The idea of the solution is that it gives personal feedback both on selected well-being metrics through a wearable self-tracking device and on relevant production metrics through the machinery of the factory, enabling the workers to see potential connections between these and thus empowering operators to make behavioral changes.

All aspects of design and evaluation framework were addressed during the development process. Before the actual participatory design, the usability and user experience of several wearable self-tracking devices were evaluated in the expert evaluation by the researchers and the potential ethical issues were identified in an expert workshop (the study is described in details in [48]). Before designing the first prototype, also factory workers were involved to give their initial feedback on the acceptability of the concept idea. We also visited two factories (pilot sites of the project) to gain understanding on the work context, which helped us in considering the safety aspects and the potential user acceptance. Based on user understanding, we first designed the user experience goals highlighting for example the role of positive feedback (empowering user experience) and the importance of not disturbing the user’s work tasks (ensuring safety). The key design implications were derived from these goals.

When initially evaluating the first prototype, we showed it to factory workers, who could comment it freely, and express their potential concerns and further ideas. With a short questionnaire, we assessed usability related issues (e.g. whether it is easy to understand the data shown), user acceptance related questions (e.g. whether the solution seems to provide value to users through interesting or useful content), and ethical issues related to the solution (whether the solution seems questionable). Based on the user feedback, refinements to the design were made.

The design and evaluation framework will be further utilized when piloting the refined functional prototype as a part of actual factory work. We are planning to conduct a 2-month usage pilot, which gives us understanding of the real user acceptance and user experience, as well as usability issues during long-term use. Based on the long-term use, we can evaluate whether using the solution may involve safety risks and can see what kind of ethical issues are brought up by the users. The design and evaluation framework will be utilized when defining the questionnaires and interview questions for the pilot.

7 Conclusions and Future Work

We have presented a framework to guide design, evaluation and impact assessment activities so that they support creating Operator 4.0 solutions that have a positive impact on work well-being. We have started to utilize the framework in our research project and we have found that the framework has supported defining a common vision and high-level goals for the development work. In the evaluation processes, it is beneficial to integrate different perspectives. However, in practice we have faced challenges as evaluation activities are different and not all the chosen viewpoints can always be included due to the complexity of the framework and the variety of tools required for the practical usage of the framework. During the early phases of the project, impact assessment has been based on foreseen impacts by different stakeholders. During the forthcoming long-term pilots, we will see if we indeed can show impacts in work well-being based on our Operator 4.0 solutions.