The aim of the design and evaluation framework is to support design and evaluation activities throughout the project. Now that we have defined the framework, our aim is to continue utilizing it in practice, analyze the experiences of use, and update the framework accordingly.
In the design activities, systems usability puts the focus of the design to new work practices and how the new tools could best support smooth ways to work. UX covers the design aspect of how people feel at work. Both systems usability and UX support early design phases where different stakeholders of the project build a common vision of the solutions to be defined. Drivers and obstacles for user acceptance should be identified early in the design to find solutions that will be accepted. Safety and ethics viewpoints emphasize the consideration of those aspects early in the design, and from the worker’s point of view. In this way, they bring in a positive, forward-looking viewpoint that helps finding safe and ethically sustainable solutions.
The evaluation perspectives included in the framework are all using different methods. Integrating these methods is challenging but necessary to avoid having too many parallel evaluations that would be difficult to organize in practice. Furthermore, parallel evaluations might result in contradicting results, the implications of which would be difficult to interpret. The chosen perspectives should be integrated in evaluation so that we address all the necessary perspectives. Depending on the evaluation activity, the focus may be on higher-level issues or on details. Using similar questionnaires in all evaluation activities allows following the progress of the development work. These questionnaires can be extended by further items and complemented with interviews for individual evaluations to focus on selected issues. The interviews can be evaluation specific even if the evaluation themes follow the perspectives of the framework.
When Operator 4.0 tools are developed, at least as prototypes, they can be tested in real-life working contexts, where a more realistic impact assessment can be gained. In the impact assessment, measuring worker’s well-being before and after the introduction of new tools and practices allows for direct comparisons. This gives us the opportunity to draw conclusions about the impact of the new work tools and practices on well-being. Impact assessment can be continued later on in actual use. We emphasize that impact assessment can also be part of the early phases of a project. Foreseen impacts, assessed by the workers themselves, other company representatives and external experts give valuable insight to the potential of the new solutions already during the design phase. From the participating companies’ point of view, early feedback on foreseen impacts has been seen especially valuable, as it helps assessing which solutions are most promising regarding business benefits.
When performing evaluation, validated and widely used measures are beneficial as they are solid means producing clear results, easy to compare with the results of other studies using the same measures. However, the set of various perspectives is rather large in our project. Using validated questionnaires would mean hundreds of questions for each worker. Such a load cannot be set to the users. As we feel it to be important to include all the perspectives of the framework, a compact set of questions, with just a few questions for each perspective is needed. Thus, in practice, the questionnaires used and to be used in this project are tailored for this purpose. Of course, validated tests can still be used in some individual evaluations if needed for some specific goal. Furthermore, the questionnaires are complemented with solution specific questions. Using the common questionnaires allows comparing different solutions, while solution specific questions and complementary results of user interviews give detailed user feedback.
In the following, we describe our first experiences of using the framework in design and evaluation activities in the Factory2Fit project.
6.1 Overall Concept Design
The Factory2Fit project started with a series of co-design activities that involved all the project partners. First, factory workers of the project’s pilot factories were interviewed and their work was observed. Then, a series of workshops were conducted in which the results of the factory worker studies were analyzed and discussed among project partners. Industrial partners, representing the pilot companies, presented their expectations related to the new tools and the foreseen impacts. In the workshops, a common vision for the project was gradually set, and the vision was complemented with descriptions of new concepts for work tools and related work practices. The common vision was based on how the future work should feel and based on the vision, two high-level UX goals were defined in the workshops: empowerment and engagement. UX experts analyzed these high-level goals and based on the results of the user studies and the design workshops, they proposed more detailed experience goals (Table 1). The experience goals will guide the design of the Operator 4.0 solutions, as they will be analyzed to identify design implications for each individual concept and tool.
Table 1. User experience goals in Factory2Fit project.
To support ethics by design, we defined ethical guidelines (Table 2) based on the ethical guidelines for ambient intelligence by Ikonen, Kaasinen and Niemelä [46]. We chose five out of the six ethical themes (we left out “benefit for society” as it was felt too abstract in our case) and for each of the five ethical themes, we modified the original guideline so that it reflected what the guideline would concretely mean in Operator 4.0 solutions in factory environments. In the design phase, the guidelines form a checklist, with which the design team can assess proposed solutions.
Table 2. Ethical guidelines in Factory2Fit project.
Safety issues are typically studied by expert assessments, and many standards give instructions on safety issues. While these safety activities are important, in the framework, we wanted to focus on the workers’ point of view. Based on safety standards [42], we defined safety requirements to guide design (Table 3). In the design process, the proposed solutions and implementations have been checked against these requirements.
Table 3. Safety requirements in Factory2Fit project.
By now, systems usability and user acceptance have been utilized in design activities indirectly: they have been included as the evaluation viewpoints in evaluation activities that support human-centered iterative design. In the next sub section, we describe our experiences of the evaluation activities.
6.2 Using the Framework in Individual Design and Evaluation Activities
In the following, we describe three case studies of utilizing the design and evaluation framework in the design and evaluation activities in the Factory2Fit project. We describe how we used the framework and we discuss related benefits and challenges.
Case 1: Evaluation of Four Factory2Fit Concepts
The design and evaluation framework was utilized in the evaluation of four concepts that were developed in the project. The concepts aimed to engage workers for participatory design of factory operations, knowledge sharing and training. The concepts were utilizing augmented and virtual reality (AR/VR) and context-aware social media. Factory workers evaluated the concepts in two workshops. The concepts were presented one by one to the participants as demonstrators and/or videos. After each presentation, the participants filled in a questionnaire that guided them to consider the chosen evaluation themes (user acceptance, UX and safety). In the questionnaire, participants also evaluated foreseen impacts on job satisfaction. After the participants had filled in the questionnaires, we had a follow-up group interview. In the interview, they were asked about the benefits, the challenges and development ideas of the presented concept. This revealed attitudes and expectations to explain the results gathered with the questionnaires. After the concepts were evaluated in this way, the group discussed about the presented entity of concepts and their potential to support their work.
Besides workers’ experience and acceptance of the concepts as well as safety issues and expected impacts on job satisfaction (i.e., aspects of the framework), concepts’ main benefits, challenges and development ideas were formulated by the factory workers. They could point at diverse issues related to usability, safety or ethical issues. In general, workers’ feedback of the concepts was positive.
Our aim is to develop the framework while using it. That is why we analyzed the benefits and challenges of the framework after the case study. As a benefit, the framework guides the data collection in evaluations and makes data comparable with other research partners. This is important in a large research project with several evaluation activities carried out by different partners in different countries. The framework also supports keeping in mind the main aim of the project - to influence work well-being. Even if direct impacts cannot be measured until the solutions are in actual use, focusing on foreseen impacts in the evaluations helps developing the solutions to support this aim. During the design process, we have several different evaluation activities. The framework needs to be utilized adaptively, so that the focus is chosen according to the current design phase. The perspectives of the evaluation should then be chosen accordingly. However, in the user interviews, we should be open for all the perspectives. For instance in this study we did not focus on usability as we were dealing with early concepts. Still the users were commenting about the expected usability challenges with the forthcoming solutions. We also left ethics out, as we considered that the concepts did not include ethically sensitive issues. However, in the interviews the users pointed out foreseen ethical problems. Based on these results, we decided to include in the forthcoming user studies all the perspectives, even if with a smaller set of questions.
Case 2: Evaluation of a Social Media Platform
The design and evaluation framework was applied in the evaluation of a social media platform. The social media platform integrates a social media messaging system with a production environment so that the discussions can be connected to physical elements of the production line or to the status of the line such as a certain error situation. The purpose of this evaluation was to get feedback to the design from workers and other stakeholders. Nine participants (factory workers and other factory stakeholders) participated in the evaluation. The prototype system was first introduced to them and then they were divided into two groups, where they could try out the system guided by the facilitators. When trying out the system, the participants were asked about problems, improvement needs as well as pros and cons related to the system. In the end, the participants filled in a questionnaire. The questionnaire included UX, user acceptance and usability perspectives from the framework. Impacts were assessed with two questions in the questionnaire regarding anticipated changes in knowledge sharing practices. Results show that the participants liked the system and agreed that the usability was on a good level. They suggested many important improvements for the system.
As such, this study could be regarded as a basic usability research. However, in the workshop also impacts and development ideas were discussed. For example, the integration of workers discussions with production data could improve performance and job satisfaction. In addition, the possibility to share knowledge could engage workers more with their work and work community. The gamification feature could motivate some people to share knowledge more actively.
As a benefit, by using the design and evaluation framework it was possible to enhance traditional usability studies towards considering impacts of a system on work well-being and on productivity. The results pointed on the high value of qualitative data gathered via interviews or group discussions as many possible impacts were raised in the discussions but in the questionnaire, we were only asking about impacts on knowledge sharing. Similar to Case 1, this case points on the importance of qualitative methods for complementing questionnaires for assessing the various aspects of the framework in the future.
Case 3: Design and Evaluation of Worker Feedback Dashboard
The design and evaluation framework was utilized in the participatory design of Worker Feedback Dashboard, a web-based solution giving personal data-based feedback to machine operators at a factory (the design process is described in [47]). The idea of the solution is that it gives personal feedback both on selected well-being metrics through a wearable self-tracking device and on relevant production metrics through the machinery of the factory, enabling the workers to see potential connections between these and thus empowering operators to make behavioral changes.
All aspects of design and evaluation framework were addressed during the development process. Before the actual participatory design, the usability and user experience of several wearable self-tracking devices were evaluated in the expert evaluation by the researchers and the potential ethical issues were identified in an expert workshop (the study is described in details in [48]). Before designing the first prototype, also factory workers were involved to give their initial feedback on the acceptability of the concept idea. We also visited two factories (pilot sites of the project) to gain understanding on the work context, which helped us in considering the safety aspects and the potential user acceptance. Based on user understanding, we first designed the user experience goals highlighting for example the role of positive feedback (empowering user experience) and the importance of not disturbing the user’s work tasks (ensuring safety). The key design implications were derived from these goals.
When initially evaluating the first prototype, we showed it to factory workers, who could comment it freely, and express their potential concerns and further ideas. With a short questionnaire, we assessed usability related issues (e.g. whether it is easy to understand the data shown), user acceptance related questions (e.g. whether the solution seems to provide value to users through interesting or useful content), and ethical issues related to the solution (whether the solution seems questionable). Based on the user feedback, refinements to the design were made.
The design and evaluation framework will be further utilized when piloting the refined functional prototype as a part of actual factory work. We are planning to conduct a 2-month usage pilot, which gives us understanding of the real user acceptance and user experience, as well as usability issues during long-term use. Based on the long-term use, we can evaluate whether using the solution may involve safety risks and can see what kind of ethical issues are brought up by the users. The design and evaluation framework will be utilized when defining the questionnaires and interview questions for the pilot.