1 Introduction

Developing new systems requires deep understanding of the context and expectations of the intended users. In HCI research, a diary study is a qualitative technique for collecting data about the user experience (UX) or behavior when interacting with a system or within a certain context [4]. A UX diary study often contains a log of the participants’ experience with an interactive system or during a specific incident, which could serve as a quality-assurance tool to verify that the system is what the users want and is fulfilling their needs [5, 6].

What makes diary studies stand out when compared to other HCI tools/methods is that they offer insight into the UX through temporal and longitudinal information about the users, products and contexts of use in a natural context of interaction [1, 3–5]. Participants in a diary study log data immediately at the occurrence of the event by answering a set of questions to help researchers get feedback about a certain experience. Alternatively, some other diary studies ask participants to capture momentary data with minimum logging and use the artifact collected to trigger participants’ memory when interviewed later on by the researchers [2].

2 Background

Complex systems are systems that involve large numbers of components and interconnect multiple interfaces together, such as real-time embedded systems and interactive platforms [8]. They do not follow a linear progress in which the output of one system is the input of another; they require an interdependent mode interaction between the different subsystems [9]. Designing user interfaces (UIs) for such systems can be a complex process for developers and designers. The literature shows different design processes followed by designers to produce prototypes for the UI of complex systems [10–12]. For example, Bonnie E. John suggested following the GOMS model for UI design and evaluation, which is a widely known theoretical concept in HCI that predicts user interaction behavior with proposed UI designs for complex systems and it stands for Goals, Operators, Methods and Selection rules [11].

Interconnected engineering systems require a special consideration when designing the interfaces given the specificity of the information displayed to the user and the level of accuracy. Using social-science concepts in HCI provides a systematic framework for the UI design of complex systems. In [10], the author provided a set of five questions related to UI design challenges that the designer should answer during the design process. These questions are based on classic graphic user interface (GUI) conventions and recent research into innovative interaction techniques to provide general design accomplishments for different interaction machines.

Researchers have used diary studies as a method to elicit requirements and insights by exploring processes and observing workflows. For instance, in [4], the authors were trying to understand the different activities conducted by information workers and the different complexities and variations of those activities. Specifically, they were using a diary study to observe how interruptions affect those activities and how office workers switch between tasks. In [5], the authors explored using a diary study as an evaluation tool to capture the system-design process and UX. It was used as a communication tool to validate users’ needs and communicate them to the designers and the researchers.

Previous work used diary study to understand the behavior and interactions of end-users interacting with a complex system. However, considering the designers of complex systems as the main focus of study, and capturing their experiences while prototyping and logging their process is the gap we have found in the literature. In our paper, we are evaluating the use of diary studies as a tool to explore the creative process in designing complex engineering systems. In addition to the diary-study logs, the participants were interviewed and asked about their insights regarding their experience prototyping each artifact. The following sections will present the methodology and the insights gathered from the study.

3 Diary Study in Complex Systems

3.1 Method

To understand the added value of diary studies in the context of interaction design for complex systems, a diary study was conducted to gather insights from the designers and developers. In this study, an online diary tool was used to collect instant data from designers involved in the design of a complex system in the form of an interactive visualization platform. Given the nature of this agile development environment, the diary approach allowed us to overcome the issue of interrupting the natural flow of work for our users (designers). The use of an online diary to capture snippets of the prototyping experience provided more flexibility by allowing more logs and a greater number of designers to join the study when the project required [7]. This flexibility would have not been possible if the study facilitator had to be physically present in each design feedback session.

The diary study was applied to explore the design process within the development of a Web-based interactive visualization platform, which is in its early stages of development as part of a project at the Center for Complex Engineering SystemsFootnote 1. The project’s team consisted of engineers who were developing the mathematical models behind the engine, and the platform’s designers and developers who were developing the interactive visualization platform. Three designers of this platform participated in the study, which spanned a time frame of four months. In the following subsections, we provide an overview explaining the diary process, the artifacts as communication tools, and the analysis that was conducted as part of this study.

3.2 Process

At the beginning of the diary-study process, the facilitator briefed the designers on the tasks required and how to use the diary-logging tool. The tool used was DscoutFootnote 2, an app for moment-based research to collect quantifiable data from users. The app allowed collecting photos, text and numbers momentarily from the designers after each design feedback session. In this study, the goal was to integrate the diary-logging task seamlessly into the design process without causing an interruption to the workflow of the designer. As depicted in Fig. 1, the process started with the designer sharing a prototype with the team to gain feedback. The designer then logged the feedback in the diary tool. The tool collected the data and provided insights for the study facilitator. An interview was conducted when the designers indicated that they faced a problem in the design of the artifact or if the type of artifact has changed from the last feedback session.

Fig. 1.
figure 1

The framework of the diary study

Full size image

Each time the designers logged in their artifact; the diary asked them a set of questions regarding the feedback session. The questions were defined to help the researchers understand the design process for this specific project, and were general enough to accommodate other projects. The questions were chosen to minimize the effort of logging by limiting the number of text inputs and instead giving the participant options and lists to choose from. Figure 2 shows some of these questions.

Fig. 2.
figure 2

Examples of the questions we have specified through the Dscout tool

Full size image

The interface was designed to capture media files in the form of images or videos of the prototype that show the context in which the artifact was shared, whether it was through a Web conference or an in-person feedback session. This approach supported the engagement and participation of our designers in the diary study over a longer duration of time. The Dscout tool was selected as a diary tool because it provided a convenient interface for the designers to capture images of the artifacts in-situ and a streamlined process for creating diary entries and aligning them in a storyboard across the project’s timeline. Figure 3 shows the diary tool’s interfaces from the perspective of the designer and the study facilitator. The app logs the data and sends them to the website on which the facilitator could access all the logs and conduct further analysis.

Fig. 3.
figure 3

Diary study tool and the designer/facilitator interfaces

Full size image

There are several time spans of UX with prototypes depending on the moment of usage in the design process for complex systems, as depicted in Fig. 4. While some methods for studying the UX shed light on one or more phases of the UX (e.g., user testing provides an insight in the “during usage” time span, focus groups can elicit insights before, during or after usage, SUS surveys provide an insight in the “after usage” time span), the diary-study method provides longitudinal information on UX that spans these four time frames [6, 7].

Fig. 4.
figure 4

Time spans of UX

Full size image

In addition to reporting activities and documenting experiences with artifacts used in the design process, the UX diary in this study was also used by the participants (designers) to discuss design issues or information about the front end of the complex systems’ interfaces. The type of artifact often impacts the participants’ experience while logging data or being interviewed and the researcher’s ability to analyze the data [2]. In this study, there were three types of artifacts used in the design process: sketches, digital prototypes (Axure and Illustrator) and coded prototypes (HTML5/D3.js). Table 1 shows how different types of artifacts (Fig. 5) were used to communicate the designs to the developer teams through the different UX time spans.

Table 1. Artifacts during different UX time spans
Full size table
Fig. 5.
figure 5

Examples of collected artifacts

Full size image

4 Analysis

To capture the design process and the role of artifacts used to communicate ideas during design sessions, the researchers went through the collected diary logs periodically to identify patterns. The diary study was supported with feedback interviews conducted with the participants (designers and developers) when the log of the participant indicated a noticeable pattern. These interviews were triggered based on two different incidents that were noticed: first, when designers indicated that they had faced a problem in designing a certain artifact and second, when designers changed the artifact they were using from one logged session to another.

For every feedback session logged, the type of artifact, the experience of the designer using that artifact and the number of designers participating in the session and with whom the designers shared their designs all were captured in the diary. In addition to that, the feedback that they received during each session was documented. Figure 6 shows some quantified data on these metrics from the study. The analysis revealed the variation in the number of participants in the feedback sessions, as they varied from few to many members. It also provided insights into the prototyping process, the issues that the designers face temporal dynamics of perceived usability of the system during the design process. Finally, the study also shows the variation in frequency of usage of artefacts (e.g. digital prototyping was the type of artifact that was mostly used to communicate the designs).

Fig. 6.
figure 6

Examples of quantified data collected from the study

Full size image

5 Reflection

Through this work, the goal was to explore the benefits of using digital diaries as user research tools in complex engineering systems. The objective of the diary study was twofold: first, to capture the design process of complex systems and the use of artifacts as communication tools using a diary study and second, to understand the influence of the diary study on the designer’s flow and designs. The insights we gathered were from two different perspectives: the perspective of the designer and the perspective of the researcher observing the design process. The data collected and the interviews provided insights into the early stage prototyping process and the exploratory phases in which the researcher observed the design process for complex systems. On the other hand, the diary study was used as a method to improve the user experience of communication artifacts (low-high fidelity) that are often considered by designers in prototyping complex systems.

5.1 The Design Process for Complex Systems

When designing for complex systems, the design process tends to adapt to accommodate the complexity of the product to be delivered. Within the interactive visualization platform observed in this study, an interesting pattern emerged during the design process. At the early stage, the designers started by sketching the layout of the tool and when more iterations were required for the digital prototype, the designers worked on the process in parallel. A design workflow was established to build the layout, and another workflow branched to prototype the visualization components. The visualization prototyping relied on producing coded prototypes, and the reason for this, according to the feedback interviews conducted, is that the design team wanted to speed up the design process to show a working mockup. Thus, the visualization section had to represent real data. The diary-study was effective in understanding why coding was perceived to be the best option to prototype the visualization in this specific context, since sketches or digital prototypes were not sufficient to display the complexity and interaction with such components. Other insights gathered from the data collected were related to temporal dynamics with different artefacts (e.g. findings indicated that designers spent most of the design time producing digital prototypes). After interviewing the designers to check on their progress and get their feedback, they indicated that using digital prototypes gave them more flexibility to share their designs, apply feedback and then update their designs and share them again, as opposed to sketches that were not enough to convey some of the platform’s complex elements.

5.2 Diary Study as a UX Method in Design

Designers and developers go through an iterative process of prototyping and implementation. In this study, the designers used three different artifacts (sketches, digital prototypes and coded prototypes), and for each artifact, the designers made several versions or drafts to communicate the concept and ideas to the team. As mentioned in the methods section, designers shared their designs with the project’s team members and principal investigators in co-located and remote communication settings. After every session, the designers documented or logged the designs they shared (sketches, prototypes or codes) and their perceived efficacy of the feedback they got in the diary-study’s tool. Based on the interviews we conducted with the designers, we found that this study helped them categorize their design process by going through different types of artifacts and documenting each session they conducted with their team and the feedback they received. The study asked designers to go through different steps, which helped them reflect on key design issues:

  1. 1.

    The type of feedback they got from sharing their design using different artifacts, which had a considerable impact on the next design decision they made and the artifact that they would consider in the prototyping process.

  2. 2.

    The importance of artifacts as a communication tool between designers and other stakeholders in the project.

  3. 3.

    The design considerations that the designers can gain from low-fidelity designs (e.g., sketches) and high fidelity (e.g., Azure prototype and coding).

The designers indicated in the interviews that logging their design process, the artifacts they have to create their designs, the people they shared these designs with and the feedback they received helped them in their next designs. Knowing how different artifacts worked as a communication tool between the designers and the stakeholders, the kind of feedback they received, and logging how difficult each step was, provided the designers with insights regarding what type of artifact could be used to convey their vision and communicate ideas. In addition to planning and communicating, going through the diary-study process helped the designers better understand the time requirements for each task, based on the feedback they received. Based on that, critical decisions were made regarding the artifact type they needed to use next. Going from low-fidelity prototypes to an interactive high-fidelity one was based on evaluating how challenging it was to communicate the designer’s idea to the stakeholders to elicit their feedback.

6 Conclusion

The use of an online diary for data collection from designers proved successful in the context of complex-engineering systems; its design was effective in providing a contextual understanding of artifacts and prototypes as communication tools, and its flexibility met users’ needs for minimal interruption in their workflow. Insights into patterns of feedback elicited from different types of artifacts allowed monitoring of the communication between the designers and other stakeholders within the project. In addition to the observations that the researchers gathered from the diary study, this approach also served as a documentation tool, helping the designers to track the contexts of use for their prototypes and plan the next artifact to be shared.

Nevertheless, there were also some challenges associated with using a diary approach in the context of agile development in complex-systems design. Although the tool provided a convenient way to capture and log data, designers had to be reminded several times to log their experiences with using artifacts in communicating ideas and conceptual designs. Given the nature of the agile development process, the designers could seek feedback during multiple events throughout the day, which made it challenging to select a feedback session to be logged in the diary study. The diary-study is ongoing and more insights are being collected on later stages of the design process for complex systems.