1 Introduction

Un-moderated user research studies such as Usertesting.com or UserZoom are gaining popularity with user experience research practitioners for several reasons. It is more efficient from a logistics standpoint, most platforms allow the researcher to easily collect user experience metrics and for some metrics (e.g., self-reported overall ease), the data collected is comparable to data collected in a lab study.

1.1 Un-Moderated User Experience (UX) Research

In a remote un-moderated research study, researchers use an online software program to automate their study. The research participants get a list of pre-determined tasks to perform on their own. In some testing platforms the participant’s screen and voice are recorded. Since the testing platform is cloud-based, participants are able to complete the tasks in their own environment at a time that is convenient for them. There is no moderator present; when the participant has completed the tasks, the results are sent to the researcher. Data available from these software platforms can include: task success ratios, time on task, clickstreams, heat maps, video, audio, facial expressions, mouse movements, or responses to a follow-up questionnaire. The research can access results through an dashboard, or export the results to a file format that can be read offline.

Un-moderated research is recommended for evaluative research studies where the goal is to collect feedback on specific tasks. It is important that the tasks are designed so that the participant is unlikely to navigate far from the research materials. If the research method relies on a large sample size for arriving at conclusive results, an un-moderated research study is recommended since a moderator does not need to be present during each session. Since this style of research allows the participant to participate on their own time and in their own environment, un-moderated research studies minimize the bias introduced by conducting studies in artificial laboratory setting.

1.2 Un-Moderated UX Research: Pros and Cons

User experience research practitioners receive several benefits from un-moderated user research platforms. Since the researcher does not need to be present for the duration of the study, the number of person-hours needed to execute the study is reduced. This is a significant benefit for research studies requiring large sample sizes. The “on-demand” aspect of the study allows multiple sessions to facilitated concurrently. As a result, it becomes more feasible to collect large sample sizes which, in turn, reduces the margin of error and allows the researcher to arrive at conclusions with a higher degree of significance.

In addition to making it easier to execute a research study with large sample size, the logistics of setting up the study become easier. There’s no need to schedule appointments or travel to a research facility to conduct the sessions. By eliminating travel it becomes easier to get broader geographic coverage in the research study participant profile. By allowing participants to participate at a time that is convenient for them and on their choice of equipment, the chance to observe more natural behavior is higher compared to an artificial lab setting with a researcher observing the participant complete the tasks.

There are several limitations to this approach for answering research questions. First, there is an emphasis on using un-moderated research studies for evaluative research, where the objective is to get feedback on an existing design or information architecture. Common types of activities include: task-based usability testing, prototype and wireframe testing, click testing, timeout testing (e.g., 5-s test), card sorting and tree testing. These platforms have not been used for conducting formative research studies. In these studies, the objective is often to collect data to inform the design of an experience, to explore a new product concept or direction with users or to learn more about a specific target user. Examples of formative research methods can include: semi-structured interviews, focus groups, diary studies, collaging, feature prioritization exercises, stakeholder mapping and experience diagramming.

One reason why un-moderated research platforms are not used for formative research lies in the mechanisms that these systems have for collecting feedback from the participants. In evaluative research, common metrics include objective measures such as task success rate, time on task and subjective measures such as the participant’s self-reported response to follow-up survey questions after a task. Un-moderated research platforms are well-designed to collect these measures. In formative research, the data is less structured. For example, in an experience diagramming exercise, the participant produces a visual artifact showing a progression of events and a reflection of the experience at each event. In a focus group, multiple participants are engaged in conversation in real-time. In a stakeholder map, the participant produces a visual artifact representing a network of people and the connections between them. As a consequence of the un-structured nature of the data in formative research activities, researchers shy away from cloud based un-moderated platforms because it is difficult to collect the participant data. Un-moderated research platforms today are optimized for collecting structured data. In this paper, we propose an existing tool that eases some of the tensions introduced by the limitations of today’s un-moderated research platforms. We present examples of how this brainstorming tool can be used to conducted un-moderated formative user experience research.

2 MURAL as a Platform for Un-Moderated UX Research

MURAL (http://mural.ly) is a visual workspace used by creative teams to share inspiration, discover new insights, brainstorm and organize ideas and define solutions. In this paper, we describe how MURAL can be used as a platform to facilitate un-moderated formative user experience research activities. While MURAL already supports traditional formative research and analysis methods such as experience diagramming, feature prioritization and affinity clustering, we present innovative ways of using the platform.

Card Sorting

Card sorting is a technique to understand and assess expectations and understanding about a set of topics. The participant is a given a set of topics/concepts/ideas as “cards” and organizes the cards into categories which can be defined by the participant or the researcher.

Example.

In Fig. 1, participants are asked to drag a set of concepts into one of two categories:

Fig. 1.
figure 1

Card sorting exercise facilitated using MURAL

Full size image

  • Table Stakes: features or experiences that earn trust and respect from customers as a provider of design software and services across desktop, web and mobile

  • Differentiating Value: opportunities to develop unique industry expertise and deliver unique value for customers

While there are other web-based platforms for doing online card sorting (e.g., OptimalSort), the service provides a limited set of capabilities for providing feedback (if any). Using MURAL’s platform for conducting a card sort exercise, participants can create additional sticky notes to explain how they sorted the cards. They can also draw shapes and lines to more visually describe their sorting rationale (Figs. 2 and 3).

Fig. 2.
figure 2

Use of MURAL to conduct a Stakeholder Mapping exercise

Full size image
Fig. 3.
figure 3

Use of MURAL to conduct an experience diagramming exercise

Full size image

Stakeholder Mapping

Stakeholder mapping [needs ref] is a way of diagramming the network of people who are involved in a given work process. For example, an industrial designer needs to work with several types of stakeholders: other industrial designers, a manager/executive persona, a mechanical engineer and external vendors such as physical prototypers and manufacturers. Stakeholder maps allow the researcher to see the entire landscape of people involved in a complex process. This may influence where UX effort is focused or it may uncover UX opportunities that were not visible due to a limited view of the entire process.

In the above example, the stakeholder mapping exercise is presented in two stages. In the first stage, the participant is asked to brainstorm a list of people that are part of a specific process (e.g., if the participant is an industrial designer, who are people that you have to interact with at some point in the lifecycle of a project?). The participant creates one sticky note for each person. Then the sticky notes are arranged in a way that the participant can draw connections using the line tools. These lines can also be labeled to describe the relationship between two people. In a more detailed stakeholder diagram, the participant can use different colored sticky notes to denote different types of stakeholders. After constructing the diagram, the participant can use an additional sticky note as a speech-bubble to describe the stakeholder’s mindset or the participant’s most common pain point when interacting with this stakeholder.

Experience Diagramming

Experience diagramming [needs ref] is a method to visualize a process or workflow that can be broken down into phases or events. When used as a formative research method, experience diagramming helps the researcher understand the impact of each event from the user’s perspective.

In the above example, the experience diagramming exercise is being used to understand how data moves between applications across the duration of a project. The researcher prepares the workspace by placing “START” and “END” sticky notes on the right and left sides of the MURAL. The participant is asked to brainstorm and generate a list of major software tools used in a project for design or collaboration. Each individual application is written on a sticky note and placed sequentially along the start/end timeline. Next, the participant draws arrows to the direction of data flow. Finally, the participant uses changes the stroke color of the arrows to green to indicate a smooth transition. Similarly, the stroke color of the arrows is changed to red to indicate when a transition is inefficient or frustrating. When needed, the participant can give more detailed explanations on additional green sticky notes.

3 Conclusion

As un-moderated research grows in popularity among the user experience researcher community, an open area of investigation is in which research methods can be executed using an un-moderated research platform. In this paper we describe how MURAL can be used as a platform for conducting formative user research activities. Because MURAL supports richer forms of feedback than most other platforms, the flexibility makes it a better tool than existing platforms for conducting formative user experience research. As a case study, we demonstrate how MURAL can be used to execute three different types of research activities: card sorting, stakeholder mapping and experience diagramming.