Keywords

1 Introduction

Simulation is a very powerful technique used in different fields to explore the behavior of complex systems as, e.g., the flow of ground water, the human walking behavior, and the world’s climate. Due to its applicability to many problems, simulation is one of the most used techniques in research and management sciences [4].

The results of simulations are uncertain due to, e.g., assumptions made in the modeling process or the parameter choice. Non-experts, who have no specialized knowledge about simulations, are confronted with simulation results and uncertain data everyday, e.g., when looking at the weather forecast that shows the possible temperature for the next day. They rely on uncertain data to implicitly predict the future, plan activities, and make decisions. These decisions could be even unintentionally manipulated by choosing a specific representation.

Although several aspects of the communication and visualization of uncertain data were examined before, still work has to be done to get more insights in how different representations influence decision-making. From existing work, we selected twelve representations (three textual and nine graphical representations) used for uncertainty communication for experts and non-experts. This leads to a wide range of used representations with different degrees of included uncertainty information.

First, we present the results of an online survey where participants had to rate the representations according to the perceived help for decision support in a farming scenario, familiarity, easiness to understand, and visual appeal. We found significant differences in the rating of the representations and also correlations between ratings. Surprisingly, we did not find any strong correlations between subjective ratings and the degree of uncertainty information included in a representation.

Second, we present the results of a small experiment with a turn-based online game. The four representations that performed best in the online survey were used to display a weather forecast. Decisions of players were analyzed on the basis of optimal decisions. The function graph of a probability distribution function performed best, but the representation with no information about the included uncertainty did not perform dramatically worse.

The contribution of our work is twofold. First, we show that people do not judge representations for the degree of uncertainty information they show. Other factors are more important for their judgment. Second, we show that a higher degree of uncertainty information does not necessarily lead to better decisions and that people are able to make good decisions with the help of a probability density function.

2 Related Work

The topic of uncertainty visualization is well explored for experts. Multiple areas such as vector field, surface and glyph visualizations are for example explored by Pang et al. [9] and Zuk and Carpendale [15]. Additionally, new versions of basic representations (e.g., box plots) are for example developed by Potter et al. [11]. In our paper, we do not take into account such visualizations and focus on basic representations developed for experts and non-experts.

For communicating uncertainty to non-experts, usually quantitative information, especially probabilities are used. One problem when using quantitative information is the inability of even well-educated adults to solve easy numeracy probability questions [5]. But also qualitative information, e.g., labels such as low uncertainty or low risk, could be misleading, as already examined by Wallsten et al. [14]. Additionally, the formulation of a risk or uncertainty, whether negative or positive, has a huge influence on decision-making [6].

One strand of work investigated uncertainty information in weather forecasts. Morss et al. [7] found that most people are aware of the uncertainty in deterministic weather forecasts, although the range of this uncertainty was perceived very differently. Additionally, 70 % of the people preferred forecasts that contained information about the uncertainty of the forecast. Studies by Roulston et al. [12] and Joslyn et al. [3] showed that people make better decisions when having information about the uncertainty of a forecast and that information about the uncertainty also increases the trust in a forecast. They used a small number of alternative representations in a decision task.

Another strand of work investigated and compared visualizations including uncertainty information. Ibrekk et al. [2] compared nine visualizations for uncertainty by giving non-experts specific tasks (e.g., finding the mean). They suggest displaying a normal probability distribution function together with a cumulative probability distribution function. Pappenberger et al. [10] made a study with experts in meteorology and asked them about their preferred representation for a probabilistic forecast. The most used representation were quantiles.

Additional studies focus on one specific visualization or aspect. Olston et al. [8] examined visualizations for presenting bounded uncertainty by adjusting the visual elements and including transparency in the visualizations and Tak et al. [13] used seven variations of a line graph to investigate the perceived certainty. Correll and Gleicher [1] proposed a redesign of bar charts and found that less well-known visualizations improve performance for interferential tasks.

Previous work mainly focused either on a small number or representations, variations of one representation, or a very specific task (e.g., finding the mean). They found that uncertainty information leads to better decisions, but it is not clear how this conclusion relates to different degrees of uncertainty information and how aggregated uncertainty information influence decision-making. In contrast to previous work, we decided to take a wide range of basic representations and concentrate our research on the degree of uncertainty information that is included in these representations.

3 Online Survey

To understand if the presented degree of uncertainty changes the perceived value of a representation for decision support and the easiness to understand the representation, we conducted an online survey. Building upon prior research, we selected twelve representations (see Fig. 1) with different properties for communicating uncertainty information. All representations show the expected rainfall for the next three days. Three representations use a textual representation of the information, whilst the other nine representations are graphical. We use a line chart, a box-and-whisker plot, bar charts, stacked bar charts, stacked area diagrams, shaded bars, and function graphs. The representations also communicate different degrees of uncertainty information, from no information about uncertainty at all up to detailed information. The displayed degree of uncertainty information for each representation is depicted in Table 1.

Fig. 1.
figure 1

All 12 representations compared in the online survey. Explanations: 1–3 Textual representations, 4 – Line chart with area diagram, 5 – Box-and-whisker plot, 6–7 Bar charts, 8 –Stacked bar charts, 9 – Area chart, 10 – Shaded horizontal bars, 11–12 Function graph

Full size image
Table 1. Degree of uncertainty information included in the representations.
Full size table

3.1 Questionnaire

On the first page of the questionnaire, we asked participants for demographic information: age, gender, their highest degree, and their field of work.

We then displayed a scenario description that told participants to imagine that they are a farmer and want to grow plants. The plants need a certain amount of water to grow and survive. A weather forecast will be available to support participants in making a decision, but it will be uncertain.

Participants then had to navigate through twelve pages with one representation on each page. The order of the representations was randomized across participants. For each representation, participants had to indicate their level of agreement on a five-level Likert scale from totally disagree to totally agree with four statements:

  1. 1.

    The representation supports me in making a decision.

  2. 2.

    I am familiar with the representation.

  3. 3.

    The representation is easy to understand.

  4. 4.

    The representation is visually appealing.

3.2 Participants

We invited participants through social networks to participate in our survey. In total, 90 participants (36 female, 54 male) fully answered our online survey. Participants’ age ranged from 18 to 82 years with a mean of 31 years (sd: 12.6). 45 % of the participants had a university degree, 28 % had a high school diploma, further 20 % finished a vocational training, and all other participants had a minor degree or no degree at all. Participants worked in different field such as computer science, economics, commerce, teaching, mechanics, services, and others. They had no specialized knowledge about simulations.

3.3 Results

For each representation, we calculated the mean for each of the four statements and the overall mean (see Table 2). For each statement, we also conducted a Friedman test and for the post hoc analysis Wilcoxon signed-rank tests with an applied Bonferroni correction. The Friedman test showed that there is a statistically significant difference between the twelve representations for each statement.

Table 2. Calculated mean values for the level of agreement on a five-level Likert scale from totally disagree (1) to totally agree (5) with the statements: S1 – The representation supports me in making a decision., S2 – I am familiar with the representation., S3 – The representation is easy to understand., S4 – The representation is visually appealing., and O – the overall mean values for all 12 representations.
Full size table

The Wilcoxon singed-rank tests showed that representation 1, 4, 7, and 11 performed significantly better than the majority of other representations in at least one judgment each. Representation 3 was rated significantly worse than the majority of representations on three scales.

We ran a Spearman’s rank-order correlation to determine relationships between our 1080 Likert items and the degree of uncertainty information of the different representations. As expected, we found strong, positive correlations between all pairs of Likert items, which were all statistically significant (p < 0.0005), see Table 3 for detailed values. Surprisingly, we did not find any significant positive or negative correlation between the Likert items and the degree of uncertainty information of the representations, except one moderate positive correlation with the Likert items for visual appeal (see Table 3). We assume that this correlation occurred because we used textual representations with low degrees of uncertainty information.

Table 3. Spearmen’s rho for a Spearman’s rank-order correlation between Likert scale items of the online survey and the degree of uncertainty information of the presented representations. Statistically significant values are marked with asterisk(s). **p < 0.01, *p > 0.0005
Full size table

4 Experiment

Based on the results of the online survey, we implemented a turn-based online game that displayed a weather forecast using the four representations that performed best in our online survey. With the help of the weather forecast for the next three days, players had to decide which crops they want to plant. Crops needed specific values for rainfall, wind, and sun that were always displayed and gave different amounts of money when fully grown and harvested. Players had the goal to get as much money as possible. If one weather condition for a planted crop was violated, it withered and it gave no money at all.

We conducted a controlled experiment with 12 participants (4 female, 8 male), recruited with the help of social networks, who had to play the game with each of the four representations. The order of the representations was changed for each participant and randomly assigned to them. Each game lasted for 10 rounds. Participants were not trained and did not have any specialized knowledge about simulations. For the analysis, we logged all relevant information, which included the forecasted weather, the real weather, and all crops that were planted.

Participants played in total 480 rounds of the game, 120 rounds with each representation. For our analysis, we only considered rounds in which the players at least clicked on the button to open the weather forecasts once. This resulted in 442 rounds.

For each round, we calculated the optimal decision based on the weather forecast displayed and compared it with players’ decisions. With representation 11, participants made the most optimal decisions, in 69 % of the rounds, with representation 7 in 64 % of the rounds, with representation 1 in 60 % of the round and with representation 4 in 57 % of the rounds.

At the end, we asked participants which representation they liked most. 8 participants selected representation 11, three participants selected representation 7, 1 participant selected representation 4, and no participant selected representation 1.

5 Discussion & Conclusion

The results of our online survey show that the representations were judged very differently. Surprisingly, the four best-rated representations taking the overall mean show a different degree of uncertainty information each. Correlating the judgments and the degree of uncertainty information provided by the representations, we found that there are no significant correlations between the perceived support in decision-making and the degree of uncertainty information. Thus, we showed that participants do not judge the perceived support of decision making for a representation in regard to the degree of uncertainty information presented. In contrast, factors such as familiarity, easiness to understand, and visual appeal have a huge influence on the judgment. Our work indicates that other factors besides the presented degree of uncertainty information have to be considered when displaying uncertain data.

Comparing results from the online survey and the experiment, we found that the line chart with the highest rating in the survey performed worse than the other representations in the experiment. Although other studies showed that participants make better decision when having uncertainty information, our experiment suggests that aggregated uncertainty information do not provide enough details to make better decisions than only information about the expected values. We assume that the aggregated uncertainty information, the confidence interval, was not enough information to make a better decision but instead alienated participants. Nevertheless, most participants in our experiment preferred using the representation of the probability distribution function. Additionally, participants made very good decisions using this probability density function although earlier studies suggest that people are not very good at intuitively interpreting statistical information. This indicates that a probability function should be used to communicate uncertain data for non-experts and that too much aggregation of uncertainty information should be avoided.

Our experiment is clearly limited by the small number of participants and the scenario, but nevertheless shows interesting factors that are relevant for uncertainty communication. Future work on different scenarios with more participants will help to generalize the findings.