# Measuring higher order ambiguity preferences

- 358 Downloads

## Abstract

We report the results from an experiment designed to measure attitudes towards ambiguity beyond ambiguity aversion. In particular, we implement recently-proposed model-free preference conditions of ambiguity prudence and ambiguity temperance. Ambiguity prudence has been shown to play an important role in precautionary behavior and the mere presence of ambiguity averse agents in markets. We observe that the majority of individuals’ decisions are consistent with ambiguity aversion, ambiguity prudence and ambiguity temperance. This finding confirms the prediction of many popular (specifications of) ambiguity models and has important implications for models of prevention behavior.

### Keywords

Higher-order preferences Prudence Ambiguity attitude Risk attitude### JEL Classification

C91 D81## 1 Introduction

Risk aversion governs many parts of economic behavior. However, it has to be complemented by the higher-order risk preferences *prudence* and *temperance* if one wants to explain important economic phenomena such as precautionary saving or prevention efforts. Consider a decision maker (DM) whose income is the same now and at a future point in time, and who just learned that she will face a zero-mean risk at the later point in time. According to expected utility (EU) theory, the DM should reduce current consumption and increase savings (i.e., save precautionary) if his future expected marginal utility is higher than his present marginal utility, i.e. if he has a convex marginal utility (*u*′′′ > 0), a property typically called (risk) prudence (Leland 1968; Kimball 1990). This result illustrates the importance of risk attitudes in economic decision making beyond risk aversion.

For long, the literature on higher order risk preferences remained purely theoretical (Leland 1968; Sandmo 1970; Kimball 1990, 1993). Eeckhoudt and Schlesinger (2006) were the first to propose behavioral, testable conditions of higher order risk preferences that made experimental investigation of these attitudes possible. For instance, one can test whether the DM is risk prudent if he prefers adding a zero-mean risk to a high income state rather than to a low income state. The DM is risk temperate (fourth order, *u* ^{(4)} < 0 under EU) if he prefers to disaggregate independent zero-mean risks across equiprobable states of nature. The conditions proposed by Eeckhoudt and Schlesinger (2006) led to many measurements in the laboratory and in the field, generally finding strong evidence for risk prudence and, to a lesser extent, for risk temperance (Tarazona-Gomez 2004; Deck and Schlesinger 2010, 2014; Ebert and Wiesen 2011, 2014; Noussair et al. 2014; Maier and Rüger 2012).

The literature mentioned above only concerns objectively known probabilities. However, such probabilities are unavailable in many important economic situations (Knight 1921). People have been found to dislike the absence of objective probabilities, exhibiting ambiguity aversion. Since Ellsberg (1961) proposed his famous paradox, empirical studies of ambiguity preferences have virtually all been restricted to studying the causes and prevalence of ambiguity aversion (e.g., see Trautmann and van de Kuilen 2015).

This paper explores attitudes toward ambiguity beyond aversion. *Ambiguity prudence* is a preference for combining ambiguity with states of the world yielding a high chance of a good outcome, rather than with states yielding a low chance of a good outcome. An *ambiguity temperate* DM dislikes facing two sources of ambiguity at the same time, and prefers to disaggregate them (for details, see Sect. 2).

Experimental evidence about higher order ambiguity attitudes is necessary to complement the recent theoretical literature that highlighted the importance of these concepts. For example, ambiguity prudence has been found to play a key role in models of survival of ambiguity averse agents on a market (Guerdjikova and Sciubba 2015) and of prevention behavior (Baillon 2017). Ambiguity prudence is necessary and sufficient for future ambiguity to trigger an increase in prevention efforts, e.g., an ambiguity prudent DM would rather get a vaccine against a new (“ambiguous”) disease than against a well-known disease with the same average severity and incidence (Baillon 2017).^{1} Moreover, specifications of popular ambiguity models (Gilboa and Schmeidler 1989; Schmeidler 1989; Hansen and Sargent 2001; Ghirardato et al. 2004; Klibanoff et al. 2005) that are widely used in applications imply ambiguity prudence. A policy maker who makes *robust* decisions in the sense of Hansen and Sargent (2007) is ambiguity prudent and ambiguity temperate. Despite the theoretical importance of higher order ambiguity attitudes for economic decision making, there is no evidence regarding the prevalence of higher order ambiguity attitudes in the population.

This paper is the first to investigate higher-order ambiguity preferences empirically. First, we develop an experimental methodology to implement the preference conditions proposed by Baillon (2017). Second, we embed our tests in a more general experiment, also measuring ambiguity aversion and higher order risk preferences (risk aversion, risk prudence and risk temperance). This allows us to investigate the relationship between these attitudes, and to test whether both these preferences stem from combining good outcomes with bad outcomes, as recently conjectured by Crainich et al. (2013) and observed by Deck and Schlesinger (2014). Our results show clear support for ambiguity prudence, thereby confirming the predictions of several ambiguity models. Our findings further support, to a lesser extent, ambiguity temperance.

The remainder of this paper is organized as follows. In the next section, we present the theoretical foundation of our measurements of higher order ambiguity attitudes. Section 3 describes the design of our experiment. The results are presented in Sect. 4 and are discussed in Sect. 5. Section 6 concludes.

## 2 Higher order preferences

### 2.1 Order 2: aversion

A DM is facing a choice between two 50–50 lotteries A_{r2} ≡ (*x* – *k* _{1} – *k* _{2}, *x*) and B_{r2} ≡ (*x* – *k* _{1}, *x* – *k* _{2}) for some *x*, *k* _{1}, *k* _{2} > 0.^{2} In lottery B_{r2}, the two losses (two “harms”) –*k* _{1} and –*k* _{2} are “disaggregated”, i.e., one of them occurs only if the other does not. By contrast, the harms are aggregated in lottery A_{r2}. *Risk aversion* can be defined as preferring B_{r2} to A_{r2} for all *W*, *k* _{1}, and *k* _{2}. Hence, risk averters prefer disaggregating the two harms. Under EU, it is equivalent to *u*″ < 0. *Risk lovers* always prefer A_{r2} to B_{r2} and therefore satisfy *u*″ > 0 under EU. We used 50–50 lotteries in the definition to show the similarity with the definitions that will follow. Of course, A_{r2} can be any lottery and B_{r2} its expected value. We write (*p*: *x*, *y*) to refer to binary lotteries yielding *x* with probability *p* and *y* otherwise. If *p* is omitted, then it is 50%.

Let ({*p*, *q*}: *x*, *y*) describe a bet in which one state of nature (e.g., one possible composition of the deck) gives a probability *p* of winning *x* (and *y* < *x* otherwise) and the only other possible state of nature gives a probability *q* of winning the same prize *x* (and *y* otherwise). In the remainder, when using the notation {*p*, *q*}, we will assume *informational symmetry*: the DM does not have information favoring one of the two states over the other. Bet A can be represented by ({0%,100%}: €30, €0) and bet B by ({50%, 50%}: €30, €0) (whatever the state of nature, i.e., whatever the color composition of the deck, the probability of winning is 50%). Preferring bet B, with the known probability of winning *x*, to bet A is interpreted as ambiguity aversion. Consider the pair of bets A_{a2} and B_{a2} defined by A_{a2} ≡ ({*p* – *q* _{1} – *q* _{2}, *p*}: *x*, *y*) and B_{a2} ≡ ({*p* – *q* _{1}, *p* – *q* _{2}}: *x*, *y*), with *x* > *y*, 0 ≤ *p* ≤ 1, *q* _{1} > 0, *q* _{2} > 0, and *p* – *q* _{1} – *q* _{2} ≥ 0. We call –*q* _{1} and –*q* _{2} *probability losses*, i.e. reductions of the probability of winning. *Ambiguity aversion* can be characterized as preferring B_{a2} to A_{a2} for all such pairs of bets. It can be seen as aversion toward mean preserving spreads in winning probabilities. The opposite preference is called *ambiguity loving*.

Many models have been proposed to accommodate ambiguity aversion. Among others, maxmin expected utility (Gilboa and Schmeidler 1989), Choquet expected utility with convex capacities (Schmeidler 1989), variational preferences (Maccheroni et al. 2006) all predict a preference for B_{a2} in the choice described above. Klibanoff et al. (2005) introduced a smooth model of ambiguity attitudes, which is close in spirit to EU. Consider a pair of bets A_{a2} and B_{a2}. The smooth model involves three functions: a von Neumann and Morgenstern utility function *u* that we scale such that *u*(*y*) = 0 and *u*(*x*) = 1; a subjective probability function *μ* over the possible states of nature, and a smooth ambiguity attitude function *φ.* According to the smooth model, A_{a2} and B_{a2} are evaluated by *μ*(*p* – *q* _{1} – *q* _{2})*φ*(*p* – *q* _{1} – *q* _{2}) + *μ*(*p*)*φ*(*p*) and *μ*(*p* – *q* _{1})*φ*(*p* – *q* _{1}) + *μ*(*p* – *q* _{2})*φ*(*p* – *q* _{2}) respectively. From informational symmetry, we obtain *μ*(*p*) = *μ*(*p* – *q* _{1} – *q* _{2}) = *μ*(*p* – *q* _{1}) = *μ*(*p* – *q* _{2}) = 0.5 and therefore, that ambiguity aversion is equivalent to *φ*″ < 0.

### 2.2 Order 3: prudence

Eeckhoudt and Schlesinger (2006) showed how to define higher order risk attitudes as a preference for disaggregating harms. Consider a choice between A_{r3} ≡ (*x*, *x* + \(\tilde{\varepsilon }\) _{1} – *k* _{2}) and B_{r3} ≡ (*x* + \(\tilde{\varepsilon }\) _{1}, *x* – *k* _{2}), with \(\tilde{\varepsilon }\) _{1} a zero-mean risk. The two 50–50 lotteries involve a loss – *k* _{2} and another “harm”, the zero-mean risk \(\tilde{\varepsilon }\) _{1}. *Risk prudence* can be defined as always preferring B_{r3} to A_{r3} for all *x*, \(\tilde{\varepsilon }\) _{1}, and *k* _{2}, hence preferring disaggregating the two harms. The DM would rather bear the loss in the situation in which he does not bear the risk. Under EU, it is equivalent to *u*′′′ > 0. A DM always preferring A_{r3} to B_{r3} is called *risk imprudent* and will satisfy *u*′′′ < 0.

*plus*an additional chance of winning depending on two independent states of nature, i.e., the absence or presence of jacks {0, 50%}. Now imagine that the DM must give up a possibility of winning: either he does not win if a red king is drawn

*or*he does not win if a black king is drawn from the deck. If he gives up “red king” the bet becomes ({50%, 0% + {0%, 50%}}: €30, €0). If he gives up “black king” it becomes ({25%, 25% + {0%, 50%}}: €30, €0). In other words, the DM has to choose whether he should aggregate or disaggregate a 25% probability loss with the presence of ambiguity ({0, 50%}). An ambiguity prudent DM would rather disaggregate the two harms (probability loss and jack-related ambiguity) and therefore chooses to give up the black king.

Consider the general pair of bets A_{a3} and B_{a3} defined by A_{a3} ≡ ({*p*, *p* – *q* _{2} + {–*t*, *t*}}: *x*, *y*) and B_{a3} ≡ ({*p* – *q* _{2}, *p* + {–*t*, *t*}}: *x*, *y*) with *x* > *y*, 0 ≤ *p* ≤ 1, *q* _{2} > 0, *t* > 0, *p* + *t* ≤ 1 and *p* – *q* _{2} – *t* ≥ 0. *Ambiguity prudence* is characterized by preferring B_{a3} to A_{a3} for all such pairs of bets. The opposite preference is called *ambiguity imprudence*. Baillon (2017) showed that ambiguity prudence is equivalent to *φ*′′′ > 0 in the smooth model and that some popular specifications of other ambiguity models predict ambiguity prudence: e.g., Hansen and Sargent’s (2001) multiplier preferences, Chateauneuf et al.’s (2007) Choquet expected utility with neoadditive capacities, maxmin expected utility (Gilboa and Schmeidler 1989) for some specific sets of priors.

### 2.3 Order 4: temperance

To define the fourth order of risk attitude, let us consider two independent zero-mean risks \(\tilde{\varepsilon }\) _{1} and \(\tilde{\varepsilon }\) _{2} and define A_{r4} ≡ (*x*, *x* + \(\tilde{\varepsilon }\) _{1} + \(\tilde{\varepsilon }\) _{2}) and B_{r4} ≡ (*x* + \(\tilde{\varepsilon }\) _{1}, *x* + \(\tilde{\varepsilon }\) _{2}). A DM is *risk temperate* if and only if he prefers B_{r4} to A_{r4} for all *x*, \(\tilde{\varepsilon }\) _{1}, and \(\tilde{\varepsilon }\) _{2}. Such a DM rather disaggregates the two (harmful) risks. The opposite preference is called *risk intemperance.* Risk (in)temperance is equivalent to *u* ^{(4)} < (>) 0 under EU (Eeckhoudt and Schlesinger 2006).

More generally, define a pair of bets A_{a4} and B_{a4} by A_{a4} ≡ ({*p*, *p* + {–*t*, *t*} + {–*s*, *s*}}: *x*, *y*) and B_{a4} ≡ ({*p* + {–*t*, *t*}, *p* + {–*s*, *s*}}: *x*, *y*) with *x* > *y*, 0 ≤ *p* ≤ 1, *t* > 0, *s* > 0, *p* + *t* + *s* ≤ 1 and *p* – *t* – *s* ≥ 0. *Ambiguity temperance* is defined as preferring B_{a4} to A_{a} _{4} for all such pairs of bets. The opposite preference is called *ambiguity intemperance*. Ambiguity temperance is equivalent to *φ* ^{(4)} < 0 in the smooth model.

In the definitions of ambiguity aversion, prudence, and temperance, we used binary spreads in probabilities (e.g. {–*t*, *t*}). Baillon (2017) showed that the definitions can also be adapted to series of such spreads. For instance, in the example for ambiguity prudence, we described a deck in which four cards (out of eight) were either jacks or aces and we denoted the ambiguity related to betting on jacks {0, 50%}. We can replace this example by a deck in which each of the four cards can be either an ace or a jack. In such a case, we will write [0, 50%], to indicate that intermediary probabilities are also possible.

### 2.4 Mixed attitudes

We presented the risk apportionments of order 2, 3, and 4 and their equivalent for ambiguity, thereby illustrating the link between risk and ambiguity for each order. We now turn to the link between the various orders. The definitions of aversion, prudence, and temperance all relied on a preference for disaggregating harms. Under EU for risk, risk apportionment of order *n* for *n* = 2 to 4 is equivalent to *sgn u* ^{(n)} = (−1)^{n+1} as shown by Eeckhoudt and Schlesinger (2006). A function satisfying this property for all *n* was called *mixed risk averse* by Caballé and Pomansky (1996). Crainich et al. (2013) called *mixed risk loving* a utility function whose successive derivatives are all positive. They show that such a utility is equivalent to always preferring combining good with good (and bad with bad). At order 2, it implies risk seeking (preferring the aggregation of the losses in Fig. 1a). Hence, risks are “good.” At order 3, it surprisingly implies risk prudence because the DM prefers combining the “good” risk with the high outcome. At order 4, it implies risk intemperance (preferring the aggregation of the “good” risks).

Let us extend the analysis to the domain of ambiguity in the framework of the smooth model. In particular, let us call *mixed ambiguity averse* a smooth ambiguity function *φ* satisfying *sgn φ* ^{(n)} = (−1)^{n+1} and *mixed ambiguity loving* a smooth ambiguity function *φ* satisfying *sgn φ* ^{(n)} = 1. Mixed ambiguity aversion implies ambiguity aversion, prudence, and temperance and can be explained by a preference for combining good with bad. Mixed ambiguity loving implies ambiguity loving, prudence, and intemperance and can be explained by a preference for combining good with good.

Combining good with bad or good with good

Prefer combining good with bad | Prefer combining good with good |
---|---|

Mixed risk averse Risk averse ( Risk prudent ( Risk temperate ( | Mixed risk loving Risk loving ( Risk prudent ( Risk intemperate ( |

Mixed ambiguity averse Ambiguity averse ( Ambiguity prudent ( Ambiguity temperate ( | Mixed ambiguity loving Ambiguity loving ( Ambiguity prudent ( Ambiguity intemperate ( |

## 3 Experimental design

One hundred ninety-nine students from Erasmus University Rotterdam participated in a computerized experiment, which was conducted at the ESE-econlab. Subjects were recruited among a pool of volunteers and were told the experiment could last up to 1 h 15 min. They were told they would receive a €5 participation fee and that they could additionally earn from €0 to €45. The subjects could expect earning €23.50 in total. Each session involved between 23 and 28 subjects. The mean age of subjects was 22 years, 62% were studying economics or finance (38% other social sciences), and 38% were female.

The experiment consisted of 2 parts and each part of 15 choices. One part involved the elicitation of attitudes towards risk, the other part involved the elicitation of attitudes towards ambiguity. Each part started with instructions, read out aloud by the experimenter. In four sessions, subjects first made the risky choices and then the ambiguity choices. In the other four sessions, the order of the parts was reversed. Within each part, the order of the choices varied across sessions, but this order was the same for all subjects in a given session.

Choice tasks

Task | Dom | Order | Option A | Option B | EV |
---|---|---|---|---|---|

1 | Risk | 2 | (1/2: €30, €0) | €15 | €15 |

2 | Risk | 2 | (1/2: €45, €15) = 1A + €15 | 1B + €15 | €30 |

3 | Risk | 2 | (1/2: €45, €0) = 1A × 1.5 | 1B × 1.5 | €22.5 |

4 | Risk | 2 | (1/3: €30, €0) | €10 | €10 |

5 | Risk | 2 | (2/3: €30, €0) | €20 | €20 |

6 | Risk | 3 | (1/2: €15, €7.5 + \(\tilde{\varepsilon }\) | (1/2: €15 + \(\tilde{\varepsilon }\) | €11.25 |

7 | Risk | 3 | 6A + €15 | 6B + €15 | €26.25 |

8 | Risk | 3 | 6A × 2 | 6B × 2 | €22.5 |

9 | Risk | 3 | (1/2: €10, €5 + \(\tilde{\varepsilon }\) | (1/2: €10 + \(\tilde{\varepsilon }\) | €7.5 |

10 | Risk | 3 | (1/2: €25, €10 + \(\tilde{\varepsilon }\) | (1/2: €25 + \(\tilde{\varepsilon }\) | €17.5 |

11 | Risk | 4 | (1/2: €15, €15 + \(\tilde{\varepsilon }\) | (1/2: €15 + \(\tilde{\varepsilon }\) | €15 |

12 | Risk | 4 | 11A + €15 | 11B + €15 | €30 |

13 | Risk | 4 | 11A × 1.5 | 11B × 1.5 | €22.5 |

14 | Risk | 4 | (1/2: €10, €10 + \(\tilde{\varepsilon }\) | (1/2: €10 + \(\tilde{\varepsilon }\) | €10 |

15 | Risk | 4 | (1/2: €20, €20 + \(\tilde{\varepsilon }\) | (1/2: €20 + \(\tilde{\varepsilon }\) | €20 |

16 | Amb | 2 | ([0, 1]: €30, €0) | 1A | €15 |

17 | Amb | 2 | 16A + €15 | 2A | €30 |

18 | Amb | 2 | ({0, 1}: €45, €0) | 3A | €22.5 |

19 | Amb | 2 | ([0, 2/3]: €30, €0) | 4A | €10 |

20 | Amb | 2 | ([1/3, 1]: €30, €0) | 5A | €20 |

21 | Amb | 3 | ({1/2, 1/4 + [−1/4, 1/4]}: €30, €0) | ({1/4, 1/2 + [−1/4, 1/4]}: €30, €0) | €11.25 |

22 | Amb | 3 | 21A + €15 | 21B + €15 | €26.25 |

23 | Amb | 3 | 21A × 1.5 | 21B × 1.5 | €16.88 |

24 | Amb | 3 | ({1/3, 1/6 + [−1/6, 1/6]}: €30, €0) | ({1/6, 1/3 + [−1/6, 1/6]}: €30, €0) | €7.5 |

25 | Amb | 3 | ({2/3, 1/2 + [−1/6, 1/6]}: €30, €0) | ({1/2, 2/3 + [−1/6, 1/6]}: €30, €0) | €17.5 |

26 | Amb | 4 | ({1/2, 1/2 + [−1/8, 1/8] + [−1/8, 1/8]}: €30, €0) | ({1/2 + [−1/8, 1/8], 1/2 + [−1/8, 1/8]}: €30, €0) | €15 |

27 | Amb | 4 | 26A + €15 | 26B + €15 | €30 |

28 | Amb | 4 | 26A × 1.5 | 26B × 1.5 | €22.5 |

29 | Amb | 4 | ({1/3, 1/3 + [−1/6, 1/6] + [−1/6, 1/6]}: €30, €0) | ({1/3 + [−1/6, 1/6], 1/3 + [−1/6, 1/6]}: €30, €0) | €10 |

30 | Amb | 4 | ({2/3, 2/3 + [−1/6, 1/6] + [−1/6, 1/6]}: €30, €0) | ({2/3 + [−1/6, 1/6], 2/3 + [−1/6, 1/6]}: €30, €0) | €20 |

Presenting reduced lotteries is a departure from most previous experimental investigations of higher order risk preferences. We did so because Halevy (2007) and Abdellaoui et al. (2015) found that ambiguity aversion is associated (strongly for the former study, weakly for the latter) with non-reduction of compound lotteries. Hence, we expected that presenting the lotteries in a non-reduced form could contaminate higher order risk preferences with violations of lottery reduction, and artificially inflate the association between higher order risk preferences and higher order ambiguity preferences. An important consequence of using reduced forms is that losses were not displayed as such but integrated into the final outcome (for instance, in Fig. 5a, the ±€15 risk never implies a net loss). Haering et al. (2017) found that DMs exhibit less prudence and temperance when lotteries are presented in their reduced form.

For task 23, the subjects saw on their computer screen both Figs. 4b and 5b (and some explanation). In both options, they would win €45 if a black King or a red 9 was drawn. In Option L (representing Option A of Table 2), they would also win if a red 8 was drawn and in Option R, if a black 8 was drawn. If the deck of Fig. 4b was composed of red cards, then Option L had a winning probability of 1/2, but if the deck is composed of black cards, the deck had a winning probability between 0 (if there was no king) and 1/2 (if there were four Kings). This implements ({1/2, 1/4 + [−1/4,1/4]}: €45, €0). Option R similarly implements ({1/4, 1/2 + [−1/4,1/4]}: €45, €0).

A potential problem in ambiguity tasks is suspicion: subjects may think that the experimenters have manipulated the likelihood that particular unknown events are realized to minimize the cost of the experiment. To avoid suspicion playing a role in the experiment, subjects were allowed to permute all ambiguous events. For example, in task 23, subjects could permute Kings and Queens and/or red and black. Since the decks were prepared before the experiment, the experimenters could not be suspected of strategic behavior. This possibility was also crucial to ensure that the subjects consider the various ambiguities to be informationally symmetric. They had no reason to believe that one color was more likely to be drawn than the other (see Sect. 3 for a further discussion).

Subjects were paid according to one of their choices in one randomly selected task. The implementation of random incentives can create issues in ambiguity experiments by offering the possibility for the subjects to hedge against ambiguity (e.g., Oechssler and Roomets 2014). In our experiment, each task concerned a different deck of cards and therefore, a different source of ambiguity. Hence, the random incentives did not provide a hedging possibility.

The incentives were implemented as follows. At the beginning of the session, one envelope was randomly drawn (by a subject) from a pile of 30 envelopes containing a decision task each. Subjects were told that the envelope (signed by the subject to identify it) would be opened at the end of the session and that the task it contains would be played for real money. This approach can enhance isolation (Johnson et al. 2014), making it clear that only one task is used for payment. Once the envelope was opened, the corresponding deck was mixed, and a subject drew a card from the deck. Payoffs were then determined and implemented.

## 4 Results

### 4.1 Order 2: aversion

First, as the figure shows and in line with previous literature, the majority of choices in the experiment is consistent with risk aversion. On average, subjects chose the risk averse option 4.20 out of 5 times, which is significantly different from the average that would be observed if subjects chose randomly (Wilcoxon signed-rank test,^{3} *p* value < 0.01). More than half of the subjects (57%) chose the risk averse option in all 5 choices. In addition, the observed distribution of risk averse choices is significantly different what would be observed if subjects chose randomly (*χ* ^{2} test, *p*-value < 0.01).

As the right panel of the Fig. 6 suggests, we also observe ambiguity aversion. On average, subjects chose the ambiguity averse option 2.95 out of 5 times, which is significantly different from random choice (Wilcoxon signed-rank test, *p*-value < 0.01). In addition, the observed distribution of ambiguity averse choices is significantly different from the distribution that would be observed if subjects chose randomly (*χ* ^{2} test, *p*-value < 0.01).

### 4.2 Order 3: prudence

First, we corroborate the common finding in the field that decision makers are risk prudent. On average, subjects chose the risk prudent option 3.54 out of 5 times, which is significantly different from the average that would be observed if subjects chose randomly (Wilcoxon signed-rank test, *p*-value < 0.01). In addition, the observed distribution of risk prudent choices is significantly different what would be observed if subjects chose randomly (*χ* ^{2} test, *p*-value < 0.01).

More interestingly, we confirm a strong preference for prudent options for ambiguity. On average, subjects chose the ambiguity prudent lottery 3.10 out of 5 times, which is significantly different from random choice (Wilcoxon signed-rank test, *p*-value = 0.01). In addition, the observed distribution of ambiguity prudent choices is significantly different what would be observed if subjects chose randomly (*χ* ^{2} test, *p*-value < 0.01). Thus, on average, subjects prefer to disaggregate a fixed probability loss and an informationally symmetric ambiguity, i.e., they can better cope with ambiguity if the wining probability is high, which implies that *φ*′′′ > 0 in the smooth model.

### 4.3 Order 4: temperance

*in*temperance. Subjects chose the risk temperate lottery 2.14 times on average, which is significantly less than the average that would be expected if subjects chose randomly (Wilcoxon signed-rank test,

*p*-value < 0.01). Moreover, the observed distribution of risk temperate choices is significantly different what would be observed if subjects chose randomly (

*χ*

^{2}test,

*p*-value < 0.01).

The observed preference for aggregating risks does not extend to ambiguity, as the right panel of Fig. 9 shows. On average, subjects chose the ambiguity temperate option 2.91 out of 5 times, which is significantly different from random choice (Wilcoxon signed-rank test, *p*-value < 0.01), and the observed distribution of ambiguity temperate choices is significantly different from the distribution that would be observed if subjects chose randomly (*χ* ^{2} test, *p*-value < 0.01). Our results thus indicate that subjects exhibit a preference to disaggregate independent informationally symmetric ambiguities, implying a negative fourth derivative of *φ* in the smooth model.

### 4.4 Regression analysis

^{4}on choices, with a preference for Option A coded as 0 and for Option B coded as 1. Hence the regression results can be interpreted in terms of probability to choose the averse/prudent/temperate option. Our regression models include no constant, i.e., all coefficients equal to zero would imply random choice. Table 3 reports the results in terms of marginal effects.

Analysis of the choices with logistic regressions, with and without control variables

Model 1 | Model 2 | Model 3 | Model 4 | Model 5 | Model 6 | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Risk averse | 0.36*** | (0.02) | 0.34*** | (0.04) | 0.40*** | (0.04) | 0.41*** | (0.04) | 0.35*** | (0.04) | 0.41*** | (0.06) |

Risk prud. | 0.19*** | (0.02) | 0.18*** | (0.02) | 0.20*** | (0.02) | 0.19*** | (0.03) | 0.17*** | (0.03) | 0.17*** | (0.05) |

Risk temp. | −0.06*** | (0.02) | −0.04* | (0.03) | −0.01 | (0.03) | −0.01 | (0.03) | −0.04 | (0.03) | 0.09* | (0.05) |

Amb. avers. | 0.08*** | (0.02) | 0.08*** | (0.02) | 0.07*** | (0.02) | 0.08*** | (0.03) | 0.10*** | (0.03) | 0.09** | (0.04) |

Amb. prud. | 0.11*** | (0.02) | 0.13*** | (0.02) | 0.12*** | (0.02) | 0.12*** | (0.03) | 0.13*** | (0.03) | 0.20*** | (0.04) |

Amb. temp. | 0.07*** | (0.02) | 0.10*** | (0.02) | 0.09*** | (0.02) | 0.04 | (0.03) | 0.08*** | (0.03) | 0.11** | (0.04) |

Impact of order (ambiguity first) | ||||||||||||

Risk averse × order | 0.03 | (0.05) | 0.03 | (0.05) | ||||||||

Risk prud. × order | 0.02 | (0.04) | 0.02 | (0.04) | ||||||||

Risk temp. × order | −0.04 | (0.04) | −0.05 | (0.04) | ||||||||

Amb. avers. × order | 0.01 | (0.03) | 0.01 | (0.03) | ||||||||

Amb. prud. × order | −0.06* | (0.03) | −0.06** | (0.03) | ||||||||

Amb. temp. × order | −0.06** | (0.03) | −0.07** | (0.03) | ||||||||

Impact of the position | ||||||||||||

Risk averse × pos. | −0.07 | (0.06) | −0.07 | (0.05) | ||||||||

Risk prud. × pos. | −0.02 | (0.04) | −0.02 | (0.04) | ||||||||

Risk temp. × pos. | −0.10*** | (0.04) | −0.11*** | (0.04) | ||||||||

Amb. avers. × pos. | 0.02 | (0.03) | 0.02 | (0.03) | ||||||||

Amb. prud. × pos. | −0.04 | (0.03) | −0.04 | (0.03) | ||||||||

Amb. temp. × pos. | −0.03 | (0.03) | −0.03 | (0.03) | ||||||||

Impact of gender (male) | ||||||||||||

Risk averse × male | −0.08 | (0.05) | −0.08 | (0.06) | ||||||||

Risk prud. × male | −0.01 | (0.04) | −0.01 | (0.04) | ||||||||

Risk temp. × male | −0.09** | (0.04) | −0.09** | (0.04) | ||||||||

Amb. avers. × male | −0.01 | (0.03) | 0.01 | (0.03) | ||||||||

Amb. prud. × male | −0.03 | (0.03) | −0.02 | (0.03) | ||||||||

Amb. temp. × male | 0.05 | (0.03) | 0.05 | (0.03) | ||||||||

Impact of study field (economics or finance) | ||||||||||||

Risk averse × econ | 0.02 | (0.05) | 0.04 | (0.06) | ||||||||

Risk prud. × econ | 0.03 | (0.04) | 0.03 | (0.04) | ||||||||

Risk temp. × econ | −0.04 | (0.04) | −0.03 | (0.04) | ||||||||

Amb. avers. × econ | −0.03 | (0.03) | −0.03 | (0.03) | ||||||||

Amb. prud. × econ | −0.04 | (0.03) | −0.04 | (0.03) | ||||||||

Amb. temp. × econ | −0.02 | (0.03) | −0.03 | (0.03) | ||||||||

| 411.98*** | 446.38*** | 446.02*** | 435.79*** | 424.77*** | 508.32*** | ||||||

N | 5970 | 5970 | 5970 | 5970 | 5940 | 5940 |

In Model 1, we regress choices on binary variables capturing whether these choices concern aversion, prudence or temperance, under risk or under ambiguity. It confirms the result of the non-parametric analysis, in the sense that subjects prefer the averse/prudent/temperate option in all choices, except the risk temperate ones, where we observe risk intemperance. In Model 2, we interact the variables of Model 1 with a variable capturing whether the ambiguity part preceded the risk part or not. Starting with the ambiguity choices reduced ambiguity prudence and ambiguity temperance but did not reverse the choices. Model 3 shows that having the temperate option on the left made subjects choose it less often. Models 4 and 5 interact the variables of Model 1 with gender (being a male) and study field (studying economics or finance). It shows that the preference for risk intemperance is gender-specific: men are significantly intemperate. Thus, the often-observed gender difference in risk aversion, with men being inclined to take more risks than women (e.g., Croson and Gneezy 2009), appears to extend to temperance: compared to females, males exhibit a stronger preference towards aggregating independent zero mean risks. This result is consistent with Noussair et al. (2014), who observe that women are significantly more temperate than men in a representative sample of the Dutch population. Model 6 includes all variables. It confirms the effects previously described. The coefficient for risk temperance is positive (marginally significant) while it was negative for Model 1. This means that the reference category exhibited slight risk temperance but this concerns a very small sample.^{5}

^{6}and the subjects (Model 7 in Table 4). In all the choices of the experiment, both options had the same expected value. However, the higher this shared expected value, the less likely the subjects were to choose the averse/prudent/temperate option. Unsurprisingly, subjects were more likely to choose these options, the higher the standard deviation of the alternative option. The third variable of Model 7 is the skewness of the averse/prudent/temperate option minus the skewness of the alternative option. The larger the skewness difference, the more subjects chose the averse/prudent/temperate option. This is consistent with preferring positively skewed options to negatively skewed ones, as observed by others (e.g., Ebert and Wiesen 2011). Finally, subjects disliked higher kurtosis. We found a small gender effect, and having the averse/prudent/temperate option displayed on the left made subjects choose it less. From Table 3, we know that these effects are mainly driven by temperance.

Determinants of risk averse/prudent/temperate choices

Model 7 | ||
---|---|---|

Expected value of both options | −0.01*** | (0.00) |

Standard deviation of alternative option | 0.03*** | (0.00) |

Skewness difference | 0.18*** | (0.02) |

Kurtosis difference | 0.06*** | (0.01) |

Age | −0.00 | (0.00) |

Male | −0.05** | (0.03) |

Studying economics or finance | 0.01 | (0.02) |

Ambiguity first | −0.01 | (0.02) |

Averse/prudent/temperate option on the left | −0.07*** | (0.02) |

| 271.48*** | |

N | 2970 |

### 4.5 Mixed attitudes

To test the predictions regarding the link between the various orders of risk aversion and ambiguity aversion summarized in Table 1, we examine the behavior of risk- and ambiguity averters and risk- and ambiguity seekers separately. We classify subjects as follows. Those that choose the averse option 4 or 5 (0 or 1) out of 5 times are considered to be averse (seeking) for both risk and ambiguity. Under this classification, 10 subjects are classified as risk seeking and 157 are risk averse, while 24 subjects are classified as being ambiguity seeking and 70 subjects are ambiguity averse.

*p*-value < 0.01) while risk seekers are not significantly so (

*p*-value = 0.12). However, risk averters are temperate neutral (

*p*-value

**=**0.09), while risk seekers are risk intemperate (

*p*-value = 0.03).

Based on Wilcoxon signed-rank tests, ambiguity averters are ambiguity prudent (*p*-value < 0.01), and ambiguity temperate (*p*-value < 0.01), while ambiguity seekers are ambiguity prudent neutral (*p*-value = 0.16) and ambiguity temperate neutral (*p*-value = 0.80).

Deck and Schlesinger (2014) and Ebert and Wiesen (2014) provide further evidence for mixed risk averse behavior by considering the correlation of individual behavior between tasks of different orders. In our sample, our measurement of the degree of risk aversion is indeed positively correlated with the degree of risk temperance (Spearman ρ = 0.22; *p*-value < 0.01), which corroborates the findings of Deck and Schlesinger (2014) and Ebert and Wiesen (2014). However, we also observe a significant positive correlation between the degree of risk prudence and the degree of risk temperance (Spearman ρ = 0.24; *p*-value < 0.01), which is not in line with the mixed risk aversion pattern. For ambiguity, we only observe a significant correlation between our measurement of ambiguity prudence and ambiguity temperance (Spearman ρ = 0.19; *p*-value < 0.01).

Even though ambiguity models are silent about the link between risk and ambiguity attitudes, one may wonder whether they are correlated: if subjects consistently combine good with bad for risk and ambiguity, such a correlation should be observed. Only the correlation between risk- and ambiguity aversion is significantly positive (Spearman ρ = 0.22; *p*-value = 0.002). It was not different from 0 for the higher orders.^{7} Previous empirical findings about the correlation between risk and ambiguity aversion are mixed. Most studies found positive correlations, a few negative correlations, and some no correlations at all (e.g., Sutter et al. 2013; Trautmann and van de Kuilen 2015).

To conclude, the evidence regarding the hypothesis that DMs consistently combine good with bad for ambiguity is mixed. Even though ambiguity averters are indeed ambiguity temperate as predicted by the pattern of mixed ambiguity aversion, we do not observe a significant correlation between the *degree* of ambiguity aversion and ambiguity temperance. A potential explanation for these results is the fact that the preference to combine good with bad is not consistent over probabilities; for unlikely events, the majority of subjects prefers to combine good with good, while for likely events we observe a consistent preference to combine good with bad. More research is needed to identify the link between a preference to combine good with bad and ambiguity attitudes.

## 5 Discussion

In this study, we used an experiment to test for the prevalence of nonlinear preferences towards higher order ambiguity attitudes in a sample of 199 students. Our results show that the often-observed aversion towards ambiguity extends to prudence and temperance, which has important implications for the descriptive validity of assumptions underlying theoretical work on attitudes towards ambiguity.

Ambiguity, under which probabilities are unknown, opens up the possibility that results are affected by subjects’ beliefs. In the definitions presented in Sect. 3, we used the argument of informational symmetry, i.e., that subjects did not have any reason to expect one color to be more likely than the other if they were not explicitly told so. In ambiguity experiments, this concern is related to the risk of suspicion. A subject may suspect the experimenter to voluntarily put fewer cards of the winning color in the deck. To avoid suspicion, we made clear to subjects that they were free to change the winning color from red to black or two similarly exchange two ambiguous types of cards (kings and queens in Figs. 4b, 5b for instance). Only in 21% of all choices that option was ever chosen, suggesting that subjects were not especially suspicious, and that those that were suspicious did use the opportunity to exchange events.

Even though evidence in favor of risk prudence is strong and consistent across studies, evidence about risk temperance is mixed. Deck and Schlesinger (2010) and the present study observe intemperance, while other studies (e.g., Noussair et al. 2014; Deck and Schlesinger 2014) find evidence in favor of temperate behavior. A possible explanation for risk intemperance is probability weighting, for instance as exhibited in prospect theory (Tversky and Kahneman 1992) or rank-dependent utility (Quiggin 1981). In these models, small probabilities of high gains tend to be overweighed. For risk prudence tasks, this effect is likely to reinforce the preference for the prudent option, whose best outcome is always assigned a smaller probability than the best outcome of the imprudent option. For risk temperance, the opposite holds. By construction, the best outcome of lottery A_{r4} is assigned a smaller probability than the best outcome of lottery B_{r4}. Hence, overweighting of small probabilities points towards risk prudence and risk intemperance.

An alternative explanation for the mixed findings regarding temperance is that the temperance preference is not that strong and, hence, more likely to be affected by framing effects. The studies that have observed a strong preference for temperate lotteries clearly frame the choice between temperate lotteries as a risk apportionment task, while we present the lotteries in their reduced form. Given the theoretical importance of risk temperance and the mixed results on its prevalence obtained in the laboratory, more research into framing effects on higher-order risk attitudes is warranted.

In our experiment, the evidence for ambiguity temperance is also rather weak. This could be explained by the complexity of the tasks, which involves many sources of ambiguity (e.g. color of the cards, Aces or Jacks, and Kings or Queens in Fig. 3). The task was more cognitively demanding and therefore more prone to error or random choice. From a theoretical perspective, ambiguity temperance has been little studied but it is predicted by models like multiplier preferences (Hansen and Sargent 2001). In real life, it may influence investment strategies in the presence of many sources of ambiguity. For instance, if there are two possible states of the economy and several companies whose performance is more ambiguous in one state than in the other, an ambiguity temperate DM will compose a portfolio that does not aggregate all ambiguities in one state.

Higher order preferences are defined by relatively complex preference conditions. Our experiment, being the first to explore this domain, was designed to remain as simple as possible. Hence, we restricted ourselves to binary choices in order to detect the presence of higher order preferences, and refrained from using valuation tasks. Our conclusions therefore only address the prevalence of higher order ambiguity preferences and not to their intensity (as for instance done for higher order risk preferences by Ebert and Wiesen 2014), which is left for future research.

## 6 Conclusion

Since Ellsberg’s (1961) seminal paper, ambiguity aversion has been much studied, empirically and theoretically. Recent theoretical contributions had highlighted the role of higher order ambiguity preferences but it remained uncertain whether people actually exhibit such behavior. This paper is first to provide evidence of ambiguity prudence and ambiguity temperance. By doing so, it provided support for ambiguity models that predict ambiguity prudence. Some other ambiguity models are more general and do not specifically predict ambiguity prudence. Our results will help theorists fine-tune their models or help them choose specifications of those. For instance, in the smooth model, our results support choosing a smooth ambiguity function *φ* with a positive third derivative.

We also elicited higher risk attitudes and could analyze how each order of behavior correlates between risk and ambiguity and, within each domain, with the other orders. We found some, but limited, support for mixed behavior, i.e., the overall tendency to combine good with bad.

Finally, we have demonstrated how to measure ambiguity prudence and temperance. The tasks we introduced can be, for instance, added to survey on prevention behavior to better understand the role of ambiguity in prevention behavior.

## Footnotes

- 1.
These results can be contrasted with those obtained for ambiguity aversion. Alary et al. (2013) provided conditions in which ambiguity aversion may actually decrease prevention efforts.

- 2.
Throughout, we only consider lotteries leading to positive final wealth, to avoid bankruptcy considerations.

- 3.
All Wilcoxon signed-rank tests reported in this section are based on the number of times subjects chose the averse/prudent/temperate option compared to 2.5. All Wilcoxon signed-rank test results are replicated with Fisher-Pitman permutation tests.

- 4.
Clustering standard errors at the individual level. All results were the same with probit regressions (see Appendix C of electronic supplementary material).

- 5.
It concerns female participants who did not study finance nor economics, started with the risk tasks and had the temperate option on the right of the screen.

- 6.
The characteristics of the lotteries are reported in Appendix A of electronic supplementary material.

- 7.
ρ = 0.01,

*p*value = 0.89 for prudence; ρ = 0.06,*p*-value = 0.41 for temperance.

## Notes

### Acknowledgments

Due to his untimely death, Harris Schlesinger did not have the opportunity to (dis)approve the final version of the paper. This article is dedicated to his memory. We are grateful to Han Bleichrodt, Sebastian Ebert, the editor, and two anonymous reviewers for helpful comments and suggestions. Funding was provided by NWO (Grant No. 452-13-011 and Grant No. 452-13-013).

## Supplementary material

### References

- Abdellaoui, M., Klibanoff, P., & Placido, L. (2015). Experiments on compound risk in relation to simple risk and to ambiguity.
*Management Science,**61*(6), 1306–1322.CrossRefGoogle Scholar - Alary, D., Gollier, C., & Treich, N. (2013). The effect of ambiguity aversion on insurance and self-protection.
*Economic Journal,**123,*1188–1202.CrossRefGoogle Scholar - Baillon, A. (2017). Prudence with respect to ambiguity.
*Economic Journal*. doi: 10.1111/ecoj.12358. - Caballé, J., & Pomansky, A. (1996). Mixed risk aversion.
*Journal of Economic Theory,**71*(2), 485–513.CrossRefGoogle Scholar - Chateauneuf, A., Eichberger, J., & Grant, S. (2007). Choice under uncertainty with the best and worst in mind: Neo-additive capacities.
*Journal of Economic Theory,**137,*538–567.CrossRefGoogle Scholar - Crainich, D., Eeckhoudt, L., & Trannoy, A. (2013). Even (mixed) risk lovers are prudent.
*American Economic Review,**103*(4), 1529–1535.CrossRefGoogle Scholar - Croson, R., & Gneezy, U. (2009). Gender differences in preferences.
*Journal of Economic Literature,**47,*448–474.CrossRefGoogle Scholar - Deck, C., & Schlesinger, H. (2010). Exploring higher-order risk effects.
*Review of Economic Studies,**77,*1403–1420.CrossRefGoogle Scholar - Deck, C., & Schlesinger, H. (2014). Consistency of higher order risk preferences.
*Econometrica,**82*(5), 1913–1943.CrossRefGoogle Scholar - Ebert, S., & Wiesen, D. (2011). Testing for prudence and skewness seeking.
*Management Science,**57,*1334–1349.CrossRefGoogle Scholar - Ebert, S., & Wiesen, D. (2014). Joint measurement of risk aversion, prudence, and temperance.
*Journal of Risk and Uncertainty,**48,*231–252.CrossRefGoogle Scholar - Eeckhoudt, L., & Schlesinger, H. (2006). Putting risk in its proper place.
*American Economic Review,**96,*280–289.CrossRefGoogle Scholar - Ellsberg, D. (1961). Risk, ambiguity and the savage axioms.
*Quarterly Journal of Economics,**75,*643–669.CrossRefGoogle Scholar - Ghirardato, P., Maccheroni, F., & Marinacci, M. (2004). Differentiating ambiguity and ambiguity attitude.
*Journal of Economic Theory,**118,*133–173.CrossRefGoogle Scholar - Gilboa, I., & Schmeidler, D. (1989). Maxmin expected utility with a non-unique prior.
*Journal of Mathematical Economics,**18,*141–153.CrossRefGoogle Scholar - Guerdjikova, A., & Sciubba, E. (2015). Survival with ambiguity.
*Journal of Economic Theory,**155,*50–94.CrossRefGoogle Scholar - Haering, A., Heinrich, T., & Mayrhofer, T. (2017).
*Exploring the consistency of higher-order risk preferences*. Working paper. University of Duisburg.Google Scholar - Halevy, Y. (2007). Ellsberg revisited: An experimental study.
*Econometrica,**75,*503–536.CrossRefGoogle Scholar - Hansen, Lars P., & Sargent, Thomas J. (2001). Robust control and model uncertainty.
*American Economic Review,**91*(2), 60–66.CrossRefGoogle Scholar - Hansen, Lars P., & Sargent, Thomas J. (2007). Recursive robust estimation and control without commitment.
*Journal of Economic Theory,**136*(1), 1–27.CrossRefGoogle Scholar - Johnson, C., Baillon, A., Bleichrodt, H., Li, Z., van Dolder, D., & Wakker P. P. (2014).
*Prince: An improved method for measuring incentivized preferences*. Working paper, ESE Rotterdam.Google Scholar - Kimball, Miles S. (1990). Precautionary saving in the small and in the large.
*Econometrica,**58*(1), 53–73.CrossRefGoogle Scholar - Klibanoff, P., Marinacci, M., & Mukerji, S. (2005). A smooth model of decision under uncertainty.
*Econometrica,**73,*1849–1892.CrossRefGoogle Scholar - Knight, F. H. (1921).
*Risk, uncertainty, and profit*. New York, NY: Houghton Mifflin.Google Scholar - Leland, H. E. (1968). Saving and uncertainty: The precautionary demand for saving.
*Quarterly Journal of Economics,**82*(3), 465–473.CrossRefGoogle Scholar - Maccheroni, F., Marinacci, M., & Rustichini, A. (2006). Ambiguity aversion, robustness, and the variational representation of preferences.
*Econometrica,**74,*1447–1498.CrossRefGoogle Scholar - Maier, J., & Rüger M. (2012).
*Experimental evidence on higher order risk attitudes with real monetary losses*. Working paper, LMU Munich. https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxqb2hhbm5lc2ttYWllcnxneDpmYjY1OTllZDYzMzViZmE. - Noussair, C. N., Trautmann, S. T., & van de Kuilen, G. (2014). Higher order risk attitudes, demographics, and financial decisions.
*Review of Economic Studies,**81*(1), 325–355.CrossRefGoogle Scholar - Oechssler, J., & Roomets, A. (2014). Unintended hedging in ambiguity experiments.
*Economics Letters,**122,*243–246.CrossRefGoogle Scholar - Quiggin, J. (1981). Risk perception and risk aversion among Australian farmers.
*Australian Journal of Agricultural Economics,**25,*160–169.CrossRefGoogle Scholar - Sandmo, A. (1970). The effect of uncertainty on saving decisions.
*Review of Economic Studies,**37,*353–360.CrossRefGoogle Scholar - Schmeidler, D. (1989). Subjective probability and expected utility without additivity.
*Econometrica,**57,*571–587.CrossRefGoogle Scholar - Sutter, M., Kocher, M. G., Glätzle-Rüetzler, D., & Trautmann, S. T. (2013). Impatience and uncertainty: Experimental decisions predict adolescents’ field behavior.
*American Economic Review,**103,*510–531.CrossRefGoogle Scholar - Tarazona-Gomez, M. (2004). Are individuals prudent? An experimental approach using lottery choices.
*mimeo*. http://www2.toulouse.inra.fr/lerna/cahiers2005/05.13.177.pdf. - Trautmann, S. T., & van de Kuilen, G. (2015). Ambiguity attitudes. Forthcoming In G. Keren & G. Wu (eds.),
*Blackwell handbook of judgment and decision making*.Google Scholar - Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty.
*Journal of Risk and Uncertainty,**5,*297–323.CrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.