Journal of the Economic Science Association

, Volume 1, Issue 2, pp 146–157 | Cite as

Naive play and the process of choice in guessing games

  • Marina Agranov
  • Andrew Caplin
  • Chloe TergimanEmail author
Original Paper


There is growing evidence that not all experimental subjects understand their strategic environment. We introduce a “choice process” (CP) protocol that aids in identifying these subjects. This protocol elicits in an incentive compatible manner provisional choices as players internalize their decision making environment. We implement the CP protocol in the modified 2/3 guessing game and use it to pinpoint players that are naive by identifying those who make weakly dominated choices some time into the play. At all time horizons these players average close to 50. This is consistent with the assumption in Level-K theory that the least sophisticated subjects (the naive ones) play uniformly over the [1–100] action space. In contrast, sophisticated players show evidence of increased understanding as time passes. We find that the CP protocol mirrors play in multiple setups with distinct time constraints. Hence it may be worth deploying more broadly to understand the interaction between decision time and choice.


Experiment 2/3 Guessing game Level 0 play 

JEL Classification

D03 C81 C91 

1 Introduction

Not all experimental subjects fully understand the decision making environments in which they are placed. A striking indication of this can be found in the auction setting, in which Ivanov et al. (2010) find that many subjects are unable to best-respond even to their own past play. Similarly, Agranov et al. (2012) find that roughly 50 % of the subjects are unable to pick rationally when playing against a computer whose strategy they have been told.1

Given how prevalent they may be, it is important to identify players who are unaware of their decision making environment. We introduce a “choice process” (CP) protocol that helps identifying these players and apply it to the 2/3 guessing game (Nagel 1995; Stahl 1996; Ho et al. 1998; Camerer et al. 2004). In this game, all players choose a number between 0 and 100 and the winner is the person whose number is closest to 2/3 of the average of the numbers that were chosen by the people in the experiment. “Level-K” theory has been applied to this game to account for the significant clustering around 33 and 22 when \(0\) is the only Nash Equilibrium. This theory assumes that naive “level zero” (L0) players make choices that are uniformly distributed over the range [0, 100], hence averaging 50. Sophisticated type L2 players who pick numbers close to 22 are interpreted as best responding to type L1 players, who pick 33 in response to the assumed average of 50 associated with naive play. However, sophistication is not revealed by final choices alone: 33 could be the result of a random choice or the best response to a belief that others are averaging 50.

To overcome this identification problem, we study decision problem that mirrors the essence of the 2/3 guessing game described above. Subjects in our experiment are playing against a group of subjects who have played the standard 2/3 guessing game in the past, and, are rewarded if their selected number is closer to the 2/3 of the average number chosen by those who played the game in the past. Importantly, in addition to measuring the final choice, our CP protocol measures players’ provisional choices in the 3 min period after the task is conveyed to them. The procedure is simple: subjects make choices over a period of time and which choice matters for payment is randomly determined. Since it is no longer only the final choice that is potentially rewarded, subjects are incentivized to make what they perceive as the best decision at each point in time. Since it measures choices alone and is incentive compatible, it may be easier to map to theory than verbal reports, mouse-clicks, and physiological data of the sort that is increasingly being used to identify strategic awareness.2

We identify naive subjects in the game as those players who make weakly dominated choices (above 64) after having had some time to consider the decision.3 We find a high proportion of such naive players (approximately 45 %).4 Consistent with assumptions made in most empirical papers, we find that these players make final choices that average close to 50, and do so during much of the 3 min observation period. Our method allows us to categorize and identify naive players prior to testing best response dynamics.

The remainder of the paper is organized as follows. The experimental design can be found in Sect. 2. Results concerning the prevalence and patterns of naive and sophisticated play are in Sect. 3. We discuss the connection between the CP protocol and series of time-constrained tasks in Sect. 4. Concluding remarks are in Sect. 5.

2 The experimental design

Experiments were conducted at the laboratory of the Center for Experimental Social Science (CESS) at New York University with participants drawn from the general undergraduate population. Given our interest in how learning takes place in a novel one shot game, in each session subjects performed a 2/3 guessing task one time only and we dropped subjects who reported being familiar with guessing game in a questionnaire conducted at the end of the experiment. Our sample consists of 188 subjects after removing about 25 % of subjects who had either played the game or heard of it. All sessions lasted no longer than 40 min and subjects earned between $10 and $15.

We conducted three different treatments: the control treatment of 30 s, the control treatment of 180 s and the CP treatment. Full instructions for the CP treatment are presented in the Online Appendix. In all treatments, the rules of the game and the task were as follows:

Rules of the game A few days ago eight undergraduate students like yourselves played the following game. Each of the 8 students had 180 s to choose an integer between 1 and 100 inclusive, which they wrote on a piece of paper. After 180 s, we collected the papers. The winner was the person whose number was closest to two thirds of the average of everyone’s numbers. That is, the eight students played among themselves and their goal was to guess two thirds of the average of everyone’s numbers. The winner won $10 and in case of a tie the prize was split.

Your task You will have 30 (180) s to choose an integer between 1 and 100 inclusive. You win $10 if you are “better than” those eight students at determining two thirds of the average of their numbers. That is, you win $10 if your number is the closest to two thirds of the average of the numbers in the past game.

We indeed conducted an 8-player “standard” guessing game prior to these series of experiments and used the data in the way described to the subjects in this current paper.

In all sessions, the screen displayed 100 buttons, each representing an integer between 1 and 100 inclusive.5 In the control treatment, only the final choice of each subject mattered for payment. The control treatment sessions differed in the amount of time given to subjects to make their choice: 30 or 180 s. During available time subjects were free to select buttons as many times as they wished but only the final number they chose affected their payment. The longer time of 180 s was chosen since prior work suggests that it is enough time for most subjects to reason through the game, while the shorter time of 30 s was chosen to cut short such reasoning. We conducted these control sessions not only to gauge the importance of decision time in the outcome of the game, but also to provide benchmarks with which to compare the main CP treatment. In total, 66 subjects participated in the 30 s control treatment and 62 participated in the 180 s control treatment.

Our main treatment, the CP treatment, lasted for 180 s and included 60 subjects. In CP treatment, subject’s payment was determined based on the number selected by subject at a random second; the exact second that mattered for payment was drawn by the computer at the end of the task and, thus, not known in advance to subjects. In other words, it was a choice of a subject at a random second that was compared to the choices of eight subjects that played the game previous to the experiment. The CP payment mechanism incentivizes subjects to make a quick and intuitive first estimate of two-thirds of the average final number picked by the group that had played the guessing game previously. Whenever further reflection causes this best estimate to change, subjects are incentivized immediately to make the corresponding change in their guess to reduce the likelihood that their previous guess (which they came to realize was inferior to the current estimate) be chosen for payment.

2.1 Discussion of the CP methodology

The main advantages of the CP protocol is that it is a choice experiment, which is simple to explain to subjects and it allows to elicit in an incentive compatible manner the whole path of individual thought in addition to the final choice. One may, however, wonder if this protocol nudges subjects to change their guesses beyond their natural inclination to do so given that instructions emphasize that subjects can change their estimates as often as they wish. Our results presented in Sect. 4 show that the CP protocol does not suffer from demand effect of this sort (see Figure 2 in Online Appendix which presents the distribution of subjects’ final choices in the CP and in the control treatment).

Another potential concern is probability matching—the tendency of some people to probability match and choose actions proportionally especially in the environments in which the best action is probabilistic (see Rubinstein 2002). Such behavior may be exaggerated by the CP protocol if subjects are uncertain about the average number chosen by the eight players that played the game previously since subjects are asked to make the same choice over and over (in every second of the 3 min interval). While it is impossible to rule this out for subjects who change their choices during the experiment, we point out that our main classification of naive players is one that does not rely on any assumption regarding switching patterns of play. Further, in Sect. 4 (see Finding 5) we describe an alternative method for identifying naive types, which yields results comparable with the ones obtained using the main classification criterion.

The third issue is whether the stakes in our CP treatment were high enough to ensure that the desire for the monetary prize dominated systematic non-pecuniary preferences, which is an important premise of induced value theory. Indeed, since every second of the 3 min interval is equally likely to be selected for payment, subjects face arguably small stakes at any second during the CP treatment. Our results, however, show that the distribution of final choices in the 30 (180) s control treatment and the CP treatment at second 30 (180) are statistically indistinguishable, which suggests that the change of stakes had no significant effect on the observed behavior.

Finally, we note that our experiment is the individual decision problem rather than the game. Indeed, our subjects are guessing the 2/3 of the average number chosen by eight subjects in the past and, therefore, choices made by subjects in our main treatments do not affect payoffs of those who played the game in the past. This transformation from the game into the individual decision problem is necessary to guarantee that the CP protocol is incentive compatible and, importantly, it preserves the essence of the 2/3 guessing game both in terms of subjects’ goal and the framing of the task.

3 Naive and sophisticated players

In standard choice data, that is in data that only asks for final choice alone, one has no information as to whether the decision made by a subject was the result of random choice or the result of a deliberate thought process. In other words, final choice does not adequately summarize strategy, particularly if learning is taking place. To illustrate this point, Fig. 1 presents three individuals whose final choice is the same (\(33\)), and who would be classified as L1 thinkers if only their final choice was observed. However, the manner in which they arrived at this final choice is dramatically different, and may contain information of value in understanding their behavior.6
Fig. 1

Is final choice enough? Paths of choice of three subjects with the same final choice

Finding 1 More than 45 % of the population is naive.

As we described earlier choosing a number above 64 is a weakly dominated action. We use this strong theory-driven cutoff to define naive players. Figure 2 presents the histogram of the last time each subject chose numbers above 64. For instance, someone who never chose numbers above 64 is counted in the 0 bin on this graph, while a person whose final choice is above 64 appears in the 180 bin.
Fig. 2

Last time each subject chose numbers above 64 in the CP experiments. Note for example, if a subject is in the “0” bin it means that he has never chosen a number above 64. A subject is in the “20” bin if the last time he chose a number above 64 was at second 20

About 40 % of subjects never chose a number above 64. However, it is plausible that some subjects make first choices that are instinctive, before even internalizing the structure of the game. Thus, we define naive subjects as those who chose above 64 at some point 40 s or more into the experiment.7

According to our definition, 46.7 % of subjects are naive players. This may underestimate the true number of naive subjects in the population since such subjects may never choose numbers above 64. Yet the proportion is consistent with other work that studies different games, such as Costa-Gomes and Weizsacker (2008), Ivanov et al. (2010) and Agranov et al. (2012).

One may ask at this point whether we really need CP data to identify naive players. The answer is unequivocal: if one were to look only at the last choice made by subjects, all that is possible with standard choice data, about 10 % (5 subjects) of subjects made choices above 64. Hence, without the CP data, one is likely to dramatically underestimate the proportion of naive individuals in the population.8

Finding 2 Naive players average close to 50 (a) at the end of the game and (b) throughout the game.
Fig. 3

Choices of naive players

Consistent with the standard assumption on naive play, 50 is within the 95 % confidence interval for the mean final choice (the mean is just above 43.5, and the median is 41).9 We now consider the behavior of these players throughout the game.

Summary statistics on naive play are presented in Table 1. Over the course of the entire experiment, the average choice of naive players is 51.6 and the average choice of naive players remains fairly stable over the course of the experiment. In addition, the group of naive players spends close to a third of the time on choices above 64 throughout the whole experiment.

The first three panels of Fig. 3 depict the histograms of all the choices made by naive players in the first 60, second 60 and third 60 s of the experiment, treating each choice for a given subject as an independent observation. In the last panel, we present the average choice of naive players during the course of the experiment along with the 95 % confidence interval for the mean choice.
Table 1

Choices of naive players


Naive players

Mean choice

Time above 64 (%)

Seconds 1–60

53.2 (15.6)


Seconds 61–120

50.3 (11)


Seconds 121–180

51 (14.4)


Standard deviations in parentheses

It is quite striking that 50 is within the 95 % confidence interval for average play of the naive types over 90 % of the time. These players average close to 50 throughout the experiment. Hence they fit the intuitive conception of naive players through most of the period of play.

Our criterion to determine the naive subjects is the play of a weakly dominated strategy. This does not in itself have strong implications for the path of choice. Certainly, it does not imply that choices for the naive types should remain constant at close to 50 over the course of the experiment. We interpret the result as providing robust support for the assumption that naive players choices average 50 not only at the end of the game but throughout the game.

Finding 3 Sophisticated players’ choices decrease over time.

In this section we focus on those subjects who never or almost never choose weakly dominated actions. We refer to these subjects as the sophisticated subjects.

Figure 4 displays the average choice as a function of consideration time as well as the fitted regression lines using fractional-polynomial formulations in the CP treatment for those two groups. The coefficient on time in a random effects linear regression model shows that sophisticated players substantially decrease their choices over the course of the experiment (the coefficient on time is is \(-\)4.6 % and significant at the 1 % level), while naive players decrease their choices only slightly (the coefficient on time is \(-\)1.7 % and significant at the 1 % level).
Fig. 4

Choice over time: average data with fractional polynomial regressions superimposed

Table 2

Some statistics on sophisticated and naive players


Naive players

Sophisticated players

Average choice

51.6 (21.3)

33.6 (16.4)

Average time of first choice

6.5 (3.5)

7.8 (5.6)

Average total number of switches

51.2 (56)

9.3 (18.6)

Average time of last choice

162.9 (29.4)

97.4 (68)

Table 2 presents some statistics on the sophisticated and naive types. What is clear from Table 2 and Fig. 4 is that these types differ in aspects that are in some way removed from the defining criterion of playing a weakly dominated strategy. While time of first choice does not differ between these groups, naive subjects “change their minds” and switch numbers much more often than the sophisticated subjects. Further, the naive subjects keep changing their minds almost to the end of the experiment, which is not the case for the sophisticated ones. The statistical analyses of the above claims were conducted using Wilcoxon rank sum and Kolmogorov–Smirnov tests, with one observation per subject and a significance level of 5 %. In other words, our sophisticated subjects, while not defined that way, seem to converge faster to a final decision, making their decisions possibly more deliberate than those of the naive ones.

4 CP and other beauty contest games

In this section we present findings suggesting the value of the CP data in other settings. We first compare behavior in the CP treatment with that in 2/3 guessing tasks with distinct time constraints and show that the CP protocol does not interfere with choice. Finally, we present results suggesting that the CP protocol may be able to identify naive players in a broad class of games for which there is no weakly dominated strategy. We refer the reader to the Online Appendix for a detailed literature review of the papers that explore the connection between contemplation time and choice in guessing games. The most related papers are those of Weber (2003), Kocher and Sutter (2006) and Rubinstein (2007).

Finding 4 The CP protocol is equivalent to a series of time constrained decision problems.

By definition, a subject can play an unfamiliar game one and only one time. This poses a challenge for those seeking to understand how the contemplation period interacts with the final decision. For one so interested, the standard procedure (between subject design) requires the use of separate pools of subjects for each time constraint, which is rather costly.
Table 3

Summary statistics of choices in control and CP treatments (standard deviation in parenthesis) and in prior literature


# of obs

Mean choice

Median choice

30 s, control


42.8 (20.1)


30 s, CP


41.7 (20)


180 s, control


36.4 (20.2)


180 s, CP


36.7 (18.8)


Ho et al. (1998, \(p=0.7\))


38.9 (24.7)


Nagel (1995, \(p=2/3\))


36.7 (20)


The CP experiment is designed to elicit from one individual an entire sequence of time-constrained choices in their very first play of an unfamiliar game. The extent to which this design provides information on how time constraints impact play depends on whether or not the choices it gives rise to are different than those in the corresponding sequence of time-constrained decision problems.

Table 3 displays a powerful similarity between the CP and control experiments of equivalent horizon (\(p>0.10\) in both Wilcoxon rank sum and Kolmogorov–Smirnov tests, where we use one observation per subject).10 Finally, the results at 180 s in the CP treatment and in the 180 s control treatment are not only similar to one another, but also similar to those identified in the pioneering work of Nagel (1995) and Ho et al. (1998).

We conclude that indeed there are strong similarities between CP data and data on the corresponding sequence of time-constrained decision problems. To a first approximation, the CP treatment appears to be equivalent to multiple guessing tasks with different time constraints. In addition to allowing one to economize on the number of plays, using the CP treatment to explore the impact of time constraints removes the need to control for individual differences: the players are one and the same regardless of the time constraint.

Finding 5 The CP protocol can be used to identify naive players based on switching patterns.

Table 2 showed that naive players that we identify with our dominated strategy methods tend to make many switches and to keep changing their choices until the end of the experiment. In games where there is no weakly dominated strategy, this suggests the possibility of identifying naive players directly off the pattern of switching behavior.
Table 4

Relationship between playing above 64 and standard deviation in second half of the experiment


Correlation w/sophistication (no play above 64)

\(\%\) Who don’t play above 64 (%)

No. of observations

Sophistication (Std. dev \(=0\))




Sophistication (0 \(<\) Std. dev \(\le 10\))




Sophistication (10 \(<\) Std. dev \(\le 20\))




Sophistication (Std. dev \(>20\))




Figure 5 shows the distribution of standard deviation of choices in the second half of the experiment. As Table 4 shows, a relatively high variance in choice is negatively correlated with being sophisticated according to the weakly dominated criterion.
Fig. 5

Standard deviation of choices in the second half of the experiment

While our experiment was not designed with the “convergence” criterion in mind, sophisticated players can be identified by using behavioral switches, and it may be possible to use measures of convergence to identify naive players in other games. Such measures are likely to be game-specific and to be guided by specific theories of strategic decision making.

5 Conclusions

We introduce a new experimental protocol to provide information on provisional choices in games, and hence the process of decision-making. We implement our CP treatment in the decision problem that closely mimics the essence of 2/3 guessing game and use it to identify naive players. We find that there is a high proportion of such players, that their choices average close to 50 for most of the 3 min observation period, and that they differ in systematic fashion from the remaining sophisticated players. The CP methodology may be useful in understanding patterns of strategic play in a variety of other games.


  1. 1.

    See also Costa-Gomes and Weizsacker (2008) who find that subjects’ reported beliefs are often not consistent with chosen actions in two-person simultaneous games and De Sousa et al. (2014) who find that a sizable fraction of highly ranked chess players behave non-strategically in rather simple experimental settings.

  2. 2.

    We survey other choice procedure methods in the Online Appendix.

  3. 3.

    In our setup, action 100 is strictly dominated, while all actions between 65 and 99 are weakly dominated as there exists a strategy (playing 64 against seven other players playing between 65 and 99) which does as well or better for all possible states of the world.

  4. 4.

    See Agranov et al. (2012) for a different method of estimating the proportion of naive players in the population.

  5. 5.

    Our experimental apparatus—displaying all the possible choices on the screen—makes the restriction to integers a necessary one. Having the minimum choice be 1 guarantees that the unique optimal choice in our CP and control treatments is to select 1, provided that one believes that players in the past all select 1, which is the unique Nash equilibrium of the game.

  6. 6.

    All complete paths of choice are presented in the Online Appendix.

  7. 7.

    All the qualitative results in this paper follow through if we instead choose 60 or 90 s as the cutoff for defining naive subjects.

  8. 8.

    If subjects who are naive play uniformly over the action space, then we would expect that at the last second of the game about one third of those naive subjects choose a number above 64. In our data, we find that this fraction, 22 %, is slightly lower than one third. This indicates that while naive players average 50, they do not necessarily play uniformly.

  9. 9.

    See Online Appendix for the histogram of final choices.

  10. 10.

    More evidence is presented in the Online Appendix.



We thank Jim Andreoni, Colin Camerer, Mark Dean, John Duffy, Martin Dufwenberg, Guillaume Frechette, Drew Fudenberg, Sen Geng, P.J. Healy, Daniel Martin, Rosemarie Nagel, Muriel Niederle, Stefan Penczynski, Andy Schotter, Lise Vesterlund, Roberto Weber, the seminar participants at the Experimental Economics seminar at NYU, at the UCLA Theory Workshop, and at the Sauder School of Business at UBC.

Supplementary material

40881_2015_3_MOESM1_ESM.pdf (1015 kb)
ESM1 (PDF 1,015 kb)


  1. Agranov, M., Potamites, E., Schotter, A., & Tergiman, C. (2012). Beliefs and endogenous cognitive levels: An experimental study. Games and Economic Behavior, 75, 449–463.CrossRefGoogle Scholar
  2. Camerer, C., Ho, T.-H., & Chong, J.-K. (2004). A cognitive hierarchy model of games. The Quarterly Journal of Economics, 119(3), 861–898.CrossRefGoogle Scholar
  3. Costa-Gomes, M., & Weizsacker, G. (2008). Stated beliefs and play in normal-form games. Review of Economic Studies, 75(3), 729–762.CrossRefGoogle Scholar
  4. De Sousa, J., Hollard, G., Terracol, A. (2014). Non-strategic players are the rule rather than the exception. Working Paper.Google Scholar
  5. Ho, T.-H., Camerer, C., & Weigelt, K. (1998). Iterated dominance and iterated best-response in p-beauty contests. The American Economic Review, 88, 947–969.Google Scholar
  6. Ivanov, A., Levin, D., & Niederle, M. (2010). Can relaxation of beliefs rationalize the winner’s curse? An experimental study. Econometrica, 78(4), 1435–1452.CrossRefGoogle Scholar
  7. Kocher, M., & Sutter, M. (2006). Time is money—Time pressure, incentives, and the quality of decision-making. Journal of Economic Behavior and Organization, 61(3), 375–392.CrossRefGoogle Scholar
  8. Nagel, R. (1995). Unraveling in guessing games: An experimental study. The American Economic Review, 85(5), 1313–1326.Google Scholar
  9. Rubinstein, A. (2002). Irrational diversification in multiple decision problems. European Economic Review, 46, 1369–1378.CrossRefGoogle Scholar
  10. Rubinstein, A. (2007). Instinctive and cognitive reasoning: A study of response times. Economic Journal, 117, 1243–1259.CrossRefGoogle Scholar
  11. Stahl, D. (1996). Boundedly rational rule learning in a guessing game. Games and Economic Behavior, 16, 303–330.CrossRefGoogle Scholar
  12. Weber, R. (2003). Learning with no feedback in a competitive guessing game. Games and Economic Behavior, 44(1), 134–144.CrossRefGoogle Scholar

Copyright information

© Economic Science Association 2015

Authors and Affiliations

  1. 1.California Institute of Technology and the Center for Experimental Social SciencePasadenaUSA
  2. 2.New York University and the Center for Experimental Social ScienceNew YorkUSA
  3. 3.University of British Columbia and the Center for Experimental Social ScienceVancouverCanada

Personalised recommendations