Skip to main content
Log in

Internal and External Validity in Experimental Games: A Social Reality Check

  • Original Article
  • Published:
The European Journal of Development Research Aims and scope Submit manuscript

Abstract

The relevance of experimental games as methods in development research depends crucially on how far the results from the games can be extrapolated to real life, that is, the external validity of those results. The extent to which external validity matters depends on what you want to do with the data; some kinds of theory testing can arguably afford indifference, but many experiments are used as an indicator of behaviour in everyday life. This article takes an anthropological perspective on both the internal and external validity of social preference experiments in developing country settings, and argues for more cautious knowledge claims, triangulation of data and a broader conceptualisation of norms and behaviour.

La pertinence des jeux expérimentaux en tant que méthodes dans la recherche sur le développement dépend fondamentalement de la mesure dans laquelle les résultats obtenus à partir de ces jeux s’appliquent à la réalité, c’est à dire de la validité externe de ces résultats. La validité externe est plus ou moins importante selon ce que l’on souhaite faire avec les données. Certains types de jeux testent des théories sans se soucier de leur validité externe, mais d’autres sont tenus comme étant des indicateurs de comportements quotidiens. Cet article examine à partir d’une perspective anthropologique les validités tant internes qu’externes d’expériences de préférences sociales dans un contexte de pays en développement, et plaide en faveur de revendications théoriques plus prudentes, de triangulations de données, et d’une conceptualisation plus large des normes et des comportements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. This is derived from an index composed of ‘payoffs to cooperation’, that is, the extent of reliance on cooperation outside family level; ‘market integration’, that is, extent of dependence on markets; ‘anonymity’, that is, the extent to which people interact with strangers they may never see again; ‘privacy’, that is, how possible it is to keep secrets from others; and ‘sociopolitical complexity’, that is, how much centralised decision making occurs above the level of the household.

  2. Although Ensminger for the Orma finds that those engaged in wage labour and trade were more likely to engage in fair behaviour than subsistence producers.

  3. Henrich and Smith's players all knew each other and the researcher well, that is, questionable anonymity, yet they cooperated less than the US students interacting with strangers (2004, p. 156).

  4. For example, meat is more likely to be evaluated positively if labelled ‘75 per cent lean’ compared to ‘25 per cent fat’.

  5. For example, subjects are more willing to accept breast self-examination if the same outcomes are described negatively compared to positively (1998, p. 168).

  6. Levitt and List (2007) cite evidence behaviour experiments, bid the same as men when they are menstruating, but differently from men when experiencing higher oestrogen levels (Chen et al 2005). They argue that women who self-select as volunteers are thus possibly more pro-social than non-volunteer women.

  7. Not all games were played with money.

References

  • Alvard, M. (2004) The Ultimatum Game, fairness and cooperation among big game hunters. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 413–435.

    Chapter  Google Scholar 

  • Barr, A. (2004) Kinship, familiarity, and trust: An experimental investigation. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 305–334.

    Chapter  Google Scholar 

  • Benz, M. and Meier, S. (2008) Do people behave in experiments as in the field? Evidence from donations. Experimental Economics 11 (3): 268–281.

    Article  Google Scholar 

  • Bloch, M. and Parry, J. (eds.) (1989) Money and the Morality of Exchange. Cambridge: CUP.

    Google Scholar 

  • Camerer, C. and Fehr, E. (2004) Measuring social norms and preferences using experimental games: A guide for social scientists. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 55–95.

    Chapter  Google Scholar 

  • Cardenas, J.C. and Carpenter, J. (2008) Behavioural development economics: Lessons from field labs in the developing world. Journal of Development Studies 44 (3): 311–338.

    Article  Google Scholar 

  • Charness, G. and Kuhn, P. (2010) Lab Labor: What Can Labor Economists Learn from the Lab? National Bureau of Economic Research, Working Paper no. w15913.

  • Chen, Y., Katuscak, P. and Ozdenoren, E. (2005) Why Can’t a Woman Bid More Like a Man? Department of Economics, University of Michigan, Working Paper.

  • Croson, R. and Gneezy, U. (2009) Gender differences in preferences. Journal of Economic Literature 47 (2): 448–474.

    Article  Google Scholar 

  • Doty, R. and Silverthorne, C. (1975) Influence of menstrual cycle on volunteering behaviour. Nature 254 (5496): 139–140.

    Article  Google Scholar 

  • Douglas, M . (1966) Purity and Danger: an Analysis of Concepts of Pollution and Taboo. London: Routledge and Kegan Paul.

    Book  Google Scholar 

  • Eckel, C. (2008) The gender gap: Using the lab as a window on the market. In: F. Bettio and A. Verashchagina (eds.) Frontiers in the Economics of Gender (Routledge Siena Studies in Political Economy), Oxon: Routledge, pp. 223–242.

  • Ensminger, J. (2004) Market integration and fairness: Evidence from Ultimatum, Dictator and Public Goods Experiments in East Africa. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 356–381.

    Chapter  Google Scholar 

  • Falk, A. and Heckman, J. (2009) Lab experiments are a major source of knowledge in the social sciences. Science 326 (5952): 535–538.

    Article  Google Scholar 

  • Fletschner, D., Anderson, C.L. and Cullen, A. (2010) Are women as likely to take risks and compete? Behavioural findings from Central Vietnam. Journal of Development Studies 46 (8): 1459–1479.

    Article  Google Scholar 

  • Giddens, A. (1984) The Constitution of Society. Cambridge: Polity Press.

    Google Scholar 

  • Gil-White, F. (2004) Ultimatum Game with an ethnicity manipulation; results from Khovdiin Bulgan Sum, Mongolia. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 260–304.

    Chapter  Google Scholar 

  • Goffman, E. (1983) The interaction order: American Sociological Association, 1982 Presidential Address. American Sociological Review 48 (1): 1–17.

    Article  Google Scholar 

  • Goffman, E. (1961) Encounters: Two Studies in the Sociology of Interaction. Indianapolis, IN: Bobbs Merrill.

    Google Scholar 

  • Guala, F. (2005) The Methodology of Experimental Economics. Cambridge: CUP.

    Book  Google Scholar 

  • Guardian. (2010) Colombia tries to stop people drawing money. 10 April.

  • Gurven, M. (2004) Does market exposure affect economic game behaviour? The Ultimatum Game and Public Goods Game among the Tsimane’ of Bolivia. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 194–321.

    Chapter  Google Scholar 

  • Haley, K. and Fessler, D. (2005) Nobody's watching? Subtle cues affect generosity in an anonymous economic game. Evolution and Human Behaviour 26 (3): 245–256.

    Article  Google Scholar 

  • Hamoudi, A. and Thomas, D. (2006) Do You Care? Altruism and Inter-generational Exchanges in Mexico. California Centre for Population Research, On-line Series of Working Papers CCPR-008-06.

  • Henrich, J. and Smith, N. (2004) Comparative experimental evidence from Machiguenga, Mapuche, Huinca and American populations. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 125–167.

    Chapter  Google Scholar 

  • Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E. and Gintis, H. (eds.) (2004a) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press.

    Book  Google Scholar 

  • Henrich, J. et al (2004b) Overview and synthesis. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 8–55.

    Chapter  Google Scholar 

  • Hill, K. and Gurven, M. (2004) Economic experiments to examine fairness and cooperation among the Ache Indians of Paraguay. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, p. 382.

    Chapter  Google Scholar 

  • Hirschman, A . (1982) Rival Interpretations of market society: civilising, destructive or feeble. Journal of Economic Literature 20 (4): 1463–1484.

    Google Scholar 

  • Hoffman, E., McCabe, K., Sachet, K. and Smith, V. (1994) Preference, property rights and anonymity in bargaining games. Games and Economic Behaviour 7 (3): 346–380.

    Article  Google Scholar 

  • Iversen, V., Jackson, C., Kebede, B., Munro, A. and Verschoor, A . (2011) Do spouses realise cooperative gains? Experimental evidence from rural Uganda. World Development. 39 (4): 569–578.

    Article  Google Scholar 

  • Jackson, C. (2009) Researching the researched: Gender, reflexivity and actor-orientation in an experimental game. European Journal of Development Research 21: 772–791.

    Article  Google Scholar 

  • Jackson, C . (2011) Research with experimental games: Questioning practice and interpretation. Progress in Development Studies 11 (4): 229–241.

    Article  Google Scholar 

  • Jacobson-Wilding, A. (1999) Chaos and creativity: The transformative symbolism of fused categories. In: H. Moore, T. Sanders and B. Kaare (eds.) Those Who Play with Fire: Gender Fertility and Transformation in East and Southern Africa, LSE Monographs in Social Anthropology London and New Jersey: Athlone Press, pp. 283–293.

    Google Scholar 

  • Levin, I., Schneider, S. and Gaeth, G. (1998) All frames are not created equal: A typology and critical analysis of framing effects. Organisational Behaviour and Human Decision Processes 76 (2): 149–188.

    Article  Google Scholar 

  • Levitt, S. and List, J. (2007) What do laboratory experiments measuring social preferences reveal about the real world? The Journal of Economic Perspectives 21 (2): 153–174.

    Article  Google Scholar 

  • Long, N. (2001) Development Sociology: Actor Perspectives. London: Routledge.

    Book  Google Scholar 

  • Marlowe, F. (2004) Dictator games and Ultimatums in an egalitarian society of hunter-gatherers: The Hadza of Tanzania. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 168–193.

    Chapter  Google Scholar 

  • Mazar, N. and Chen-Bo, Z. (2010) Do green products make us better people? Psychological Science 21 (4): 494–498.

    Article  Google Scholar 

  • Miller, D. (ed.) (2005) Materiality. USA: Duke University Press.

    Book  Google Scholar 

  • McElreath, R. (2004) Community structure, mobility, and the strength of norms in an African society: The Sangu of Tanzania. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 335–355.

    Chapter  Google Scholar 

  • Monin, B. and Miller, D. (2001) Moral credentials and the expression of prejudice. Journal of Personality and Social Psychology 81 (1): 33–43.

    Article  Google Scholar 

  • Patton, J. (2004) Coalitional effects on reciprocal fairness in the Ultimatum Game: A case from the Ecuadorian Amazon. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 96–124.

    Chapter  Google Scholar 

  • Read, D. (2005) Monetary incentives, what are they good for? Journal of Economic Methodology 12 (2): 265–276.

    Article  Google Scholar 

  • Shipton, P. (1989) Bitter Money: Cultural Economy and Some African Meanings of Forbidden Commodities. American Ethnological Society Monograph Series No. 1, American Anthropological Association.

  • Smith, V. (1976) Experimental economics: Induced value theory. The American Economic Review 66 (2): 274–279.

    Google Scholar 

  • Strathern, M . (1972) Women in Between: Female Roles in a Male World: Mount Hagen, New Guinea. Boston: Rowman and Littlefield.

    Google Scholar 

  • Swedberg, R. (2001) Sociology and game theory: Contemporary and historical perspectives. Theory and Society 30: 301–355.

    Article  Google Scholar 

  • Tracer, D. (2004) Market integration, reciprocity and fairness in rural Papua New Guinea: Results from a two village Ultimatum Game experiment. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 232–259.

    Chapter  Google Scholar 

  • Tversky, A. and Kahneman, D. (1981) The framing of decisions and the psychology of choice. Science 211 (4481): 453–458.

    Article  Google Scholar 

  • Weiss, B. (1996) The Making and Unmaking of the Haya Lived World: Consumption, Commoditisation and Everyday Practice. Durham, NC and London: Duke University Press.

    Google Scholar 

Download references

Acknowledgements

I would like to thank the EJDR reviewers for their insightful comments, and the ESRC for research funding during the period when this article was written.

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jackson, C. Internal and External Validity in Experimental Games: A Social Reality Check. Eur J Dev Res 24, 71–88 (2012). https://doi.org/10.1057/ejdr.2011.47

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/ejdr.2011.47

Keywords

Navigation