Abstract
The relevance of experimental games as methods in development research depends crucially on how far the results from the games can be extrapolated to real life, that is, the external validity of those results. The extent to which external validity matters depends on what you want to do with the data; some kinds of theory testing can arguably afford indifference, but many experiments are used as an indicator of behaviour in everyday life. This article takes an anthropological perspective on both the internal and external validity of social preference experiments in developing country settings, and argues for more cautious knowledge claims, triangulation of data and a broader conceptualisation of norms and behaviour.
La pertinence des jeux expérimentaux en tant que méthodes dans la recherche sur le développement dépend fondamentalement de la mesure dans laquelle les résultats obtenus à partir de ces jeux s’appliquent à la réalité, c’est à dire de la validité externe de ces résultats. La validité externe est plus ou moins importante selon ce que l’on souhaite faire avec les données. Certains types de jeux testent des théories sans se soucier de leur validité externe, mais d’autres sont tenus comme étant des indicateurs de comportements quotidiens. Cet article examine à partir d’une perspective anthropologique les validités tant internes qu’externes d’expériences de préférences sociales dans un contexte de pays en développement, et plaide en faveur de revendications théoriques plus prudentes, de triangulations de données, et d’une conceptualisation plus large des normes et des comportements.
Similar content being viewed by others
Notes
This is derived from an index composed of ‘payoffs to cooperation’, that is, the extent of reliance on cooperation outside family level; ‘market integration’, that is, extent of dependence on markets; ‘anonymity’, that is, the extent to which people interact with strangers they may never see again; ‘privacy’, that is, how possible it is to keep secrets from others; and ‘sociopolitical complexity’, that is, how much centralised decision making occurs above the level of the household.
Although Ensminger for the Orma finds that those engaged in wage labour and trade were more likely to engage in fair behaviour than subsistence producers.
Henrich and Smith's players all knew each other and the researcher well, that is, questionable anonymity, yet they cooperated less than the US students interacting with strangers (2004, p. 156).
For example, meat is more likely to be evaluated positively if labelled ‘75 per cent lean’ compared to ‘25 per cent fat’.
For example, subjects are more willing to accept breast self-examination if the same outcomes are described negatively compared to positively (1998, p. 168).
Levitt and List (2007) cite evidence behaviour experiments, bid the same as men when they are menstruating, but differently from men when experiencing higher oestrogen levels (Chen et al 2005). They argue that women who self-select as volunteers are thus possibly more pro-social than non-volunteer women.
Not all games were played with money.
References
Alvard, M. (2004) The Ultimatum Game, fairness and cooperation among big game hunters. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 413–435.
Barr, A. (2004) Kinship, familiarity, and trust: An experimental investigation. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 305–334.
Benz, M. and Meier, S. (2008) Do people behave in experiments as in the field? Evidence from donations. Experimental Economics 11 (3): 268–281.
Bloch, M. and Parry, J. (eds.) (1989) Money and the Morality of Exchange. Cambridge: CUP.
Camerer, C. and Fehr, E. (2004) Measuring social norms and preferences using experimental games: A guide for social scientists. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 55–95.
Cardenas, J.C. and Carpenter, J. (2008) Behavioural development economics: Lessons from field labs in the developing world. Journal of Development Studies 44 (3): 311–338.
Charness, G. and Kuhn, P. (2010) Lab Labor: What Can Labor Economists Learn from the Lab? National Bureau of Economic Research, Working Paper no. w15913.
Chen, Y., Katuscak, P. and Ozdenoren, E. (2005) Why Can’t a Woman Bid More Like a Man? Department of Economics, University of Michigan, Working Paper.
Croson, R. and Gneezy, U. (2009) Gender differences in preferences. Journal of Economic Literature 47 (2): 448–474.
Doty, R. and Silverthorne, C. (1975) Influence of menstrual cycle on volunteering behaviour. Nature 254 (5496): 139–140.
Douglas, M . (1966) Purity and Danger: an Analysis of Concepts of Pollution and Taboo. London: Routledge and Kegan Paul.
Eckel, C. (2008) The gender gap: Using the lab as a window on the market. In: F. Bettio and A. Verashchagina (eds.) Frontiers in the Economics of Gender (Routledge Siena Studies in Political Economy), Oxon: Routledge, pp. 223–242.
Ensminger, J. (2004) Market integration and fairness: Evidence from Ultimatum, Dictator and Public Goods Experiments in East Africa. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 356–381.
Falk, A. and Heckman, J. (2009) Lab experiments are a major source of knowledge in the social sciences. Science 326 (5952): 535–538.
Fletschner, D., Anderson, C.L. and Cullen, A. (2010) Are women as likely to take risks and compete? Behavioural findings from Central Vietnam. Journal of Development Studies 46 (8): 1459–1479.
Giddens, A. (1984) The Constitution of Society. Cambridge: Polity Press.
Gil-White, F. (2004) Ultimatum Game with an ethnicity manipulation; results from Khovdiin Bulgan Sum, Mongolia. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 260–304.
Goffman, E. (1983) The interaction order: American Sociological Association, 1982 Presidential Address. American Sociological Review 48 (1): 1–17.
Goffman, E. (1961) Encounters: Two Studies in the Sociology of Interaction. Indianapolis, IN: Bobbs Merrill.
Guala, F. (2005) The Methodology of Experimental Economics. Cambridge: CUP.
Guardian. (2010) Colombia tries to stop people drawing money. 10 April.
Gurven, M. (2004) Does market exposure affect economic game behaviour? The Ultimatum Game and Public Goods Game among the Tsimane’ of Bolivia. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 194–321.
Haley, K. and Fessler, D. (2005) Nobody's watching? Subtle cues affect generosity in an anonymous economic game. Evolution and Human Behaviour 26 (3): 245–256.
Hamoudi, A. and Thomas, D. (2006) Do You Care? Altruism and Inter-generational Exchanges in Mexico. California Centre for Population Research, On-line Series of Working Papers CCPR-008-06.
Henrich, J. and Smith, N. (2004) Comparative experimental evidence from Machiguenga, Mapuche, Huinca and American populations. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 125–167.
Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E. and Gintis, H. (eds.) (2004a) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press.
Henrich, J. et al (2004b) Overview and synthesis. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 8–55.
Hill, K. and Gurven, M. (2004) Economic experiments to examine fairness and cooperation among the Ache Indians of Paraguay. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, p. 382.
Hirschman, A . (1982) Rival Interpretations of market society: civilising, destructive or feeble. Journal of Economic Literature 20 (4): 1463–1484.
Hoffman, E., McCabe, K., Sachet, K. and Smith, V. (1994) Preference, property rights and anonymity in bargaining games. Games and Economic Behaviour 7 (3): 346–380.
Iversen, V., Jackson, C., Kebede, B., Munro, A. and Verschoor, A . (2011) Do spouses realise cooperative gains? Experimental evidence from rural Uganda. World Development. 39 (4): 569–578.
Jackson, C. (2009) Researching the researched: Gender, reflexivity and actor-orientation in an experimental game. European Journal of Development Research 21: 772–791.
Jackson, C . (2011) Research with experimental games: Questioning practice and interpretation. Progress in Development Studies 11 (4): 229–241.
Jacobson-Wilding, A. (1999) Chaos and creativity: The transformative symbolism of fused categories. In: H. Moore, T. Sanders and B. Kaare (eds.) Those Who Play with Fire: Gender Fertility and Transformation in East and Southern Africa, LSE Monographs in Social Anthropology London and New Jersey: Athlone Press, pp. 283–293.
Levin, I., Schneider, S. and Gaeth, G. (1998) All frames are not created equal: A typology and critical analysis of framing effects. Organisational Behaviour and Human Decision Processes 76 (2): 149–188.
Levitt, S. and List, J. (2007) What do laboratory experiments measuring social preferences reveal about the real world? The Journal of Economic Perspectives 21 (2): 153–174.
Long, N. (2001) Development Sociology: Actor Perspectives. London: Routledge.
Marlowe, F. (2004) Dictator games and Ultimatums in an egalitarian society of hunter-gatherers: The Hadza of Tanzania. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 168–193.
Mazar, N. and Chen-Bo, Z. (2010) Do green products make us better people? Psychological Science 21 (4): 494–498.
Miller, D. (ed.) (2005) Materiality. USA: Duke University Press.
McElreath, R. (2004) Community structure, mobility, and the strength of norms in an African society: The Sangu of Tanzania. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 335–355.
Monin, B. and Miller, D. (2001) Moral credentials and the expression of prejudice. Journal of Personality and Social Psychology 81 (1): 33–43.
Patton, J. (2004) Coalitional effects on reciprocal fairness in the Ultimatum Game: A case from the Ecuadorian Amazon. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 96–124.
Read, D. (2005) Monetary incentives, what are they good for? Journal of Economic Methodology 12 (2): 265–276.
Shipton, P. (1989) Bitter Money: Cultural Economy and Some African Meanings of Forbidden Commodities. American Ethnological Society Monograph Series No. 1, American Anthropological Association.
Smith, V. (1976) Experimental economics: Induced value theory. The American Economic Review 66 (2): 274–279.
Strathern, M . (1972) Women in Between: Female Roles in a Male World: Mount Hagen, New Guinea. Boston: Rowman and Littlefield.
Swedberg, R. (2001) Sociology and game theory: Contemporary and historical perspectives. Theory and Society 30: 301–355.
Tracer, D. (2004) Market integration, reciprocity and fairness in rural Papua New Guinea: Results from a two village Ultimatum Game experiment. In: J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr and H. Gintis (eds.) Foundations of Human Sociality: Economic Experiments and Ethnographic Evidence from Fifteen Small-scale Societies. Oxford: Oxford University Press, pp. 232–259.
Tversky, A. and Kahneman, D. (1981) The framing of decisions and the psychology of choice. Science 211 (4481): 453–458.
Weiss, B. (1996) The Making and Unmaking of the Haya Lived World: Consumption, Commoditisation and Everyday Practice. Durham, NC and London: Duke University Press.
Acknowledgements
I would like to thank the EJDR reviewers for their insightful comments, and the ESRC for research funding during the period when this article was written.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Jackson, C. Internal and External Validity in Experimental Games: A Social Reality Check. Eur J Dev Res 24, 71–88 (2012). https://doi.org/10.1057/ejdr.2011.47
Published:
Issue Date:
DOI: https://doi.org/10.1057/ejdr.2011.47