Skip to main content
SpringerLink
Log in

Large-Scale Social Experimentation in the United States

  • Conference paper
A Celebration of Statistics

Abstract

Randomized controlled experimentation has a long history. But the transition from statistical theory to statistical practice remains a difficult one, especially when the experiments are large in scale. In this paper, we review several major randomized social experiments carried out in the United States beginning in the early 1970s. We describe briefly the policy settings out of which these experiments grew, and we indicate how this type of experimentation fits into the historical development of randomized experimentation more generally. We give detailed design features for five different sets of social experiments, and we briefly summarize some of the analyses of the experimental data. We end the paper with a discussion of analytical strategies for social experiments, and possible alternatives to large-scale experimentation, including combining the results of several experiments, and evolutionary designs. Throughout, we place special emphasis on the possible roles of models and of social and economic theory in both the design and the analysis of experiments, and we describe several open research problems requiring the attention of mathematical statisticians and economists.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  • Acton, J. P. et al. (1982). Promoting Energy Efficiency Through Improved Electricity Pricing: A Mid-Project Report, N-1843-HF/FF/NSF. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Acton, J. P., Manning, W. G., Jr., and Mitchell, B. M. (1978). Lessons to be Learned from the Los Angeles Rate Experiment in Electricity. Report R-2113-DWP. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Acton, J. P. and Mitchell, B. M. (1979). Evaluating Time-of-Day Electricity Rates for Residential Customers. Report R-2509-DWP. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Aigner, D. J. (1979). “A brief introduction to the methodology of optimal experimental design.” J. Econometrics, 11, 7–27.

    Article  MathSciNet  MATH  Google Scholar 

  • Aigner, D. J. (1982). “The residential electricity time-of-use pricing experiments: What have we learned?” In J. A. Hausman and D. Wise (eds.), Social Experimentation, NBER conference volume.

    Google Scholar 

  • Aigner, D.J. and Learner, E. (1984). “Estimation of time-of-use pricing response in the absence of experimental data: An application of methodology of data transferability.” J. Econometrics, 26, 202–227.

    Google Scholar 

  • Archibald, R. W. and Newhouse, J. P. (1980). Social Experimentation: Some Whys and Hows. Report R-2479-HEW. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Atkinson, A. C. (1985). “An introduction to the optimum design of experiments.” In this volume, Chapter 20.

    Google Scholar 

  • Berry, S. H. (1979). Conducting a Survey Using the Client’s Staff: Evaluation of Interviewer performance in the Electricity Rate Study. Report R-2223-DWP. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Becker, N. G. and Kersting, G. (1983). “Design problems for the pure birth process.” Adv. in Appl. Probab., 15, 255–273.

    Article  MathSciNet  MATH  Google Scholar 

  • Bliss, C.I. (1952). The Statistics of Bioassay. New York: Academic.

    Google Scholar 

  • Borjas, G. J. and Heckman, J. J. (1978). “Labor supply estimates for public policy evaluation.” In Proceedings of 31st Annual Meeting of Industrial Relations Research Association, 320–331.

    Google Scholar 

  • Boruch, R. F. (1975). “On common contentions about randomized field experiments.” In R. F. Boruch and H. W. Riecken (eds.), Experimental Tests of Public Policy. Boulders, CO: Westview, 108–145.

    Google Scholar 

  • Box, G. E. P. (1978). “Experimental design: Response surface methodology.” In W. H. Kruskal and J. M. Tanur (eds.), International Encyclopedia of Statistics, 1. New York: Free Press, 294–299.

    Google Scholar 

  • Box, G. E. P. and Draper, N. R. (1969). Evolutionary Operation: A Statistical Method for Process Improvement. New York: Wiley.

    Google Scholar 

  • Box, G. E. P. and Wilson, K. B. (1951). “On experimental attainment of optimal conditions.” J. Roy. Statist. Soc. Ser. B, 13, 1–45.

    MathSciNet  Google Scholar 

  • Bradbury, K. and Downs, A. (eds.) (1981). Do Housing Allowances Work? Washington, D.C.: Brookings Institution.

    Google Scholar 

  • Cain, G. G. and Watts, H. W. (eds.) (1973). Income Maintenance and Labor Supply. Chicago: Markham.

    Google Scholar 

  • Campbell, D. T. and Stanley, J. C. (1963). “Experimental and quasi-experimental designs for research on teaching.” In N. L. Gage (ed.), Handbook of Research on Teaching. Chicago: Rand Mally, 171–246.

    Google Scholar 

  • Cochran, W. G. (1937). “Problems arising in the analysis of a series of similar experiments.” J. Roy. Statist. Soc. Suppl., 4, 102–118.

    Article  Google Scholar 

  • Cochran, W. G. (1951). “Improvement by means of selection.” In Second Berkeley Symposium on Mathematical Statistics and Probability, 3, Berkeley, CA: Univ. of California Press, 449–470.

    Google Scholar 

  • Cochran, W. G. (1954). “The combination of estimates from different experiments.” Biometrics, 7, 101–129.

    Article  Google Scholar 

  • Cohen, E. R., Crowe, K. M., and Dumond, J. W. M. (1957). Fundamental Constants of Physics. New York:Mraw-Hill.

    Google Scholar 

  • Conlisk, J. and Watts, H. W. (1969). “A model for optimizing experimental designs for estimating response surfaces.” In Proceedings of the Social Statistics Section, American Statistical Association, 150–156.

    Google Scholar 

  • Conlisk, J. and Watts, H. W. (1979). “A model for optimizing experimental designs for estimating response surfaces.” J. Econometrics, 11, 27–42.

    Article  MATH  Google Scholar 

  • Conlisk, J. and Kurz, M. (1972). The Assignment Model of the Seattle and Denver Income Maintenance Experiment. Research Memo 15. Menlo Park, CA: Stanford Research Institute, Center for the Study of Welfare Policy.

    Google Scholar 

  • Cosslett, S. R. (1983). “Distribution-free maximum likelihood estimator of the binary choice model.” Econometrica, 51, 765–782.

    Article  MathSciNet  MATH  Google Scholar 

  • Cox, D. R. (1972). “Regression models and life tables (with discussion).” J. Roy. Statist. Soc. Ser. B, 34, 187–220.

    MathSciNet  MATH  Google Scholar 

  • Cox, G. M. (1958). “Forward.” In Experimental Designs in Industry, V. Chew (ed.), New York: Wiley.

    Google Scholar 

  • Cronbach, L. J. (1982). Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass.

    Google Scholar 

  • Duan, N., Manning, W. G., Jr., Morris, C. N., and Newhouse, J. P. (1983). “A comparison of alternative models of the demand for medical care.” J. Business and Economics, 1, 115–126.

    Article  Google Scholar 

  • Douchel, W. H. and Harris, J. E. (1983). “Bayes methods for combining results of cancer studies in humans and other species (with discussion).” J. Amer. Statist. Assoc., 78, 293–315.

    Article  MathSciNet  Google Scholar 

  • Ericsson, N. R. and Hendry, D. F. (1985). “Econometric modeling of the market for new housing in the United Kingdom.” In this volume, Chapter 11.

    Google Scholar 

  • Ferber, R. and Hirsch, W. Z. (1979). “Social experiments in economics.” J. Econometrics, 11, 77–115.

    Article  Google Scholar 

  • Ferber, R. and Hirsch, W. Z. (1982). Social Experimentation and Economic Policy. Cambridge: Cambridge U.P.

    Google Scholar 

  • Finney, D. J. (1945). “The fractional replication of factorial arrangements.” Ann. Eugenics, 12, 291–301.

    Article  MathSciNet  Google Scholar 

  • Fisher, R. A. (1935). The Design of Experiments. Edinburgh: Oliver and Boyd.

    Google Scholar 

  • Fisk, J. D. and Roth, D. M. (1980). Work Disincentives and Income Maintenance Programs: Review of the Empirical Evidence. Congressional Research Service, Libary of Congress, Congress of the United States.

    Google Scholar 

  • Gallant, A. R. and Koenker, R. W. (1983). Some Welfare Econometrics of Peak-Load Pricing of Electricity: A Continuous Time Approach. Unpublished technical report. Murray Hill, N. J.: Bell Laboratories.

    Google Scholar 

  • Gilbert, J. P., Light, R. J., and Mosteller, F. (1975). “Assessing social innovations: An empirical base for policy.” In C. A. Bennett and A. A. Lumsdaine (eds.), Evaluation and Experiment: Some Critical Issues in Assessing Social Programs. New York: Academic, 39–193.

    Google Scholar 

  • Glass, G. V., Maw, B., and Smith, M. L. (1981). Meta-analysis in Social Research. Beverly Hills, CA: Sage.

    Google Scholar 

  • Haggstrom, G. W. (1976). “The pitfalls of manpower experiments.” In H. W. Sinaiko and L. A. Broedling (eds.), Perspectives on Attitude Assessment: Surveys and Their Alternatives. Champaign, IL: Pendleton.

    Google Scholar 

  • Haggstrom, G. W., Blaschke, T. J., Chow, W. K., and Lisowski, Wm. (1981). The Multiple Option Recruiting Experiment: Report R-2671-MRAL. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Hannan, M., Tuma, N., and Groeneveld, L. (1976). The Impact of Income Maintenance on the Making and Breaking of Marital Unions: Interim Report. Research Memorandum 28. Menlo Park, CA: Stanford Research Institute, Center for the Study of Welfare Policy.

    Google Scholar 

  • Hannan, M., Tuma, N., and Groeneveld, L. (1977). “Income and marital events: Evidence from an income maintenance experiment.” Amer. J. Sociology, 82, 1186–1211.

    Article  Google Scholar 

  • Hausman, J. A. (1981). “The design and analysis of social and economic experiments.” Proc Internat. Statist. Inst., 43rd Session, 49 (Book 2), 1084–1108.

    MathSciNet  Google Scholar 

  • Heckman, J. J. (1978). “Comment on The labor supply response of wage earners,’ by O. Ashenfelter.” In J. Palmer and J. Pechman (eds.) Welfare in Rural Areas. Washington: Brookings Institution, 138–147.

    Google Scholar 

  • Heckman, J. and Singer, B. (1984). “Econometric duration analysis.” Econometrics, 24, 63–132.

    Article  MathSciNet  MATH  Google Scholar 

  • Hedges, L. V. and Olkin, I. (1981). “Analyses, reanalyses, and meta-analysis: Review of Glass, Maw, and Smith.” Contemporary Education Review, 1, 157–165.

    Google Scholar 

  • Hendricks, W. and Koenker, R. (1979). “Demand for electricity by time-of-day: An evaluation of experimental results.” Paper presented at Rutgers Conference, The Demand for and Pricing of Public Utility Services, October 1979.

    Google Scholar 

  • Hermalin, A. and Mason, W. (1980). “A strategy for the comparative analysis of WFS data with illustrative examples.” In The United Nations Program for Comparative Analysis of World Fertility Survey Data. New York: United Nations Fund for Population Activities, 90–168.

    Google Scholar 

  • Immer, F. R., Hayes, H. K., and Powers, L. (1934). “Statistical determination of barley varietal adaptation.” J. Amer. Soc. Agronomy, 26, 403–419.

    Article  Google Scholar 

  • Kalbfleisch, J. D. and Prentice, R. L. (1980). The Statistical Analysis of Failure Time Data. New York: Wiley.

    MATH  Google Scholar 

  • Keeler, E. and Rolph, J. (1982). The Demand for Episodes of Medical Treatment. Report R-2829-HHS. Santa Monica, CA: The Rand Corporation.

    Google Scholar 

  • Keeley, M. C. and Robins, P. K. (1978). The Design of Social Experiments: A Critique of the Conlisk-Watts Assignment Model. Research Memorandum 57. Menlo Park, CA: SRI International.

    Google Scholar 

  • Kershaw, D. N. and Fair, J. (1976). The New Jersey Income-Maintenance Experiment. Vol. 1, Operations, Survey, and Administration. New York: Academic.

    Google Scholar 

  • Lancaster, T. and Nickell, S. (1980). “The analysis of reemployment probabilities for the unemployed.” J. Roy. Statist. Soc. Ser. A, 143, 141–165.

    Article  MATH  Google Scholar 

  • Lillard, L. A. and Action, J. P. (1980). Seasonal Electricity Demand and Pricing Analysis with a Variable Response Model. Report R-2425-DWP. Santa Monica, CA: Rand Corporation.

    Google Scholar 

  • Lillard, L. A. and Aigner, D. J. (1984). “Time-of-day electricity consumption response to temperature and the ownership of air conditioning appliances.” J. Business and Economic Statistics, 2, No. 1, 40–54.

    Article  Google Scholar 

  • Madansky, A. (1980). “Response surface exploration in social experiments.” Paper presented at Second Annual Public Policy Conference, Boston, MA, October 24–25, 1980.

    Google Scholar 

  • Manning, W. G., and Acton, J. P. (1980). Residential Electricity Demand Under Time-of-Day Pricing: Exploratory Data Analysis from the Los Angeles Rate Study. Report R-2426-DWP/HP. Santa Monica, CA: Rand Corporation.

    Google Scholar 

  • Manning, W. G., Mitchell, B. M., and Acton, J. P. (1979). “Design of the Los Angeles peak-load pricing experiment for electricity.” J. Econometrics, 11, 131–194.

    Article  Google Scholar 

  • Mason, W., Wong, G., and Entwistle, B. (1984). “Contextual analysis through the multilevel linear model.” In S. Leinhardt (ed.), Sociological Methodology 1983–1984. San Francisco: Jossey-Bass, 72–103.

    Google Scholar 

  • Meier, P. (1975). “Statistics and medical experimentation.” Biometrics, 31, 511–529.

    Article  MATH  Google Scholar 

  • Morris, C. (1979). “A finite selection model for experimental design of the health insurance study.” J. Econometrics, 11, 43–61.

    Article  MATH  Google Scholar 

  • Mosteller, F. (1978). “Errors: Nonsampling errors.” In W. H. Kruskal and J. M. Tanur (eds.), International Encyclopedia of Statistics, 1. New York: Free Press, 208–229.

    Google Scholar 

  • Mosteller, F. and Tukey, J. W. (1982). “Combination of results of stated precision: I. The optimistic case.” Utilitas Mathematics, 21, 155–178.

    MathSciNet  Google Scholar 

  • Mosteller, F. and Tukey, J. W. (1984). “Combination of results of stated precision: II. A more realistic case.” In P.S.R.S. Rao and J. Sedransk (eds.), W. G. Cochran’s Impact on Statistics. New York: Wiley, 223–252.

    Google Scholar 

  • Neuberg, L. (1985). Social Control Experimentation: Towards a Rationalist and Historical Critique of Empiricist Methodology. Cambridge, England: Cambridge U.P.

    Google Scholar 

  • Newhouse, J. P. (1974). “A design for a health insurance experiment.” Inquiry, 11, 5–27.

    Google Scholar 

  • Newhouse, J. P., Marquis, K. H., Morris, C. N., et al. (1979). “Measurement issues in the second generation of social experiments: The Health Insurance Study.” J. Econometrics, 11, 117–130.

    Article  Google Scholar 

  • Newhouse, J. P., Manning, W. G., Morris, C. N., et al. (1981). “Some interim results from a controlled trial of cost-sharing in health insurance.” New England J. Medicine 305, 1501–1507.

    Article  Google Scholar 

  • Pottinger, G. (1983). “Explanation, rationality, and microeconomic theory.” Behavioral Sci., 28, 109–125.

    Article  Google Scholar 

  • Riecken, H. W., Boruch, R. F., Campbell, D. T., Caplan, N, Glennan, T. K., Pratt, J. W., Rees, A., and Williams, W. (1974). Social Experiments: A Method for Planning and Evaluating Social Programs. New York: Seminar Press.

    Google Scholar 

  • Rivlin, A. M. (1974). “How can experiments be more useful?” Amer. Econ. Rev., 64, 346–354.

    Google Scholar 

  • Rosenthal, R. and Rubin, D. B. (1978). “Interpersonal expectancy effects: The first 345 studies.” Behavioral and Brain Sciences, 3, 377–415.

    Article  Google Scholar 

  • Rossi, P. H., Berk, R. A., and Lenihan, K. J. (1980). Money, Work, and Crime. New York: Academic.

    Google Scholar 

  • Rossi, P. H. and Lyall, K. C. (1976). Reforming Public Welfare: A Critique of the Negative Income Experiment. New York: Russell Sage Foundation.

    Google Scholar 

  • Scriven, M. (1969). “Evaluating educational programs.” Urban Rev., 3, 20–22.

    Article  Google Scholar 

  • Simon, H. A. (1957). Models of Man. New York: Wiley.

    MATH  Google Scholar 

  • Student (1931). “The Lanarkshire milk experiment.” Biometrika, 23, 398–406.

    Article  Google Scholar 

  • Tharp, R. G. and Gallimore, R. (1979). “The ecology of program research and evaluation: A model of evaluation succession.” In L. Sechrest et al. (eds.), Evaluation Studies Review Annual, 4. Beverly Hills, CA: Sage.

    Google Scholar 

  • Titterington, D. M. (1980). “Aspects of optimal design in dynamic systems.” Technometrics, 22, 289–299.

    MathSciNet  Google Scholar 

  • University Group Diabetes Program (1970). “A study of the effects of hypoglycemic agents in vascular complications in patients with adult-onset diabetes.” Diabetes (Suppl. 2), 19, 747–830.

    Google Scholar 

  • U.S. Department of Energy (1978). Analytical Master Plan for the Analysis of Data from the Electric Utility Rate Demonstration Projects. Prepared by Research Triangle Institute, Research Triangle Park, NC.

    Google Scholar 

Download references

Authors
  1. Stephen E. Fienberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Burton Singer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Judith M. Tanur
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Imperial College of Science and Technology, SW7 2BZ, London, England

    Anthony C. Atkinson

  2. Carnegie-Mellon University, 15213, Pittsburgh, Pennsylvania, USA

    Stephen E. Fienberg

Rights and permissions

Reprints and permissions

Copyright information

© 1985 Springer-Verlag New York Inc.

About this paper

Cite this paper

Fienberg, S.E., Singer, B., Tanur, J.M. (1985). Large-Scale Social Experimentation in the United States. In: Atkinson, A.C., Fienberg, S.E. (eds) A Celebration of Statistics. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-8560-8_12

Download citation

  • DOI: https://doi.org/10.1007/978-1-4613-8560-8_12

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4613-8562-2

  • Online ISBN: 978-1-4613-8560-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation