Planning an Observational Study

  • Paul R. Rosenbaum
Part of the Springer Series in Statistics book series (SSS)


Many observational studies do not succeed in providing tangible, enduring, and convincing evidence about the effects caused by treatments, and those that do succeed often exhibit great care in their design. Particularly at the early stages of design, this care consists of choices that determine the circumstances of the study and the data to be collected. A convincing observational study is the result of active observation, an active search for those rare circumstances in which tangible evidence may be obtained to distinguish treatment effects from the most plausible biases. In an experiment, treatment effects are seen clearly because the environment is tightly controlled, whereas in a compelling observational study, control is, to a large extent, replaced by choice—the environment is carefully chosen.


Minimum Wage Successful Theory Broad Theory Minimum Wage Increase Male Teenager 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allison, P. D. (1990) Change scores as dependent variables in regression analyses. In: Sociological Methodology, C. C. Clogg, ed., Oxford: Basil Blackwell, pp. 93–114.Google Scholar
  2. Altonji, J. G. and Dunn, T. A. (1996) Using siblings to estimate the effect of school quality on wages. Review of Economics and Statistics, 77, 665–671.CrossRefGoogle Scholar
  3. Anderson, E. (2000) Code of the Streets: Decency, Violence and the Moral Life of the Inner City. New York: Norton.Google Scholar
  4. Angrist, J. D. and Krueger, A. B. (1999) Empirical strategies in labor economics. In: Handbook of Labor Economics, III, New York: Elsevie, Chapter 23.Google Scholar
  5. Angrist, J. D. and Lavy, V. (1999) Using Maimonides’ rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 533–575.Google Scholar
  6. Ashenfelter, O. A. and Krueger, A. B. (1994) Estimates of the economic return to schooling from a new sample of twins. American Economic Review, 84, 1157–1173.Google Scholar
  7. Becker, H. S. (1972) Outsiders: Studies in the Sociology of Deviance. New York: The Free Press.Google Scholar
  8. Behrman, J., Rosenzweig, M., and Taubman, P. (1996) College choice and wages: Estimates using data on female twins. The Review of Economics and Statistics, 672–685.Google Scholar
  9. Bosk, C. L. (1981) Forgive and Remember: Managing Medical Failure. Chicago: University of Chicago Press.Google Scholar
  10. Box, G. E. P. (1966) The use and abuse of regression. Technometrics, 8, 625–629.MathSciNetMATHCrossRefGoogle Scholar
  11. Bronars, S. G. and Grogger, J. (1994) The economic consequences of unwed motherhood: Using twin births as a natural experiment. American Economic Review, 84, 1141–1156.Google Scholar
  12. Bross, I. D. J. (1960). Statistical criticism. Cancer, 13, 394–400. Reprinted in: The Quantitative Analysis of Social Problems, E. Tufte, ed., Reading, MA: Addison-Wesley, pp. 97–108.CrossRefGoogle Scholar
  13. Campbell, D. T. (1957) Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.CrossRefGoogle Scholar
  14. Campbell, D. T. (1984, 1988) Can we be scientific in applied social science? Evaluation Studies Review Annual, 9, 26–48. Reprinted in: D. Campbell, Methodology and Epistemology for Social Science: Selected Papers. Chicago: University of Chicago Press, pp. 315–333.Google Scholar
  15. Campbell, D. and Stanley, R. (1963) Experimental and Quasi-Experimental Designs for Research. Chicago: Rand McNally.Google Scholar
  16. Card, D. (1990) The impact of the Mariel Boatlift on the Miami labor market. Industrial and Labor Relations Review, 43, 245–257.CrossRefGoogle Scholar
  17. Card, D., Katz, L., and Krueger, A. (1994) Comment. Industrial and Labor Relations Review, 48, 487–496.CrossRefGoogle Scholar
  18. Card, D. and Krueger, A. (1994) Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84, 772–793.Google Scholar
  19. Card, D. and Krueger, A. (1995) Myth and Measurement: The New Economics of the Minimum Wage. Princeton, NJ: Princeton University Press.Google Scholar
  20. Card, D. and Krueger, A. (2000) Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania: Reply. American Economic Review, 90, 1397–1420.CrossRefGoogle Scholar
  21. Chamberlin, T. C. (1890, 1965) The method of multiple working hypotheses. Originally in Science 1890, 15, 92. Reprinted in Science 1965, 148, 754–759.Google Scholar
  22. Cook, T. D., Campbell, D. T. and Peracchio, L. (1990) Quasi-experimentation. In: Handbook of Industrial and Organizational Psychology, M. Dunnette and L. Hough, Palo Alto, eds., CA: Consulting Psychologists Press, Chapter 9, pp. 491–576.Google Scholar
  23. Cook, T. D. and Shadish, W. R. (1994) Social experiments: Some developments over the past fifteen years. Annual Review of Psychology, 45, 545–580.CrossRefGoogle Scholar
  24. Dawes, R. (1996) The purpose of experiments: Ecological validity versus comparing hypotheses. Behavioral and Brain Sciences, 19, 20.CrossRefGoogle Scholar
  25. Deere, D., Murphy, K., and Welch, F. (1995) Employment and the 1990–1991 minimum-wage hike. American Economic Review, 85, 232–237.Google Scholar
  26. Diggle, P. J., Liang, K.Y., and Zeger, S. L. (1994) Analysis of Longitudinal Data. New York: Oxford University Press.Google Scholar
  27. Duff, C. (1996) New minimum wage makes few waves: Employers offset 50-cent raise with minor shifts. Wall Street Journal, 20 November 1996, pp. 2–4.Google Scholar
  28. Engle, R., Hendry, D., and Richard, J. (1983) Exogeneity. Econometrica, 51, 277–304.MathSciNetMATHCrossRefGoogle Scholar
  29. Estroff, S. E. (1985) Making It Crazy: An Ethnography of Psychiatric Clients in an American Community. Berkeley, CA: University of California Press.CrossRefGoogle Scholar
  30. Feyerabend, P. (1968) How to be a good empiricist-a plea for tolerance in matters epistemological. In: The Philosophy of Science (Oxford Readings in Philosophy), P. H. Nidditch, ed., New York: Oxford University Press, pp. 12–39.Google Scholar
  31. Feyerabend, P. (1975) Against Method. London: Verso.Google Scholar
  32. Fisher, R. A. (1935) The Design of Experiments. Edinburgh: Oliver and Boyd.Google Scholar
  33. Freedman, D. (1997) From association to causation via regression. Advances in Applied Mathematics, 18, 59–110.MathSciNetMATHCrossRefGoogle Scholar
  34. French, S. A., Story, M., and Jeffery, R. W. (2001) Environmental influences on eating and physical activity. Annual Review of Public Health, 22, 309–335.CrossRefGoogle Scholar
  35. Friedman, M. (1953) The methodology of positive economics. In: Essays in Positive Economics, Chicago: University of Chicago Press, pp. 3–43.Google Scholar
  36. Gastwirth, J. L., Krieger, A. M., and Rosenbaum, P. R. (1997) Hypotheticals and hypotheses. American Statistician, 51, 120–121.Google Scholar
  37. Giddens, A. (1979) Central Problems in Social Theory. Berkeley: University of California Press.Google Scholar
  38. Holland, P. (1993) Which comes first, cause or effect? In: A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues, G. Keren and C. Lewis, eds., Hillsdale, NJ: Lawrence Erlbaum, pp. 273–282.Google Scholar
  39. Holland, P. & Rubin, D. (1983) On Lord’s paradox. In: Principles of Psychological Measurement: A Festschrift for Frederic Lord, H. Wainer and S. Messick, eds., Hillsdale, NJ: Lawrence Erlbaum, pp. 3–25.Google Scholar
  40. Joffe, M., Hoover, D., Jacobson, L., Kingsley, L., Chmiel, J., Visscher, B., and Robins, J. (1998) Estimating the effect of zidovudine on Kaposi’s sarcoma from observational data using a rank preserving structural failure-time model. Statistics in Medicine, 17, 1073–1102.CrossRefGoogle Scholar
  41. Kessler, R. C. (1997) The effects of stressful life events on depression. Annual Review of Psychology, 48, 191–214.CrossRefGoogle Scholar
  42. Kuczmarski, R. J., Flegal, K. M., Campbell, S. M., and Johnson, C. L. (1994) Increasing prevalence of overweight among US adults. The National Health and Nutrition Examination Surveys, 1960 to 1991. Journal of the American Medical Association, 272, 205–211CrossRefGoogle Scholar
  43. Lakatos, I. (1970) Falsification and the methodology of scientific research programs. In: Criticism and the Growth of Knowledge, I. Lakatos and A. Musgrave, eds., New York: Cambridge University Press, pp. 91–196. Reprinted in: I. Lakatos (1978) Philosophical Papers, Volume 1, New York, Cambridge University Press, pp. 8–101.Google Scholar
  44. Lakatos, I. (1981) History of science and its rational reconstructions. In: Scientific Revolutions, I. Hacking, ed., New York: Oxford University Press, pp. 107–127. Reprinted from Boston Studies in the Philosophy of Science, VIII, 1970.Google Scholar
  45. Lehman, D., Wortman, C., and Williams, A. (1987) Long-term effects of losing a spouse or a child in a motor vehicle crash. Journal of Personality and Social Psychology, 52, 218–231.CrossRefGoogle Scholar
  46. Li, Y., Propert, K. J., and Rosenbaum, P. R. (2001) Balanced risk set matching. Journal of the American Statistical Association, 96, September, to appear..Google Scholar
  47. Lichtenstein, P., Gatz, M., Pedersen, N., Berg, S., and McClearn, G. (1996) A co-twin-control study of response to widowhood. Journal of Gerontology: Psychological Sciences, 51B, 279–289.CrossRefGoogle Scholar
  48. Manski, C. (1999) Comment. Statistical Science, 14, 279–281.Google Scholar
  49. Meehi, P. (1978) Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806–834. Reprinted in: P. Meehl, Selected Philosophical and Methodological Papers, Minneapolis: University of Minnesota Press, 1991, pp. 1–42.CrossRefGoogle Scholar
  50. Meyer, B. D. (1995) Natural and quasi-experiments in economics. Journal of Business and Economic Statistics, 13, 151–161.Google Scholar
  51. Meyer, M. and Fienberg, S, eds. (1992) Assessing Evaluation Studies: The Case of Bilingual Education Strategies. Washington, DC: National Academy Press.Google Scholar
  52. Morton, D., Saah, A., Silberg, S., Owens, W., Roberts, M., and Saah, M. (1982) Lead absorption in children of employees in a lead-related industry. American Journal of Epidemiology, 115, 549–555.Google Scholar
  53. Neumark, D. and Wascher, W. (1992 Employment effects of minimum and subminimum wages: Panel data on state minimum wage laws. Industrial and Labor Relations Review, 46, 55–81.CrossRefGoogle Scholar
  54. Neumark, D. and Wascher, W. (1994) Reply. Industrial and Labor Relations Review, 48, 497–512.CrossRefGoogle Scholar
  55. Peirce, C. S. (1903) On selecting hypotheses. In: Collected Papers of Charles Sanders Peirce, C. Hartshorne and P. Weiss, eds., Volume 5, Cambridge, MA: Harvard University Press, 1960, pp. 413–422.Google Scholar
  56. Peto, R., Pike, M., Armitage, P., Breslow, N., Cox, D., Howard, S., Mantel, N., McPherson, K., Peto, J., and Smith, P. (1976) Design and analysis of randomized clinical trials requiring prolonged observation of each patient, I: Introduction and design. British Journal of Cancer, 34, 585–612.CrossRefGoogle Scholar
  57. Platt, J. (1964) Strong inference. Science, 146, 347–352.CrossRefGoogle Scholar
  58. Polanyi, M. (1946, 1964) Science, Faith and Society. New York: Oxford University Press. Reprinted, Chicago: University of Chicago Press.Google Scholar
  59. Popper, K. (1965) Conjectures and Refutations. New York: Harper and Row.Google Scholar
  60. Popper, K. (1968) The Logic of Scientific Discovery. New York: Harper and Row. (English translation of Popper’s 1934 Logik der Forschung.) Google Scholar
  61. Putnam, H. (1995) Pragmatism. Oxford: Blackwell.Google Scholar
  62. Quine, W. (1951) Two dogmas of empiricism. Philosophical Review. Reprinted in: W. Quine (1980) From a Logical Point of View, Cambridge, MA: Harvard University Press, pp. 20–46.Google Scholar
  63. Robins, J. (1989) The control of confounding by intermediate variables. Statistics in Medicine, 8, 679–701.CrossRefGoogle Scholar
  64. Robins, J. (1992 Estimation of the time-dependent accelerated failure time model in the presence of confounding factors. Biometrika, 79, 321–334.MathSciNetMATHCrossRefGoogle Scholar
  65. Robins, J. (1998) Correction for non-compliance in equivalence trials. Statistics in Medicine, 17, 269–302.CrossRefGoogle Scholar
  66. Robins, J. (1999) Comment. Statistical Science, 14, 281–293.Google Scholar
  67. Robins, J., Rotnitzky, A. and Zhao, L. (1995) Analysis of semiparametric regression models for repeated outcomes in the presence of missing data. Journal of the American Statistical Association, 90,106–121.MathSciNetMATHCrossRefGoogle Scholar
  68. Rosenbaum, P. R. (1984) The consequences of adjustment for a concomitant variable that has been affected by the treatment. Journal of the Royal Statistical Society, Series A, 147, 656–666.CrossRefGoogle Scholar
  69. Rosenbaum, P. R. (1986) Dropping out of high school in the United States: An observational study. Journal of Educational Statistics, 11, 207 – 224.CrossRefGoogle Scholar
  70. Rosenbaum, P. R. (1999) Choice as alternative to control in observational studies (with discussion) . Statistical Science, 14, 259–304.MATHCrossRefGoogle Scholar
  71. Rosenbaum, P. R. (2001a) Stability in the absence of treatment. Journal of the American Statistical Association, 96, 210–219.MathSciNetMATHCrossRefGoogle Scholar
  72. Rosenbaum, P. R. (2001b) Replicating effects and biases. American Statistician, 55, 223–227.MathSciNetCrossRefGoogle Scholar
  73. Rosenzweig, M. and Wolpin, K. (1980) Testing the quantity-quality fertility model: The use of twins as a natural experiment. Econometrica, 48, 227–240.MATHCrossRefGoogle Scholar
  74. Rosenzweig, M. R. and Wolpin, K. I. (2000) Natural “natural experiments” in economics. Journal of Economic Literature, 38, 827–874.CrossRefGoogle Scholar
  75. Rubin, D. B. (1991) Practical implications of modes of statistical inference for causal inference and the critical role of the assignment mechanism. Biometrics, 47, 1213–1234.MathSciNetMATHCrossRefGoogle Scholar
  76. Samuelson, P. (1947, 1983) Foundations of Economic Analysis. Cambridge, MA: Harvard University Press.Google Scholar
  77. Shadish, W. R. and Cook, T. D. (1999) Comment-Design rules: More steps toward a complete theory of quasi-experimentation. Statistical Science, 14, 294–300.Google Scholar
  78. Shadish, W. R., Cook, T. D., and Campbell, D. T. (2002) Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.Google Scholar
  79. Shafer, G. (1996) The Art of Causal Conjecture. Cambridge, MA: MIT Press.MATHGoogle Scholar
  80. Sobel, M. (1995) Causal inference in the social and behavioral sciences. In: Handbook of Statistical Modelling for the Social and Behavioral Sciences, G. Arminger, C. Clogg, and M. Sobel, eds., New York: Plenum, pp. 1–38.Google Scholar
  81. Sowell, T. (1995) Repealing the law of gravity. Forbes, 22 May, p. 82.Google Scholar
  82. Susser, M. (1987) Falsification, verification and causal inference in epidemiology: Reconsideration in the light of Sir Karl Popper’s philosophy. In: M. Susser, Epidemiology, Health and Society: Selected Papers, New York: Oxford, pp. 82–93.Google Scholar

Copyright information

© Springer Science+Business Media New York 2002

Authors and Affiliations

  • Paul R. Rosenbaum
    • 1
  1. 1.Department of Statistics, The Wharton SchoolUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations