Skip to main content

Addressing confounding errors when using non-experimental, observational data to make causal claims

Abstract

In their recent book, Is Inequality Bad for Our Health?, Daniels, Kennedy, and Kawachi claim that to “act justly in health policy, we must have knowledge about the causal pathways through which socioeconomic (and other) inequalities work to produce differential health outcomes.” One of the central problems with this approach is its dependency on “knowledge about the causal pathways.” A widely held belief is that the randomized clinical trial (RCT) is, and ought to be the “gold standard” of evaluating the causal efficacy of interventions. However, often the only data available are non-experimental, observational data. For such data, the necessary randomization is missing. Because the randomization is missing, it seems to follow that it is not possible to make epistemically warranted claims about the causal pathways. Although we are not sanguine about the difficulty in using observational data to make warranted causal claims, we are not as pessimistic as those who believe that the only warranted causal claims are claims based on data from (idealized) RCTs. We argue that careful, thoughtful study design, informed by expert knowledge, that incorporates propensity score matching methods in conjunction with instrumental variable analyses, provides the possibility of warranted causal claims using observational data.

This is a preview of subscription content, access via your institution.

References

  1. Angrist J.D., Imbens G.W., Rubin D.B. (1996). Identification of effects using instrumental variables. Journal of the American Statistical Association 91(434): 444–455

    Article  Google Scholar 

  2. Angrist J.D., Krueger A.B. (2001). Instrumental variables and the search for identification: From supply and demand to natural experiments. Journal of Economic Perspectives 15(4): 69–85

    Article  Google Scholar 

  3. Austin P.C., Grootendorst P., Anderson G.M. (2007). A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Statistics in Medicine 26(4): 734–753

    Article  Google Scholar 

  4. Bellman R. (1961). Adaptive control processes. Princeton University Press, Princeton, NJ

    Google Scholar 

  5. Berk R.A. (2004). Regression analysis: A constructive critique. Sage Publications, Thousand Oaks, CA

    Google Scholar 

  6. Bond S.J., White I.R., Walker A.S. (2007). Instrumental variables and interactions in the causal analysis of a complex clinical trial. Statistics in Medicine 26(7): 1473–1496

    Article  Google Scholar 

  7. Clogg C.C., Haritou A. (1997). The regression method of causal inference and a dilemma confronting this method. In: McKim V., Turner S. (eds) Causality in crisis? Statistical methods and the search for causal knowledge in the social sciences. University of Notre Dame, Press, Notre Dame IN, pp 83–112

    Google Scholar 

  8. D’Agostino R.B., Jr. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine 17(19): 2265–2281

    Article  Google Scholar 

  9. D’Agostino R.B., Jr., Rubin D.B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association 95(451): 749–759

    Article  Google Scholar 

  10. Daniels D., Kennedy B., Kawachi I. (2000). Justice is good for our health. In: Cohen J., Rogers J. (eds) Is inequality bad for our health?. Beacon Press, Boston, MA, pp 3–33

    Google Scholar 

  11. Davidson R., MacKinnon J.G. (1993). Estimation and inference in econometrics. Oxford University Press, New York, NY

    Google Scholar 

  12. DiPrete T.A., Gangl M. (2004). Assessing bias in the estimation of causal effects: Rosenbaum bounds on matching estimators and instrumental variables estimation with imperfect instruments. Sociological Methodology 34: 271–310

    Article  Google Scholar 

  13. Freedman D.A. (1999). From association to causation: Some remarks on the history of statistics. Statistical Science 14(3): 243–258

    Article  Google Scholar 

  14. Freedman D.A. (2005). Statistical models: Theory and practice. Cambridge University Press, Cambridge

    Google Scholar 

  15. Glymour M.M. (2006). Natural experiments and instrumental variable analyses in social epidemiology. In: Oakes J.M., Kaufman J. (eds) Methods in social epidemiology. Jossey-Bass, San Francisco, CA, pp 429–460

    Google Scholar 

  16. Greenland S. (1990). Randomization, statistics, and causal inference. Epidemiology 1: 421–429

    Article  Google Scholar 

  17. Greenland S. (2000). An introduction to instrumental variables for epidemiologists. International Journal of Epidemiology 29: 722–729

    Article  Google Scholar 

  18. Greenland S., Robins J.M. (1986). Identifiability, exchangeability, and epidemiological confounding. International Journal of Epidemiology 15(3): 413–419

    Article  Google Scholar 

  19. Haukoos J.S., Newgard C.D. (2007). Advanced statistics: Missing data in clinical research – Part 1: An introduction and conceptual framework. Academic Emergency Medicine 14(7): 662–668

    Google Scholar 

  20. Heckman J.J. (1997). Instrumental variables: A study of implicit behavioral assumptions used in making program evaluations. The Journal of Human Resources 32(3): 441–462

    Article  Google Scholar 

  21. Heckman, J. J. (2005). The scientific model of causality. In R. Stolzenberg (Ed.), Sociological methodology (Vol. 35, pp. 1–97). Oxford: Basil Blackwell (for the American Sociological Association).

  22. Hernán M.A. (2004). A definition of causal effect for epidemiological research. Journal of Epidemiology and Community Health 58: 265–271

    Article  Google Scholar 

  23. Hernán, M.A., Robins J.M. (2006). Instruments for causal inference: An epidemiologist’s dream? Epidemiology 17(4): 360–372

    Article  Google Scholar 

  24. Hintikka J. (1975). The intentions of intentionality and other new models for modalities. D. Reidel Publishing Co., Dordrecht

    Google Scholar 

  25. Holland P. (1986). Statistics and causal inference. Journal of the American Statistical Association 81(396): 945–960

    Article  Google Scholar 

  26. Humphreys P. (1986). Causation in the social sciences: An overview. Synthese 68: 1–12

    Google Scholar 

  27. Imbens G.W., Angrist J.D. (1994). Identification and estimation of local average treatment effects. Econometrica 62(2): 467–475

    Article  Google Scholar 

  28. Imbens G.W., Rosenbaum P.R. (2005). Robust, accurate confidence intervals with a weak instrument: Quarter of birth and education. Journal of the Royal Statistical Society, Series A 168(part 1): 109–126

    Google Scholar 

  29. Kaufman J.S., Kaufman S., Poole C. (2003). Causal inference from randomized trials in social epidemiology. Social Science and Medicine 57: 2397–2409

    Article  Google Scholar 

  30. Linden A., Adams J.L. (2006). Evaluating disease management programme effectiveness: An introduction to instrumental variables. Journal of Evaluation in Clinical Practice 12(2): 148–154

    Article  Google Scholar 

  31. Little R.J., Rubin D.B. (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and approaches. Annual Review of Public Health 21: 121–145

    Article  Google Scholar 

  32. Luellen J.K., Shadish W.R., Clark M.H. (2005). Propensity scores: An introduction and experimental test. Evaluation Review 29(6): 530–558

    Article  Google Scholar 

  33. Maldonado G., Greenland S. (2002). Estimating causal effects. International Journal of Epidemiology 31: 422–429

    Article  Google Scholar 

  34. Manski, C. F. (1993). Identification problems in the social sciences. In P. Marsden (Ed.), Social methodology (Vol. 23, pp. 1–56). Oxford: Basil Blackwell (for the American Sociological Association).

  35. Manski C.F. (1995). Identification problems in the social sciences. Harvard University Press, Cambridge, MA

    Google Scholar 

  36. Moffitt R. (2005). Remarks on the analysis of causal relationships in population research. Demography 42(1): 91–108

    Article  Google Scholar 

  37. Newgard C.D., Haukoos J.S. (2007). Advanced statistics: Missing data in clinical research – Part 2: Multiple imputation. Academic Emergency Medicine 14(7): 669–678

    Google Scholar 

  38. Newgard C.D., Hedges J.R., Arthur M., Mullins R.J. (2004). Advanced statistics: The propensity score – A method for estimating treatment effect in observational research. Academic Emergency Medicine 11(9): 953–961

    Google Scholar 

  39. Newhouse J.P., McClellan M. (1998). Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health 19: 17–34

    Article  Google Scholar 

  40. Newman S.C. (2004). Commonalities in the classical, collapsibility and counterfactual concepts of confounding. Journal of Clinical Epidemiology 57: 325–329

    Article  Google Scholar 

  41. Oakes M.J. (2004). The (Mis)estimation of neighborhood effects: Causal inference for a practicable social epidemiology. Social Science and Medicine 58(10): 1929–1952

    Article  Google Scholar 

  42. Oakes M.J., Johnson P.J. (2006). Propensity score matching for social epidemiology. In: Oakes J.M., Kaufman J. (eds) Methods in social epidemiology. Jossey-Bass, San Francisco, CA, pp 370–392

    Google Scholar 

  43. Pearl J. (2000). Causality: Models, reasoning, and inference. Cambridge University Press, Cambridge

    Google Scholar 

  44. Pearl J. (2001). Causal inference in health sciences: A conceptual introduction. Health Services and Outcomes Research Methodology 2: 189–220

    Article  Google Scholar 

  45. Randall Jr. J.H. (1940). The making of the modern mind – Revised edition. Houghton Mifflin Company, Boston, MA

    Google Scholar 

  46. Reiter J. (2000). Using statistics to determine causal relationships. The American Mathematical Monthly 107(1): 24–32

    Article  Google Scholar 

  47. Robins J.M., Scheines R., Spirtes P., Wasserman L. (2003). Uniform consistency in causal inference. Biometrika 90(3): 491–515

    Article  Google Scholar 

  48. Rosenbaum P.R. (2002). Observational studies (2nd ed). Springer, New York, NY

    Google Scholar 

  49. Rosenbaum P.R. (2004). Matching in observational studies. In: Gelman A., Meng X.-L. (eds) Applied Bayesian modeling and causal inference from incomplete-data perspectives. Wiley and Sons, Ltd, West Sussex, pp 15–24

    Chapter  Google Scholar 

  50. Rosenbaum P.R., Rubin D.B. (1983). The central role of propensity scores in observational studies for causal effects. Biometrika 70(1): 41–55

    Article  Google Scholar 

  51. Rosenbaum P.R., Rubin D.B. (1984). On the nature and discovery of structure: Comment. Journal of the American Statistical Association 79(385): 26–28

    Article  Google Scholar 

  52. Rosenbaum P.R., Rubin D.B. (1985a). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician 39(1): 33–38

    Article  Google Scholar 

  53. Rosenbaum P.R., Rubin D.B. (1985b). The bias due to incomplete matching. Biometrics 41(1): 103–116

    Article  Google Scholar 

  54. Rothman K. J. (2002). Epidemiology: An introduction. Oxford University Press, Oxford

    Google Scholar 

  55. Rubin D.B. (1986). Statistics and causal inference: Comment: Which ifs have causal answers. Journal of the American Statistical Association 81(396): 961–962

    Article  Google Scholar 

  56. Rubin D.B. (1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine 127(8): 757–763

    Google Scholar 

  57. Rubin D.B. (2004). On principles for modeling propensity scores in medical research. Pharmacoepidemiology and Drug Safety 13(12): 855–857

    Article  Google Scholar 

  58. Rubin D.B. (2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine 26(1): 20–36

    Article  Google Scholar 

  59. Rubin D.B., Thomas N. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics 52(1): 249–264

    Article  Google Scholar 

  60. Shadish W.R., Cook T.D., Campbell D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin Company, Boston, MA

    Google Scholar 

  61. Smith H.L. (1997). Matching with multiple controls to estimate treatment effects in observational studies. Sociological Methodology 27: 325–353

    Article  Google Scholar 

  62. Smith H.L. (2003). Some thoughts on causation as it relates to demography and population studies. Population and Development Review 29(3): 459–469

    Article  Google Scholar 

  63. Smith J.A., Todd P.E. (2001). Reconciling conflicting evidence on the performance of propensity-score matching methods. American Economics Review 91(2): 112–118

    Article  Google Scholar 

  64. Sobel, M. E. (2005). Discussion: The scientific model of causality. In R. Stolzenberg (Ed.), Social methodology (Vol. 35, pp. 99–133). Oxford: Basil Blackwell (for the American Sociological Association).

  65. Urbach P. (1985). Randomization and the design of experiments. Philosophy of Science 52(2): 256–273

    Article  Google Scholar 

  66. Weitzen S., Lapane K.L., Toledano A.Y., Hume A.L., Mor V. (2005). Weaknesses of goodness-of-fit tests for evaluating propensity score models: The case of the omitted confounder. Pharmacoepidemiology and Drug Safety 14(4): 227–238

    Article  Google Scholar 

  67. Winship C., Morgan S.L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology 25: 659–706

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Andrew Ward.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Ward, A., Johnson, P.J. Addressing confounding errors when using non-experimental, observational data to make causal claims. Synthese 163, 419–432 (2008). https://doi.org/10.1007/s11229-007-9292-4

Download citation

Keywords

  • Causal inference
  • Confounding
  • Social epidemiology
  • Propensity scores
  • Instrumental variables
  • Methodology