Abstract
For a half-century, sociology and allied social sciences have worked with a model of research design founded on a distinction between internal validity, the capacity of designs to support statements about cause and effect, and external validity, the extent to which the results from specific studies can be generalized beyond the batch of data on which they are founded. The distinction is conceptually useful and has great pedagogic value, that is, the association of the experimental model with internal validity, and random sampling with external validity. The advent of the potential outcomes model of causation, by emphasizing the definition of a causal effect at the unit level and the heterogeneity of causal effects, has made it clear how indistinct (and interpenetrated) are these “twin pillars” of research design. This is the theme of this chapter, which inveighs against the idea of a hierarchy of research design desiderata, with causal inference at the peak. Rather, I adopt the design typology of Leslie Kish (1987), which advocates an appropriate balance of randomization, representation, and realism, and illustrate how all three elements (and not just randomization, the internal validity design mechanism) are integrated aspects of meaningful causal analysis. What is meaningful causal analysis? It depends first and foremost on getting straight why we are doing what we are doing. Understanding why something has happened may tell us a lot about what will happen if we were actually to do something, but this is not necessarily so.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
My treatment is necessarily selective. These are not the only eminent statisticians who have put research design at the forefront of thinking about causation.
- 2.
It is less well known that his Ph.D. was in sociology, part of a fascinating intellectual and personal background (Frankel and King 1996).
- 3.
This includes the possible overrepresentation of certain domains for their theoretical salience, their meager share of the population notwithstanding (Smith 1990: 68).
- 4.
Bollen and Pearl (Chap. 15, this volume) take explicit issue with Freedman’s characterizations of, in particular, recursive path models.
- 5.
A well-known piece of Freedman’s mockery—“The Modelers’ Responses”—appears on the same page.
- 6.
If the latter, then this is a meal that, from the get-go, sociologists have been disinclined to eat on its own (Chap. 2 by Barringer, Leahey, and Eliason, this volume).
- 7.
This is related to the issue of support for inference, mentioned above, and also to the manipulation criterion, to be discussed below (Smith 1997: 333–334).
- 8.
The related question of what an experiment does or does not tell us about causal mechanisms will be taken up two sections hence, in The Experiment as the Model for Research Design. The definitions of causal mechanisms here, as per Rosenbaum (1984: 42), differ in some ways from canonical sociological treatments of causal mechanisms and theory (e.g., Hedström and Swedberg 1998). Attempts to either integrate or differentiate these perspectives appear in Goldthorpe (2001), Morgan and Winship (2007: 219–242), Smith (n.d.: 33–35), and—especially—Knight and Winship (Chap. 14, this volume).
- 9.
Kish (1987), for example, incorporates measures of bias, of stochastic (e.g., sampling) variation, and of cost (fixed and unit-specific) in the same equations, hence in comparable metrics.
- 10.
- 11.
The “uniformity among units” is with respect to the effect of a treatment.
- 12.
There are discussions of heterogeneity and interactions with respect to treatment effects in the foundational work on randomized experiments (Fisher [1925] 1951). But heterogeneity was with reference to variance among subjects in other factors related to the response but independent of assignment to treatment, hence on the efficiency of experimental designs (pp. 107–109), and interactions in effects of factors were with respect to other factors in the design of the experiment, not to characteristics of the units subject to randomization (e.g., pp. 93–99).
- 13.
Either that or that it is only white male psychiatrists who know how to use correctly this information on the empirical incidence of psychological disorders conditional on the gender and race of the patient.
- 14.
- 15.
I am indebted to Xiaolu Wang for developing, clarifying, and drawing out the following points.
- 16.
Thus, parolees could substitute these payments for the money that would otherwise be derived from work. In contrast, charter schools do not require additional subvention, so the conditional cash transfer is a net plus for those who are offered it and send their children to charter schools. This distinguishes the charter school experiment from a similar conditional cash transfer scheme in which vouchers that can be used to pay Catholic school fees are randomly tendered (Morgan and Winship 2007).
- 17.
This is without considering the possibility that heterogeneity in treatment effects obtains not only with respect to families but also to school types—what Morgan and Winship (2012) term compositional heterogeneity.
- 18.
“No causation without manipulation” is the third of eight “myths” addressed by Bollen and Pearl (Chap. 15, this volume). They tend to hold with the critics of the manipulation criterion, but at the core their myth busting targets the irrelevance of this criterion for the practice of causal analysis via structural equation models.
References
Ackoff, R. L. (1953). The design of social research. Chicago: The University of Chicago Press.
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455.
Arceneaux, K., Gerber, A. S., & Green, D. P. (2010). A cautionary note on the use of matching to estimate causal effects: An empirical example comparing matching estimates to an experimental benchmark. Sociological Methods & Research, 39(2), 256–282.
Babbie, E. (2010). The practice of social research (12th ed.). Belmont: Wadsworth.
Berk, R. A. (1991). Toward a methodology for mere mortals. In P. V. Marsden (Ed.), Sociological methodology (pp. 315–324). Oxford: Basil Blackwell.
Berk, R. A. (2005). Randomized experiments as the bronze standard. Journal of Experimental Criminology, 1(4), 417–433.
Berk, R. A., & Sherman, L. W. (1988). Police responses to family violence incidents: An analysis of an experimental design with incomplete randomization. Journal of the American Statistical Association, 83(401), 70–76.
Bickel, P. J., & Freedman, D. A. (1981). Some asymptotic theory for the bootstrap. The Annals of Statistics, 9(6), 1196–1217.
Blalock, H. M., Jr. (1991). Are there really any constructive alternatives to causal modeling. In P. V. Marsden (Ed.), Sociological methodology (pp. 325–335). Oxford: Basil Blackwell.
Bongaarts, J., & Potter, R. G. (1983). Fertility, biology, and behavior: An analysis of the proximate determinants. New York: Academic.
Boruch, R. (Ed.). (2005). Place randomized trials: Experimental tests of public policy. Annals of the American Academy of Political and Social Science 599.
Brand, J. E., & Xie, Y. (2007). Identification and estimation of causal effects with time-varying treatments and time-varying outcomes. In Y. Xie (Ed.), Sociological methodology (pp. 393–434). Boston/Oxford: Blackwell.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally & Company.
Card, D. (1999). The causal effect of education on earnings. In O. Ashenfelter & D. Card (Eds.), Handbook of labor economics (Vol. 5, pp. 1801–1863). New York: North-Holland.
Cheslack-Postava, K., Liu, K., & Bearman, P. S. (2011). Closely spaced pregnancies are associated with increased odds of autism in California sibling births. Pediatrics, 127(2), 246–253.
Cook, T. D. (2002). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24(3), 175–199.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally & Company.
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
Dawid, A. P. (2000). Causal inference without counterfactuals. Journal of the American Statistical Association, 95(450), 407–424.
Dawid, A. P., & Fienberg, S. E. (2011, July). The causes of effects. In Plenary talk presented at 8th international conference on forensic inference statistics, Seattle, WA.
de Brauw, A., & Hoddinott, J. (2011). Must conditional cash transfer programs be conditioned to be effective? The impact of conditioning transfers on school enrollment in Mexico. Journal of Development Economics, 96(2), 359–370.
Diaconis, P., & Freedman, D. (1986). On the consistency of Bayes estimates. The Annals of Statistics, 14(1), 1–26.
Duncan, O. D. (1974). Developing social indicators. Proceedings of the National Academy of Sciences, 71(12), 5096–5102.
Duncan, O. D. (1984). Notes on social measurement: Historical and critical. New York: Russell Sage.
Farewell, V. T. (1979). Some results on the estimation of logistic models based on retrospective data. Biometrika, 66(1), 27–32.
Firebaugh, G. (1978). A rule for inferring individual-level relationships from aggregate data. American Sociological Review, 43(4), 557–572.
Fisher, R. A. ([1925] 1951). Statistical methods for research workers (6th ed.). New York: Hafner Publishing Company.
Fisher, R. A. ([1935] 1958). The design of experiments (13th ed.). New York: Hafner Publishing Company, Inc.
Fountain, C., & Bearman, P. (2011). Risk as social context: Immigration policy and autism. Sociological Forum, 26(2), 215–240.
Frangakis, C. E., & Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics, 58(1), 21–29.
Frankel, M., & King, B. (1996). A conversation with Leslie Kish. Statistical Science, 11(1), 65–87.
Freedman, D. A. (1981). Bootstrapping regression models. The Annals of Statistics, 9(6), 1218–1228.
Freedman, D. A. (1983). Markov chains. New York: Springer.
Freedman, D. A. (1985). Statistics and the scientific method. In W. M. Mason & S. E. Fienberg (Eds.), Cohort analysis in social research: Beyond the identification problem (pp. 343–366). New York: Springer.
Freedman, D. A. (1991a). Statistical models and shoe leather. In P. V. Marsden (Ed.), Sociological methodology 1991 (pp. 291–313). Oxford: Basil Blackwell.
Freedman, D. A. (1991b). A rejoinder to Berk, Blalock, and Mason. In P. V. Marsden (Ed.), Sociological methodology 1991 (pp. 353–358). Oxford: Basil Blackwell.
Freedman, D. A. (2005). Statistical models: Theory and practice. Cambridge: Cambridge University Press.
Freedman, D., Pisani, R., & Purves, R. (1978). Statistics. New York: W. W. Norton.
Gage, N. L. (Ed.). (1963). Handbook of research on teaching. Chicago: Rand McNally & Company.
Gangl, M. (2010). Causal inference in sociological research. Annual Review of Sociology, 36, 21–47.
Goldthorpe, J. H. (2001). Causation, statistics, and sociology. European Sociological Review, 17(1), 1–20.
Greiner, D. J., & Rubin, D. B. (2011). Causal effects of perceived immutable characteristics. The Review of Economics and Statistics, 93(3), 775–785.
Härdle, W. (1990). Applied nonparametric regression. Cambridge: Cambridge University Press.
Heckman, J. J. (2001). Micro data, heterogeneity, and the evaluation of public policy: Nobel lecture. Journal of Political Economy, 109(4), 673–748.
Heckman, J. J., & Smith, J. A. (1995). Assessing the case for social experiments. Journal of Economic Perspectives, 9(2), 85–110.
Hedström, P., & Swedberg, R. (1998). Social mechanisms: An introductory essay. In P. Hedström & R. Swedberg (Eds.), Social mechanisms: An analytical approach to social theory (pp. 1–31). Cambridge: Cambridge University Press.
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960.
Holland, P. W. (2008). Causation and race. In T. Zuberi & E. Bonilla-Silva (Eds.), White logic, white methods: Racism and methodology (pp. 93–109). Lanham: Rowman & Littlefield.
Jick, H., & Kaye, J. A. (2003). Epidemiology and possible causes of autism. Pharmacotherapy, 23(12), 1524–1530.
Johnson-Hanks, J. A., Bachrach, C. A., Morgan, S. P., & Kohler, H.-P. (2011). Understanding family change and variation: Toward a theory of conjunctural action. New York: Springer.
Kadane, J. B. (2011). Principles of uncertainty. Boca Raton: Chapman & Hall.
King, M., & Bearman, P. (2009). Diagnostic change and the increased prevalence of autism. International Journal of Epidemiology, 38(5), 1224–1234.
King, M. D., Fountain, C., Dakhlallah, D., & Bearman, P. S. (2009). Estimated autism and older reproductive age. American Journal of Public Health, 99(9), 1673–1679.
Kish, L. (1965). Survey sampling. New York: Wiley.
Kish, L. (1987). Statistical design for research. New York: Wiley.
Kong, A., Frigge, M. L., Masson, G., Besenbacher, S., Sulem, P., Magnusson, G., Gudjonsson, S. A., Sigurdsson, A., Jonasdottir, A., Jonasdottir, A., Wong, W. S. W., Sigurdsson, G., Walters, G. B., Steinberg, S., Helgason, H., Thorleifsson, G., Gudbjartsson, D. F., Helgason, A., Magnusson, O. T., Thorsteinsdottir, U., & Stefansson, K. (2012). Rate of de novo mutations and the importance of father’s age to disease risk. Nature, 488, 471–475.
Lieberson, S., & Lynn, F. B. (2002). Barking up the wrong branch: Scientific alternatives to the current model of sociological science. Annual Review of Sociology, 28, 1–19.
Liu, K.-Y., King, M., & Bearman, P. S. (2010a). Social influence and the autism epidemic. The American Journal of Sociology, 115(5), 1387–1434.
Liu, K., Zerubavel, N., & Bearman, P. (2010b). Social demographic change and autism. Demography, 47(2), 327–343.
Loring, M., & Powell, B. (1988). Gender, race, and DSM-III: A study of objectivity of psychiatric diagnostic behavior. Journal of Health and Social Behavior, 29(1), 1–22.
Ludwig, J., Liebman, J. B., Kling, J. R., Duncan, G. J., Katz, L. F., Kessler, R. C., & Sanbonmatsu, L. (2008). What can we learn about neighborhood effects from the moving to opportunity experiment? The American Journal of Sociology, 114(1), 144–188.
Marini, M. M., & Singer, B. (1988). Causality in the social sciences. In C. C. Clogg (Ed.), Sociological methodology 1988 (pp. 347–409). Washington, DC: American Sociological Association.
Merli, M. G., Qian, Z., & Smith, H. L. (2004). Adaptation of a political bureaucracy to economic and institutional change under socialism: The Chinese state family planning system. Politics and Society, 31(2), 231–256.
Mill, J. S. (1843). A system of logic, ratiocinative and inductive, being a connected view of the principles of evidence, and the methods of scientific investigation. London: John W. Parker.
Morgan, S. L., & Winship, C. (2007). Counterfactuals and causal inference: Methods and principles for social research. New York: Cambridge University Press.
Morgan, S. L. & Winship, C. (2012). Bringing context and variability back in to causal analysis, Chapter 14. In H. Kincaid (Ed.), The Oxford handbook of the philosophy of the social sciences. New York: Oxford University Press.
Ní Bhrolcháin, M., & Dyson, T. (2007). On causation in demography: Issues and illustrations. Population and Development Review, 33(1), 1–36.
O’Roak, B. J., Vives, L., Girirajan, S., Karakoc, E., Krumm, N., Coe, B. P., Levy, R., Ko, A., Lee, C., Smith, J. D., Turner, E. H., Stanaway, I. B., Vernot, B., Malig, M., Baker, C., Reilly, B., Akey, J. M., Borenstein, E., Rieder, M. J., Nickerson, D. A., Bernier, R., Shendure, J., & Eichler, E. E. (2012). Sporadic autism exomes reveal a highly interconnected protein network of de novo mutations. Nature, 485(7397), 246–250.
Rosenbaum, P. R. (1984). From association to causation in observational studies: The role of tests of strongly ignorable treatment assignment. Journal of the American Statistical Association, 79, 41–48.
Rosenbaum, P. R. (2002). Observational studies (2nd ed.). New York: Springer.
Rosenbaum, P. R. (2009). Design of observational studies. New York: Springer.
Rosenbaum, P. R., & Rubin, D. B. (1983a). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
Rosenbaum, P. R., & Rubin, D. B. (1983b). Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society, Series B, 45(2), 212–218.
Rossi, P. H., Berk, R. A., & Lenihan, K. J. (1982). Saying it wrong with figures: A comment on Zeisel. The American Journal of Sociology, 88(2), 390–393.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.
Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.
Russo, F., Wunsch, G., & Mouchart, M. (2010). Inferring causality through counterfactuals in observational studies: Some epistemological issues (Discussion Paper 1029). Institut de statistique, biostatistique et sciences actuarielles (ISBA), Université Catholique de Louvain. http://www.stat.ucl.ac.be/ISpub/dp/2010/DP1029.pdf
Rytand, D. A. (1980). Sutton’s or dock’s Law. The New England Journal of Medicine, 302(17), 972.
Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489–500.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston/New York: Houghton Mifflin Company.
Shelton, J. F., Tancredi, D. J., & Hertz-Picciotto, I. (2010). Independent and dependent contributions of advanced maternal and paternal ages to autism risk. Autism Research, 3(1), 30–39.
Sloan, J. H., Kellermann, A. L., Reay, D. T., Ferris, J. A., Koepsell, T., Rivara, F. P., Rice, C., Gray, L., & LoGerfo, J. (1988). Handgun regulations, crime, assaults, and homicide: A tale of two cities. The New England Journal of Medicine, 319(19), 1256–1262.
Smith, H. L. (1990). Specification problems in experimental and nonexperimental social research. In C. C. Clogg (Ed.), Sociological methodology 1990 (pp. 59–91). Cambridge, MA: Basil Blackwell.
Smith, H. L. (1997). Matching with multiple controls to estimate treatment effects in observational studies. In A. E. Raftery (Ed.), Sociological methodology 1997 (pp. 325–353). Oxford: Basil Blackwell.
Smith, H. L. (2003). Some thoughts on causation as it relates to demography and population studies. Population and Development Review, 29(3), 459–469.
Smith, H. L. (2005). Introducing new contraceptives in rural China: A field experiment. The Annals of the American Academy of Political and Social Science, 599, 246–271.
Smith, H. L. (2009). Causation and its discontents. In H. Engelhardt-Woelfler, H.-P. Kohler, & A. Fuernkranz-Prskawetz (Eds.), Causal analysis in population studies: Concepts, methods, applications. Dordrecht: Springer.
Smith, H. L. (n.d.). La causalité en sociologie et démographie. Retour sur le principe de l’action humaine.
Sobel, M. E. (2006). What do randomized studies of housing mobility demonstrate? Causal inference in the face of interference. Journal of the American Statistical Association, 101(476), 1398–1407.
Stinchcombe, A. L. (1969). Constructing social theories. New York: Harcourt, Brace & World, Inc.
Tukey, J. (1986). Sunset salvo. The American Statistician, 40(1), 72–76.
Vaupel, J. W., Carey, J. R., & Christensen, K. (2003). It’s never too late. Science, 301(5640), 1679–1681.
Vogt, T., & Kluge, F. (2012, May 5). Does public spending level mortality inequalities? — Findings from East Germany after unification. In Presented at the annual meeting of the Population Association of America, San Francisco, CA.
Vogt, T., Vaupel, J. W., & Rau, R. (2012). Health or wealth. Life expectancy convergence after the German unification. Dissertation, Max Planck Institute for Demographic Research.
Wainer, H. (Ed.). ([1986] 2000). Drawing inferences from self-selected samples. Mahwah: Lawrence Erlbaum Associates.
Wheaton, B. (1978). The sociogenesis of psychological disorder: Reexamining the causal issues with longitudinal data. American Sociological Review, 43(3), 383–403.
Winship, C., & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659–706.
Zeisel, H. (1982a). Disagreement over the evaluation of a controlled experiment. American Journal of Sociology, 88(2), 378–389.
Zeisel, H. (1982b). Hans Zeisel concludes the debate. American Journal of Sociology, 88(2), 394–396.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Smith, H.L. (2013). Research Design: Toward a Realistic Role for Causal Analysis. In: Morgan, S. (eds) Handbook of Causal Analysis for Social Research. Handbooks of Sociology and Social Research. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-6094-3_4
Download citation
DOI: https://doi.org/10.1007/978-94-007-6094-3_4
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-007-6093-6
Online ISBN: 978-94-007-6094-3
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)