Skip to main content
SpringerLink
Log in

How do social scientists reach causal inferences? A study of reception

  • Published:
Quality & Quantity Aims and scope Submit manuscript

A Correction to this article was published on 07 April 2022

This article has been updated

Abstract

In this paper we outline a new area of research, social science reception: how inferences are reached by social scientists. Specifically, we ask how the nature of the evidence affects the sort of causal inferences that social scientists draw. As an exemplar, we focus on multimethod research. Specifically, we subject respondents to an experiment in which each species of evidence constitutes a distinct treatment: (a) qualitative, (b) quantitative, or (c) multimethod. In the abstract, respondents tend to reflect widespread methodological norms about the strengths of multimethod research. However, when confronted with specific studies, scholars are not more likely to believe causal inferences backed by multimethod research than inferences backed by mono-method research. This suggests that there is a misalignment between methodological norms and on-the-ground judgments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

Data will be made available once the paper is accepted for publication.

Coda availability

Stata code will be available.

Change history

Notes

  1. Recent work on process tracing and multi-method analysis has introduced the application of formal Bayesian modeling to qualitative research (Humphreys and Jacobs 2015; Fairfield and Charman 2017); clearly, the distinction between qualitative and quantitative is multi-dimensional and allows exceptions.

  2. www.random.org/.

  3. This means, of course, that respondents are aware of when and where a section of the text has been omitted. Overcoming this feature of the experiment would involve re-setting the type and considerable editing of the articles, so as to present the qualitative and quantitative treatments as stand-alone articles. Even so, we suspect that these reconstructed treatment conditions would lack the content and texture of published articles in top journals.

  4. In two iterations of the experiment we asked respondents whether they were familiar with the article assigned to them. Only a few answered in the affirmative, so it does not appear to be a common occurrence.

  5. A causal framework provides an overarching structure to our project, but it does not prevent attention to other kinds of inferences. Descriptive or predictive inferences often play a major role in, or rely on, causal arguments; as such, several of our arguments below apply to these kinds of inferences.

  6. For reviews, see Mahoney (1979), Miller (2006). We include cognitive psychology and organizational behavior under the rubric of natural science.

  7. A few nonexperimental studies exist, though these are generally fraught with problems of identification (e.g., Crane 1967).

  8. A number of questions may be asked of respondents in order to gauge their degree of familiarity with the substantive (not methodological) context of a piece of research. One may construct a Likert scale in which respondents rate their own knowledge of an area. One may ask a series of factual questions about that area. Or one may ask whether respondents are conversant with a series of key published works in that area.

References

  • Armstrong, J.S.: Unintelligible management research and academic prestige. Interfaces 10(April), 80–86 (1980)

    Article  Google Scholar 

  • Beach, D., Pedersen, R.B.: Process-tracing methods: foundations and guidelines. University of Michigan Press, Ann Arbor, MI (2013)

    Book  Google Scholar 

  • Beck, N.: Is causal-process observation an oxymoron? Polit. Anal. 14(3), 347–352 (2006)

    Article  Google Scholar 

  • Beck, N.: Causal process ‘observation’: oxymoron or (fine) old wine. Polit. Anal. 18, 499–505 (2010)

    Article  Google Scholar 

  • Bennett, A., Checkel, J.T. (eds.): Process Tracing: From Metaphor to Analytic Tool. Cambridge University Press, Cambridge (2015)

    Google Scholar 

  • Brady, H.E., Collier, D. (eds.): Rethinking social inquiry: diverse tools, shared standards. Rowman & Littlefield, Lanham, MD (2004)

    Google Scholar 

  • Brannen, J. (ed.): Mixing methods: qualitative and quantitative research. Avebury, Aldershot (1992)

    Google Scholar 

  • Brewer, J., Hunter, A.: Foundations of Multimethod Research: Synthesizing Styles. Sage, Thousand Oaks, CA (2006)

    Book  Google Scholar 

  • Bryman, A.: Barriers to integrating quantitative and qualitative research. J. Mixed Methods Res. 1(1), 8–22 (2007)

    Article  Google Scholar 

  • Cicchetti, D.V.: The reliability of peer review for manuscript and grant submissions: a cross-disciplinary investigation. Behav. Brain Sci. 14(1), 119–186 (1991)

    Article  Google Scholar 

  • Clark, V.L., Creswell, J.W. (eds.): The Mixed Methods Reader. Sage, USA (2007)

    Google Scholar 

  • Crane, D.: The gatekeepers of science: some factors affecting the selection of articles for scientific journals. Am. Sociol. 3, 195–201 (1967)

    Google Scholar 

  • Daniel, H.-D.: Guardians of science: Fairness and reliability of peer review. Wiley, VCH (1993)

    Book  Google Scholar 

  • Fairfield, T., Charman, A.E.: Explicit Bayesian Analysis for Process Tracing: Guidelines, Opportunities, and Caveats. Polit. Anal. 25(July), 363–380 (2017)

    Article  Google Scholar 

  • Freedman, D.A.: Statistical Models and Causal Inference: A Dialogue with the Social Sciences. Cambridge University Press, Cambridge (2011)

    Google Scholar 

  • George, A.L., Bennett, A.: Case Studies and Theory Development. MIT Press, Cambridge (2005)

    Google Scholar 

  • Gerring, J.: Social Science Methodology: A Unified Framework, 2d edn. Cambridge University Press, Cambridge (2012)

    Google Scholar 

  • Gerring, J.: Case Study Research: Principles and Practices, 2d edn. Cambridge University Press, Cambridge (2017)

    Google Scholar 

  • Glassner, B., Moreno, J.D. (eds.): The Qualitative-Quantitative Distinction in the Social Sciences, p. 112. Boston Studies in the Philosophy of Science, Dordrecht (1989)

    Google Scholar 

  • Goertz, G., Mahoney, J.: A Tale of Two Cultures: Contrasting Qualitative and Quantitative Paradigms. Princeton University Press, Princeton (2012)

    Book  Google Scholar 

  • Greene, W.H.: Econometric Analysis, 5th edn. Prantice Hall, New Jersey (2002)

  • Hafner-Burton, E.M.: Trading human rights: how preferential trade agreements influence government repression. Int. Organ. 59(3), 593–629 (2005)

    Article  Google Scholar 

  • Hammersley, M.: Deconstructing the Qualitative-Quantitative Divide. In: Brannen, J. (ed.) Mixing Methods: Qualitative and Quantitative Research. Aldershot, Avebury (1992)

    Google Scholar 

  • Humphreys, M., Jacobs, A.M.: Mixing methods: a Bayesian approach. Am. Polit. Sci. Rev. 109, 653–673 (2015)

    Article  Google Scholar 

  • Johnson, S.G.B., Ahn, W.-K.: Causal Mechanisms. In: Waldmann, M.R. (ed.) The Oxford Handbook of Causal Reasoning, pp. 127–146. Oxford University Press, New York (2017)

    Google Scholar 

  • Johnson, Samuel G. B., Amanda Royka, Peter McNally, Frank Keil.: The False Promise of Sexiness: Perceived Importance, Interest, and Citation Patterns for Counterintuitive Research Findings. PsyArXiv. (2019). https://doi.org/10.31234/osf.io/45rth.

  • Kapiszewski, D., MacLean, L.M., Read, B.L.: Field Research in Political Science: Practices and Principles. Cambridge University Press, Cambridge (2015)

    Book  Google Scholar 

  • King, G., Keohane, R., Verba, S.: Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton University Press, Princeton (1994)

    Book  Google Scholar 

  • Lieberman, E.S.: Nested analysis as a mixed-method strategy for comparative research. Am. Polit. Sci. Rev. 99(3), 435–452 (2005)

    Article  Google Scholar 

  • Lieberson, S.: Making it Count: The Improvement of Social Research and Theory. University of California Press, Berkeley (1985)

    Google Scholar 

  • Lieberson, S.: Small N’s and Big Conclusions: An Examination of the Reasoning in Comparative Studies Based on a Small Number of Cases. In: Ragin, C.S., Becker, H.S. (eds.) What Is a Case? Exploring the Foundations of Social Inquiry. Cambridge University Press, Cambridge (1992)

    Google Scholar 

  • Lieberson, S.: More on the uneasy case for using mill-type methods in small-N comparative studies. Soc. Forces 72(4), 1225–1237 (1994)

    Article  Google Scholar 

  • Lukin, Stephanie M., Pranav Anand, Marilyn Walker, Steve Whittaker.: Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion. arXiv:1708.09085 [cs.CL] (2017)

  • Mahoney, M.J.: Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cogn. Ther. Res. I, 161–175 (1977)

    Article  Google Scholar 

  • Mahoney, M.J.: Psychology of the scientists: an evaluative review. Soc. Stud. Sci. 9(3), 349–375 (1979)

    Article  Google Scholar 

  • Mahoney, J., Goertz, G.: A tale of two cultures: contrasting quantitative and qualitative research. Polit. Anal. 14(3), 227–249 (2006)

    Article  Google Scholar 

  • Mahoney, M.J., Kuzdin, A.E., Kenigsberg, M.: Getting published. Cogn. Ther. Res. 2(1), 69–70 (1978)

    Article  Google Scholar 

  • Malterud, K.: Qualitative research: standards, challenges, and guidelines. The Lancet 358(9280), 483–488 (2001)

    Article  Google Scholar 

  • McLaughlin, E.: Oppositional poverty: the quantitative/qualitative divide and other dichotomies. The Sociological Review 39(May), 292–308 (1991)

    Article  Google Scholar 

  • Miller, C.C.: Peer Review in the Organizational and Management Sciences: Prevalence and Effects of Reviewer Hostility, Bias, and Dissensus. Acad. Manag. J. 49(3), 425–431 (2006)

    Article  Google Scholar 

  • Morgan, D.L.: Integrating Qualitative & Quantitative Methods: A Pragmatic Approach. Sage, Los Angeles, CA (2014)

    Book  Google Scholar 

  • Nisbett, R., Wilson, T.D.: Telling more than we can know: Verbal reports on mental processes. Psychol. Rev. 84(3), 231–259 (1977)

    Article  Google Scholar 

  • Nylenna, M., Riis, P., Karlsson, Y.: Multiple blinded reviews of the same two manuscripts. effects of referee characteristics and publication language. J. Am. Med. Assoc. 272, 149–151 (1994)

    Article  Google Scholar 

  • Oaksford, M., Chater, N.: Bayesian Rationality: The Probabilistic Approach to Human Reasoning. Oxford University Press, Oxford (2007)

    Book  Google Scholar 

  • Peters, D.P., Ceci, S.J.: Peer-reviewed practices of psychological journals: the fate of published articles, submitted again. Behav. Brain Sci. 5, 111–125 (1982)

    Article  Google Scholar 

  • Ross, M.L.: Oil, Islam and women. Am. Polit. Sci. Rev. 102(1), 107–123 (2008)

    Article  Google Scholar 

  • Rossman, G.B., Rallis, S.F.: Learning in the Field: An Introduction to Qualitative Research. Sage, Thousand Oaks, CA (1998)

    Google Scholar 

  • Seawright, J.: Multi-Method Social Science: Combining Qualitative and Quantitative Tools. Cambridge University Press, Cambridge (2016)

    Book  Google Scholar 

  • Seawright, J., Gerring, J.: Case-selection techniques in case study research: a menu of qualitative and quantitative options. Polit. Res. Quart. 61(2), 294–308 (2008)

    Article  Google Scholar 

  • Sekhon, J.S.: Quality meets quantity: case studies, conditional probability and counterfactuals. Perspect. Polit. 2(2), 281–293 (2004)

    Article  Google Scholar 

  • Shweder, R.A.: Quanta and Qualia: What is the ‘Object’ of Ethnographic Method? In: Jessor, R., Colby, A., Shweder, R.A. (eds.) Ethnography and Human Development: Context and Meaning in Social Inquiry. University of Chicago Press, Chicago (1996)

    Google Scholar 

  • Smith, R.: Peer review: a flawed process at the heart of science and journals. J. r. Soc. Med. 99(4), 178–182 (2006)

    Article  Google Scholar 

  • Snow, C.P.: The Two Cultures. Cambridge University Press, Cambridge (1959)

    Google Scholar 

  • Teddlie, C., Tashakkori, A.: Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. Sage, Thousand Oaks, CA (2009)

    Google Scholar 

  • Trejo, G.: Religious Competition and Ethnic Mobilization in Latin America: Why the Catholic Church Promotes Indigenous Movements in Mexico. Am. Polit. Sci. Rev. 103(3), 323–342 (2009)

    Article  Google Scholar 

  • Waldner, D.: Process Tracing and Causal Mechanisms. In: Kincaid, H. (ed.) Oxford Handbook of Philosophy of Social Science, pp. 65–84. Oxford University Press, Oxford (2012)

    Google Scholar 

  • Weller, N., Barnes, J.: Finding pathways: Mixed-method research for studying causal mechanisms. Cambridge University Press, Cambridge (2014)

    Book  Google Scholar 

  • White, H.: Combining Quantitative and Qualitative Approaches in Poverty Analysis. World Dev. 30(3), 511–522 (2002)

    Article  Google Scholar 

  • Wilson, E.J., Sherrell, D.L.: Source Effects in Communication and Persuasion Research: A Meta-Analysis of Effect Size. J. Acad. Mark. Sci. 21(Spring), 101–112 (1993)

    Article  Google Scholar 

  • Worchel, S., Andreoli, V., Eason, J.: Is the Medium the Message? A Study of the Effects of Media, Communicator, and Message Characteristics on Attitude Change. J. Appl. Soc. Psychol. 5, 157–172 (1975)

    Article  Google Scholar 

  • Yarkoni, Tal.: The Generalizability Crisis. PsyArXiv. (2019) https://doi.org/10.31234/osf.io/jqw35

Download references

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. National University of San Martín, Buenos Aires, Argentina

    Alejandro Avenburg

  2. University of Texas, Austin, USA

    John Gerring

  3. Northwestern University, Evanston, USA

    Jason Seawright

Authors
  1. Alejandro Avenburg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Gerring
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jason Seawright
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alejandro Avenburg.

Ethics declarations

Conflicts of interest

No conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: In the original version the authors inadvertently included a sentence which misrepresents the work of Beach and Pedersen (2013) in the section “Does method matter”. The following correction has been made to better reflect the cited work: “Work that advocates for the use qualitative methods often carries the implication that qualitative evidence is more trustworthy than quantitative evidence, at least for some purposes, e.g., for elucidating causal mechanisms (Beach and Pedersen 2013, Gerring 2017).”

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 101 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Avenburg, A., Gerring, J. & Seawright, J. How do social scientists reach causal inferences? A study of reception. Qual Quant 57, 257–275 (2023). https://doi.org/10.1007/s11135-022-01353-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-022-01353-5

Keywords

Search

Navigation