Skip to main content
SpringerLink
Log in

Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues

  • Original Paper
  • Published:
Political Behavior Aims and scope Submit manuscript

Abstract

Scholars have made considerable strides in evaluating and improving the external validity of experimental research. However, little attention has been paid to a crucial aspect of external validity – the topic of study. Researchers frequently develop a general theory and hypotheses (e.g., about policy attitudes), then conduct a study on a specific topic (e.g., environmental attitudes). Yet, the results may vary depending on the topic chosen. In this paper, we develop the idea of topic sampling – rather than studying a single topic, we randomly sample many topics from a defined population. As an application, we combine topic sampling with a classic survey experiment design on partisan cues. Using a hierarchical model, we efficiently estimate the effect of partisan cues for each policy, showing that the size of the effect varies considerably, and predictably, across policies. We conclude with advice on implementing our approach and using it to improve theory testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Modern software makes fitting the hierarchical model easy. For simple models, (restricted) maximum likelihood estimation works well (e.g., Bates et al. 2015). For more complex models, full posterior simulation offers a robust alternative (e.g., Bürkner 2017; Goodrich et al. 2020).

  2. In a recent study, Barber and Pope (2019) study 10 issues at once, providing perhaps the most generalizable findings. Yet, the study was restricted to 10 topics on which former President Trump publicly took stances on both sides of the issue. Moreover, their analysis focuses only on the average effect, while analyses in the supplementary materials suggest meaningful but unexamined heterogeneity in effect size across issues.

  3. We used 2016 rather than a later year because 2017 polls were still being added to the Roper database at the time.

  4. Specifically, the terms were “favor or oppose or for or against or should or approve or support.” Diagnostic checks suggested that this set of terms included virtually all policy attitudes measured in this time period.

  5. Questions that asked about the same policy but used different question wording were considered redundant.

  6. Of the 48 policies we sampled (see below), 40% also showed up three years later in the 2019 Roper population, suggesting considerable stability in the population over time.

  7. To validate our coding, we compared assigned values to partisan differences in the data among untreated respondents. In 77% of the cases, we observed a statistically significant difference in the expected direction. For the remaining 23% of cases, there was no significant partisan difference. Thus, our coding reliably mapped onto partisan differences.

  8. This study was reviewed and approved by the London School of Economics Research Ethics Committee. All respondents gave informed consent before participating in the study.

  9. Using the average of posterior simulations with Stan (Carpenter et al. 2017) produces nearly identical estimates of the awareness of the parties’ positions on each issue.

  10. This possibility of investigating many moderators underscores the importance of pre-registration. We also note that the topic sampling should be described in the pre-registration document just as a researcher would describe the sampling of participants.

References

  • Arceneaux, K. (2007). Can partisan cues diminish democratic accountability? Political Behavior, 30(2), 139–160.

    Article  Google Scholar 

  • Bakker, B. N., & Lelkes, Y., and Ariel Malka (2020). Understanding partisan cue receptivity: Tests of predictions from the Bounded rationality and expressive utility perspectives. Journal of Politics, 82(3), 1061–1077.

    Article  Google Scholar 

  • Bansak, K., Hainmueller, J., Hopkins, D. J., & Teppei Yamamoto. (2021). “Conjoint Survey Experiments.” Ch. 2 in Advances in Experimental Political Science, eds. Druckman, J. N., & Green, D. P. (p. 19). Cambridge: Cambridge University Press.

  • Barabas, J., and Jennifer Jerit (2010). Are Survey experiments externally valid? American Political Science Review, 104(02), 226–242.

    Article  Google Scholar 

  • Barber, M., and Jeremy C. Pope (2019). Does Party Trump ideology? Disentangling Party and ideology in America. American Political Science Review, 113(1), 38–54.

    Article  Google Scholar 

  • Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using Lme4. Journal of Statistical Software, 67(1), 1–48.

    Article  Google Scholar 

  • Berinsky, A. J., Gregory, A., Huber, & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.Com’s mechanical Turk. Political Analysis, 20(3), 351–368.

    Article  Google Scholar 

  • Brandt, M. J., Jarret, T., & Crawford (2019). Studying a heterogeneous array of Target Groups can help us understand prejudice. Current Directions in Psychological Science, 28(3), 292–298.

    Article  Google Scholar 

  • Brutger, R. (2022). “Abstraction and Detail in Experimental Design.” American Journal of Political Science,

  • Bullock, W., Imai, K., & Shapiro, J. (2011). Statistical Analysis of Endorsement Experiments: Measuring Support for Militant Groups in Pakistan. Political Analysis, 19(4), 363–384.

    Article  Google Scholar 

  • Bullock, J. G. (2011). “Elite Influence on Public Opinion in an Informed Electorate.” American Political Science Review, 105(03), 496–515.

    Article  Google Scholar 

  • Bürkner, P. C. (2017). Brms: An R Package for bayesian Multilevel Models using Stan. Journal of Statistical Software, 80(1), 1–28.

    Article  Google Scholar 

  • Butler, D. M., and Jonathan Homola (2017). An empirical justification for the Use of racially distinctive names to signal race in experiments. Political Analysis, 25(1), 122–130.

    Article  Google Scholar 

  • Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). “Stan: A Probabilistic Programming Language.” Journal of Statistical Software, 76(1): 1–32.

    Article  Google Scholar 

  • Cameron, A., Colin, J. B., Gelbach, & Miller, D. L. (2011). Robust inference with Multiway Clustering. Journal of Business & Economic Statistics, 29(2), 238–249.

    Article  Google Scholar 

  • Chong, D., and James N. Druckman (2007). Framing Public Opinion in competitive democracies. American Political Science Review, 101(04), 637–655.

    Article  Google Scholar 

  • Chong, D., and James N. Druckman (2010). Dynamic public opinion: Communication Effects over Time. American Political Science Review, 104(04), 663–680.

    Article  Google Scholar 

  • Chong, D., and James N. Druckman (2012). Counterframing Effects. The Journal of Politics, 75(01), 1–16.

    Article  Google Scholar 

  • Chung, Y., et al. (2013). “A Nondegenerate Penalized Likelihood Estimator for Variance Parameters in Multilevel Models.” Psychometrika, 78(4), 685–709.

    Article  Google Scholar 

  • Clifford, S., Sheagley, G., & Piston, S. (2021). Increasing Precision without Altering Treatment Effects: Repeated measures designs in Survey experiments. American Political Science Review, 115(3), 1048–1065.

    Article  Google Scholar 

  • Coppock, A., & Green, D. P. (2015). Assessing the correspondence between experimental results obtained in the lab and field: A review of recent Social Science Research. Political Science Research and Methods, 3(01), 113–131.

    Article  Google Scholar 

  • Coppock, A., Leeper, T. J., & Mullinix, K. J. (2018). “Generalizability of Heterogeneous Treatment Effect Estimates across Samples.” Proceedings of the National Academy of Sciences: 201808083.

  • Ditto, P. H., et al. (2019). At least Bias is bipartisan: A Meta-Analytic comparison of partisan Bias in Liberals and Conservatives. Perspectives on Psychological Science, 14(2), 273–291.

    Article  Google Scholar 

  • Druckman, J. N., Thomas, J., & Leeper (2012). Learning more from political communication experiments: Pretreatment and its Effects. American Journal of Political Science, 56(4), 875–896.

    Article  Google Scholar 

  • Franco, A., & Malhotra, N., and Gabor Simonovits (2014). Publication Bias in the Social Sciences: Unlocking the file drawer. Science (New York N Y), 345(6203), 1502–1505.

    Article  Google Scholar 

  • Goodrich, B., Gabry, J., Ali, I., & Brilleman, S. (2020). rstanarm: Bayesian applied regression modeling via Stan. R package version 2.21.1. https://mc-stan.org/rstanarm

  • Gelman, A., and Jennifer Hill (2007). Data Analysis using regression and Multilevel/Hierarchical models. New York: Cambridge University Press.

    Google Scholar 

  • Gelman, A., Jennifer Hill, and Masanao Yajima (2012). Why we (usually) don’t have to worry about multiple comparisons. Journalof Research on Educational Effectiveness, 5(2), 189–211.

    Article  Google Scholar 

  • Gilens, M. (2001). Political ignorance and collective policy preferences. American Political Science Review, 95(2), 379–396.

    Article  Google Scholar 

  • Green, D. P., & Kern, H. L. (2012). Modeling Heterogeneous Treatment Effects in Survey experiments with bayesian additive regression trees. Public Opinion Quarterly, 76(3), 491–511.

    Article  Google Scholar 

  • Grieco, J. M., Gelpi, C., & Reifler, J., and Peter D. Feaver (2011). Let’s get a second opinion: International Institutions and American Public support for War1. International Studies Quarterly, 55(2), 563–583.

    Article  Google Scholar 

  • Guess, A., & Coppock, A. (2020). Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments. British Journal of Political Science, 50(4), 1497–1515.

    Article  Google Scholar 

  • Guisinger, A., and Elizabeth N. Saunders (2017). Mapping the Boundaries of Elite Cues: How elites shape Mass Opinion across International Issues. International Studies Quarterly, 61(2), 425–441.

    Article  Google Scholar 

  • Hainmueller, J., & Hopkins, D. J., and Teppei Yamamoto (2014). Causal inference in Conjoint Analysis: Understanding multidimensional choices via stated preference experiments. Political Analysis, 22(1), 1–30.

    Article  Google Scholar 

  • Jerit, J. (2009). How predictive appeals affect policy opinions. American Journal of Political Science, 53(2), 411–426.

    Article  Google Scholar 

  • Jerit, J. (2012). Partisan perceptual bias and the information environment. The Journal of Politics, 74(03), 672–684.

    Article  Google Scholar 

  • Jerit, J., & Barabas, J. (2013). Comparing contemporaneous laboratory and field experiments on media effects. Public Opinion Quarterly, 77(1), 256–282.

    Article  Google Scholar 

  • Johnston, C. D., Lavine, H., & Federico, C. M. (2017). Open versus closed: Personality, identity, and the politics of redistribution. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Johnston, C. D. (2015). Personality dispositions and political preferences across hard and easy issues. Political Psychology, 36(1), 35–53.

    Article  Google Scholar 

  • Kam, C. D., & Trussler, M. J. (2016). At the nexus of observational and experimental research: Theory, specification, and analysis of experiments with heterogeneous treatment effects. Political Behavior, 39, 1–27.

    Google Scholar 

  • Kam, C. D. (2005). Who Toes the Party Line? Cues, Values, and Individual Differences. Political Behavior, 27, 163–182.

    Article  Google Scholar 

  • Kay, M. (2019). “Tidybayes: Tidy Data and Geoms for Bayesian Models.” https://doi.org/10.5281/zenodo.1308151

  • Kuklinski, J. H., et al. (2000). Misinformation and the currency of democratic citizenship. The Journal of Politics, 62(3), 790–816.

    Article  Google Scholar 

  • Lenz, G. S. (2009). Learning and opinion change, not priming: Reconsidering the priming hypothesis. American Journal of Political Science, 53(4), 821–837.

    Article  Google Scholar 

  • Lindner, N. M., & Nosek, B. A. (2009). Alienable Speech: Ideological variations in the application of Free-Speech Principles. Political Psychology, 30(1), 67–92.

    Article  Google Scholar 

  • McDermott, R. (2002). Experimental methodology in Political Science. Political Analysis, 10(4), 325–342.

    Article  Google Scholar 

  • Miller, J. M., & Krosnick, J. A. (2004). Threat as a motivator of political activism: A field experiment. Political Psychology, 25(4), 507–523.

    Article  Google Scholar 

  • Nicholson, S. P. (2012). “Polarizing Cues.” American Journal of Political Science, 56(1), 52–66.

    Article  Google Scholar 

  • Nyhan, B., and Jason Reifler (2010). When Corrections fail: The persistence of political Misperceptions. Political Behavior, 32(2), 303–330.

    Article  Google Scholar 

  • Ryan, T. J. (2014). “Reconsidering Moral Issues in Politics.” The Journal of Politics, 76(2), 1–18.

    Article  Google Scholar 

  • Ryan, T. J. (2017). No compromise: Political consequences of moralized attitudes. American Journal of Political Science, 61(2), 409–423.

    Article  Google Scholar 

  • Simas, E. N., & Milita, K., and John Barry Ryan (2021). Ambiguous rhetoric and legislative accountability. Journal of Politics, 83(4), 1695–1705.

    Article  Google Scholar 

  • Slothuus, R. (2016). Assessing the influence of political parties on public opinion: The challenge from pretreatment effects. Political Communication, 33(2), 302–327.

    Article  Google Scholar 

  • Swire-Thompson, Briony, J., DeGutis, & Lazer, D. (2020). Searching for the backfire effect: Measurement and design considerations. Journal of Applied Research in Memory and Cognition, 9(3), 286–299.

    Article  Google Scholar 

  • Tappin, B. M. (2022). “Estimating the Between-Issue Variability in Party Elite Cue Effects.” Public Opinion Quarterly, 86(4), 862–885

    Article  Google Scholar 

  • Tavits, M. (2007). Principle vs. pragmatism: Policy shifts and political competition. American Journal of Political Science, 51(1), 151–165.

    Article  Google Scholar 

  • Tesler, M. (2015). Priming Predispositions and changing policy positions: An account of when Mass Opinion is primed or changed. American Journal of Political Science, 59(4), 806–824.

    Article  Google Scholar 

  • Vehtari, A., & Gelman, A. (2017). Practical bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27, 1413–1432.

    Article  Google Scholar 

  • Wells, G. L. (2016). Stimulus sampling and social psychological experimentation. Personality and Social Psychology Bulletin, 25(9), 1115–1125.

    Article  Google Scholar 

  • Wood, T., & Porter, E. (2019). “The elusive backfire effect: mass attitudes’ steadfast factual adherence.” Political Behavior, 41, 135–163.

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank Brandon de la Cuesta and Brendan Nyhan for helpful comments.

Author information

Authors and Affiliations

  1. University of Houston, 3551 Cullen Blvd. Rm 447, Houston, TX, 77204, USA

    Scott Clifford

  2. London School of Economics, Houghton Street, London, UK

    Thomas J. Leeper

  3. Florida State University, 600 W. College Avenue, Tallahassee, FL, 32306, USA

    Carlisle Rainey

Authors
  1. Scott Clifford
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas J. Leeper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carlisle Rainey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Scott Clifford.

Ethics declarations

Conflict of interest

This study was funded by the Danish Council for Independent Research award DFF–4003-00192B. The authors declare they have no financial interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 9914.4 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Clifford, S., Leeper, T.J. & Rainey, C. Generalizing Survey Experiments Using Topic Sampling: An Application to Party Cues. Polit Behav (2023). https://doi.org/10.1007/s11109-023-09870-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11109-023-09870-1

Keywords

Search

Navigation