Skip to main content
SpringerLink
Log in

Public opinion on climate change in the USA: to what extent can it be nudged by questionnaire design features?

  • Published:
Climatic Change Aims and scope Submit manuscript

Abstract

Policymakers may wish to take into account public opinion on climate change as they craft legislation, but if public opinion changes substantially in response to seemingly trivial changes in survey questionnaire design, perhaps such reliance would be unwise. This paper examines 110 experiments implemented in surveys of truly random samples of American adults between 2012 and 2018 (N = 4414), exploring the extent to which answers to questions were influenced by order and wording manipulations. Of 144 tests, 31 (22%) yielded statistically significant effects. Adjustments for multiple hypothesis tests reduced this percentage to between 7 and 9%. The effect sizes are routinely small. These results are consistent with the conclusion that survey results on climate change issues are relatively robust, so policymakers can take them seriously if they wish to do so.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abeles A, Howe L, Krosnick JA, MacInnis B (2019) Perception of public opinion on global warming and the role of opinion deviance. J Env Psych 63:118–129

    Article  Google Scholar 

  • American Clean Energy and Security Act, H.R. 2454, 111th Cong. (2009)

  • Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 57(1):289–300

    Google Scholar 

  • Bishop G, Smith A (2001) Response-order effects and the early Gallup split-ballots. Public Opin Quart 65(4):479–505

    Article  Google Scholar 

  • Cantril H (1944) Gauging public opinion. Princeton Univ Press, Princeton

  • Chang L, Krosnick JA (2009) National surveys Via RDD telephone interviewing versus the Internet. Public Opin Quart 73:641–678

    Article  Google Scholar 

  • Climate Stewardship and Innovation Act, S. 280, 110th Cong. (2007)

  • Converse P (1964) The nature of belief systems in mass publics. Crit Rev 18(1-3):1–74

    Article  Google Scholar 

  • Converse P (1974) Comment: the status of nonattitudes. Am Polit Sci Rev 68(2):650–660

  • Cristea I, Ioannidis J (2018) P values in display items are ubiquitous and almost invariably significant: a survey of top science journals. PLOS ONE 13:e0197440

    Article  Google Scholar 

  • Dahl R (1989) Democracy and its critics. Yale Univ Press, New Haven

  • Erikson RS, Tedin KL (2015) American public opinion: its origins, content and impact. Routledge, New York

  • Fanelli D (2010) Do pressures to publish increase scientists' bias? An empirical support from US states data. PLOS ONE 5:e10271

    Article  Google Scholar 

  • Fraley R, Vazire S (2014) The n-pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLOS ONE 9:e109019

    Article  Google Scholar 

  • Franco A, Malhotra N, Simonovits G (2014) Publication bias in the social sciences: unlocking the file drawer. Science 345(6203):1502–1505

    Article  Google Scholar 

  • Gardner G, Stern P (2008) The short list: the most effective actions U.S. households can take to curb climate change. Environment: Science and Policy for Sustainable Development 50:12–25

    Google Scholar 

  • Holbrook A, Green M, Krosnick JA (2003) Telephone versus face-to-face interviewing of national probability samples with long questionnaires. Public Opin Quart 67:79–125

    Article  Google Scholar 

  • Holbrook A, Krosnick JA, Moore D, Tourangeau R (2007) Response order effects in dichotomous categorical questions presented orally. Public Opin Quart 71:325–348

    Article  Google Scholar 

  • Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat:65–70

  • Hommel G (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75(2):383–386

    Article  Google Scholar 

  • Kalton G, Collins M, Brook L (1978) Experiments in wording opinion questions. J R Stat Soc C 27(2):149–161

    Google Scholar 

  • Krosnick JA (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys. Appl Cognitive Psych 5:213–236

    Article  Google Scholar 

  • Krosnick JA (1999) Survey research. Annu Rev Psychol 50(1):537–567

    Article  Google Scholar 

  • Krosnick JA, Alwin DF (1987) An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opin Quart 51:201–219

    Article  Google Scholar 

  • Krosnick JA, MacInnis B (2020) Climate insights 2020: overall trends. Resources for the future. https://www.rff.org/publications/reports/climateinsights2020/. Accessed 25 February 2021

  • Krosnick JA, Presser S (2010) Question and questionnaire design. In: Wright JD, Marsden PV (eds) The handbook of survey research. Emerald Publishing, West Yorkshire, pp 269–319

    Google Scholar 

  • Kyselá E, Ščasný M, Zvěřinová I (2019) Attitudes toward climate change mitigation policies: a review of measures and a construct of policy attitudes. Clim Policy 19:878–892

    Article  Google Scholar 

  • Lavrakas P (2008) Surveys by Telephone. In: Donsbach Wolfgang, Traugott Michael W. (eds) The SAGE Handbook of Public Opinion Research. SAGE Publications Ltd, pp 249–261

    Chapter  Google Scholar 

  • Lee T, Markowitz E, Howe P, Ko C, Leiserowitz A (2015) Predictors of public climate change awareness and risk perception around the world. Nat Clim Change 5:1014–1020

    Article  Google Scholar 

  • Lumley T (2020) survey: analysis of complex survey samples. R package version 3.37. https://cran.r-project.org/web/packages/survey/index.html

  • MacInnis B, Krosnick JA, Ho AS, Cho MJ (2018) The accuracy of measurements with probability and nonprobability survey samples: replication and extension. Public Opin Quart 82(4):707–744

    Article  Google Scholar 

  • MacInnis B, Miller JM, Krosnick JA, Below C, Lindner M (2021) Candidate name order effects in New Hampshire: evidence from primaries and from general elections with party column ballots. PLOS ONE 3:e0248049

    Article  Google Scholar 

  • Malhotra N (2008) Completion time and response order effects in web surveys. Public Opin Quart 72(5):914–934

    Article  Google Scholar 

  • Malhotra N, Krosnick JA (2007) The effect of survey mode and sampling on inferences about political attitudes and behavior: comparing the 2000 and 2004 ANES to Internet surveys with nonprobability samples. Polit Anal 15(3):286–323

    Article  Google Scholar 

  • Marcott SA, Shakun JD, Clark PU, Mix AC (2013) A reconstruction of regional and global temperature for the past 11,300 years. Science 339(6124):1198–1201

    Article  Google Scholar 

  • Miller R (1981) Simultaneous statistical inference. Springer, New York

  • Motta M, Chapman D, Stecula D, Haglin K (2019) An experimental examination of measurement disparities in public climate change beliefs. Climatic Change 154(1):37–47

    Article  Google Scholar 

  • Napolitano J, Johnson K (2018) Universities should lead efforts to slow climate change, if the federal government won’t. The Washington Post. https://www.washingtonpost.com/education/2018/09/17/universities-should-lead-efforts-slow-climate-change-if-federal-government-wont/

  • Narayan S, Krosnick JA (1996) Education moderates some response effects in attitude measurement. Public Opin Quart 60(1):58–88

    Article  Google Scholar 

  • National Surveys on Energy and Environment (2014) American acceptance of global warming retreats in wake of winter 2014. Michigan Univ Center for Local, State, and Urban Policy http://closup.umich.edu/issues-in-energy-and-environmental-policy/12/american-acceptance-of-global-warming-retreats-in-wake-of-winter-2014. Accessed 11 May 2019

  • Page B, Shapiro R (2010) The rational public: fifty years of trends in Americans' policy preferences. Univ. of Chicago Press, Chicago

  • Pasek J, Schneider D, Krosnick JA, Tahk A, Ophir E, Milligan C (2014) Prevalence and moderators of the candidate name-order effect: evidence from statewide general elections in California. Public Opin Quart 78(2):416–439

    Article  Google Scholar 

  • Pasek J, Tahk A, Culter G, Schwemmle M (2020). Weights: weighting and weighted statistics. R package version 1.0.1. https://CRAN.R-project.org/package=weights

  • Payne S (1951) The art of asking questions. Princeton Univ Press, Princeton

  • Political Psychology Research Group (2020) American public opinion on global warming. Stanford Univ. http://climatepublicopinion.stanford.edu/

  • Rothstein H, Sutton A, Borenstein M (2006) Publication bias in meta-analysis. John Wiley & Sons, New York

  • Schuman H, Presser S (1981) Experiments in question wording, form and context in attitude surveys. Academic, New York

  • Schuman H, Presser S (1996) Questions and answers in attitude surveys: experiments on question form, wording, and context. Sage, Thousand Oaks

  • Schwarz N, Bles H (1992) Scandals and the public's trust in politicians: assimilation and contrast effects. Pers Soc Psychol B 18:574–579

    Article  Google Scholar 

  • Sterman J (2008) Risk communication on climate: mental models and mass balance. Science 322:532–533

    Article  Google Scholar 

  • Streiner DL (2015) Best (but oft-forgotten) practices: the multiple problems of multiplicity—whether and how to correct for many statistical tests. Am J Clin Nutr 102(4):721–728

    Article  Google Scholar 

  • Tvinnereim E, Fløttum K (2015) Explaining topic prevalence in answers to open-ended survey questions about climate change. Nat Clim Change 5:744–747

  • Yeager DS, Krosnick JA, Chang L, Javitz HS, Levendusky MS, Simpser A, Wang R (2011) Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opin Quart 75(4):709–747

    Article  Google Scholar 

Download references

Data and code availability

Analyses were performed using R v.4.0.2 (https://www.R-project.org/). Data and code needed to replicate the analyses are available at: https://osf.io/9k3cp

Funding

The 2012 data collection was supported by the National Science Foundation grant 1042938. The 2013 data collection was supported by Stanford University, Resources for the Future, and USA Today. The 2013 data collection was supported by Stanford University and University of Arizona. The 2015 data collection was supported by Stanford University, Resources for the Future, and The New York Times. The 2018 data collection was supported by Stanford University and Resources for the Future.

Author information

Authors and Affiliations

  1. Department of Communication, Stanford University, Stanford, CA, 94305, USA

    Catherine Chen & Bo MacInnis

  2. Department of Political Science, Stanford University, Stanford, USA

    Matthew Waltman

  3. Departments of Communication, Political Science, and Psychology, Stanford University, Stanford, USA

    Jon A. Krosnick

Authors
  1. Catherine Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo MacInnis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew Waltman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jon A. Krosnick
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The first, second, and fourth authors wrote the manuscript. The first and second authors analyzed the data. The first, second, and fourth authors developed the study idea. The third author participated in manuscript preparation.

Corresponding author

Correspondence to Catherine Chen.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

IRB approval number for each of surveys

2012: IRB-23464

2013: IRB-29317

2014: IRB-31688

2015: IRB-32562

2018: IRB-46124

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

ESM 1

(DOCX 328 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, C., MacInnis, B., Waltman, M. et al. Public opinion on climate change in the USA: to what extent can it be nudged by questionnaire design features?. Climatic Change 167, 35 (2021). https://doi.org/10.1007/s10584-021-03194-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10584-021-03194-x

Keywords

Search

Navigation