Skip to main content
SpringerLink
Log in

Considering Mixed Mode Surveys for Questions in Political Behavior: Using the Internet and Mail to Get Quality Data at Reasonable Costs

  • Original Paper
  • Published:
Political Behavior Aims and scope Submit manuscript

Abstract

Telephone surveys have been a principle means of learning about the attitudes and behaviors of citizens and voters. The single mode telephone survey, however, is increasingly threatened by rising costs, the declining use of landline telephones, and declining participation rates. One solution to these problems has been the introduction of mixed-mode surveys. However, such designs are relatively new and questions about their representativeness and the intricacies of the methodology remain. We report on the representativeness of a post election mixed-mode (Internet and mail) survey design of 2006 general election voters. We compare sample respondent means to sample frame means on key demographic characteristics and examine how mail and Internet respondents differed in terms of attitudes, behaviors and demographics. We find that overall the Internet respondents were representative of the population and that respondent choice of mode did not influence item response. We conclude that mixed-mode designs may allow researchers to ask important questions about political behavior from their desktops.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Examples such as the American National Election Studies (ANES) and the General Social Survey (GSS) to the contrary.

  2. http://www.electionstudies.org/studypages/2008_2009panel/anes2008_2009panel.htm.

  3. Also see: http://web.mit.edu/polisci/portl/cces/sampledesign.html for a description of Polimetrix’s sampling method.

  4. See the Pew Internet and Life report at: http://www.pewinternet.org/PPF/r/182/report_display.asp.

  5. We thank New Mexico Secretary of State Rebecca Vigil-Giron and Colorado Secretary of State Gigi Dennis who provided us with the voter registration files for free.

  6. A frequency report of the Election Administration Survey can be found at: www.unm.edu/~atkeson.

  7. We strongly recommend a FAQ because the phone calls about this issue stopped after we included it on the website.

  8. Seventeen mail surveys were returned, all of them voters. In NM1 10% (n = 5) and in COCD1 about 5% (n = 3) of sample letters were returned, even though we had received no returned post cards from these addresses in previous contacts. Thus, lower mail delivery quality may be an important concern in terms of sample recruitment and in evaluating the quality of the sample. Therefore, when implementing this type of survey researchers should carefully monitor the returned mail and use first class mail to ensure return delivery.

  9. Because of page constraints one question had to be moved elsewhere.

  10. Many unreachable were inactive voters who are sent an official mailing from the County Clerk or the Secretary of State that is returned as undeliverable. This suggests that removing inactive registered voters from the population frame would be a prudent and cost effective research design measure in any future study iteration.

  11. A partial survey is less than 50% of substantive questions completed with something other than don’t know or refused.

  12. This is the minimum response rate (RR2) as defined by the AAPOR (AAPOR 2008).

  13. The cooperation rate is the minimum cooperation rate (COOP2) by AAPOR (AAPOR 2008).

  14. Our higher voter validation compared to the ANES may be due to the fact that the population frame is registered voters while the ANES population frame is citizens.

  15. We note there may be some error in our matching. There is also likely some error in the voter registration files (Cassel 2004; Presser et al. 1990). For example, the official 2006 canvass for Bernalillo County reported 198,611 voters, but the voter record file we have indicated that only 194,582 voters cast ballots, a difference of 4,029 votes. We also know that when voters move their vote history follow them to their new voting location, but the voting location for those votes does not. Therefore, some errors are created in the way in which the file is administered. Voter files are not well characterized yet (but see McDonald 2007).

  16. Because Colorado had very few Hispanic voters, we matched Hispanic surname with the New Mexico sample only (See Word and Perkins 1996).

  17. Spanish language contact letters, postcards and survey were not included in our study. Theoretically this choice could hurt our response rate among Spanish-speaking sample members. In a 2008 phone study, we provided this option and 1.8% of sample respondents chose to take the survey in Spanish suggesting the problem is relatively small. In addition, 80% of Hispanics speak English at home or describe themselves as fluent in English in Bernalillo County (U.S. Census Bureau 2008). According to the U.S. Census Bureau, New Mexico Hispanics have the highest English fluency across the states. Thus, any impact is likely very small.

  18. House campaign activities include: attending a rally, fundraising, canvassing, convincing others, writing a letter to a magazine/newspaper or Internet site, advertising (e.g. sign, bumper sticker, button), and contributing money. We also counted the number of ways respondents learned about the US House candidates in their district including: met him/her personally, attended a meeting or rally, contacted in-person by candidate, party or interest group, received an e-mail from candidate, party or interest groups, contacted by phone by candidate, party or interest group, visited candidate, party or interest group web site, or read online or local newspaper.

  19. We substituted the income mean for system-missing codes because of the large number of missing data.

  20. These include: No One Encouraged Respondent to Vote, Did Not Have to Show ID to Vote, Never Had a Problem with Voting and the Ideology of Democratic House Candidate, Trust in Government, Satisfaction with the 2006 Fall Election, and Political Discussion.

  21. The quality of voter registration files varies by states and localities and this may influence the ability to adequately pursue this type of design.

References

  • Atkeson, L. R., & Adams, A. N. (2010). A comparison of data quality in post election mixed mode and phone surveys. Presented at the Western Political Science Association’s Annual Meeting, April 1–3, San Francisco, California.

  • Atkeson, L. R., & Tafoya, L. (2008a). Close, but not close enough: Democrats lose again by the slimmest of margins in New Mexico’s first congressional district. In D. Magleby & K. Patterson (Eds.), War games: Issues and resources in the battle for control of congress. Boulder: Paradigm Publishers.

  • Atkeson, L. R., & Tafoya, L. (2008b). Surveying political activists: An examination of the effectiveness of a mixed-mode (Internet and mail) survey design. Journal of Elections Public Opinion and Parties, 18(4), 367–386.

    Article  Google Scholar 

  • Barreto, M. A., Streb, M. J., Marks, M., & Guerra, F. (2006). Do absentee voters differ from polling place voters? Public Opinion Quarterly, 70(2), 224–234.

    Article  Google Scholar 

  • Belli, R. F., Traugott, M. W., & Beckmann, M. N. (2001). What leads to voting overreports? Contrasts of overreporters and admitted non voters in the American National Election Studies. Journal of Official Statistics, 17(4), 479–498.

    Google Scholar 

  • Blumberg, S. J., & Luke, J. V. (2009). Wireless substitution: Early release of estimates from the National Health Interview Survey, July–December 2008. http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless200905.htm.

  • Bullock III, C. S., Hood III, M. V., & Gonzalez, J. (2007). Examining methods for identifying Latino voters. Election Law Journal, 6(2), 202–208.

    Article  Google Scholar 

  • Cassel, C. A. (2004). Voting records and validated voting studies. Public Opinion Quarterly, 68(1), 102–108.

    Article  Google Scholar 

  • Chang, L. C., & Krosnick, J. (2010). National Surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly, 73(4), 641–678.

    Article  Google Scholar 

  • Christian, L. M., & Dillman, D. A. (2004). The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly, 68(1), 57–80.

    Article  Google Scholar 

  • Cobanoglu, C., Warde, B., & Morec, P. J. (2001). A comparison of mail, fax, and web survey methods. International Journal of Market Research, 43, 441–452.

    Google Scholar 

  • Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64, 413–428.

    Article  Google Scholar 

  • De Leeuw, E. (2005). To mix or not to mix: Data collection modes in surveys. Journal of Official Statistics, 21, 233–255.

    Google Scholar 

  • De Leeuw, E., & Van der Zouwen J. (1988). Data quality in telephone and face to face surveys: A comparative meta-analysis. In R. Groves, P. P. Bimer, L. Lyberg, I. T. Massey, W. L. Nicholls, & J. Waksberg. Telephone survey methodology. New York: Wiley.

  • Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: Wiley.

    Google Scholar 

  • Dillman, D. A., & Christian, L. M. (2005). Survey mode as a source of instability in responses across surveys. Field Methods, 17(1), 30–52.

    Article  Google Scholar 

  • Dillman, D. A., R. Sangster, J. Tarnai, & T. Rockwood (1996). In M. T. Braverman, J. K. Slater (Eds.), Current issues in survey research: New directions for evaluation, #70 (Chap. 4, pp. 45–62). San Francisco: Jossey-Bass.

  • Fowler, F. J., Roman, A. M., Jr., & Di, Z. X. (1998). Mode effects in a survey of medicare prostate surgery patients. Public Opinion Quarterly, 62(1), 29–46.

    Article  Google Scholar 

  • Groves, R. M., & Couper, M. P. (1998). Non-response in Household Interview Surveys. New York: Wiley.

    Google Scholar 

  • Groves, R. M., Presser, S., & Dipko, S. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31.

    Article  Google Scholar 

  • Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing non-response in a national telephone survey. Public Opinion Quarterly, 64, 125–148.

    Article  Google Scholar 

  • Lensvelt-Mulders, G. J. L. M., P. Lugtig, & M. Hubregtse (2009). Separating selection bias and non-coverage in Internet panels using propensity matching. Survey Practice. http://surveypractice.org/.

  • Malhotra, N., & Krosnick, J. A. (2007). The effect of survey mode and sampling on inferences about political attitudes and behavior: Comparing the 2000 and 2004 ANES to Internet surveys with nonprobability samples. Political Analysis, 15, 286–323.

    Article  Google Scholar 

  • McDermott, M. L. (1998). Race and gender cues in low-information elections. Political Research Quarterly, 51(4), 895–918.

    Google Scholar 

  • McDonald, M. P. (2007). The true electorate: A cross validation of voter registration files and election survey demographics. Public Opinion Quarterly, 71(4), 588–602.

    Article  Google Scholar 

  • Merkle, D. M., & Edelman M. (2002). Non-response in exit polls: A comprehensive analysis. In R. M. Groves, D. A. Dillman, J. L. Eltinge, R. J. A. Little (Eds.), Survey non-response (pp. 243–258). New York: Wiley.

  • Presser, S., Traugott, M. W., & Traugott, S. (1990). Vote ‘Over’ reporting in surveys: The records or the respondents. Technical Report No. 39. Ann Arbor, MI: National Election Studies.

  • Rosen, R., & Gomes, T. (2004). Converting CES reporters from TDE to web data collection. In Paper presented at the Joint Statistical Meetings, Toronto, Canada.

  • Sanders, D., Clarke, H. D., Stewart, M. C., & Whitely, P. (2007). Does mode matter for modeling political choice? Evidence from the 2005 British Election Study. Political Analysis, 15, 257–286.

    Article  Google Scholar 

  • Schuman, H. (1992). Context effects: State of the past/state of the art. In N. Schwarz, S. Sudman (Eds.), Context effects in social and psychological research (pp. 5–20). New York: Springer-Verlag.

  • Schuman, H., & Presser, S. (1981). Questions and answers in attitude survey: Experiments on question form, wording and context. New York: Academic Press.

    Google Scholar 

  • Smyth, J. D., Dillman, D., Christian, L. M., & Stern, M. J. (2006). Effects of using visual design principles to group response options in web survey. International Journal of Internet Science, 1, 6–16.

    Google Scholar 

  • Sudman, S., Bradburn, N. M., & Schwarz, Norbert. (1996). Thinking about answers. San Francisco: Josey-Bass.

    Google Scholar 

  • The American Association for Public Opinion Research (AAPOR). (2008). Standard definitions: Final dispositions of case codes and outcome rates for survey (5th ed.). Lenexa, Kansas: AAPOR. http://www.aapor.org/Resources_for_Researchers.htm.

  • Tourangeau, R., Couper, M., & Conrad, F. (2004). Spacing, position and order: Interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368–393.

    Article  Google Scholar 

  • Traugott, S. (1989). Validating self-reported vote: 1964–1988. ANES technical report series, No. nes010152. Ann Arbor Michigan: National Election Studies.

  • U.S. Census Bureau. (2008). American Community Survey 2006–2008 three year estimates. Data profile: New Mexico: Bernalillo County: Hispanics (All). Accessed 15 Jan 2010.

  • Visser, P. S., Krosnick, J. A., Marquette, J., & Curtin, M. (1996). Mail surveys for election forecasting? An evaluation of the Columbus dispatch poll. Public Opinion Quarterly, 60(2), 181–227.

    Article  Google Scholar 

  • Word, D. L., Perkins Jr., R. C. (1996). Building a Spanish surname list for the 1990s—a new approach to an old problem. US Census Bureau, March 4, 1996. http://www.census.gov/genealogy/www/spanname.html.

Download references

Acknowledgements

The authors would like to thank Wendy Hansen, Tim Krebs, Walter Stone and the editors for their helpful comments, edits, and feedback. We would like to thank the Resource Allocation Committee at the University of New Mexico and the College of Arts and Sciences at the University of New Mexico for providing us with the funding to complete this study. Of course any errors are our own.

Author information

Authors and Affiliations

  1. Department of Political Science, MSC05-3070, University of New Mexico, 1 University of New Mexico, Albuquerque, NM, 87131-0001, USA

    Lonna Rae Atkeson, Alex N. Adams & Lisa A. Bryant

  2. Center for Development and Disability, University of New Mexico, Albuquerque, NM, USA

    Luciana Zilberman

  3. Department of Political Science, 1782, Campus Delivery, Colorado State University, Ft Collins, CO, 80523-1782, USA

    Kyle L. Saunders

Authors
  1. Lonna Rae Atkeson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex N. Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lisa A. Bryant
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Luciana Zilberman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kyle L. Saunders
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lonna Rae Atkeson.

Appendices

Appendix A

Sample Contact Letter

November 7, 2006

Dear NAME & ADDRESS:

This election year was very important and exciting as we decided which party would control Congress and which candidates would best represent our national and local interests. Your part in this drama as a registered voter, whether or not you participated in the election, is especially important to the health of our democracy. We would like your help in assessing the congressional campaign in your district and your experiences with the election process. Given recent electoral controversies and changes in election law it is important to understand how well our democratic process is working. Thus, we seek your help. Only a few citizens from your district were randomly chosen to participate in this study and, therefore, your participation is extremely important to the success of this valuable research, which will be given to policymakers through our research reports.

We have placed an Internet survey online at: http://votenewmexico.unm.edu (please note there is no www or @ sign, just type this address into your web browser, such as Internet Explorer). We would greatly appreciate if you would take the 15 min to complete our survey.

If you do not have access to a computer, we still need the vital information you possess! Please request a mail survey by calling our toll free number at 1-(866)-568-6455. We will mail it out immediately at no cost to you.

You will notice at the top of this form an identification number next to your name (ID#: XXXX), please use this number when the survey asks for it. This identification number is for internal purposes only so we may check your name off of the list when your survey is complete. Please be assured that your answers will be held in complete confidentiality. We will only use your answers in a statistical summary and your answers will never be associated with your name. Your participation is, of course, completely voluntary, so if there are questions that you would prefer not to answer, simply skip them and go on with the survey.

If you have questions or need assistance in any way, please call us at the toll free number above, or our offices at 1-505-277-7592 or 1-970-491-5751, or e-mail us at atkeson@unm.edu or kyle.saunders@colostate.edu.

If you have other concerns, please contact the Institutional Review Board at the University of New Mexico, 1717 Roma NE, Room 205, Albuquerque, NM 87131, (505) 277-2257 or toll free at 1-866-844-9018.

We look forward to hearing from you! Thanks!

Lonna Atkeson    Kyle Saunders

Professor    Assistant Professor

University of New Mexico    Colorado State University

Appendix B

Sample Reminder/Contact Postcard

Dear New Mexico Registered Voter:

A couple of weeks ago we sent you a letter telling you about our 2006 Election Administration Survey. We know you are extremely busy, but you are part of a select group of people we have asked to participate and therefore your response is extremely important to the success of our project.

Please assist us by taking a few moments to complete our fun Internet survey! The study’s web address appears below. Please note there is no www or @ sign in the web address. Also, please be assured that your answers are confidential. So we can check your name off our list, please use your identification code, which is located right above your name on the mailing label of this post card (e.g. ID#XXXX).

http://votenewmexico.unm.edu/

Your participation is completely voluntary. If there are questions that you prefer not to answer simply skip them. If you have any question about this project, or would prefer a mail survey, please call me toll free at 1-866-568-6455, or at my work phone at 1-505-277-7592 or contact me via e-mail at atkeson@unm.edu.

We look forward to hearing from you!

Thank you,

Lonna Atkeson

Professor

University of New Mexico

Rights and permissions

Reprints and permissions

About this article

Cite this article

Atkeson, L.R., Adams, A.N., Bryant, L.A. et al. Considering Mixed Mode Surveys for Questions in Political Behavior: Using the Internet and Mail to Get Quality Data at Reasonable Costs. Polit Behav 33, 161–178 (2011). https://doi.org/10.1007/s11109-010-9121-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11109-010-9121-1

Keywords

Search

Navigation