Skip to main content
SpringerLink
Log in

Non-Response Bias with a Web-Based Survey of College Students: Differences from a Classroom Survey About Carrying Concealed Handguns

  • Original Paper
  • Published:
Journal of Quantitative Criminology Aims and scope Submit manuscript

Abstract

Internet-based and e-mail surveys represent viable administrative methods for efficiently collecting data. These methods appear to be particularly well-suited for studying college student populations, a group that has gained attention from criminologists interested in testing theories. An important concern with administering surveys with the Internet and via e-mail is that of non-response bias. Despite the appeal of online surveys, nonresponse bias associated with these methods has not been sufficiently investigated. The study described here estimates nonresponse bias associated with a web-administered survey that measured opinions about changing concealed handgun carrying laws on college campuses, items likely to elicit polarizing opinions. Results show important substantive differences between web-administered and in-class versions of the survey. Students who responded to the web survey expressed more extreme opinions and behavioral responses to a proposed policy that would allow concealed handgun carrying on campus. Survey researchers who utilize web-based administrative methods should consider using multiple sources of leverage when soliciting participation and must carefully evaluate sample representativeness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. See Payne and Chappell (2008) for a description of criminological studies that have utilized college student samples.

  2. Manfreda et al. (2008) conducted a meta-analysis of 45 studies and report that, among students, response rates to Web surveys are not very different from their response rates to surveys administered via different methods, including telephone, mail, and e-mail. Their investigation did not capture any study that compared Web surveys to surveys administered in classroom meetings.

  3. There was one significant national incident involving a mass shooting that occurred during the course of data collection. The shooting occurred on November 5th, 2009 at Fort Hood, a United States Army base located in Killeen, TX. Of the in-class surveys, only one class (with 16 students) was surveyed prior to the shooting on the afternoon of Nov. 5th. The online version of the survey had 127 students complete the survey after the incident (Nov. 6th and beyond). Thus, most students in the in-class sample completed surveys following the Fort Hood shooting and most students in the on-line sample completed surveys prior to the Fort Hood shooting. As a consequence of this timing, the type of the survey utilized in this study was highly collinear with the media coverage of the Fort Hood shooting (Chi-square = 3087.7, p < 0.00001; Cramer’s V = −0.919), although the precise nature of the relationship between the shooting and participation in this study is unclear. Due to concerns about multi-collinearity it was not possible, for example, to include a variable to control for the timing of the shooting in multivariate models, nor to provide an estimate of the magnitude of this potential effect.

  4. It was not possible to measure the proportion of students who read the participation request that was delivered via e-mail. Anecdotal evidence obtained through discussions with students in classes suggests that an important percentage of students will not even open e-mails sent to them by someone they do not know. Thus, some portion of students remain unaware of survey participation requests. Manfreda et al. (2008) point to this as a potentially important reason for lower response rates with Web-based surveys.

  5. A decision was made to use 1,414 surveys in calculating the response rate. This figure is calculated by removing 17 blank surveys and 34 non-student surveys from the total 1,465 surveys that were returned. This 1,414 value includes the 79 students who indicated that they had already completed the survey in another class. These are included in the numerator during the computation of response rates because these students reflect the extent to which in-class surveys result in participation responses. The 79 surveys were excluded from substantive analyses because on these surveys students reported they had already completed the survey in a previous class; duplicating their response might unduly influence the results.

  6. Consistent with recommendations from Ferguson (2009), we report Cramer’s V as an indicator of strength of association in addition to standard Chi-square statistics and p values, which may show statistical but not necessarily practical significance.

  7. Identical analyses were also performed with dependent variables using slightly different recoded values to reflect individuals with moderate opinions (0 and 100 compared to 50; 0 and 100 compared to 1 through 99). The results using these alternate dependent variables were substantively consistent with those presented here.

  8. The authors thank the anonymous reviewers for contributing details about this issue.

References

  • Bandilla W, Bosnjak M, Altdorfer P (2003) Survey administration effects? A comparison of web-based and traditional written self-administered surveys using the ISSP environment module. Soc Sci Comput Rev 21:235–243

    Article  Google Scholar 

  • Bichler-Roberston G, Potchak MC, Tibbetts S (2003) Low self-control, opportunity, and strain in student reported cheating behavior. J Crim Justice 26:23–53

    Google Scholar 

  • Birnholtz JP, Horn DB, Finholt TA, Bae SJ (2004) The effects of cash, electronic, and paper gift certificates as respondent incentives for a Web-based survey of technologically sophisticated respondents. Soc Sci Comput Rev 22:355–362

    Article  Google Scholar 

  • Bouffard JA (2002) The influence of sexual arousal on rational decision making in sexual aggression. J Crim Justice 30:121–134

    Article  Google Scholar 

  • Bouffard JA, Bry J, Smith S, Bry R (2008) Beyond the “science of sophomores:” Does the rational choice explanation of crime generalize from university students to an actual offender sample? Int J Offender Ther Comp Criminol 52:698–721

    Article  Google Scholar 

  • Carini RM, Hayek JC, Kuh GD, Kennedy JM, Ouimet JA (2003) College student responses to web and paper surveys: does mode matter? Res High Educ 44:1–19

    Article  Google Scholar 

  • Carmichael S, Piquero AR (2004) Sanctions, perceived anger, and criminal offending. J Quant Criminol 20:371–393

    Article  Google Scholar 

  • Clodfelter TA, Turner MG, Hartman JL, Kuhns JB (2010) Sexual harassment victimization during emerging adulthood: a test of routine activities theory and a general theory of crime. Crime Delinq 56:455–481

    Article  Google Scholar 

  • Cole ST (2005) Comparing mail and web-based survey distribution methods: results of surveys to leisure travel retailers. J Trav Res 43:422–430

    Article  Google Scholar 

  • Cook PJ, Ludwig J (1997) Guns in America: results of a comprehensive survey on private firearms ownership and use. Police Foundation, Washington, DC

  • Couper MP (2000) Web surveys: a review of issues and approaches. Public Opin Q 64:464–494

    Article  Google Scholar 

  • Curtin R, Presser S, Singer E (2000) The effects of response rate changes on the index of consumer sentiment. Public Opin Q 64:413–428

    Article  Google Scholar 

  • Daigle LE, Fisher BS, Cullen FT (2008) The violent and sexual victimization of college women: is repeat victimization a problem? J Interpers Violence 23:1296–1313

    Article  Google Scholar 

  • Dillman DA (2007) Mail and internet surveys: the tailored design method 2007 update with new internet, visual, and mixed-mode guides, 2nd edn. Wiley, Hoboken

  • Dillman DA, Smyth JD, Christian LM (2009) Internet, mail, and mixed-mode surveys: the tailored design method, 3rd edn. Wiley, Hoboken

    Google Scholar 

  • Farnworth M, Longmire DR, West VM (1998) College students views on criminal justice. J Crim Justice Educ 9:39–57

    Article  Google Scholar 

  • Ferguson CJ (2009) An effect size primer: a guide for clinicians and researchers. Prof Psychol Res Pr 40(5):532–538

    Article  Google Scholar 

  • Ferguson CJ, Rueda S, Cruz A, Ferguson D, Fritz S, Smith S (2008) Violent video games and aggression: causal relationship or byproduct of family violence and intrinsic violence motivation? Crim Justice Behav 35:311–332

    Article  Google Scholar 

  • Fisher BS, Sloan JJ, Cullen FT, Lu C (1998) Crime in the ivory tower: the level and sources of student victimization. Criminology 36:671–710

    Article  Google Scholar 

  • Gibbs JJ, Giever D, Martin JS (1998) Parental management and self-control: an empirical test of Gottfredson and Hirschi’s general theory. J Res Crime and Delinq 35:40–70

    Article  Google Scholar 

  • Gottfredson MR, Hirschi T (1990) A general theory of crime. Stanford University Press, Stanford

  • Grasmick HG, Tittle CR, Bursik RJ, Arneklev BJ (1993) Testing the core empirical implications of Gottfredson and Hirschi’s general theory of crime. J Res Crime and Delinq 30:5–329

    Google Scholar 

  • Groves RM (2004) Survey errors and survey costs. Wiley, Hoboken

    Google Scholar 

  • Groves RM (2006) Nonresponse rates and nonresponse bias in household surveys. Public Opin Q 70:646–675

    Article  Google Scholar 

  • Groves RM, Singer E, Corning A (2000) Leverage-saliency theory of survey participation: description and an illustration. Public Opin Q 64:299–308

    Article  Google Scholar 

  • Groves RM, Dillman DA, Eltinge JL, Little RJA (2002) Survey nonresponse. Wiley, New York

    Google Scholar 

  • Groves RM, Presser S, Dipko S (2004) The role of topic interest in survey participation decisions. Public Opin Q 68:2–31

    Article  Google Scholar 

  • Hayslett MM, Wildemuth BM (2003) Pixels or pencils? The relative effectiveness of web-based versus paper surveys. Libr Inf Sci Res 26:73–93

    Article  Google Scholar 

  • Heerwegh D, Loosveldt G (2008) Face to face versus web surveying in a high-internet-coverage population: differences in response quality. Public Opin Q 72:836–846

    Article  Google Scholar 

  • Higgins GE, Wolfe SE, Marcum CD (2008) Digital piracy: an examination of three measurements of self-control. Deviant Behav 29:440–460

    Article  Google Scholar 

  • Hilinski CM (2009) Fear of crime among college students: a test of the shadow of sexual assault hypothesis. Am J Crim Justice 34:84–102

    Article  Google Scholar 

  • Hirschi T (2004) Self-control and crime. In: Baumeister RF, Vohs KD (eds) Handbook of self-regulation: research, theory, and applications. Guilford Press, New York

    Google Scholar 

  • Kaysen D, Neighbors C, Martell J, Fossos N, Larimer ME (2006) Incapacitated rape and alcohol use: a prospective analysis. Addict Behav 31:1820–1832

    Google Scholar 

  • Ketter S, Miller C, Kohut A, Groves RM, Presser S (2000) Consequences of reducing nonresponse in a national telephone survey. Public Opin Q 64:125–148

    Article  Google Scholar 

  • Mallicoat SL, Brown GC (2008) The impact of race and ethnicity on student opinions of capital punishment. J Ethn Crim Justice 6:255–280

    Article  Google Scholar 

  • Manfreda KL, Bosnjak M, Berzelak J, Haas I, Vehovar V (2008) Web surveys versus other survey modes: a meta-analysis comparing response rates. Int J Market Res 50:79–104

    Google Scholar 

  • McCabe SE, Diez A, Boyd CJ, Nelson TF, Weitzman ER (2006) Comparing web and mail responses in a mixed mode survey in college alcohol use research. Addict Behav 31:1619–1627

    Article  Google Scholar 

  • Mertler CA, Earley MA (2002) The mouse or the pencil? A psychometric comparison of web-based and traditional survey methodologies. Paper presented at the annual meeting of the mid-western educational research association, Columbus

  • Mertler CA, Earley MA (2003) A comparison of the psychometric qualities of surveys administered by web and traditional methods. Paper presented at the annual meeting of the American educational research association, Chicago

  • Messman-Moore TL, Coates AA, Gaffey KJ, Johnson CF (2008) Sexuality, substance use, and susceptibility to victimization: risk for rape and sexual coercion in a prospective study of college women. J Interpers Violence 23:1730–1746

    Article  Google Scholar 

  • Miller M, Hemenway D, Wechsler H (2002) Guns and gun threats at college. J Am Coll Health 51:57–65

    Article  Google Scholar 

  • Mustaine EE, Tewksbury R (1998) Predicting risks of larceny theft victimization: a routine activity analysis using refined lifestyle measures. Criminology 36:829–858

    Article  Google Scholar 

  • Nagin DS, Pogarsky G (2001) Integrating celerity, impulsivity, and extralegal sanction threats into a model of general deterrence: theory and evidence. Criminology 39:865–882

    Article  Google Scholar 

  • Parks KA, Romosz AM, Bradizza CM, Hsieh Y (2008) A dangerous transition: women’s drinking and related victimization from high school to the first year at college. J Stud Alcohol Drugs 69:65–74

    Google Scholar 

  • Payne BK, Chappell A (2008) Using student samples in criminological research. J Crim Justice Educ 19:175–192

    Article  Google Scholar 

  • Perkins GH, Yuan H (2001) A comparison of web-based and paper-and-pencil library satisfaction survey results. Coll Res Libr 62:369–377

    Google Scholar 

  • Pew Internet & American Life Project (2009) Home broadband adoption increases sharply in 2009 with big jumps among seniors, low-income households, and rural residents even though prices have risen since last year. Retrieved April 16, 2010 from: http://www.pewinternet.org/Press-Releases/2009/Home-broadband-adoption-increases-sharply-in-2009.aspx

  • Piquero AR, Bouffard JA (2007) Something old, something new: a preliminary investigation of Hirschi’s redefined self-control. Justice Q 24:1–27

    Article  Google Scholar 

  • Piquero AR, MacIntosh R, Hickman M (2000) Does self-control affect survey response? Applying exploratory, confirmatory, and item response theory analysis to Grasmick et al.’s self control scale. Criminology 38:897–930

    Article  Google Scholar 

  • Rayburn NR, Earleywine M, Davison GC (2003) Base rates of hate crime victimization among college students. J Interpers Violence 18:1209–1221

    Article  Google Scholar 

  • Ricketts ML, Higgins GE (2007) A preliminary study of the factors that influence college students perceptions of nonmedical use of prescription drugs: criminal justice versus noncriminal justice students. Crim Justice Stud 20:407–422

    Article  Google Scholar 

  • Sellers CS, Cochran JK, Branch KA (2005) Social learning theory and partner violence: a research note. Deviant Behav 26:379–395

    Article  Google Scholar 

  • Shih TH, Fan X (2008) Comparing response rates from web and mail surveys: a meta-analysis. Field Method 20:249–271

    Article  Google Scholar 

  • Singleton RA Jr, Straits BC (2005) Approaches to social research, 4th edn. Oxford University Press, New York

    Google Scholar 

  • Smith TW (2001) 2001 National gun policy survey of the national opinion research center. National Opinion Research Center, Chicago

    Google Scholar 

  • Spitzberg BH, Veksler AE (2007) The personality of pursuit: personality attributions of unwanted pursuers and stalkers. Violence Vict 22:275–289

    Article  Google Scholar 

  • Sue VM, Ritter LA (2007) Conducting online surveys. Sage, Los Angeles

    Google Scholar 

  • Tewksbury R, Mustaine EE (2003) College students’ lifestyles and self-protective behaviors: further considerations of the guardianship concept in routine activity theory. Crim Justice Behav 30:302–327

    Article  Google Scholar 

  • Wilcox P, Jordan CE, Pritchard AJ (2007) A multidimensional examination of campus safety: victimization, perceptions of danger, worry about crime, and precautionary behavior among college women in the Post-Clery era. Crime Delinq 53:219–254

    Article  Google Scholar 

  • Yang S, Wyckoff LA (2010) Perceptions of safety and victimization: does survey construction affect perceptions? J Exp Crim 6:293–323

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Criminal Justice, Sam Houston State University, Huntsville, TX, 77341–2296, USA

    William Wells, Michael R. Cavanaugh, Jeffrey A. Bouffard & Matt R. Nobles

Authors
  1. William Wells
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael R. Cavanaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeffrey A. Bouffard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matt R. Nobles
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William Wells.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wells, W., Cavanaugh, M.R., Bouffard, J.A. et al. Non-Response Bias with a Web-Based Survey of College Students: Differences from a Classroom Survey About Carrying Concealed Handguns. J Quant Criminol 28, 455–476 (2012). https://doi.org/10.1007/s10940-011-9148-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10940-011-9148-4

Keywords

Search

Navigation