Estimates of Non-Heterosexual Prevalence: The Roles of Anonymity and Privacy in Survey Methodology
When do people feel comfortable enough to provide honest answers to sensitive questions? Focusing specifically on sexual orientation prevalence—a measure that is sensitive to the pressures of heteronormativity—the present study was conducted to examine the variability in U.S. estimates of non-heterosexual identity prevalence and to determine how comfortable people are with answering questions about their sexual orientation when asked through commonly used survey modes. We found that estimates of non-heterosexual prevalence in the U.S. increased as the privacy and anonymity of the survey increased. Utilizing an online questionnaire, we rank-ordered 16 survey modes by asking people to rate their level of comfort with each mode in the context of being asked questions about their sexual orientation. A demographically diverse sample of 652 individuals in the U.S. rated each mode on a scale from −5 (very uncomfortable) to +5 (very comfortable). Modes included anonymous (name not required) and non-anonymous (name required) versions of questions, as well as self-administered and interviewer-administered versions. Subjects reported significantly higher mean comfort levels with anonymous modes than with non-anonymous modes and significantly higher mean comfort levels with self-administered modes than with interviewer-administered modes. Subjects reported the highest mean comfort level with anonymous online surveys and the lowest with non-anonymous personal interviews that included a video recording. Compared with the estimate produced by an online survey with a nationally representative sample, surveys utilizing more intrusive methodologies may have underestimated non-heterosexual prevalence in the U.S. by between 50 and 414%. Implications for public policy are discussed.
KeywordsSexual orientation Prevalence of non-heterosexuality Survey methodology Heteronormativity
Earlier versions of this article were presented at the 2013 annual meeting of the Society for the Scientific Study of Sexuality and at the 2014 annual meeting of the Western Psychological Association. The authors would like to thank Brian W. Ward, Debby Herbenick, Randy Sell, and Tom W. Smith for their helpful comments and suggestions.
Compliance with Ethical Standards
Conflict of interest
All authors declare no conflicts of interest.
Human and Animal Rights
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Informed consent was obtained from all individual participants included in the study.
- Badgett, M. V. (2009). Best practices for asking questions about sexual orientation on surveys. Los Angeles, CA: The Williams Institute.Google Scholar
- Baker, R. P., Bradburn, N. M., & Johnson, R. A. (1995). Computer-assisted personal interviewing: An experimental evaluation of data quality and cost. Journal of Official Statistics, 11, 413–431.Google Scholar
- Beatty, P., & Herrmann, D. (2002). To answer or not to answer: Decision processes related to survey item nonresponse. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 71–85). New York, NY: Wiley.Google Scholar
- Black, M. C., Basile, K. C., Breiding, M. J., Smith, S. G., Walters, M. L., Merrick, M. T., & Stevens, M. R. (2011). National intimate partner and sexual violence survey. Atlanta, GA: Centers for Disease Control and Prevention.Google Scholar
- Bradburn, N. M., Frankel, M. R., Hunt, E., Ingels, J., Schoua-Glusberg, A., Wojcik, M., & Pergamit, M. R. (1991, May 16–19). A comparison of computer-assisted personal interviews (CAPI) with personal interviews in the national longitudinal survey of labor market behavior-youth cohort. Lecture presented at the annual meeting of the American Association for Public Opinion Research, Phoenix, AZ.Google Scholar
- Bradford, J. B., & Mayer, K. H. (2008). Demography and the LGBT population: What we know, don’t know, and how the information helps to inform clinical practice. In H. Makadon, K. Mayer, J. Potter, & H. Goldhammer (Eds.), The Fenway guide to lesbian, gay, bisexual, and transgender health (pp. 25–41). Philadelphia, PA: American College of Physicians.Google Scholar
- Bradford, J., & Mustanski, B. (2014). Health disparities among sexual minority youth: The value of population data. American Journal of Public Health, 104, 197. doi: 10.2105/AJPH.2013.301818.
- Brim, O. G., Baltes, P. B., Bumpass, L. L., Cleary, P. D., Featherman, D. L., Hazzard, W. R., & Shweder, R. A. (2011). National Survey of Midlife Development in the United States (MIDUS), 1995–1996 (ICPSR02760-v8). Ann Arbor, MI: Inter-University Consortium for Political and Social Research.Google Scholar
- Burkill, S., Copas, A., Couper, M. P., Clifton, S., Prah, P., Datta, J., et al. (2016). Using the web to collect data on sensitive behaviours: A study looking at mode effects on the British National Survey of Sexual Attitudes and Lifestyles. PLoS One, 11(2), e0147983. doi: 10.1371/journal.pone.0147983.PubMedPubMedCentralCrossRefGoogle Scholar
- CDC. (n.d.-a). National health interview survey questionnaires, datasets, and related documentation 1997 to the present. Retrieved September 2, 2015 from http://www.cdc.gov/nchs/nhis/quest_data_related_1997_forward.htm.
- CDC. (n.d.-b). National health and nutritional examination survey (NHANES) 2011–2012 questionnaire data overview. Retrieved from http://www.cdc.gov/nchs/nhanes/nhanes2011-2012/quexdoc_g.htm.
- Centers for Disease Control and Prevention (CDC) National Center for Health Statistics (NCHS). (2009–2012). National health and nutrition examination survey questionnaire (or examination protocol, or laboratory protocol). Hyattsville, MD: U.S. Department of Health and Human Services, CDC.Google Scholar
- Chandra, A., Mosher, W. D., & Copen, C. E. (2013). Sexual behavior, sexual attraction, and sexual identity in the United States: Data from the 2006–2010 national survey of family growth. In A. K. Baumle (Ed.), International handbook on the demography of sexuality (pp. 45–66). Dordrecht: Springer.CrossRefGoogle Scholar
- Chang, L., & Krosnick, J. A. (2002). A comparison of the random digit dialing telephone survey methodology with Internet survey methodology as implemented by Knowledge Networks and Harris Interactive. Columbus, OH: Department of Psychology, Ohio State University.Google Scholar
- Cochran, S. D., & Mays, V. M. (2006). Estimating prevalence of mental and substance-using disorders among lesbians and gay men from existing national health data. In A. M. Omoto & H. S. Kurtzman (Eds.), Sexual orientation and mental health: Examining identity and development in lesbian, gay, and bisexual people (pp. 143–165). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
- Cochran, S. D., & Mays, V. M. (2007). Physical health complaints among lesbians, gay men, and bisexual and homosexually experienced heterosexual individuals: Results from the California Quality of Life Survey. American Journal of Public Health, 97, 2048–2055.PubMedPubMedCentralCrossRefGoogle Scholar
- Crowne, D. P., & Marlowe, D. (1964). The approval motive: Studies in evaluative dependence. New York: Wiley.Google Scholar
- Dahlgreen, W., & Shakespeare, A. (2015, August 16). 1 in 2 young people say they are not 100% heterosexual. Retrieved from https://yougov.co.uk/news/2015/08/16/half-young-not-heterosexual/.
- Dahlhamer, J. M., Galinsky, A. M., Joestl, S. S., & Ward, B. W. (2014). Sexual orientation in the 2013 National Health Interview Survey: A quality assessment. Vital and Health Statistics. Series 2. Data Evaluation and Methods Research, 169, 1–32.Google Scholar
- DeMaio, T. J. (1984). Social desirability and survey measurement: A review. In C. Turner & E. Martin (Eds.), Surveying subjective phenomena (Vol. 2, pp. 257–282). New York, NY: Sage.Google Scholar
- Diamond, L. M. (2008). Sexual fluidity: Understanding women’s love and desire. Cambridge, MA: Harvard University Press.Google Scholar
- Dilley, J. A., Simmons, K. W., Boysun, M. J., Pizacani, B. A., & Stark, M. J. (2010). Demonstrating the importance and feasibility of including sexual orientation in public health surveys: Health disparities in the Pacific Northwest. American Journal of Public Health, 100, 460–467.PubMedPubMedCentralCrossRefGoogle Scholar
- Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: Wiley.Google Scholar
- Gates, G. J. (2014a). LGBT demographics: Comparisons among population-based surveys. Los Angeles, CA: The Williams Institute.Google Scholar
- Gates, G. J. (2014b). In U.S., LGBT more likely than non-LGBT to be uninsured. Retrieved from http://www.gallup.com/poll/175445/lgbt-likely-non-lgbt-uninsured.aspx.
- Gates, G. J., & Newport, F. (2012). Special report: 3.4% of U.S. adults identify as LGBT. Retrieved from http://www.gallup.com/poll/158066/special-report-adults-identify-lgbt.aspx.
- Gay and Lesbian Medical Association (GLMA). (2001). Healthy People 2010: Companion document for lesbian, gay, bisexual and transgender (LGBT) health. San Francisco, CA: GLMA.Google Scholar
- Gilman, S. E., Cochran, S. D., Mays, V. M., Hughes, M., Ostrow, D., & Kessler, R. C. (2001). Risk of psychiatric disorders among individuals reporting same-sex sexual partners in the national comorbidity survey. American Journal of Public Health, 91, 933–939.PubMedPubMedCentralCrossRefGoogle Scholar
- Goffman, E. (1963). Stigma: Notes on the management of spoiled identity. New York, NY: Simon and Schuster.Google Scholar
- Graham, R., Berkowitz, B., Blum, R., Bockting, W., Bradford, J., de Vries, B., … Makadon, H. (2011). The health of lesbian, gay, bisexual, and transgender people: Building a foundation for better understanding. Washington, DC: Institute of Medicine.Google Scholar
- Hinkins, S., Crego, A., English, N., Pedlow, S., & Wolter, K. M. (n.d.). 2010 national sample frame. Retrieved from http://www.norc.org/Research/Projects/Pages/2010-national-sample-frame.aspx.
- Internet Live Stats. (2016). Internet users in the U.S.A. (2016*). Retrieved from http://www.internetlivestats.com.
- Itaborahy, L., & Zhu, J. (2014). State-sponsored homophobia, a world survey of laws: Criminalisation, protection and recognition of same-sex love (9th ed.). Geneva: International Lesbian, Gay, Bisexual and Intersex Associates.Google Scholar
- Joloza, T., Evans, J., O’Brien, R., & Potter-Collins, A. (2010). Measuring sexual identity: An evaluation report. Newport, UK: Office for National Statistics.Google Scholar
- Kann, L., Olsen, E. O., McManus, T., Kinchen, S., Chyen, D., Harris, W. A., & Wechsler, H. (2011). Sexual identity, sex of sexual contacts, and health-risk behaviors among students in grades 9–12-youth risk behavior surveillance, selected sites, United States, 2001–2009. Morbidity and Mortality Weekly Report. Surveillance Summaries, 60(7), 1–133.PubMedGoogle Scholar
- Kanuha, V. (1999). The social process of passing to manage stigma: Acts of internalized oppression or acts of resistance? Journal of Sociology and Social Welfare, 26(4), 27–46.Google Scholar
- Keeter, S., McGeeney, K., Igielnik, R., Mercer, A., & Mathiowetz, N. (2015, May 13). From telephone to the web: The challenge of mode of interview effects in public opinion polls. Retrieved from http://www.pewresearch.org/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/.
- Lee, R. M. (1993). Doing research on sensitive topics. Thousand Oaks, CA: Sage.Google Scholar
- Locke, S. D., & Gilbert, B. O. (1995). Method of psychological assessment, self-disclosure, and experiential differences: A study of computer, questionnaire, and interview assessment formats. Journal of Social Behavior and Personality, 10, 255–263.Google Scholar
- Miller, J. D. (1984). A new survey technique for studying deviant behavior. Doctoral dissertation, George Washington University.Google Scholar
- Moore, P. (2015). A third of young Americans say they aren’t 100% heterosexual. Retrieved from https://today.yougov.com/news/2015/08/20/third-young-americans-exclusively-heterosexual/?sid=5388f1ffdd52b8ed110008bc&wpsrc=slatest_newsletter.
- Mustanski, B., Van Wagenen, A., Birkett, M., Eyster, S., & Corliss, H. L. (2014). Identifying sexual orientation health disparities in adolescents: Analysis of pooled data from the Youth Risk Behavior Survey, 2005 and 2007. American Journal of Public Health, 104, 211–217.PubMedPubMedCentralCrossRefGoogle Scholar
- National Institute on Alcohol Abuse and Alcoholism (NIAAA). (2010). U.S. alcohol epidemiological data reference manual (2nd ed., Vol. 8). Retrieved from http://pubs.niaaa.nih.gov/publications/NESARC_DRM2/NESARC2DRM.pdf.
- National Survey of Family Growth (NSFG). (n.d). 2006–2010 NSFG questionnaires. Retrieved from http://www.cdc.gov/nchs/nsfg/nsfg_2006_2010_questionnaires.htm.
- NORC at the University of Chicago. (n.d.). 2010 national sample frame. Retrieved from http://www.norc.org/Research/Projects/Pages/2010-national-sample-frame.aspx.
- Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 51–77). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
- Pew Research Center. (2013, June 13). A survey of LGBT Americans. Retrieved from http://www.pewsocialtrends.org/2013/06/13/a-survey-of-lgbt-americans/.
- Pew Research Center. (n.d.). Global views on morality. Retrieved from http://www.pewglobal.org/2014/04/15/global-morality/.
- Ryff, C., Almeida, D. M., Ayanian, J. S., Carr, D. S., Cleary, P. D., Coe, C., … Mroczek, D. K. (2012). National Survey of Midlife Development in the United States (MIDUS II), 2004–2006 (ICPSR04652-v6). Ann Arbor, MI: Inter-University Consortium for Political and Social Research.Google Scholar
- Smith, T. W., Marsden, P. V., & Hout, M. (2015). General social surveys, 1972–2014 [data file]. Retrieved from http://publicdata.norc.org:41000/gss/documents/BOOK/GSS_Codebook.pdf.
- Tourangeau, R., & Smith, T. W. (1998). Collecting sensitive information with different modes of data collection. In M. P. Couper, R. P. Baker, J. Bethlehem, C. Z. Clark, J. Martin, W. L. Nicholls II, et al. (Eds.), Computer assisted survey information collection (pp. 431–454). New York, NY: Wiley.Google Scholar
- Turner, C. F., Forsyth, B. H., O’Reilly, J. M., Cooley, P. C., Smith, T. K., Rogers, S. M., & Miller, H. G. (1998). Automated self-interviewing and the survey measurement of sensitive behaviors. In M. P. Couper, R. P. Baler, J. Bethlehem, C. Z. Clark, J. Martin, W. Nichols, & J. M. O’Reilly (Eds.), Computer assisted survey information collection (pp. 455–473). New York, NY: Wiley.Google Scholar
- Turner, C. F., Ku, L., Sonenstein, F. L., & Pleck, J. H. (1996). Impact of ACASI on reporting of male-male sexual contacts: Preliminary results from the 1995 national survey of adolescent males. In R. B. Warnecke (Ed.), Health survey research methods (pp. 171–176). Hyattsville, MD: National Center for Health Statistics.Google Scholar
- Turner, C. F., Lessler, J. T., & Devore, J. (1992). Effects of mode of administration and wording on reporting of drug use. In C. F. Turner, J. T. Lessler, & J. C. Groerer (Eds.), Survey measurement of drug use: Methodological studies (pp. 177–220). Rockville, MD: National Institute on Drug Abuse.Google Scholar
- Walters, M. L., Chen, J., & Breiding, M. J. (2013). The national intimate partner and sexual violence survey (NISVS): 2010 findings on victimization by sexual orientation. Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.Google Scholar
- Ward, B. W., Dahlhamer, J. M., Galinsky, A. M., & Joestl, S. S. (2014). Sexual orientation and health among U.S. adults: National health interview survey, 2013. National Health Statistics Reports (no. 77). Atlanta, GA: Centers for Disease Control and Prevention.Google Scholar
- Warner, M. (Ed.). (1993). Fear of a queer planet: Queer politics and social theory (Vol. 6). Minneapolis, MN: University of Minnesota Press.Google Scholar
- YouGov U.S. (n.d.). YouGov Omnibus. Retrieved from https://today.yougov.com/find-solutions/omnibus/.