Skip to main content
SpringerLink
Log in

Estimates of Non-Heterosexual Prevalence: The Roles of Anonymity and Privacy in Survey Methodology

  • Original Paper
  • Published:
Archives of Sexual Behavior Aims and scope Submit manuscript

Abstract

When do people feel comfortable enough to provide honest answers to sensitive questions? Focusing specifically on sexual orientation prevalence—a measure that is sensitive to the pressures of heteronormativity—the present study was conducted to examine the variability in U.S. estimates of non-heterosexual identity prevalence and to determine how comfortable people are with answering questions about their sexual orientation when asked through commonly used survey modes. We found that estimates of non-heterosexual prevalence in the U.S. increased as the privacy and anonymity of the survey increased. Utilizing an online questionnaire, we rank-ordered 16 survey modes by asking people to rate their level of comfort with each mode in the context of being asked questions about their sexual orientation. A demographically diverse sample of 652 individuals in the U.S. rated each mode on a scale from −5 (very uncomfortable) to +5 (very comfortable). Modes included anonymous (name not required) and non-anonymous (name required) versions of questions, as well as self-administered and interviewer-administered versions. Subjects reported significantly higher mean comfort levels with anonymous modes than with non-anonymous modes and significantly higher mean comfort levels with self-administered modes than with interviewer-administered modes. Subjects reported the highest mean comfort level with anonymous online surveys and the lowest with non-anonymous personal interviews that included a video recording. Compared with the estimate produced by an online survey with a nationally representative sample, surveys utilizing more intrusive methodologies may have underestimated non-heterosexual prevalence in the U.S. by between 50 and 414%. Implications for public policy are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Note that we have only included estimates from surveys that asked respondents directly about their sexual orientation. Data sources such as the American Community Survey and the US Census, which can be used to estimate sexual orientation preference inferentially, are omitted from our analysis. Such estimates are obtained by comparing data on the sex of the head of a given household, the sex of other members of the household, and the relationship between those other members of the household and the head of the household.

  2. National studies measuring non-heterosexual prevalence conducted in other countries, such as the UK (Dahlgreen & Shakespeare, 2015; Joloza, Evans, O’Brien, & Potter-Collins, 2010), were not included in our study. There is reason to believe, however, survey issues regarding sexual orientation are similar in other countries to the issues we raise in the present paper (e.g., see Burkill et al., 2016; Langhaug, Sherr, & Cowan, 2010).

References

  • Aquilino, W. S. (1994). Interview mode effects in surveys of drug and alcohol use: A field experiment. Public Opinion Quarterly, 58, 210–240.

    Article  Google Scholar 

  • Aquilino, W. S., & LoSciuto, L. A. (1990). Effects of interview mode on self-reported drug use. Public Opinion Quarterly, 54, 362–393.

    Article  Google Scholar 

  • Badgett, M. V. (2009). Best practices for asking questions about sexual orientation on surveys. Los Angeles, CA: The Williams Institute.

    Google Scholar 

  • Baker, R. P., Bradburn, N. M., & Johnson, R. A. (1995). Computer-assisted personal interviewing: An experimental evaluation of data quality and cost. Journal of Official Statistics, 11, 413–431.

    Google Scholar 

  • Barnett, J. (1998). Sensitive questions and response effects: An evaluation. Journal of Managerial Psychology, 13, 63–76.

    Article  Google Scholar 

  • Beatty, P., & Herrmann, D. (2002). To answer or not to answer: Decision processes related to survey item nonresponse. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey nonresponse (pp. 71–85). New York, NY: Wiley.

    Google Scholar 

  • Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368.

    Article  Google Scholar 

  • Black, M. C., Basile, K. C., Breiding, M. J., Smith, S. G., Walters, M. L., Merrick, M. T., & Stevens, M. R. (2011). National intimate partner and sexual violence survey. Atlanta, GA: Centers for Disease Control and Prevention.

    Google Scholar 

  • Bonett, D. G. (2008). Confidence intervals for standardized linear contrasts of means. Psychological Methods, 13, 99–109.

    Article  PubMed  Google Scholar 

  • Booth-Kewley, S., Larson, G. E., & Miyoshi, D. K. (2007). Social desirability effects on computerized and paper-and-pencil questionnaires. Computers in Human Behavior, 23, 463–477.

    Article  Google Scholar 

  • Bostwick, W. B., Boyd, C. J., Hughes, T. L., & McCabe, S. E. (2010). Dimensions of sexual orientation and the prevalence of mood and anxiety disorders in the United States. American Journal of Public Health, 100, 468–475.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bradburn, N. M., Frankel, M. R., Hunt, E., Ingels, J., Schoua-Glusberg, A., Wojcik, M., & Pergamit, M. R. (1991, May 16–19). A comparison of computer-assisted personal interviews (CAPI) with personal interviews in the national longitudinal survey of labor market behavior-youth cohort. Lecture presented at the annual meeting of the American Association for Public Opinion Research, Phoenix, AZ.

  • Bradburn, N., & Mason, W. (1964). The effect of question order on responses. Journal of Marketing Research, 1, 57–61.

    Article  Google Scholar 

  • Bradford, J. B., & Mayer, K. H. (2008). Demography and the LGBT population: What we know, don’t know, and how the information helps to inform clinical practice. In H. Makadon, K. Mayer, J. Potter, & H. Goldhammer (Eds.), The Fenway guide to lesbian, gay, bisexual, and transgender health (pp. 25–41). Philadelphia, PA: American College of Physicians.

    Google Scholar 

  • Bradford, J., & Mustanski, B. (2014). Health disparities among sexual minority youth: The value of population data. American Journal of Public Health, 104, 197. doi:10.2105/AJPH.2013.301818.

  • Brim, O. G., Baltes, P. B., Bumpass, L. L., Cleary, P. D., Featherman, D. L., Hazzard, W. R., & Shweder, R. A. (2011). National Survey of Midlife Development in the United States (MIDUS), 1995–1996 (ICPSR02760-v8). Ann Arbor, MI: Inter-University Consortium for Political and Social Research.

    Google Scholar 

  • Buchanan, T. (2000). Potential of the Internet for personality research. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 121–140). San Diego, CA: Academic Press.

    Chapter  Google Scholar 

  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5.

    Article  PubMed  Google Scholar 

  • Burkill, S., Copas, A., Couper, M. P., Clifton, S., Prah, P., Datta, J., et al. (2016). Using the web to collect data on sensitive behaviours: A study looking at mode effects on the British National Survey of Sexual Attitudes and Lifestyles. PLoS One, 11(2), e0147983. doi:10.1371/journal.pone.0147983.

    Article  PubMed  PubMed Central  Google Scholar 

  • Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29, 2156–2160.

    Article  Google Scholar 

  • CDC. (n.d.-a). National health interview survey questionnaires, datasets, and related documentation 1997 to the present. Retrieved September 2, 2015 from http://www.cdc.gov/nchs/nhis/quest_data_related_1997_forward.htm.

  • CDC. (n.d.-b). National health and nutritional examination survey (NHANES) 2011–2012 questionnaire data overview. Retrieved from http://www.cdc.gov/nchs/nhanes/nhanes2011-2012/quexdoc_g.htm.

  • Centers for Disease Control and Prevention (CDC) National Center for Health Statistics (NCHS). (2009–2012). National health and nutrition examination survey questionnaire (or examination protocol, or laboratory protocol). Hyattsville, MD: U.S. Department of Health and Human Services, CDC.

  • Cerny, J. A., & Janssen, E. (2011). Patterns of sexual arousal in homosexual, bisexual, and heterosexual men. Archives of Sexual Behavior, 40, 687–697.

    Article  PubMed  Google Scholar 

  • Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53–81.

    Article  PubMed  Google Scholar 

  • Chandra, A., Mosher, W. D., & Copen, C. E. (2013). Sexual behavior, sexual attraction, and sexual identity in the United States: Data from the 2006–2010 national survey of family growth. In A. K. Baumle (Ed.), International handbook on the demography of sexuality (pp. 45–66). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Chang, L., & Krosnick, J. A. (2002). A comparison of the random digit dialing telephone survey methodology with Internet survey methodology as implemented by Knowledge Networks and Harris Interactive. Columbus, OH: Department of Psychology, Ohio State University.

    Google Scholar 

  • Chang, L., & Krosnick, J. A. (2009). National surveys via RDD telephone interviewing versus the Internet comparing sample representativeness and response quality. Public Opinion Quarterly, 73, 641–678.

    Article  Google Scholar 

  • Chang, L., & Krosnick, J. A. (2010). Comparing oral interviewing with self-administered computerized questionnaires: An experiment. Public Opinion Quarterly, 74, 154–167.

    Article  Google Scholar 

  • Chivers, M. L., Rieger, G., Latty, E., & Bailey, J. M. (2004). A sex difference in the specificity of sexual arousal. Psychological Science, 15, 736–744.

    Article  PubMed  Google Scholar 

  • Chivers, M. L., Seto, M. C., & Blanchard, R. (2007). Gender and sexual orientation differences in sexual response to sexual activities versus gender of actors in sexual films. Journal of Personality and Social Psychology, 93, 1008–1121.

    Article  Google Scholar 

  • Cochran, S. D., & Mays, V. M. (2006). Estimating prevalence of mental and substance-using disorders among lesbians and gay men from existing national health data. In A. M. Omoto & H. S. Kurtzman (Eds.), Sexual orientation and mental health: Examining identity and development in lesbian, gay, and bisexual people (pp. 143–165). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Cochran, S. D., & Mays, V. M. (2007). Physical health complaints among lesbians, gay men, and bisexual and homosexually experienced heterosexual individuals: Results from the California Quality of Life Survey. American Journal of Public Health, 97, 2048–2055.

    Article  PubMed  PubMed Central  Google Scholar 

  • Coffman, K. B., Coffman, L. C., & Ericson, K. M. M. (2016). The size of the LGBT population and the magnitude of anti-gay sentiment are substantially underestimated. Management Science. doi:10.1287/mnsc.2016.2503.

    Google Scholar 

  • Conron, K. J., Mimiaga, M. J., & Landers, S. J. (2010). A population-based study of sexual orientation identity and gender differences in adult health. American Journal of Public Health, 100, 1953–1960.

    Article  PubMed  PubMed Central  Google Scholar 

  • Crowne, D. P., & Marlowe, D. (1964). The approval motive: Studies in evaluative dependence. New York: Wiley.

    Google Scholar 

  • Curtin, R., Presser, S., & Singer, E. (2000). The effects of response rate changes on the index of consumer sentiment. Public Opinion Quarterly, 64, 413–428.

    Article  PubMed  Google Scholar 

  • Dahlgreen, W., & Shakespeare, A. (2015, August 16). 1 in 2 young people say they are not 100% heterosexual. Retrieved from https://yougov.co.uk/news/2015/08/16/half-young-not-heterosexual/.

  • Dahlhamer, J. M., Galinsky, A. M., Joestl, S. S., & Ward, B. W. (2014). Sexual orientation in the 2013 National Health Interview Survey: A quality assessment. Vital and Health Statistics. Series 2. Data Evaluation and Methods Research, 169, 1–32.

    Google Scholar 

  • DeMaio, T. J. (1984). Social desirability and survey measurement: A review. In C. Turner & E. Martin (Eds.), Surveying subjective phenomena (Vol. 2, pp. 257–282). New York, NY: Sage.

    Google Scholar 

  • Diamond, L. M. (2008). Sexual fluidity: Understanding women’s love and desire. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Diaz, R. M., Ayala, G., Bein, E., Henne, J., & Marin, B. V. (2001). The impact of homophobia, poverty, and racism on the mental health of gay and bisexual Latino men: Findings from 3 U.S. cities. American Journal of Public Health, 91, 927–932.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dilley, J. A., Simmons, K. W., Boysun, M. J., Pizacani, B. A., & Stark, M. J. (2010). Demonstrating the importance and feasibility of including sexual orientation in public health surveys: Health disparities in the Pacific Northwest. American Journal of Public Health, 100, 460–467.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: Wiley.

    Google Scholar 

  • Durant, L. E., Carey, M. P., & Schroder, K. E. (2002). Effects of anonymity, gender, and erotophilia on the quality of data obtained from self-reports of socially sensitive behaviors. Journal of Behavioral Medicine, 25, 439–467.

    Article  Google Scholar 

  • Dwight, S. A., & Feigelson, M. E. (2000). A quantitative review of the effect of computerized testing on the measurement of social desirability. Educational and Psychological Measurement, 60, 340–360.

    Article  Google Scholar 

  • Epstein, R., McKinney, P., Fox, S., & Garcia, C. (2012). Support for a fluid-continuum model of sexual orientation: A large-scale Internet study. Journal of Homosexuality, 59, 1356–1381.

    Article  PubMed  Google Scholar 

  • Feinstein, B. A., Dyar, C., Bhatia, V., Latack, J. A., & Davila, J. (2016). Conservative beliefs, attitudes toward bisexuality, and willingness to engage in romantic and sexual activities with a bisexual partner. Archives of Sexual Behavior, 45, 1535–1550.

    Article  PubMed  Google Scholar 

  • Fortson, B. L., Scotti, J. R., Ben, K. S. D., & Chen, Y. C. (2006). Reliability and validity of an Internet traumatic stress survey with a college student sample. Journal of Traumatic Stress, 19, 709–720.

    Article  PubMed  Google Scholar 

  • Fredriksen-Goldsen, K. I., Kim, H. J., & Barkan, S. E. (2012). Disability among lesbian, gay, and bisexual adults: Disparities in prevalence and risk. American Journal of Public Health, 102, e16–e21.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fu, H., Darroch, J. E., Henshaw, S. K., & Kolb, E. (1998). Measuring the extent of abortion underreporting in the 1995 national survey of family growth. Family Planning Perspectives, 30, 128–138.

    Article  PubMed  Google Scholar 

  • Gates, G. J. (2013a). Demographics and LGBT health. Journal of Health and Social Behavior, 54, 72–74.

    Article  PubMed  Google Scholar 

  • Gates, G. J. (2013b). Geography of the LGBT population. In A. K. Baumle (Ed.), International handbook on the demography of sexuality (pp. 229–242). Dordrecht: Springer.

    Chapter  Google Scholar 

  • Gates, G. J. (2014a). LGBT demographics: Comparisons among population-based surveys. Los Angeles, CA: The Williams Institute.

    Google Scholar 

  • Gates, G. J. (2014b). In U.S., LGBT more likely than non-LGBT to be uninsured. Retrieved from http://www.gallup.com/poll/175445/lgbt-likely-non-lgbt-uninsured.aspx.

  • Gates, G. J., & Newport, F. (2012). Special report: 3.4% of U.S. adults identify as LGBT. Retrieved from http://www.gallup.com/poll/158066/special-report-adults-identify-lgbt.aspx.

  • Gay and Lesbian Medical Association (GLMA). (2001). Healthy People 2010: Companion document for lesbian, gay, bisexual and transgender (LGBT) health. San Francisco, CA: GLMA.

    Google Scholar 

  • Gilman, S. E., Cochran, S. D., Mays, V. M., Hughes, M., Ostrow, D., & Kessler, R. C. (2001). Risk of psychiatric disorders among individuals reporting same-sex sexual partners in the national comorbidity survey. American Journal of Public Health, 91, 933–939.

    Article  PubMed  PubMed Central  Google Scholar 

  • Gnambs, T., & Kaspar, K. (2015). Disclosure of sensitive behaviors across self-administered survey modes: A meta-analysis. Behavior Research Methods, 47, 1237–1259.

    Article  PubMed  Google Scholar 

  • Goffman, E. (1963). Stigma: Notes on the management of spoiled identity. New York, NY: Simon and Schuster.

    Google Scholar 

  • Graham, R., Berkowitz, B., Blum, R., Bockting, W., Bradford, J., de Vries, B., … Makadon, H. (2011). The health of lesbian, gay, bisexual, and transgender people: Building a foundation for better understanding. Washington, DC: Institute of Medicine.

    Google Scholar 

  • Graham, A., & Papandonatos, G. (2008). Reliability of Internet- versus telephone-administered questionnaires in a diverse sample of smokers. Journal of Medical Internet Research, 10, e8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Gruskin, E. P., Greenwood, G. L., Matevia, M., Pollack, L. M., & Bye, L. L. (2007). Disparities in smoking between the lesbian, gay, and bisexual population and the general population in California. American Journal of Public Health, 97, 1496–1502.

    Article  PubMed  PubMed Central  Google Scholar 

  • Herbenick, D., Reece, M., Schick, V., Sanders, S. A., Dodge, B., & Fortenberry, J. D. (2010). Sexual behavior in the United States: Results from a national probability sample of men and women ages 14–94. Journal of Sexual Medicine, 7, 255–265.

    Article  PubMed  Google Scholar 

  • Hinkins, S., Crego, A., English, N., Pedlow, S., & Wolter, K. M. (n.d.). 2010 national sample frame. Retrieved from http://www.norc.org/Research/Projects/Pages/2010-national-sample-frame.aspx.

  • Holbrook, A. L., & Krosnick, J. A. (2010). Social desirability bias in voter turnout reports tests using the item count technique. Public Opinion Quarterly, 74, 37–67.

    Article  Google Scholar 

  • Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression. New York: Wiley.

    Book  Google Scholar 

  • Hyman, H. (1944). Do they tell the truth? Public Opinion Quarterly, 8, 557–559.

    Article  Google Scholar 

  • Internet Live Stats. (2016). Internet users in the U.S.A. (2016*). Retrieved from http://www.internetlivestats.com.

  • Itaborahy, L., & Zhu, J. (2014). State-sponsored homophobia, a world survey of laws: Criminalisation, protection and recognition of same-sex love (9th ed.). Geneva: International Lesbian, Gay, Bisexual and Intersex Associates.

    Google Scholar 

  • Johnson, T., & Fendrich, M. (2005). Modeling sources of self-report bias in a survey of drug use epidemiology. Annals of Epidemiology, 15, 381–389.

    Article  PubMed  Google Scholar 

  • Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires. Behavior Research Methods, Instruments, & Computers, 31, 433–438.

    Article  Google Scholar 

  • Joinson, A. N., Woodley, A., & Reips, U. D. (2007). Personalization, authentication and self-disclosure in self-administered Internet surveys. Computers in Human Behavior, 23, 275–285.

    Article  Google Scholar 

  • Joloza, T., Evans, J., O’Brien, R., & Potter-Collins, A. (2010). Measuring sexual identity: An evaluation report. Newport, UK: Office for National Statistics.

    Google Scholar 

  • Jones, E. F., & Forrest, J. D. (1992). Underreporting of abortion in surveys of U.S. women: 1976 to 1988. Demography, 29, 113–126.

    Article  PubMed  Google Scholar 

  • Jones, R. K., & Kost, K. (2007). Underreporting of induced and spontaneous abortion in the United States: An analysis of the 2002 National Survey of Family Growth. Studies in Family Planning, 38, 187–197.

    Article  PubMed  Google Scholar 

  • Kann, L., Olsen, E. O., McManus, T., Kinchen, S., Chyen, D., Harris, W. A., & Wechsler, H. (2011). Sexual identity, sex of sexual contacts, and health-risk behaviors among students in grades 9–12-youth risk behavior surveillance, selected sites, United States, 2001–2009. Morbidity and Mortality Weekly Report. Surveillance Summaries, 60(7), 1–133.

    PubMed  Google Scholar 

  • Kanuha, V. (1999). The social process of passing to manage stigma: Acts of internalized oppression or acts of resistance? Journal of Sociology and Social Welfare, 26(4), 27–46.

    Google Scholar 

  • Katz-Wise, S. L. (2015). Sexual fluidity in young adult women and men: Associations with sexual orientation and sexual identity development. Psychology & Sexuality, 6, 189–208.

    Article  Google Scholar 

  • Keeter, S., McGeeney, K., Igielnik, R., Mercer, A., & Mathiowetz, N. (2015, May 13). From telephone to the web: The challenge of mode of interview effects in public opinion polls. Retrieved from http://www.pewresearch.org/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/.

  • Keeter, S., Miller, C., Kohut, A., Groves, R. M., & Presser, S. (2000). Consequences of reducing nonresponse in a national telephone survey. Public Opinion Quarterly, 64, 125–148.

    Article  PubMed  Google Scholar 

  • Kitzinger, C. (2005). Heteronormativity in action: Reproducing the heterosexual nuclear family in after-hours medical calls. Social Problems, 52, 477–498.

    Article  Google Scholar 

  • Kleck, G., & Roberts, K. (2012). What survey modes are most effective in eliciting self-reports of criminal or delinquent behavior? In L. Gideon (Ed.), Handbook of survey methodology for the social sciences (pp. 417–439). New York, NY: Springer.

    Chapter  Google Scholar 

  • Kreuter, F., Presser, S., & Tourangeau, R. (2008). Social desirability bias in CATI, IVR, and Web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72, 847–865.

    Article  Google Scholar 

  • Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201–219.

    Article  Google Scholar 

  • Krumpal, I. (2013). Determinants of social desirability bias in sensitive surveys: A literature review. Quality & Quantity, 47, 2025–2047.

    Article  Google Scholar 

  • Langhaug, L. F., Sherr, L., & Cowan, F. M. (2010). How to improve the validity of sexual behaviour reporting: Systematic review of questionnaire delivery modes in developing countries. Tropical Medicine & International Health, 15, 362–381.

    Article  Google Scholar 

  • Lee, R. M. (1993). Doing research on sensitive topics. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Lee, S., McClain, C., Webster, N., & Han, S. (2016). Question order sensitivity of subjective well-being measures: Focus on life satisfaction, self-rated health, and subjective life expectancy in survey instruments. Quality of Life Research, 25, 2497–2510.

    Article  PubMed  Google Scholar 

  • Lensvelt-Mulders, G. J., Hox, J. J., & Van Der Heijden, P. G. (2005). How to improve the efficiency of randomised response designs. Quality & Quantity, 39, 253–265.

    Article  Google Scholar 

  • Locke, S. D., & Gilbert, B. O. (1995). Method of psychological assessment, self-disclosure, and experiential differences: A study of computer, questionnaire, and interview assessment formats. Journal of Social Behavior and Personality, 10, 255–263.

    Google Scholar 

  • Lucas, G. M., Gratch, J., King, A., & Morency, L. P. (2014). It’s only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior, 37, 94–100.

    Article  Google Scholar 

  • Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.

    Article  PubMed  Google Scholar 

  • Mayer, K. H., Bradford, J. B., Makadon, H. J., Stall, R., Goldhammer, H., & Landers, S. (2008). Sexual and gender minority health: What we know and what needs to be done. American Journal of Public Health, 98, 989–995.

    Article  PubMed  PubMed Central  Google Scholar 

  • McLaughlin, K. A., Hatzenbuehler, M. L., & Keyes, K. M. (2010). Responses to discrimination and psychiatric disorders among Black, Hispanic, female, and lesbian, gay, and bisexual individuals. American Journal of Public Health, 100, 1477–1484.

    Article  PubMed  PubMed Central  Google Scholar 

  • Millar, M. M., & Dillman, D. A. (2011). Improving response to Web and mixed-mode surveys. Public Opinion Quarterly, 75, 249–269.

    Article  Google Scholar 

  • Miller, J. D. (1984). A new survey technique for studying deviant behavior. Doctoral dissertation, George Washington University.

  • Mock, S. E., & Eibach, R. P. (2012). Stability and change in sexual orientation identity over a 10-year period in adulthood. Archives of Sexual Behavior, 41, 641–648.

    Article  PubMed  Google Scholar 

  • Moore, P. (2015). A third of young Americans say they aren’t 100% heterosexual. Retrieved from https://today.yougov.com/news/2015/08/20/third-young-americans-exclusively-heterosexual/?sid=5388f1ffdd52b8ed110008bc&wpsrc=slatest_newsletter.

  • Morral, A. R., McCaffrey, D., & Iguchi, M. Y. (2000). Hardcore drug users claim to be occasional users: Drug use frequency underreporting. Drug and Alcohol Dependence, 57, 193–202.

    Article  PubMed  Google Scholar 

  • Mustanski, B., Van Wagenen, A., Birkett, M., Eyster, S., & Corliss, H. L. (2014). Identifying sexual orientation health disparities in adolescents: Analysis of pooled data from the Youth Risk Behavior Survey, 2005 and 2007. American Journal of Public Health, 104, 211–217.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nathanson, A. T., & Reinert, S. E. (1999). Windsurfing injuries: Results of a paper-and Internet-based survey. Wilderness & Environmental Medicine, 10, 218–225.

    Article  Google Scholar 

  • National Institute on Alcohol Abuse and Alcoholism (NIAAA). (2010). U.S. alcohol epidemiological data reference manual (2nd ed., Vol. 8). Retrieved from http://pubs.niaaa.nih.gov/publications/NESARC_DRM2/NESARC2DRM.pdf.

  • National Survey of Family Growth (NSFG). (n.d). 2006–2010 NSFG questionnaires. Retrieved from http://www.cdc.gov/nchs/nsfg/nsfg_2006_2010_questionnaires.htm.

  • NORC at the University of Chicago. (n.d.). 2010 national sample frame. Retrieved from http://www.norc.org/Research/Projects/Pages/2010-national-sample-frame.aspx.

  • O’Reilly, J., Hubbard, M., Lessler, J., Biemer, P., & Turner, C. F. (1994). Audio computer assisted self-interviewing: New technology for data collection on sensitive issues and special populations. Journal of Official Statistics, 10, 197–214.

    PubMed  PubMed Central  Google Scholar 

  • Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.

    Article  Google Scholar 

  • Parry, H. J., & Crossley, H. M. (1950). Validity of responses to survey questions. Public Opinion Quarterly, 14, 61–80.

    Article  Google Scholar 

  • Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46, 598.

    Article  Google Scholar 

  • Paulhus, D. L. (1986). Self-deception and impression management in test responses. In A. Angleitner & J. S. Wiggins (Eds.), Personality assessment via questionnaires (pp. 143–165). New York, NY: Springer.

    Chapter  Google Scholar 

  • Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 51–77). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031.

    Article  PubMed  Google Scholar 

  • Pew Research Center. (2013, June 13). A survey of LGBT Americans. Retrieved from http://www.pewsocialtrends.org/2013/06/13/a-survey-of-lgbt-americans/.

  • Pew Research Center. (n.d.). Global views on morality. Retrieved from http://www.pewglobal.org/2014/04/15/global-morality/.

  • Preisendörfer, P., & Wolter, F. (2013). Asking sensitive questions: An evaluation of the randomized response technique versus direct questioning using individual validation data. Sociological Methods & Research, 42, 321–353.

    Article  Google Scholar 

  • Ramo, D. E., Hall, S. M., & Prochaska, J. J. (2011). Reliability and validity of self-reported smoking in an anonymous online survey with young adults. Health Psychology, 30, 693–701.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rasinski, K. A., Willis, G. B., Baldwin, A. K., Yeh, W., & Lee, L. (1999). Methods of data collection, perceptions of risks and losses, and motivation to give truthful answers to sensitive survey questions. Applied Cognitive Psychology, 13, 465–484.

    Article  Google Scholar 

  • Remafedi, G., French, S., Story, M., Resnick, M. D., & Blum, R. (1998). The relationship between suicide risk and sexual orientation: Results of a population-based study. American Journal of Public Health, 88, 57–60.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rhodes, S. D., Bowie, D. A., & Hergenrather, K. C. (2003). Collecting behavioural data using the World Wide Web: Considerations for researchers. Journal of Epidemiology and Community Health, 57, 68–73.

    Article  PubMed  PubMed Central  Google Scholar 

  • Roberts, A. L., Austin, S. B., Corliss, H. L., Vandermorris, A. K., & Koenen, K. C. (2010). Pervasive trauma exposure among U.S. sexual orientation minority adults and risk of posttraumatic stress disorder. American Journal of Public Health, 100, 2433–2441.

    Article  PubMed  PubMed Central  Google Scholar 

  • Russell, S. T., & Joyner, K. (2001). Adolescent sexual orientation and suicide risk: Evidence from a national study. American Journal of Public Health, 91, 1276–1281.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rust, P. R. (2002). Bisexuality: The state of the union. Annual Review of Sex Research, 13, 180–240.

    PubMed  Google Scholar 

  • Ryff, C., Almeida, D. M., Ayanian, J. S., Carr, D. S., Cleary, P. D., Coe, C., … Mroczek, D. K. (2012). National Survey of Midlife Development in the United States (MIDUS II), 2004–2006 (ICPSR04652-v6). Ann Arbor, MI: Inter-University Consortium for Political and Social Research.

    Google Scholar 

  • Savin-Williams, R. C. (2006). Who’s gay? Does it matter? Current Directions in Psychological Science, 15, 40–44.

    Article  Google Scholar 

  • Sell, R. L., & Becker, J. B. (2001). Sexual orientation data collection and progress toward Healthy People 2010. American Journal of Public Health, 91, 876–882.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sell, R. L., & Holliday, M. L. (2014). Sexual orientation data collection policy in the United States: Public health malpractice. American Journal of Public Health, 104, 967–969.

    Article  PubMed  PubMed Central  Google Scholar 

  • Singer, E., Von Thurn, D. R., & Miller, E. R. (1995). Confidentiality assurances and response a quantitative review of the experimental literature. Public Opinion Quarterly, 59, 66–77.

    Article  Google Scholar 

  • Smith, T. W., Marsden, P. V., & Hout, M. (2015). General social surveys, 1972–2014 [data file]. Retrieved from http://publicdata.norc.org:41000/gss/documents/BOOK/GSS_Codebook.pdf.

  • Stocké, V., & Hunkler, C. (2007). Measures of desirability beliefs and their validity as indicators for socially desirable responding. Field Methods, 19, 313–336.

    Article  Google Scholar 

  • Tourangeau, R., Couper, M. P., & Conrad, F. G. (2013). “Up means good”: The effect of screen position on evaluative ratings in web surveys. Public Opinion Quarterly, 77, 69–88.

    Article  PubMed  PubMed Central  Google Scholar 

  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, MA: Cambridge University Press.

    Book  Google Scholar 

  • Tourangeau, R., & Smith, T. W. (1996). Asking sensitive questions the impact of data collection mode, question format, and question context. Public Opinion Quarterly, 60, 275–304.

    Article  Google Scholar 

  • Tourangeau, R., & Smith, T. W. (1998). Collecting sensitive information with different modes of data collection. In M. P. Couper, R. P. Baker, J. Bethlehem, C. Z. Clark, J. Martin, W. L. Nicholls II, et al. (Eds.), Computer assisted survey information collection (pp. 431–454). New York, NY: Wiley.

    Google Scholar 

  • Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883.

    Article  PubMed  Google Scholar 

  • Trau, R. N., Härtel, C. E., & Härtel, G. F. (2013). Reaching and hearing the invisible: Organizational research on invisible stigmatized groups via web surveys. British Journal of Management, 24, 532–541.

    Article  Google Scholar 

  • Tsuchiya, T., Hirai, Y., & Ono, S. (2007). A study of the properties of the item count technique. Public Opinion Quarterly, 71, 253–272.

    Article  Google Scholar 

  • Turner, C. F., Forsyth, B. H., O’Reilly, J. M., Cooley, P. C., Smith, T. K., Rogers, S. M., & Miller, H. G. (1998). Automated self-interviewing and the survey measurement of sensitive behaviors. In M. P. Couper, R. P. Baler, J. Bethlehem, C. Z. Clark, J. Martin, W. Nichols, & J. M. O’Reilly (Eds.), Computer assisted survey information collection (pp. 455–473). New York, NY: Wiley.

    Google Scholar 

  • Turner, C. F., Ku, L., Sonenstein, F. L., & Pleck, J. H. (1996). Impact of ACASI on reporting of male-male sexual contacts: Preliminary results from the 1995 national survey of adolescent males. In R. B. Warnecke (Ed.), Health survey research methods (pp. 171–176). Hyattsville, MD: National Center for Health Statistics.

    Google Scholar 

  • Turner, C. F., Lessler, J. T., & Devore, J. (1992). Effects of mode of administration and wording on reporting of drug use. In C. F. Turner, J. T. Lessler, & J. C. Groerer (Eds.), Survey measurement of drug use: Methodological studies (pp. 177–220). Rockville, MD: National Institute on Drug Abuse.

    Google Scholar 

  • Villarroel, M. A., Turner, C. F., Eggleston, E., Al-Tayyib, A., Rogers, S. M., Roman, A. M., … Gordek, H. (2006). Same-gender sex in the United States impact of T-ACASI on prevalence estimates. Public Opinion Quarterly, 70, 166–196.

    Article  PubMed  PubMed Central  Google Scholar 

  • Vrangalova, Z., & Savin-Williams, R. C. (2012). Mostly heterosexual and mostly gay/lesbian: Evidence for new sexual orientation identities. Archives of Sexual Behavior, 41, 85–101.

    Article  PubMed  Google Scholar 

  • Walters, M. L., Chen, J., & Breiding, M. J. (2013). The national intimate partner and sexual violence survey (NISVS): 2010 findings on victimization by sexual orientation. Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.

    Google Scholar 

  • Ward, B. W., Dahlhamer, J. M., Galinsky, A. M., & Joestl, S. S. (2014). Sexual orientation and health among U.S. adults: National health interview survey, 2013. National Health Statistics Reports (no. 77). Atlanta, GA: Centers for Disease Control and Prevention.

    Google Scholar 

  • Warner, M. (Ed.). (1993). Fear of a queer planet: Queer politics and social theory (Vol. 6). Minneapolis, MN: University of Minnesota Press.

    Google Scholar 

  • Wright, K. B. (2005). Researching Internet-based populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of Computer-Mediated Communication. doi:10.1111/j.1083-6101.2005.tb00259.x.

    Google Scholar 

  • Wright, D. L., Aquilino, W. S., & Supple, A. J. (1998). A comparison of computer-assisted and paper-and-pencil self-administered questionnaires in a survey on smoking, alcohol, and drug use. Public Opinion Quarterly, 62, 331–353.

    Article  Google Scholar 

  • YouGov U.S. (n.d.). YouGov Omnibus. Retrieved from https://today.yougov.com/find-solutions/omnibus/.

Download references

Acknowledgements

Earlier versions of this article were presented at the 2013 annual meeting of the Society for the Scientific Study of Sexuality and at the 2014 annual meeting of the Western Psychological Association. The authors would like to thank Brian W. Ward, Debby Herbenick, Randy Sell, and Tom W. Smith for their helpful comments and suggestions.

Author information

Authors and Affiliations

  1. American Institute for Behavioral Research and Technology, 1035 East Vista Way, Ste. 120, Vista, CA, 92084, USA

    Ronald E. Robertson, Felix W. Tran, Lauren N. Lewark & Robert Epstein

Authors
  1. Ronald E. Robertson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Felix W. Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lauren N. Lewark
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert Epstein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ronald E. Robertson.

Ethics declarations

Conflict of interest

All authors declare no conflicts of interest.

Human and Animal Rights

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

The data and code used in this paper are available at: https://github.com/gitronald/nh-estimates.

Appendix: Questionnaire Content

Appendix: Questionnaire Content

Mode

Questionnaire item

Online

In any setting where Internet access is available to you (on your personal computer at home, a public library computer, etc.), you read questions about your sexual orientation on a computer screen and respond by entering your answer on a keyboard or a touch screen. No interviewer is present, and you are not required to give your name or contact information

Online (name required)

In any setting where Internet access is available to you (on your personal computer at home, a public library computer, etc.), you read questions about your sexual orientation on a computer screen and respond by entering your answer on a keyboard or a touch screen. No interviewer is present, but you are required to give your name and contact information

CASI

In a university setting or research facility—somewhere you’ve never been before—you read questions about your sexual orientation on a computer screen. You respond by entering your answers on a keyboard or touch screen. No interviewer is present, and no one in that setting knows your name

CASI (name required)

In a university setting or research facility—somewhere you’ve never been before—you read questions about your sexual orientation on a computer screen. You respond by entering your answers on a keyboard or touch screen. No interviewer is present, but you gave your name and contact information when you entered the setting

ACASI

In a university setting or research facility—somewhere you’ve never been before—you are presented with questions about your sexual orientation on a computer screen as you simultaneously hear an audio recording of the questions through a pair of headphones. You respond by entering your answer on a keyboard or a touch screen. No interviewer is present, and no one in that setting knows your name

ACASI (name required)

In a university setting or research facility—somewhere you’ve never been before—you are presented with questions about your sexual orientation on a computer screen as you simultaneously hear an audio recording of the questions through a pair of headphones. You respond by entering your answer on a keyboard or a touch screen. No interviewer is present, but you gave your name and contact information when you entered the setting

SAQ

In a university setting or research facility—somewhere you’ve never been before—you answer questions about your sexual orientation on a piece of paper and return it to the administrator when you are done. You were not required to write your name or contact information on the paper, and no one in that setting knows your name

SAQ (name required)

In a university setting or research facility—somewhere you’ve never been before—you answer questions about your sexual orientation on a piece of paper and return it to the administrator when you are done. You were required to write your name and contact information on the paper

CATI

At home, you are called by an interviewer who says that he or she works for a university or a research organization. You are not asked to provide your name. The caller asks you questions about your sexual orientation and enters your answers into a computer

CATI (name required)

At home, you are called by an interviewer who says that he or she works for a university or a research organization. The caller asks you for your name, and you give it. The caller then asks you questions about your sexual orientation and enters your answers into a computer

CAPI

In a university setting or research facility—somewhere you’ve never been before—you are face-to-face with an interviewer whom you don’t know. He or she asks you questions about your sexual orientation and enters your answers into a computer. No one in that setting knows your name

CAPI (name required)

In a university setting or research facility—somewhere you’ve never been before—you are face-to-face with an interviewer whom you don’t know. He or she asks you questions about your sexual orientation and enters your answers into a computer. You gave your name and contact information when you entered that setting

CAPI (video recorded)

In a university setting or research facility—somewhere you’ve never been before—you are face-to-face with an interviewer whom you don’t know. He or she asks you questions about your sexual orientation and enters your answers into a computer. No one in that setting knows your name, but a video recording is being made of your session

CAPI (name required and video recorded)

In a university setting or research facility—somewhere you’ve never been before—you are face-to-face with an interviewer whom you don’t know. He or she asks you questions about your sexual orientation and enters your answers into a computer. You gave your name and contact information as part of the interview, and a video recording is being made of your session

Face-to-face

In a university setting or research facility—somewhere you’ve never been before—you are face-to-face with an interviewer whom you don’t know. He or she asks you questions about your sexual orientation and takes notes as you answer. No one in that setting knows your name

Face-to-face (name required)

In a university setting or research facility—somewhere you’ve never been before—you are face-to-face with an interviewer whom you don’t know. He or she asks you questions about your sexual orientation and takes notes as you answer. You gave your name and contact information as part of the interview

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Robertson, R.E., Tran, F.W., Lewark, L.N. et al. Estimates of Non-Heterosexual Prevalence: The Roles of Anonymity and Privacy in Survey Methodology. Arch Sex Behav 47, 1069–1084 (2018). https://doi.org/10.1007/s10508-017-1044-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10508-017-1044-z

Keywords

Search

Navigation