Abstract
In this study, cognitive interviewing methods were used to test targeted questionnaire items from a battery of quantitative instruments selected for a large multisite trial of supported housing interventions for homeless individuals with mental disorders. Most of the instruments had no published psychometrics in this population. Participants were 30 homeless adults with mental disorders (including substance use disorders) recruited from service agencies in Vancouver, Winnipeg, and Toronto, Canada. Six interviewers, trained in cognitive interviewing methods and using standard interview schedules, conducted the interviews. Questions and, in some cases, instructions, for testing were selected from existing instruments according to a priori criteria. Items on physical and mental health status, housing quality and living situation, substance use, health and justice system service use, and community integration were tested. The focus of testing was on relevance, comprehension, and recall, and on sensitivity/acceptability for this population. Findings were collated across items by site and conclusions validated by interviewers. There was both variation and similarity of responses for identified topics of interest. With respect to relevance, many items on the questionnaires were not applicable to homeless people. Comprehension varied considerably; thus, both checks on understanding and methods to assist comprehension and recall are recommended, particularly for participants with acute symptoms of mental illness and those with cognitive impairment. The acceptability of items ranged widely across the sample, but findings were consistent with previous literature, which indicates that “how you ask” is as important as “what you ask.” Cognitive interviewing methods worked well and elicited information crucial to effective measurement in this unique population. Pretesting study instruments, including standard instruments, for use in special populations such as homeless individuals with mental disorders is important for training interviewers and improving measurement, as well as interpreting findings.
Similar content being viewed by others
Notes
In the main study, homelessness is defined as being “absolute” (having no fixed place to stay for at least the previous seven nights and little likelihood of getting a place in the upcoming month) or “precariously housed.” No fixed place to stay includes living rough in a public or private place not ordinarily used as a regular sleeping accommodation for a human being (e.g., outside on the streets, in parks or on the beach, in doorways, in parked vehicles, squats, or parking garages), as well as those whose primary night-time residence is a supervised public or private emergency accommodation (e.g., a shelter or hostel). Those currently being discharged from an institution, prison, jail, or hospital with no accommodation also qualify as absolutely homeless. Precarious housing is defined as having a room in a single room occupancy facility, a rooming house, or hotel/motel as a primary residence, and two or more episodes of being absolutely homeless in the past year. The criteria for presence of a mental disorder includes two of five observed behaviors, one of five functional impairment items, written documentation of a diagnosed disorder or psychiatric inpatient admission, and/or an indication on the Mini International Neuropsychiatric Interview28 of the presence of current major depression, bipolar disorder, posttraumatic stress disorder, panic disorder, or psychotic disorder (more details available from the authors).
References
Hwang SW. Homelessness and health. Can Med Assoc J. 2001; 164: 229–233.
Canadian Institute for Health Information. Improving the Health of Canadians: Mental Health and Homelessness. Ottawa, Canada: CIHI; 2007.
Goering P, Tolomiczenko G, Sheldon T, Boydell K, Wasylenki D. Characteristics of persons who are homeless for the first time. Psychiatr Serv. 2002; 53(11): 1472–1474.
Aubry T, Klodawsky F, Hay E. Panel Study on Persons Who Are Homeless in Ottawa: Phase 1 Results. Centre for Research on Community Services. Ottawa, Canada: University of Ottawa; 2003.
Patterson M, Somers JM, McIntosh K, Shiell A, Frankish CJ. Housing and Support for Adults with Severe and/or Mental Illness in British Columbia. British Columbia, Canada: Centre for Applied Research in Mental Health and Addiction, Simon Fraser University; 2008.
Pinkney S, Ewing S. The Costs and Pathways of Homelessness: Developing Policy-Relevant Economic Analyses for the Australian Homelessness Service System. Melbourne, Australia: Institute for Social Research, Swinburne University of Technology; 2006.
Hopper K, Baumohl J. Redefining the Cursed Word: a Historical Interpretation of American Homelessness. Westport, CT: Oryx; 1996.
Boydell K, Goering P, Morrell-Bellai T. Narratives of identity: re-presentation of self in people who are homeless. Qual Health Res. 2000; 10(1): 26–38.
Kessell ER, Bhatia R, Bamberger JD, Kushel MB. Public health care utilization in a cohort of homeless adult applicants to a supportive housing program. J Urban Health. 2006; 83(5): 860–873.
Hwang SW, Ueng JJM, Chui S, et al. Universal health insurance and health care access for homeless persons. Am J Public Health. 2010; 100: 1454–1461.
Wright JD. Address Unknown: the Homeless in America. New York, NY: Transaction Publishers; 2009.
Tsemberis S. From streets to homes: an innovative approach to supported housing for homeless adults with psychiatric disabilities. J Community Psychol. 1999; 27: 225–241.
Tsemberis S, Eisenberg RF. Pathways to housing: supported housing for street-dwelling homeless individuals. Psychiatr Serv. 2000; 51: 487–493.
Pearson CL, Locke G, Montgomery AE, Buron L. The Applicability of Housing First Models to Homeless Persons with Serious Mental Illness. Washington, DC: US Department of Housing and Urban Development Office of Policy Development and Research; 2007.
Tsemberis S, Gulcur L, Nakae M. Housing first, consumer choice, and harm reduction for homeless individuals with a dual diagnosis. A J Public Health. 2004; 94(4): 651–656.
Sadowski LS, Kee RA, VanderWeele TJ. Effect of a housing and case management program on emergency department visits and hospitalizations among critically ill homeless adults: a randomized trial. JAMA. 2009; 301(17): 1771–1778.
Larimer ME, Malone DK, Gardner MD. Health care and public service use and costs before and after provision of housing for chronically homeless persons with severe alcohol problems. JAMA. 2009; 301(13): 1349–1357.
Kertesz SG, Crouch K, Milby JB, Cusimano RE, Schumacher JE. Housing first for homeless persons with active addiction: are we overreaching? Milbank Q. 2009; 87(2): 495–534.
Yanos PT, Barrow SM, Tsemberis S. Community integration in the early phase of housing among homeless persons diagnosed with severe mental illness: successes and challenges. Community Ment Health J. 2004; 40(2): 133–150.
Gelberg L, Siecke N. Accuracy of homeless adults’ self-reports. Med Care. 1997; 35(3): 287–290.
Goldfinger SM, Schutt RK, Seidman LJ, et al. Self-report and observer measures of substance abuse among homeless mentally ill persons in the cross-section and over time. J Nerv Ment Dis. 1996; 184(11): 667–672.
Rosen MI, McMahon TJ, Rosenheck RA. Homeless people whose self-reported SSI/DI status is inconsistent with social security administration records. Soc Secur Bull. 2007; 67(1): 53–62.
Mallinson S. Listening to respondents: a qualitative assessment of the Short-Form 36 Health Status Questionnaire. Soc Sci Med. 2002; 54: 11–21.
Collins D. Pre-testing survey instruments: an overview of cognitive methods. Qual Life Res. 2003; 12: 229–238.
Willis GB. Cognitive Interviewing: a Tool for Improving Questionnaire Design. New York, NY: SAGE; 2005.
Tourangeau R, Rasinski K. Psychol Bull. 1988; 103: 299–314.
Matter R, Kline S, Cook KF, Amtmann D. Measuring pain in the context of homelessness. Qual Life Res. 2009; 18: 863–872.
Sheehan DV, Lecrubier Y, Harnett-Sheehan K, et al. The Mini International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview. J Clin Psychiatry. 1998; 59(suppl 20): S22–S33.
Shern DL, Wilson NZ, Saranga Coen A, et al. Client outcomes II: longitudinal client data from the Colorado treatment outcome study. Milbank Q. 1994; 72(1): 123–148.
Dennis ML, Chan Y, Funk RR. Development and validation of the GAIN short screener for internalizing, externalizing and substance use disorders and crime/violence problems among adolescents and adults. Am J Addict. 2006; 15(suppl 1): S80–S91.
Latimer EA, Lecomte T, Becker DR, et al. Generalisability of the individual placement and support model of supported employment: results of a Canadian randomised controlled trial. Br J Psychiatry. 2006; 189: 65–73.
Canadian Community Health Survey 4.1. http://www.statcan.gc.ca/cgi-bin/imdb/p2SV.pl?Function=getSurvey&SurvId=3226&SurvVer=1&InstaId=15282&InstaVer=4&SDDS=3226&lang=en&db=imdb&adm=8&dis=2. Accessed August 20, 2010.
National Population Health Survey. http://www.statcan.gc.ca/concepts/nphs-ensp/index-eng.htm. Accessed August 20, 2010.
Mares AS, Rosenheck RA. HUD/HHS/VA Collaborative Initiative to Help end Chronic Homelessness National Performance Outcomes Assessment: Preliminary Client Outcomes Report. West Haven, CT: Northeast Program Evaluation Center; 2007.
Segal SP, Aviram U. The Mentally Ill in Community Based Sheltered Care: a Study of Community Care and Social Integration. New York, NY: Wiley; 1978.
Aubry T, Myner J. Community integration and quality of life: a comparison of persons with psychiatric disabilities in housing programs and community residents who are neighbors. Can J Commun Ment Health. 1996; 15: 5–20.
Chavis DM, Hogge JH, McMillan DW, Wandersman A. Sense of community through Brunswick’s lens: a first look. J Community Psychol. 1986; 14: 24–40.
Boothroyd RA, Chen HJ. The psychometric properties of the Colorado Symptom Index. Adm Policy Ment Health. 2008; 35(5): 370–378.
Conrad KJ, Yagelka JR, Matters MD, et al. Reliability and validity of a modified Colorado Symptom Index in a national homeless sample. Ment Health Serv Res. 2001; 3(3): 141–153.
Corrigan PW, Salzer M, Ralph RO, Sangster Y, Keck L. Examining the factor structure of the recovery assessment scale. Schizophr Bull. 2004; 30(4): 1035–1041.
Lehman AF. Measures of quality of life among persons with severe and persistent mental disorders. Soc Psychiatry Psychiatr Epidemiol. 1996; 31: 78–88.
Uttaro T, Lehman A. Graded response modeling of the quality of life interview. Eval Program Plann Spring. 1999; 22: 41–52.
Kavanaugh K, Moro TT, Savage T, Mehendale R. Enacting a theory of caring to recruit and retain vulnerable participants for sensitive research. Res Nurs Health. 2006; 29: 244–252.
Edwards M, Thomsen SC, Toroitich-Ruto C. Thinking aloud to create better condom use questions. Field Methods. 2005; 17: 183–199.
Bradburn NM, Sudman S, Wansink B. Asking Questions: the Definitive Guide to Questionnaire Design for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco, CA: Jossey-Bass; 2004.
Acknowledgments
Appreciation is extended to Kimberley Lewis-Ng and Rebecca Godderis who assisted with the planning stages of this study. Susan Mulligan, Verena Strehlau, and Melinda Markey are also thanked for conducting some of the interviews. ACH was supported for this work through the University of Toronto “Comprehensive Research Experience for Medical Students.” The results of the study were presented at the University of Toronto Medical Student Research Day in February 2010. This, and the main study, was made possible through a financial contribution from Health Canada. The Mental Health Commission of Canada is the oversight organization for the study. The views expressed herein solely represent those of the authors and not of the afore-named organizations.
Author information
Authors and Affiliations
Consortia
Corresponding author
Additional information
The At Home/Chez Soi Project Team includes Jayne Barker, Ph.D., VP Research Initiatives, Mental Health Commission of Canada; Cameron Keller, M.C., Director At Home/Chez Soi; and approximately 40 investigators from across Canada and the USA. In addition, there are five site coordinators (one for each city where the study is carried out) and numerous lead service and housing providers. CEA is lead for quantitative measurement and data collection at the national level; MLP and KSM are site research coordinators; SWH is lead investigator for the Toronto site; ACH was an undergraduate medical research student for the summer of 2009; and PNG is the national research lead.
Rights and permissions
About this article
Cite this article
Adair, C.E., Holland, A.C., Patterson, M.L. et al. Cognitive Interviewing Methods for Questionnaire Pre-Testing in Homeless Persons with Mental Disorders. J Urban Health 89, 36–52 (2012). https://doi.org/10.1007/s11524-011-9632-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11524-011-9632-z