Skip to main content
SpringerLink
Log in

Reliability of Self-Reported Health Service Use: Evidence from the Women with Co-occurring Disorders, and Violence Study

  • Regular Paper
  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

In behavioral health services research, self-reporting provides comprehensive information on service use, but may have limited reliability because of recall bias and misclassification. This study examines test–retest reliability of self-reported health service use, factors affecting reliability, and the impact of inconsistent reporting on the robustness of cost estimates using the test–retest data from the Women, Co-occurring Disorders, and Violence Study (n = 186). Reliability varies widely across service types: moderate to substantial (k = 0.65–0.94) for any use; slight to substantial (ICC = 0.12–0.93) for quantity of use; and none to moderate (k = −0.06–0.79) for service content, but is not affected by psychiatric symptom severity. Cost estimates do not differ according to the use of test or retest data. Findings suggest that self-reporting provides reliable data on service quantity and is adequate for economic evaluations. However, self-reporting of treatment content in highly specified service categories (e.g., individual counseling during residential treatment) may not be reliable.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1

Similar content being viewed by others

References

  1. Ganiats TG, Sieber WJ, Weisman M. Health-related quality of life. Best Practices and Benchmarking in Healthcare. 1997;2:57–62.

    CAS  PubMed  Google Scholar 

  2. Brown JB, Adams ME. Patients as reliable reporters of medical care process: recall of ambulatory encounter events. Medical Care. 1992;30:400–411.

    Article  CAS  PubMed  Google Scholar 

  3. Veysey BM, Steadman HJ, Morrissey JP, et al. In search of the missing linkages: continuity of care in U.S. jails. Behavioral Sciences & the Law. 1997;15(4):383–397.

    Article  CAS  Google Scholar 

  4. Hargreaves W, Shumway M, Hu TW, et al. (eds). Cost-outcome Methods for Mental Health. San Diego: Academic Press, 1998.

    Google Scholar 

  5. Del Boca FK, Noll JA. 2000 Truth or consequences: the validity of self-report in health services research on addictions. Addiction. 2000;95:347–360.

    Article  Google Scholar 

  6. Rozario PA, Morrow-Howell N, Proctor E. Comparing the congruency of self-report and provider records of depressed elder’s service use by provider type. Medical Care. 2004;42:952–959.

    Article  PubMed  Google Scholar 

  7. Booth BM, Kirchner JE, Fortney SM, et al. Measuring use of health services for at-risk drinkers: how brief can you get. The Journal of Behavioral Health Services & Research. 2006;33:254–264.

    Article  Google Scholar 

  8. Fendrich M, Johnson T, Wislar JS, et al. Accuracy of parent mental health service reporting: results from a reverse record-check study. Journal of the American Academy of Child and Adolescent Psychiatry. 1999;38(2):147–155.

    Article  CAS  PubMed  Google Scholar 

  9. Rhodes AE, Fung K. Self-reported use of mental health services versus administrative records: care to recall. International Journal of Methods in Psychiatric Research. 2004;13(3):165–175.

    Article  PubMed  Google Scholar 

  10. Beebe TJ, McRae JA, Barnes SA. A comparison of self-reported use of behavioral health services with Medicaid agency records in Minnesota. Psychiatric Services. 2006;57(11):1652–1654.

    Article  PubMed  Google Scholar 

  11. Klabunde CN, Reeve BB, Harlan LC, et al. Do patients consistently report comorbid conditions over time? Results from the Prostate Cancer Outcomes Study. Medical Care. 2005;43:391–400.

    Article  PubMed  Google Scholar 

  12. Netert PJ, French MT, Kirchner J, et al. Health services utilization and cost for at-risk drinkers: rural and urban comparisons. Journal of Studies on Alcohol. 2004;65(3):352–362.

    Google Scholar 

  13. Beckett M, Weinstein M, Goldman N, et al. Do health interview surveys yield reliable data on chronic illness among older respondents. American Journal of Epidemiology. 2000;151:315–323.

    CAS  PubMed  Google Scholar 

  14. Horwitz SM, Hoagwood K, Stiffman AR, et al. Reliability of the services assessment for children and adolescents. Psychiatric Services. 2001;52(8):1088–1094.

    Article  CAS  PubMed  Google Scholar 

  15. Hoagwood EK, Jensen PS, Arnold LE, et al. Reliability of the services for children and adolescents-parent interview. Journal of the American Academy of Child and Adolescent Psychiatry. 2004;43(11):1345–1354.

    Article  Google Scholar 

  16. Canino G, Shrout PE, Alegria M, et al. Methodological challenges in assessing children’s mental health services utilization. Mental Health Services Research. 2002;4(2):97–107.

    Article  PubMed  Google Scholar 

  17. Bean DL, Rotheram-Borus MJ, Leibowitz A, et al. Spanish-language services assessment for children and adolescents (SACA): reliability of parent and adolescent reports. Journal of the American Academy of Child & Adolescent Psychiatry. 2003;42(2):241–248.

    Article  Google Scholar 

  18. Farmer E, Angold A, Burns B, et al. Reliability of self-reported service use: test–retest consistency of children’s responses to the Child and Adolescent Services Assessment (CASA). Journal of Child and Family Studies. 1994;3(3):307–325.

    Article  Google Scholar 

  19. Goldberg RW, Seybolt DC, Lehman A. Reliable self-report of health service use by individuals with serious mental illness. Psychiatric Services. 2002;53:879–881.

    Article  PubMed  Google Scholar 

  20. Santelli AJ, Klein J, Graff C, et al. Reliability in adolescent reporting of clinician counseling, health care use, and health behaviors. Medical Care. 2002;40(1):26–37.

    Article  PubMed  Google Scholar 

  21. Nelson DE, Holtzman D, Bolen J, et al. Reliability and validity of measures from the Behavioral Risk Factor Surveillance System (BRFSS). Sozial- und Präventivmedizin. 2001;46(S1):S3–S42.

    PubMed  Google Scholar 

  22. Roberts AR. The organizational structure and function of shelters for battered women and their children: a national survey. In: Roberts AR, ed. 2nd edn. Battered Women and their Families. New York, NY: Springer Publishing; 1998:58–75.

    Google Scholar 

  23. McHugo GJ, Kammerer N, Jackson EW, et al. Women, Co-occurring Disorders, and Violence Study: evaluation design and study population. Journal of Substance Abuse Treatment. 2005;28(2):91–107.

    Article  CAS  PubMed  Google Scholar 

  24. Derogatis LR. Brief Symptom Inventory (BSI): Administration, Scoring and Procedures Manual. 4th edn. Minneapolis, MN: National Computer Systems; 1993.

    Google Scholar 

  25. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37–46.

    Article  Google Scholar 

  26. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin. 1979;86:420–428.

    Article  PubMed  CAS  Google Scholar 

  27. Shrout PE. Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research. 1998;7:301–317.

    Article  CAS  PubMed  Google Scholar 

  28. Spitznagel EL, Helzer JE. A proposed solution to the base rate in the kappa statistic. Archives of General Psychiatry. 1985;42:725–728.

    CAS  PubMed  Google Scholar 

  29. Kraemer HC. Ramifications of a population model for k as a coefficient of reliability. Psychometrika. 1979;44:461–472.

    Article  Google Scholar 

  30. Carsjo K, Thorslund M, Warneryd B. The validity of survey data on utilization of health and social services among the very old. Journals of Gerontology. Series B, Social Sciences. 1994;49:S156–S164.

    CAS  Google Scholar 

  31. Domino ME, Morrissey JP, Chung S, et al. Twelve-month service use and costs for women with co-occurring mental and substance use disorders and a history of violence. Psychiatric Services. 2005;56:1223–1232.

    Article  PubMed  Google Scholar 

  32. Schwarz N. Self-reports: How the questions shape the answers. The American Psychologist. 1999;54(2):93–105.

    Article  Google Scholar 

  33. Clark RE, Ricketts SK, McHugo GJ. Measuring hospital use without claims: a comparison of patient and provider reports. Health Services Research. 1996;31:153–169.

    CAS  PubMed  Google Scholar 

  34. Kashner TM. Agreement between administrative files and written medical records—A case of the Department of Veterans Affairs. Medical Care. 1998;36(9):1324–1336.

    Article  CAS  PubMed  Google Scholar 

  35. Calsyn RJ, Allen G, Morse GA, et al. Can you trust self-report data provided by homeless mentally ill individuals. Evaluation Review. 1993;17(3):353–366.

    Article  Google Scholar 

  36. Tsemberis S, McHugo G, Williams V, et al. Measuring homelessness and residential stability: the residential time-line follow-back inventory. Journal of Community Psychology. 2007;35(1):29–42.

    Article  Google Scholar 

  37. Klinkenberg WD, Calsyn RJ, Morse GA, et al. Consistency of recall of sexual and drug-using behaviors for homeless persons with dual diagnosis. AIDS and Behavior. 2002;6(4):295–307.

    Article  Google Scholar 

  38. Nieves K, Draine J, Solomon P. The validity of self-reported criminal arrest history among clients of a psychiatric probation and parole service. Journal of Offender Rehabilitation. 2000;30(3–4):133–151.

    Article  Google Scholar 

  39. Sohler N, Colson PW, Meyer-Bahlburg HF, et al. Reliability of self-reports about sexual risk behavior for HIV among homeless men with severe mental illness. Psychiatric Services. 2000;51(6):814–816.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgment

This study was funded by the grant, number TI-00-003, from Substance Abuse and Mental Health Services Administration’s three centers: the Center for Substance Abuse Treatment, the Center for Mental Health Services, and the Center for Substance Abuse Prevention. This grant was entitled “Cooperative Agreement to Study Women with Alcohol, Drug Abuse and Mental Health Disorders who have Histories of Violence: Phase II”. The abstract of this paper was presented at the Academy Health’s Annual Research Meeting held June 26–28, 2005, at Boston, MA. Additional support was received by Dr. Chung and Dr. Domino from the National Institute of Mental Health: T32-MH-0182-61 and K01-MH-0656-39, respectively.

The assistance of project staff at the following participating sites (listed in alphabetical order by state) is gratefully acknowledged: Los Angeles, California: PROTOTYPES Systems Change Center, Vivian Brown, Principal Investigator; Stockton, California: Allies: An Integrated System of Care, Jennie Heckman, Principal Investigator; Thornton, Colorado: Arapahoe House – New Directions for Families, Nancy Van DeMark, Principal Investigator; Washington, DC: District of Columbia Trauma Collaboration Study, Roger Fallot, Principal Investigator; Avon Park, Florida: Triad Women’s Project, Margo Fleisher-Bond, Co-Principal Investigator, Colleen Clark, Co- Principal Investigator; Boston, Massachusetts: Boston Consortium of Services for Families in Recovery, Hortensia Amaro, Principal Investigator; Cambridge, Massachusetts: Women Embracing Life and Living (WELL) Project, Norma Finkelstein, Principal Investigator; Greenfield, Massachusetts: Franklin County Women’s Research Project, Rene Andersen, Principal Investigator; New York, New York: Portal Project, Sharon Cadiz, Principal Investigator. The Coordinating Center is operated by Policy Research Associates (PRA), located in Delmar, New York, in coordination with the National Center on Family Homelessness of Newton, Massachusetts and the Cecil G. Sheps Center for Health Services Research at the University of North Carolina at Chapel Hill, North Carolina. The interpretations and conclusions contained in this publication do not necessarily represent the position of the WCDVS Coordinating Center, participating study sites, participating Consumer/Survivor/Recovering persons, or the Substance Abuse and Mental Health Services Administration.

Author information

Authors and Affiliations

  1. Department of Psychiatry, University of California, San Francisco, 2727 Mariposa Street, Suite 100, San Francisco, CA, 94110, USA

    Sukyung Chung PhD

  2. Department of Health Policy and Administration, School of Public Health, University of North Carolina at Chapel Hill, 1104G McGavran-Greenberg Hall, CB#7411, Chapel Hill, NC, 27599-7411, USA

    Marisa Elena Domino PhD

  3. Innovation, Research and Training, Inc., 1415 NC Highway 54 West, Building 300, Suite 121, Durham, NC, 27707, USA

    Elizabeth W. Jackson PhD

  4. Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, 725 Martin Luther King Jr. Blvd., CB#7590, Chapel Hill, NC, 27599-7590, USA

    Joseph P. Morrissey PhD

Authors
  1. Sukyung Chung PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marisa Elena Domino PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elizabeth W. Jackson PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joseph P. Morrissey PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sukyung Chung PhD.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chung, S., Domino, M.E., Jackson, E.W. et al. Reliability of Self-Reported Health Service Use: Evidence from the Women with Co-occurring Disorders, and Violence Study. J Behav Health Serv Res 35, 265–278 (2008). https://doi.org/10.1007/s11414-007-9105-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-007-9105-z

Keywords

Search

Navigation