Skip to main content

Advertisement

SpringerLink
Log in

Survey Response Rate and Quality in a Mental Health Clinic Population: Results from a Randomized Survey Comparison

  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

Given the limited ability of informatics-based assessment technologies to reach individuals with serious mental health conditions, this study evaluated the feasibility and data quality of mail-out healthcare surveys in this population to assist in measure selection for a multi-site controlled implementation trial. Veterans were randomly selected from those who had received services at a mental health clinic in the Department of Veterans Affairs, and were randomly assigned to one of three questionnaire lengths. Survey length (48–127 items) was not associated with differences in response rate, percent of items missing, or data quality. However, internal consistency reliability was variable among scales and survey lengths. Additional analyses indicate the above measures of survey data quality may differ among respondents who are non-white and younger and have psychotic disorders. These results can inform survey protocols to ensure maximal representation of this vulnerable population in health planning and policy assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1

References

  1. Jones D, Kazis L, Lee A, et al. Health status assessments using the Veterans SF-12 and SF-36: methods for evaluating outcomes in the Veterans Health Administration. The Journal of Ambulatory Care Management. 2001;24(3):68–86.

    Article  CAS  PubMed  Google Scholar 

  2. McCabe R, Saidi M, Priebe S. Patient-reported outcomes in schizophrenia. The British Journal of Psychiatry Supplement. 2007;50:s21–28.

    Article  PubMed  Google Scholar 

  3. Slade M. Routine outcome assessment in mental health services. Psychological Medicine. 2002;32(8):1339–1343.

    Article  PubMed  Google Scholar 

  4. Cleary PD. The increasing importance of patient surveys. Now that sound methods exist, patient surveys can facilitate improvement. British Medical Journal. 1999;319(7212):720–721.

  5. Drain M. Quality improvement in primary care and the importance of patient perceptions. The Journal of Ambulatory Care Management. 2001;24(2):30–46.

    Article  CAS  PubMed  Google Scholar 

  6. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. Journal of the American Medical Association. 2012;307(15):1583–1584.

    Article  CAS  PubMed  Google Scholar 

  7. Shortell SM, Colla CH, Lewis VA, et al. Accountable Care Organizations: The National Landscape. Journal of Health Politics, Policy and Law. 2015;40(4):647–668.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Fan W, Yan Z. Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior. 2010;26(2):132–139.

    Article  Google Scholar 

  9. Groves RM. Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly. 2006;70(5):646–675.

    Article  Google Scholar 

  10. Biemer P, Lyberg L. Introduction to Survey Quality. Hoboken, NJ, John Wiley & Sons; 2003.

  11. Baruch Y. Response rate in academic studies--a comparative analysis. Human Relations. 1999;52(4):421–438.

    Google Scholar 

  12. Rookey BD, Le L, Littlejohn M, et al. Understanding the resilience of mail-back survey methods: An analysis of 20 years of change in response rates to national park surveys. Social Science Research. 2012;41(6):1404–1414.

    Article  PubMed  Google Scholar 

  13. Rogelberg SG, Stanton JM. Introduction understanding and dealing with organizational survey nonresponse. Organizational Research Methods. 2007;10(2):195–209.

    Article  Google Scholar 

  14. Tourangeau R, Yan T. Sensitive questions in surveys. Psychological Bulletin. 2007;133(5):859–883.

    Article  PubMed  Google Scholar 

  15. Edwards P, Roberts I, Clarke M, et al. Increasing response rates to postal questionnaires: systematic review. British Medical Journal. 2002;324:1183.

    Article  PubMed  Google Scholar 

  16. Galesic M, Bosnjak M. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly. 2009;73(2):349–360.

    Article  Google Scholar 

  17. Sheth JN, Roscoe AM. Impact of questionnaire length, follow-up methods, and geographical location on response rate to a mail survey. Journal of Applied Psychology. 1975;60(2):252.

    Article  Google Scholar 

  18. Mond JM, Rodgers B, Hay PJ, et al. Mode of delivery, but not questionnaire length, affected response in an epidemiological study of eating-disordered behavior. Journal of Clinical Epidemiology. 2004;57(11):1167–1171.

    Article  CAS  PubMed  Google Scholar 

  19. Miller CJ, McInnes DK, Stolzmann K, et al. Interest in use of technology for healthcare among Veterans receiving treatment for mental health. Telemedicine Journal and E- Health. 2016;22(10):847–854.

    Article  PubMed  Google Scholar 

  20. Shimada SL, Brandt CA, Feng H, et al. Personal health record reach in the Veterans Health Administration: a cross-sectional analysis. Journal of Medical Internet Research. 2014;16(12):e272.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Bauer MS, Miller C, Kim B, et al. Partnering with health system operations leadership to develop a controlled implementation trial. Implementation Science. 2016;11:22.

    Article  PubMed  Google Scholar 

  22. Kazis LE, Ren XS, Lee A, et al. Health status in VA patients: results from the Veterans Health Study. American Journal of Medical Quality. 1999;14(1):28–38.

    Article  CAS  PubMed  Google Scholar 

  23. McHorney CA, Ware JE, Jr., Raczek AE. The MOS 36-Item Short-Form Health Survey (SF-36): II. Psychometric and clinical tests of validity in measuring physical and mental health constructs. Medical Care. 1993;31(3):247–263.

    Article  CAS  PubMed  Google Scholar 

  24. Ware JE Jr., Sherbourne CD. The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Medical Care. 1992;30(6):473–483.

    Article  PubMed  Google Scholar 

  25. Kazis L, Selim A, Rogers W, et al.. Veterans RAND 12 Item Health Survey (VR-12): A white paper summary. http://www.hosonline.org/surveys/hos/download/Veterans_RAND_12_Item_Health_Survey_White_Paper_Summary.pdf. Accessed April 20, 2015.

  26. Vrijhoef HJ, Berbee R, Wagner EH, et al.. Quality of integrated chronic care measured by patient survey: identification, selection and application of most appropriate instruments. Health Expectations. 2009;12(4):417–429.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Gugiu PC, Coryn C, Clark R, et al. Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument. Chronic Illness. 2009;5(4):268–276.

    Article  PubMed  Google Scholar 

  28. Rundall TG, Shortell SM, Wang MC, et al. As good as it gets? Chronic care management in nine leading US physician organisations. British Medical Journal. 2002;325(7370):958–961.

    Article  PubMed  Google Scholar 

  29. Schmittdiel J, Mosen DM, Glasgow RE, et al. Patient Assessment of Chronic Illness Care (PACIC) and improved patient-centered outcomes for chronic conditions. Journal of General Internal Medicine. 2008;23(1):77–80.

    Article  PubMed  Google Scholar 

  30. Bauer MS, McBride L, Williford WO, et al. Collaborative care for bipolar disorder: part I. Intervention and implementation in a randomized effectiveness trial. Psychiatric Services. 2006;57(7):927–936.

    Article  PubMed  Google Scholar 

  31. Bush K, Kivlahan DR, McDonell MB, et al. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Archives of Internal Medicine. 1998;158(16):1789–1795.

    Article  CAS  PubMed  Google Scholar 

  32. Bradley KA, Bush KR, Epler AJ, et al. Two brief alcohol-screening tests from the Alcohol Use Disorders Identification Test (AUDIT): validation in a female Veterans Affairs patient population. Archives of Internal Medicine. 2003;163(7):821–829.

    Article  PubMed  Google Scholar 

  33. Bauer MS, Krawczyk L, Tuozzo K, et al. Implementing and sustaining team-based telecare for bipolar disorder: lessons learned from a model-guided, mixed methods analysis. Telemedicine Journal and E-Health. 2018;24(1):45–53.

    Article  PubMed  Google Scholar 

  34. Nabati L, Shea N, McBride L, et al. Adaptation of a simple patient satisfaction instrument to mental health: psychometric properties. Psychiatry Research. 1998;77(1):51–56.

    Article  CAS  PubMed  Google Scholar 

  35. Hall JA, Feldstein M, Fretwell MD, et al. Older patients’ health status and satisfaction with medical care in an HMO population. Medical Care. 1990;28(3):261–270.

    Article  CAS  PubMed  Google Scholar 

  36. Hall JA, Milburn MA, Epstein AM. A causal model of health status and satisfaction with medical care. Medical Care. 1993;31(1):84–94.

    Article  CAS  PubMed  Google Scholar 

  37. Sheehan DV, Harnett-Sheehan K, Raj BA. The measurement of disability. International Clinical Psychopharmacology. 1996;11 Suppl 3:89–95.

    Article  PubMed  Google Scholar 

  38. Williams GC, Freedman ZR, Deci EL. Supporting autonomy to motivate patients with diabetes for glucose control. Diabetes Care. 1998;21(10):1644–1651.

    Article  CAS  PubMed  Google Scholar 

  39. Williams GC, McGregor HA, Sharp D, et al. Testing a self-determination theory intervention for motivating tobacco cessation: supporting autonomy and competence in a clinical trial. Health Psychology. 2006;25(1):91–101.

    Article  PubMed  Google Scholar 

  40. Ludman EJ, Simon GE, Rutter CM, et al. A measure for assessing patient perception of provider support for self-management of bipolar disorder. Bipolar Disorders. 2002;4(4):249–253.

    Article  PubMed  Google Scholar 

  41. Eisen SV, Normand SL, Belanger AJ, et al. The Revised Behavior and Symptom Identification Scale (BASIS-R): reliability and validity. Medical Care. 2004;42(12):1230–1241.

    Article  PubMed  Google Scholar 

  42. Cameron IM, Cunningham L, Crawford JR, et al. Psychometric properties of the BASIS-24(c) (Behaviour and Symptom Identification Scale-Revised) Mental Health Outcome Measure. International Journal of Psychiatry in Clinical Practice. 2007;11(1):36–43.

    Article  CAS  PubMed  Google Scholar 

  43. Dillman DA Smyth JD, Christian LM. Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method (4th edition). Hoboken (NJ): John Wiley & Sons; 2014.

  44. American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 8th Edition. 2015; http://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions2015_8theditionwithchanges_April2015_logo.pdf. Accessed May 11,2015.

  45. Kazis LE, Lee A, Spiro A, et al. Measurement comparisons of the medical outcomes study and Veterans SF-36 health survey. Health Care Financing Review. 2004;25(4):43–58.

    PubMed  PubMed Central  Google Scholar 

  46. Kazis LE, Rogers W, Rothendler J, et al. Outcome Performance Measure Development for Persons with Multiple Chronic Conditions. Santa Monica, CA. 2017.

    Google Scholar 

  47. Cleary PD, Edgman-Levitan S, Roberts M, et al. Patients evaluate their hospital care: a national survey. Health Affairs. 1991;10(4):254–267.

    Article  CAS  PubMed  Google Scholar 

  48. VHA Office of Analytics and Business Intelligence. Patient Experiences Dimensions of Care, Inpatient Reports. http://vaww.car.rtp.med.va.gov/programs/shep/shepReportsDOCImp.aspx. Accessed October 5, 2015.

  49. Eisen SV, Schultz MR, Vogt D, et al. Mental and physical health status and alcohol and drug use following return from deployment to Iraq or Afghanistan. American Journal of Public Health. 2012;102 Suppl 1:S66–73.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Wowra SA, McCarter R. Validation of the empowerment scale with an outpatient mental health population. Psychiatric Services. 1999;50(7):959–961.

    Article  CAS  PubMed  Google Scholar 

  51. Singer SK, Kerrissey M, Fryer AK, et al. Patients’ perspectives on care integration: comparing physician organizations with more and less ACO-like characteristics. Academy of Management; Aug 10, 2015, 2015; Vancouver, Canada.

  52. Sheehan K. E-mail survey response rates: a review. Journal of Computer-Mediated Communication. 2001;6(2).

  53. Hays RD, Bjorner JB, Revicki DA, et al. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Quality of Life Research. 2009;18(7):873–880.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. Coordinating Care for Adults With Complex Care Needs in the Patient-Centered Medical Home: Challenges and Solutions. January 2012.

Download references

Author information

Authors and Affiliations

  1. Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System, 150 South Huntington Avenue, (152M), Boston, MA, 02130, USA

    Kelly Stolzmann MS, Christopher J. Miller PhD, Marjorie Nealon Seibert MBA & Mark S. Bauer MD

  2. Performance Measurement, VHA Office of Analytics and Business Intelligence (OABI), ENRM Veterans Affairs Medical Center, 200 Springs Road, Bedford, MA, 01730, USA

    Mark Meterko PhD

  3. Department of Health Law, Policy and Management, Boston University School of Public Health and Management, Boston, USA

    Mark Meterko PhD

  4. Department of Psychiatry, Harvard Medical School, Boston, USA

    Christopher J. Miller PhD & Mark S. Bauer MD

  5. Oregon Health and Sciences University, 3181 S.W. Sam Jackson Park Rd., Portland, OR, 97239-3098, USA

    Lindsay Belanger MPH

Authors
  1. Kelly Stolzmann MS
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Meterko PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher J. Miller PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lindsay Belanger MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marjorie Nealon Seibert MBA
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mark S. Bauer MD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kelly Stolzmann MS.

Ethics declarations

All study procedures were approved by the VA Central Institutional Review Board.

Conflict of Interest

The authors declare that they have no conflicts of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stolzmann, K., Meterko, M., Miller, C.J. et al. Survey Response Rate and Quality in a Mental Health Clinic Population: Results from a Randomized Survey Comparison. J Behav Health Serv Res 46, 521–532 (2019). https://doi.org/10.1007/s11414-018-9617-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-018-9617-8

Search

Navigation