Skip to main content

Satisfaction in Child and Adolescent Mental Health Services: Translating Users’ Feedback into Measurement

Abstract

The present research addressed gaps in our current understanding of validity and quality of measurement provided by patient reported experience measures. We established the psychometric properties of a freely available experience of service questionnaire (ESQ), based on responses from 7,067 families of patients across 41 UK providers of child and adolescent mental health services, using the two-level latent trait modeling. Responses to the ESQ were subject to strong ‘halo’ effects, which were thought to represent the overall positive or negative affect towards one’s treatment. Two strongly related constructs measured by the ESQ were interpreted as specific aspects of global satisfaction, namely satisfaction with care, and with environment. The Care construct was sensitive to differences between less satisfied patients, facilitating individual and service-level problem evaluation. The effects of nesting within service providers were strong, with parental reports being the most reliable source of data for the between-provider comparisons. We provide a scoring protocol for converting the hand-scored ESQ to the model-based population-referenced scores with supplied standard errors, which can be used for benchmarking services as well as individual evaluations.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Notes

  1. 1.

    Due to the small sample size at the provider level (N = 41), change in Chi-square based indices is not significant between models with different number of factors at the provider level. Residuals for correlations and SRMR are better indicators of relative model fit in this case.

  2. 2.

    Most EFA rotation methods would minimize the number of factors each item loads on—i.e. rotate to an independent clusters structure.

References

  1. Anderson, J. A., Rivera, V. R., & Kutash, K. (1998). Measuring consumer satisfaction with children’s mental health. In M. H. Epstein, K. Kutash, & A. J. Duchnowski (Eds.), Outcomes for children with emotional and behavioral disorders and their families: Program and evaluation best practices (pp. 455–482). Austin: Pro-Ed.

    Google Scholar 

  2. Attkisson, C., & Greenfield, T. (1996). The client satisfaction questionnaire (CSQ) scales and the service satisfaction scale-30 (SSS-30). In L. Sederer & B. Dickey (Eds.), Outcomes assessment in clinical practice. Baltimore: Williams & Wilkins.

    Google Scholar 

  3. Attride-Stirling, J. (2002). Development of methods to capture users’ views of CAMHS in clinical governance reviews. http://www.corc.uk.net/resources/downloads/.

  4. Ayton, A., Mooney, M., Sillifant, K., Powls, J., & Rasool, H. (2007). The development of the child and adolescent versions of the verona service satisfaction scale (CAMHSSS). Social Psychiatry and Psychiatric Epidemiology, 42, 892–901. doi:10.1007/s00127-007-0241-9.

    PubMed  Article  Google Scholar 

  5. Barber, A. J., Tischler, V. A., & Healy, E. (2006). Consumer satisfaction and child behaviour problems in child and adolescent mental health services. Journal of Child Health Care, 10(1), 9–21. doi:10.1177/1367493506060200.

    PubMed  Article  Google Scholar 

  6. Barnes, D., Devanney, C., Uglebjerg, A., Wistow, R. & Hartley, C. (2010). A profile of children’s health services, child and adolescent mental health services and maternity services in England 2008/9. Durham: Durham University. http://dro.dur.ac.uk/8220/1/8220.pdf.

  7. Biering, P., & Jensen, V. (2011). The concept of patient satisfaction in adolescent psychiatric care: A qualitative study. Journal of Child and Adolescent Psychiatric Nursing, 24, 3–10. doi:10.1111/j.1744-6171.2010.00261.x.

    PubMed  Article  Google Scholar 

  8. Brannan, A., Sonnichesen, E., & Heflinger, C. (1996). Measuring satisfaction with children’s mental health services: Validity and reliability of the satisfaction scales. Evaluation and Program Planning, 19, 131–141. doi:10.1016/0149-7189(96)00004-3.

    Article  Google Scholar 

  9. Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36, 111–150. doi:10.1207/S15327906MBR3601_05.

    Article  Google Scholar 

  10. Day, C., Michelson, D., & Hassan, I. (2011). Child and adolescent service experience (ChASE): Measuring service quality and therapeutic process. British Journal of Clinical Psychology, 50, 452–464. doi:10.1111/j.2044-8260.2011.02008.x.

    PubMed  Article  Google Scholar 

  11. Department of Health. (2009). Understanding what matters: A guide to using patient feedback to transform services. London: Department of Health. Retrieved from http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_099780.

  12. Edwards, A., & Elwyn, G. (2001). Developing professional ability to involve patients in their care: Pull or push? Quality in Health Care, 10, 129–134. doi:10.1136/qhc.0100129.

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  13. Garland, A., Saltzman, M., & Aarons, G. (2000). Adolescent satisfaction with mental health services: Development of a multidimensional scale. Evaluation and Program Planning, 23, 165–175. doi:10.1016/S0149-7189(00)00009-4.

    Article  Google Scholar 

  14. Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire (SDQ). Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337–1345. doi:10.1097/00004583-200111000-00015.

  15. Hart, A., Saunders, A., & Thomas, H. (2005). Attuned practice: a service user study of specialist child and adolescent mental health, UK. Epidemiology and Psychiatric Sciences, 14, 22–31. doi:10.1017/S1121189X00001895.

    Article  Google Scholar 

  16. Holbrook, M. B. (1983). Using a structural model of halo effect to assess perceptual distortion due to affective overtones. Journal of Consumer Research, 10, 247–252. http://www.jstor.org/stable/2488928.

  17. Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York: Routledge.

    Google Scholar 

  18. Kazdin, A., & Mazurick, J. (1994). Dropping out of child psychotherapy: Distinguishing early and late dropouts over the course of treatment. Journal of Consulting and Clinical Psychology, 62, 1069–1074. doi:10.1177/1359104596011012.

    CAS  PubMed  Article  Google Scholar 

  19. Kennedy, I. (2010). Getting it right for children and young people: Overcoming cultural barriers in the NHS so as to meet their needs. London: Department of Health. http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_119445.

  20. McNaughton, D. (1994). Measuring parent satisfaction with early childhood intervention programs. Topics in Early Childhood Special Education, 14, 26–48. doi:10.1177/027112149401400106.

    Article  Google Scholar 

  21. Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218–225. doi:10.1037/0021-9010.78.2.218.

    Article  Google Scholar 

  22. Muthén, L. K. & Muthén, B. O. (1998-2010). Mplus user’s guide. 6th Ed. Los Angeles, CA: Muthén & Muthén. www.stamodel.com.

  23. Muthén, B. & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. http://www.jstor.org/stable/271070.

  24. Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). Generalized multilevel structural equation modeling. Psychometrika, 69, 167–190. doi:10.1007/BF02295939.

    Article  Google Scholar 

  25. Reise, S., & Haviland, M. (2005). Item response theory and the measurement of clinical change. Journal of Personality Assessment, 84, 228–238. doi:10.1207/s15327752jpa8403_02.

    PubMed  Article  Google Scholar 

  26. Reise, S., & Waller, G. (2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 25–46. doi:10.1146/annurev.clinpsy.032408.153553.

    Article  Google Scholar 

  27. Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17). Richmond, VA: Psychometric Society.

  28. SDQ: Normative SDQ data from Britain. (2001). http://www.sdqinfo.com/UKNorm.html.

  29. Shapiro, J., Welker, C., & Jacobson, B. (1997). The youth client satisfaction questionnaire: Development, construct validation and factor structure. Journal of Child Clinical Psychology, 26, 87–98. doi:10.1207/s15374424jccp2601_9.

    CAS  Article  Google Scholar 

  30. Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120. doi:10.1007/s11336-008-9101-0.

    PubMed Central  PubMed  Article  Google Scholar 

  31. Stallard, P. (1995). Parental satisfaction with intervention: differences between respondents and non-respondents to postal questionnaire. British Journal of Clinical Psychology, 34, 397–405. doi:10.1111/j.2044-8260.1995.tb01474.x.

    PubMed  Article  Google Scholar 

  32. Thissen, D., & Orlando, M. (2001). Item response theory for items scored in two categories. In D. Thissen & H. Wainer (Eds.), Test Scoring (pp. 73–140). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  33. Williams, B., Coyle, J., & Healy, D. (1998). The meaning of patient satisfaction: An explanation of high reported levels. Social Science and Medicine, 47, 1351–1359. doi:10.1016/S0277-9536(98)00213-5.

    CAS  PubMed  Article  Google Scholar 

  34. CORC Measures Website http://www.corc.uk.net/index.php?contentkey=81.

  35. Young, S., Nicholson, J., & Davis, M. (1995). An overview of issues in research on consumer satisfaction with child and adolescent mental health services. Journal of Child and Family Studies, 4, 219–238. doi:10.1007/BF02234097.

    Article  Google Scholar 

Download references

Acknowledgments

We are grateful to all members of the CORC collaboration for providing the data, and to the CORC central team researchers Jenna Bradley and Halina Flannery for preparing the data for analyses. Anna Brown’s work on this article was supported by Grant RG63087 from the Isaac Newton Trust, University of Cambridge. Jessica Deighton’s work on this article was provided as part of the Department of Health Child Policy Research Unit (Reference number: 109/0001).

Conflict of interest

The authors have no competing interests. CORC is not-for-profit Company limited by guarantee. Tamsin Ford is an unpaid director of CORC (other than travel expenses). Miranda Wolpert is a paid director of CORC (3 days a week).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Anna Brown.

Appendix

Appendix

ESQ for parents/carers

Response options: Certainly TruePartly TrueNot TrueDon’t know

(“don’t know” response option is not scored).

  1. 1.

    I feel that the people who have seen my child listened to me

  2. 2.

    It was easy to talk to the people who have seen my child

  3. 3.

    I was treated well by the people who have seen my child

  4. 4.

    My views and worries were taken seriously

  5. 5.

    I feel the people here know how to help with the problem I came for

  6. 6.

    I have been given enough explanation about the help available here

  7. 7.

    I feel that the people who have seen my child are working together to help with the problem(s)

  8. 8.

    The facilities here are comfortable (e.g. waiting area)

  9. 9.

    The appointments are usually at a convenient time (e.g. don’t interfere with work, school)

  10. 10.

    It is quite easy to get to the place where the appointments are

  11. 11.

    If a friend needed similar help, I would recommend that he or she come here

  12. 12.

    Overall, the help I have received here is good

ESQ for young people aged 12–18

Response options: Certainly TruePartly TrueNot TrueDon’t know

(“don’t know” response option is not scored).

  1. 1.

    I feel that the people who saw me listened to me

  2. 2.

    It was easy to talk to the people who saw me

  3. 3.

    I was treated well by the people who saw me

  4. 4.

    My views and worries were taken seriously

  5. 5.

    I feel the people here know how to help me

  6. 6.

    I have been given enough explanation about the help available here

  7. 7.

    I feel that the people who have seen me are working together to help me

  8. 8.

    The facilities here are comfortable (e.g. waiting area)

  9. 9.

    My appointments are usually at a convenient time (e.g. don’t interfere with school, clubs, college, work)

  10. 10.

    It is quite easy to get to the place where I have my appointments

  11. 11.

    If a friend needed this sort of help, I would suggest to them to come here

  12. 12.

    Overall, the help I have received here is good

ESQ for children aged 9–11

Response options vary and are given with each question (“don’t know” response option is not scored).

  1. 1.

    Did the people who saw you listen to you? (Yes-Only a little-Not really-Don’t Know)

  2. 2.

    Was it easy to talk to the people who saw you? (Yes-Only a little-Not really-Don’t Know)

  3. 3.

    How were you treated by the people who saw you? (Very well-Ok-Not very well-Don’t Know)

  4. 4.

    Were your views and worries taken seriously? (Yes-Only a little-Not really-Don’t Know)

  5. 5.

    Do you feel that the people here know how to help you? (Yes-A little-Not really-Don’t Know)

  6. 6.

    Were you given enough explanation about the help available here? (Yes-Only a little-Not really-Don’t Know)

  7. 7.

    Do you feel that the people here are working together to help you? (Yes-Only a little-Not really-Don’t Know)

  8. 8.

    The facilities here (like the waiting area) are… (Comfortable-Ok-Uncomfortable-Don’t Know)

  9. 9.

    The time of my appointments was… (Convenient-Ok-Not convenient-Don’t Know)

  10. 10.

    The place where I had my appointments was… (Easy to get to-Ok to get to-Hard to get to-Don’t Know)

  11. 11.

    If a friend needed this sort of help, do you think they should come here? (Yes-Maybe-Not really-Don’t Know)

  12. 12.

    Has the help you got here been good? (Yes-Only a little-Not really-Don’t Know)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Brown, A., Ford, T., Deighton, J. et al. Satisfaction in Child and Adolescent Mental Health Services: Translating Users’ Feedback into Measurement. Adm Policy Ment Health 41, 434–446 (2014). https://doi.org/10.1007/s10488-012-0433-9

Download citation

Keywords

  • Patient satisfaction
  • PREM
  • Halo effects
  • Affective overtones
  • Approximated IRT scores