Abstract
The present research addressed gaps in our current understanding of validity and quality of measurement provided by patient reported experience measures. We established the psychometric properties of a freely available experience of service questionnaire (ESQ), based on responses from 7,067 families of patients across 41 UK providers of child and adolescent mental health services, using the two-level latent trait modeling. Responses to the ESQ were subject to strong ‘halo’ effects, which were thought to represent the overall positive or negative affect towards one’s treatment. Two strongly related constructs measured by the ESQ were interpreted as specific aspects of global satisfaction, namely satisfaction with care, and with environment. The Care construct was sensitive to differences between less satisfied patients, facilitating individual and service-level problem evaluation. The effects of nesting within service providers were strong, with parental reports being the most reliable source of data for the between-provider comparisons. We provide a scoring protocol for converting the hand-scored ESQ to the model-based population-referenced scores with supplied standard errors, which can be used for benchmarking services as well as individual evaluations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Due to the small sample size at the provider level (N = 41), change in Chi-square based indices is not significant between models with different number of factors at the provider level. Residuals for correlations and SRMR are better indicators of relative model fit in this case.
Most EFA rotation methods would minimize the number of factors each item loads on—i.e. rotate to an independent clusters structure.
References
Anderson, J. A., Rivera, V. R., & Kutash, K. (1998). Measuring consumer satisfaction with children’s mental health. In M. H. Epstein, K. Kutash, & A. J. Duchnowski (Eds.), Outcomes for children with emotional and behavioral disorders and their families: Program and evaluation best practices (pp. 455–482). Austin: Pro-Ed.
Attkisson, C., & Greenfield, T. (1996). The client satisfaction questionnaire (CSQ) scales and the service satisfaction scale-30 (SSS-30). In L. Sederer & B. Dickey (Eds.), Outcomes assessment in clinical practice. Baltimore: Williams & Wilkins.
Attride-Stirling, J. (2002). Development of methods to capture users’ views of CAMHS in clinical governance reviews. http://www.corc.uk.net/resources/downloads/.
Ayton, A., Mooney, M., Sillifant, K., Powls, J., & Rasool, H. (2007). The development of the child and adolescent versions of the verona service satisfaction scale (CAMHSSS). Social Psychiatry and Psychiatric Epidemiology, 42, 892–901. doi:10.1007/s00127-007-0241-9.
Barber, A. J., Tischler, V. A., & Healy, E. (2006). Consumer satisfaction and child behaviour problems in child and adolescent mental health services. Journal of Child Health Care, 10(1), 9–21. doi:10.1177/1367493506060200.
Barnes, D., Devanney, C., Uglebjerg, A., Wistow, R. & Hartley, C. (2010). A profile of children’s health services, child and adolescent mental health services and maternity services in England 2008/9. Durham: Durham University. http://dro.dur.ac.uk/8220/1/8220.pdf.
Biering, P., & Jensen, V. (2011). The concept of patient satisfaction in adolescent psychiatric care: A qualitative study. Journal of Child and Adolescent Psychiatric Nursing, 24, 3–10. doi:10.1111/j.1744-6171.2010.00261.x.
Brannan, A., Sonnichesen, E., & Heflinger, C. (1996). Measuring satisfaction with children’s mental health services: Validity and reliability of the satisfaction scales. Evaluation and Program Planning, 19, 131–141. doi:10.1016/0149-7189(96)00004-3.
Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36, 111–150. doi:10.1207/S15327906MBR3601_05.
Day, C., Michelson, D., & Hassan, I. (2011). Child and adolescent service experience (ChASE): Measuring service quality and therapeutic process. British Journal of Clinical Psychology, 50, 452–464. doi:10.1111/j.2044-8260.2011.02008.x.
Department of Health. (2009). Understanding what matters: A guide to using patient feedback to transform services. London: Department of Health. Retrieved from http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_099780.
Edwards, A., & Elwyn, G. (2001). Developing professional ability to involve patients in their care: Pull or push? Quality in Health Care, 10, 129–134. doi:10.1136/qhc.0100129.
Garland, A., Saltzman, M., & Aarons, G. (2000). Adolescent satisfaction with mental health services: Development of a multidimensional scale. Evaluation and Program Planning, 23, 165–175. doi:10.1016/S0149-7189(00)00009-4.
Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire (SDQ). Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337–1345. doi:10.1097/00004583-200111000-00015.
Hart, A., Saunders, A., & Thomas, H. (2005). Attuned practice: a service user study of specialist child and adolescent mental health, UK. Epidemiology and Psychiatric Sciences, 14, 22–31. doi:10.1017/S1121189X00001895.
Holbrook, M. B. (1983). Using a structural model of halo effect to assess perceptual distortion due to affective overtones. Journal of Consumer Research, 10, 247–252. http://www.jstor.org/stable/2488928.
Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York: Routledge.
Kazdin, A., & Mazurick, J. (1994). Dropping out of child psychotherapy: Distinguishing early and late dropouts over the course of treatment. Journal of Consulting and Clinical Psychology, 62, 1069–1074. doi:10.1177/1359104596011012.
Kennedy, I. (2010). Getting it right for children and young people: Overcoming cultural barriers in the NHS so as to meet their needs. London: Department of Health. http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_119445.
McNaughton, D. (1994). Measuring parent satisfaction with early childhood intervention programs. Topics in Early Childhood Special Education, 14, 26–48. doi:10.1177/027112149401400106.
Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78, 218–225. doi:10.1037/0021-9010.78.2.218.
Muthén, L. K. & Muthén, B. O. (1998-2010). Mplus user’s guide. 6th Ed. Los Angeles, CA: Muthén & Muthén. www.stamodel.com.
Muthén, B. & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. http://www.jstor.org/stable/271070.
Rabe-Hesketh, S., Skrondal, A., & Pickles, A. (2004). Generalized multilevel structural equation modeling. Psychometrika, 69, 167–190. doi:10.1007/BF02295939.
Reise, S., & Haviland, M. (2005). Item response theory and the measurement of clinical change. Journal of Personality Assessment, 84, 228–238. doi:10.1207/s15327752jpa8403_02.
Reise, S., & Waller, G. (2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 25–46. doi:10.1146/annurev.clinpsy.032408.153553.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometric Monograph No. 17). Richmond, VA: Psychometric Society.
SDQ: Normative SDQ data from Britain. (2001). http://www.sdqinfo.com/UKNorm.html.
Shapiro, J., Welker, C., & Jacobson, B. (1997). The youth client satisfaction questionnaire: Development, construct validation and factor structure. Journal of Child Clinical Psychology, 26, 87–98. doi:10.1207/s15374424jccp2601_9.
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120. doi:10.1007/s11336-008-9101-0.
Stallard, P. (1995). Parental satisfaction with intervention: differences between respondents and non-respondents to postal questionnaire. British Journal of Clinical Psychology, 34, 397–405. doi:10.1111/j.2044-8260.1995.tb01474.x.
Thissen, D., & Orlando, M. (2001). Item response theory for items scored in two categories. In D. Thissen & H. Wainer (Eds.), Test Scoring (pp. 73–140). Mahwah, NJ: Lawrence Erlbaum.
Williams, B., Coyle, J., & Healy, D. (1998). The meaning of patient satisfaction: An explanation of high reported levels. Social Science and Medicine, 47, 1351–1359. doi:10.1016/S0277-9536(98)00213-5.
CORC Measures Website http://www.corc.uk.net/index.php?contentkey=81.
Young, S., Nicholson, J., & Davis, M. (1995). An overview of issues in research on consumer satisfaction with child and adolescent mental health services. Journal of Child and Family Studies, 4, 219–238. doi:10.1007/BF02234097.
Acknowledgments
We are grateful to all members of the CORC collaboration for providing the data, and to the CORC central team researchers Jenna Bradley and Halina Flannery for preparing the data for analyses. Anna Brown’s work on this article was supported by Grant RG63087 from the Isaac Newton Trust, University of Cambridge. Jessica Deighton’s work on this article was provided as part of the Department of Health Child Policy Research Unit (Reference number: 109/0001).
Conflict of interest
The authors have no competing interests. CORC is not-for-profit Company limited by guarantee. Tamsin Ford is an unpaid director of CORC (other than travel expenses). Miranda Wolpert is a paid director of CORC (3 days a week).
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
ESQ for parents/carers
Response options: Certainly True—Partly True—Not True—Don’t know
(“don’t know” response option is not scored).
-
1.
I feel that the people who have seen my child listened to me
-
2.
It was easy to talk to the people who have seen my child
-
3.
I was treated well by the people who have seen my child
-
4.
My views and worries were taken seriously
-
5.
I feel the people here know how to help with the problem I came for
-
6.
I have been given enough explanation about the help available here
-
7.
I feel that the people who have seen my child are working together to help with the problem(s)
-
8.
The facilities here are comfortable (e.g. waiting area)
-
9.
The appointments are usually at a convenient time (e.g. don’t interfere with work, school)
-
10.
It is quite easy to get to the place where the appointments are
-
11.
If a friend needed similar help, I would recommend that he or she come here
-
12.
Overall, the help I have received here is good
ESQ for young people aged 12–18
Response options: Certainly True—Partly True—Not True—Don’t know
(“don’t know” response option is not scored).
-
1.
I feel that the people who saw me listened to me
-
2.
It was easy to talk to the people who saw me
-
3.
I was treated well by the people who saw me
-
4.
My views and worries were taken seriously
-
5.
I feel the people here know how to help me
-
6.
I have been given enough explanation about the help available here
-
7.
I feel that the people who have seen me are working together to help me
-
8.
The facilities here are comfortable (e.g. waiting area)
-
9.
My appointments are usually at a convenient time (e.g. don’t interfere with school, clubs, college, work)
-
10.
It is quite easy to get to the place where I have my appointments
-
11.
If a friend needed this sort of help, I would suggest to them to come here
-
12.
Overall, the help I have received here is good
ESQ for children aged 9–11
Response options vary and are given with each question (“don’t know” response option is not scored).
-
1.
Did the people who saw you listen to you? (Yes-Only a little-Not really-Don’t Know)
-
2.
Was it easy to talk to the people who saw you? (Yes-Only a little-Not really-Don’t Know)
-
3.
How were you treated by the people who saw you? (Very well-Ok-Not very well-Don’t Know)
-
4.
Were your views and worries taken seriously? (Yes-Only a little-Not really-Don’t Know)
-
5.
Do you feel that the people here know how to help you? (Yes-A little-Not really-Don’t Know)
-
6.
Were you given enough explanation about the help available here? (Yes-Only a little-Not really-Don’t Know)
-
7.
Do you feel that the people here are working together to help you? (Yes-Only a little-Not really-Don’t Know)
-
8.
The facilities here (like the waiting area) are… (Comfortable-Ok-Uncomfortable-Don’t Know)
-
9.
The time of my appointments was… (Convenient-Ok-Not convenient-Don’t Know)
-
10.
The place where I had my appointments was… (Easy to get to-Ok to get to-Hard to get to-Don’t Know)
-
11.
If a friend needed this sort of help, do you think they should come here? (Yes-Maybe-Not really-Don’t Know)
-
12.
Has the help you got here been good? (Yes-Only a little-Not really-Don’t Know)
Rights and permissions
About this article
Cite this article
Brown, A., Ford, T., Deighton, J. et al. Satisfaction in Child and Adolescent Mental Health Services: Translating Users’ Feedback into Measurement. Adm Policy Ment Health 41, 434–446 (2014). https://doi.org/10.1007/s10488-012-0433-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-012-0433-9