Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Developing and comparing methods for measuring the content of care in mental health services



Measurement of what happens in mental health services is needed to describe services, identify variation in care provision and understand service outcomes. However, there is no consensus about appropriate methods or measures for content of care. Previous research has primarily used a single information source and prioritised staff over patient perspectives on content of care. This study aims to enhance understanding of how to measure content of care by developing and evaluating four instruments, each using a different measurement method.


Development is described by four instruments—CaSPAR, CaRICE, CCCQ-S and CCCQ-P—which use researcher observation, staff-report and patient-report to measure the intensity and nature of care at services. Inter-rater reliability of CaRICE and CCCQ-S was investigated. Concordance between staff and patient perspectives was explored through assessing inter-rater agreement of CCCQ-S and CCCQ-P questionnaires. The convergence of data from the measures collected in an inpatient multi-site study was investigated.


CaRICE demonstrated good inter-rater reliability (κ = 0.71). CCCQ-S inter-rater reliability was poor. Concordance between staff and patient reports was low: there was a trend for patients to report less care received than staff reported had been provided. Results from CaSPAR, CaRICE and CCCQ-P exhibited divergence, indicating possible differences in patient, staff and researcher perspectives.


Information about content of care should be sought as close as possible to the point of delivery. There may be differing, valid perspectives about care provided by services. Further development of methods to measure content of care is required. Meanwhile, a multi-methods approach should be adopted, which allows inclusion of different perspectives (specifically including the patient’s perspective) and triangulation of results from different measurement methods. CaSPAR, CaRICE and CCCQ-P can provide multi-perspective content of care measurement in inpatient services.

This is a preview of subscription content, log in to check access.


  1. 1.

    Altman D (1991) Practical statistics for medical research. Chapman and Hall, London

  2. 2.

    Altschul A (1972) Patient–nurse interaction: a study of interaction patterns in acute psychiatric wards. Churchill Livingstone, Edinburgh

  3. 3.

    Bowie P, Mountain G (1993) Using direct observation to record the behaviour of long-stay patients with dementia. Int J Geriatr Psychiatry 8:857–864

  4. 4.

    Brekke J (1987) The model-guided method for monitoring program implementation. Eval Rev 11(3):281–299

  5. 5.

    Brugha TS, Lindsay F (1996) Quality of mental health service care: the forgotten pathway from process to outcome. Soc Psychiatry Psychiatr Epidemiol 31(2):89–98

  6. 6.

    Burns T, Priebe S (1996) Mental health care systems and their characteristics: a proposal. Acta Psychiatr Scand 94(6):381–385

  7. 7.

    Cormack D (1976) Psychiatric nursing observed. Royal College of Nursing, London

  8. 8.

    Craig P, Dieppe P, Macintyre S, Mitchie S, Nazareth I, Petticrew M (2008) Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 337:979–983

  9. 9.

    Dean R, Proudfoot R (1993) The quality of interactions schedule (QUIS): development, reliability and use in the evaluation of two domus units. Int J Geriatr Psychiatry 8:819–826

  10. 10.

    Department of Health (2005) Mental Health Policy Implementation Guide: Adult Acute Inpatient Care Provision. Department of Health

  11. 11.

    Donabedian A (1980) Explorations in quality assessment and monitoring. The definition of quality and approaches to its assessment, vol 1. Health Administration Press

  12. 12.

    Fiander M, Burns T (2000) A Delphi approach to describing service models of community mental health practice. Psychiatr Serv 51(5):656–658

  13. 13.

    Glick I, Burti L, Suzuki K, Sacks M (1991) Effectiveness in psychiatric care: 1. A cross-national study of the process of treatment and outcomes of major depressive disorder. J Nerv Mental Dis 179(2):55–63

  14. 14.

    Higgins R, Hurst K, Wistow G (1999) Nursing acute psychiatric patients: a quantitative and qualitative study. J Adv Nurs 29(1):52–63

  15. 15.

    Johnson S, Salvador-Carulla L (1998) Description and classification of mental health services: a European perspective. Eur Psychiatry 13:333–341

  16. 16.

    Johnson S, Billings J, Bingham C, Pilling S, Bebbington P, Morant N, Dalton J, McNicholas S (2001) An evaluation of Drayton Park Women’s Crisis Service, a community-based alternative to the psychiatric ward. Unpublished report

  17. 17.

    Johnson S, Gilburt H, Lloyd-Evans B, Slade M (2007) Acute inpatient psychiatry: residential alternatives to hospital admission. Psychiatr Bull 31:262–264

  18. 18.

    Johnson S, Gilburt H, Lloyd-Evans B, Boardman H, Leese M, Osborn D, Shepherd G, Thornicroft G, Slade M (2009) Alternatives to hospital admission: emerging models and populations served. Br J Psychiatry 194(5):456–463

  19. 19.

    Lloyd-Evans B, Slade M, Johnson S (2007) Assessing the content of mental health services: a review of measures. Soc Psychiatry Psychiatr Epidemiol 42(8):673–682

  20. 20.

    McCabe R, Roeder-Wanner U, Hoffman K, Priebe S (1999) Therapeutic relationships and quality of life: associations of two subjective constructs in schizophrenia patients. Int J Soc Psychiatry 45:276–283

  21. 21.

    Mechanic D (1996) Emerging issues in international mental health services research. Psychiatr Serv 47(4):371–375

  22. 22.

    Moos R (1996) Ward atmosphere scale manual, 3rd edn. Mind Garden

  23. 23.

    Paul G (1987) Rational operations in residential treatment settings through ongoing assessment of staff and client functioning. In: Peterson D, Fishman D (eds) Assessment for Decision. Rutgers University Press, pp 145–203

  24. 24.

    Quirk A, Lelliot P (2001) What do we know about life on acute psychiatric wards in the UK? A review of the research evidence. Soc Sci Med 53(12):1565–1574

  25. 25.

    Sandford D, Elzinger R, Iverson R (1990) A quantitative study of nursing staff interactions in psychiatric wards. Acta Psychiatr Scand 81:46–51

  26. 26.

    Sanson-Fisher R, Poole A, Thompson V (1979) Behavioural patterns within a general hospital psychiatric unit: an observational study. Behav Res Ther 17:317–332

  27. 27.

    Shepherd G, Richardson R (1979) Organization and interaction in psychiatric day centres. Psychol Med 9:573–579

  28. 28.

    Slade M, Phelan M, Thornicroft G (1996) The Camberwell Assessment of Need (CAN): comparison of assessments by staff and patients of the needs of the mentally ill. Soc Psychiatry Psychiatr Epidemiol 31:109–113

  29. 29.

    Tugwell P (1979) A methodological perspective on process measures of the quality of medical care. Clini Investig Med 2:113–121

  30. 30.

    Tyson G, Lambert G, Beattie L (1995) The quality of psychiatric nurses’ interactions with patients: an observational study. Int J Nurs Stud 32(1):49–58

Download references


We are very grateful to the people using, working in and managing the participating services who gave their time generously, and for advice and input from the steering group for The Alternatives Study. This report presents independent research commissioned by the National Institute for Health Research Service Delivery and Organisation R&D Programme. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The authors have no competing interests to declare.

Author information

Correspondence to B. Lloyd-Evans.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOC 36 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Lloyd-Evans, B., Slade, M., Osborn, D.P. et al. Developing and comparing methods for measuring the content of care in mental health services. Soc Psychiatry Psychiatr Epidemiol 46, 219–229 (2011). https://doi.org/10.1007/s00127-010-0192-4

Download citation


  • Process measurement
  • Content of care
  • Inpatient
  • Mental health services