Reproducibility of physiological track-and-trigger warning systems for identifying at-risk patients on the ward
Physiological track-and-trigger warning systems are used to identify patients on acute wards at risk of deterioration, as early as possible. The objective of this study was to assess the inter-rater and intra-rater reliability of the physiological measurements, aggregate scores and triggering events of three such systems.
Prospective cohort study.
General medical and surgical wards in one non-university acute hospital.
Patients and participants
Unselected ward patients: 114 patients in the inter-rater study and 45 patients in the intra-rater study were examined by four raters.
Measurements and results
Physiological observations obtained at the bedside were evaluated using three systems: the medical emergency team call-out criteria (MET); the modified early warning score (MEWS); and the assessment score of sick-patient identification and step-up in treatment (ASSIST). Inter-rater and intra-rater reliability were assessed by intra-class correlation coefficients, kappa statistics and percentage agreement. There was fair to moderate agreement on most physiological parameters, and fair agreement on the scores, but better levels of agreement on triggers. Reliability was partially a function of simplicity: MET achieved a higher percentage of agreement than ASSIST, and ASSIST higher than MEWS. Intra-rater reliability was better then inter-rater reliability. Using corrected calculations improved the level of inter-rater agreement but not intra-rater agreement.
There was significant variation in the reproducibility of different track-and-trigger warning systems. The systems examined showed better levels of agreement on triggers than on aggregate scores. Simpler systems had better reliability. Inter-rater agreement might improve by using electronic calculations of scores.
KeywordsObserver variation Reproducibility of results Critical illness Scoring systems
This study was funded by the UK National Health Service Research and Development Service Delivery and Organisation Programme (SDO/74/2004). The authors thank S. Ameeth, S. Collins, K. Ghosh, C. Rincon and J. Tobler for their help in preparing the study, obtaining consent from patients and collecting the data. We thank A. Pawley for entering data into electronic format and L. Gemmell for advising on the format and facilitating the setup of the study.
- 1.Department of Health and NHS Modernisation Agency (2003) The National Outreach Report. Department of Health, LondonGoogle Scholar
- 3.Subbe CP, Kruger M, Rutherford P, Gemmell L (2001) Patients at risk: validation of a modified Early Warning Score in medical admissions. Q J Med 94:521–526Google Scholar
- 6.Stenhouse C, Coates S, Tivey M, Allsop P, Parker T (2000) Prospective evaluation of a Modified Early Warning Score to aid earlier detection of patients developing critical illness on a general surgical ward. Br J Anaesth 84:663PGoogle Scholar
- 7.Subbe CP, Hibbs R, Williams E, Rutherford P, Gemmel L (2002) ASSIST: a screening tool for critically ill patients on general medical wards. Intensive Care Med 28:S21Google Scholar
- 8.Harrison D (2004) KAPUTIL: Stata module to generate confidence intervals and sample size calculations for the kappa-statistic. Statistical Software Components S446501, Boston College Department of Economics. (Available at http://ideas.repec.org/c/boc/bocode/s446501.html)
- 9.Armitage P, Berry G, Matthews JNS (2002) Statistical methods in medical research. Blackwell, Oxford, pp 704–707Google Scholar
- 11.Altman D (1991) Practical statistics for medical research. Chapman and Hall, London, pp 401–409Google Scholar