Pharmacy World and Science

, Volume 27, Issue 4, pp 311–315 | Cite as

Inter-rater Reliability of Explicit Indicators of Prescribing Appropriateness

  • Mary P. TullyEmail author
  • Judith A. Cantrill
Research Article


Objective: To assess the inter-rater reliability of 14 explicit indicators of appropriate long-term prescribing.

Method: All available data required for the assessment of 59 long-term prescriptions started during a hospital admission for 25 patients were transcribed from the patients’ medical records. These transcripts were presented in a standardised format and random order to four raters (two doctors and two pharmacists) who used the indicators to judge the appropriateness of each prescription. Debriefing interviews were held with each rater. An a priori level of acceptable agreement between the raters was set at a weighed kappa of 0.70.

Results: There was no apparent difference between pharmacists and doctors for all findings, so data were combined. Two indicators showed poor agreement, three showed moderate agreement, and nine showed substantial or near perfect agreement, exceeding a weighted kappa of 0.70. There was excellent positive agreement as to which prescriptions were judged appropriate by the indicators, but much worse negative agreement as to which prescriptions were judged to be inappropriate. In the interviews, the raters remarked on the difficulty of applying explicit indicators when they routinely made implicit judgements about data in the medical records.

Conclusion: Nine of the indicators achieved the required level of reliability and the negative agreement levels showed that this was the area that required greater improvement in future developments of the indicators. Further work needs to be conducted to investigate ways of the improving instructions on how to make explicit judgements and reducing the need for implicit or subjective assessments.

Key words

Appropriate prescribing Hospital Indicators In-patient prescribing Prescribing indicators Reliability United Kingdom 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Oborne, CA, Hooper, R, Li, KC, Swift, CG, Jackson, SHD. 2002An indicator of appropriate neuroleptic prescribing in nursing homesAge Ageing314359CrossRefPubMedGoogle Scholar
  2. 2.
    Naish, J, Sturdy, P, Toon, P. 1995Appropriate prescribing in asthma and its related cost in east LondonBMJ31097100PubMedGoogle Scholar
  3. 3.
    Beers, MH, Ouslander, JG, Rollingher, I, Reuben, DB, Brooks, J, Beck, JC. 1991Explicit criteria for determining inappropriate medication use in nursing home residentsArch Intern Med151182532CrossRefPubMedGoogle Scholar
  4. 4.
    Thompson, DF, Heflin, NR. 1987Frequency and appropriateness of drug prescribing for unlabeled uses in pediatric patientsAm J Hosp Pharm46794Google Scholar
  5. 5.
    Medical Economics. Physician’s Desk Reference. 58th ed. Montvale, NJ: Medical Economics Co Inc., 2004.Google Scholar
  6. 6.
    Joint Formulary Committee. British National Formulary. British Medical Association and Royal Pharmaceutical Society of Great Britain; 2003.Google Scholar
  7. 7.
    Hanlon, JT, Schmader, KE, Samsa, GP, Weinberger, M, Uttech, KM, Lewis, IK,  et al. 1992A method for assessing drug therapy appropriatenessJ Clin Epidemiol45104551CrossRefPubMedGoogle Scholar
  8. 8.
    Cantrill, JA, Sibbald, B, Buetow, S. 1998Indicators of the appropriateness of long term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility and reliabilityQual Health Care71305PubMedGoogle Scholar
  9. 9.
    Fitzgerald, L, Hanlon, JT, Shelton, P, Landsman, PB, Schmader, KE, Pulliam, C,  et al. 1997Reliability of a modified medication appropriateness index in ambulatory older personsAnn Pharmacother315438PubMedGoogle Scholar
  10. 10.
    Kassam, R, Martin, L, Farris, KB. 2003Reliability of a modified medication appropriateness index in community pharmaciesAnn Pharmacother37406CrossRefPubMedGoogle Scholar
  11. 11.
    Samsa, GP, Hanlon, JT, Schmader, KE, Weinberger, M, Clipp, EC, Uttech, KM,  et al. 1994A summated score for the medication appropriateness index: development and assessment of clinimetric properties including content validityJ Clin Epidemiol478916CrossRefPubMedGoogle Scholar
  12. 12.
    Tully, MP, Cantrill, JA. 2002Development of explicit indicators of appropriateness of prescribing (abstract)Pharm World Sci2424CrossRefGoogle Scholar
  13. 13.
    Javed, N, Tully, MP. 2002Content validity of indicators of the appropriateness of long-term prescribing started during a hospital admissionInt J Pharm Pract10(suppl)R55Google Scholar
  14. 14.
    Tully, MP, Cantrill, JA. 2003The validity of explicit indicators of appropriateness of long term prescribingInt J Pharm Pract11(suppl)R49Google Scholar
  15. 15.
    Taylor, S, Bogdan, B. 1998Introduction to Qualitative Research Methods A Guide Book and Resource3John Wiley & SonsNew YorkGoogle Scholar
  16. 16.
    Dunn, G. 1992Design and analysis of reliability studiesStat Meth Med Res112357Google Scholar
  17. 17.
    Cicchetti, DV, Feinstein, AR. 1990High agreement but low kappa: II. Resolving the paradoxesJ Clin Epidemiol435518CrossRefPubMedGoogle Scholar
  18. 18.
    Landis, JR, Koch, GG. 1977The measurement of observer agreement for categorical dataBiometrics3315974PubMedGoogle Scholar
  19. 19.
    Shelton, P, Hanlon, JT, Landsman, PB, Scott, M, Lewis, IK, Schmader, KE,  et al. 1997Reliability of drug utilisation evaluation as an assessment of medication appropriatenessAnn Pharmacother3153342PubMedGoogle Scholar
  20. 20.
    Oborne, CA, Batty, GM, Maskrey, V, Swift, CG, Jackson, SHD. 1997Development of prescribing indicators for elderly medical inpatientsBr J Clin Pharmacol43917PubMedGoogle Scholar
  21. 21.
    Berg, M. 1996Practices of reading and writing: the constitutive role of the patient record in medical workSoc Health Illness18499524CrossRefGoogle Scholar
  22. 22.
    Berg, M. 1998Medical Work and the computer-based patient record: a sociological perspectiveMeth Inf Med37294301Google Scholar
  23. 23.
    Buetow, SA, Sibbald, B, Cantrill, JA, Halliwell, S. 1997Appropriateness in health care: application to prescribingSoc Sci Med4526171CrossRefPubMedGoogle Scholar
  24. 24.
    Collins, D. 2003Pretesting survey instruments: an overview of cognitive methodsQual Life Res1222938CrossRefPubMedGoogle Scholar

Copyright information

© Springer 2005

Authors and Affiliations

  1. 1.School of Pharmacy and Pharmaceutical SciencesUniversity of ManchesterManchesterUK

Personalised recommendations