Skip to main content
SpringerLink
Log in

Reliability of Untrained and Experienced Raters on FEES: Rating Overall Residue is a Simple Task

  • Original Article
  • Published:
Dysphagia Aims and scope Submit manuscript

Abstract

The purpose of this study was to investigate the reliability of residue ratings on Fiberoptic Endoscopic Evaluation of Swallowing (FEES). We also examined rating differences based on experience to determine if years of experience influenced residue ratings. A group of 44 raters watched 81 FEES videos representing a wide range of residue severities for thin liquid, applesauce, and cracker boluses. Raters were untrained on the rating scales and simply rated their overall impression of residue amount on a visual analog scale (VAS) and a five-point ordinal scale in a randomized fashion across two sessions. Intra-class correlation coefficients, kappa coefficients, and ANOVAs were used to analyze agreement and differences in ratings. Residue ratings on both the VAS and ordinal scales had acceptable inter- and intra-rater reliability. Inter-rater agreement was acceptable (ICC > 0.7) for all comparisons. Intra-rater agreement was excellent on the VAS scale (rc = 0.9) and good on the ordinal scale (k = 0.78). There was no significant difference between expert ratings and other raters based on years of experience for cracker ratings (p = 0.2119) and applesauce ratings (p = 0.2899), but there was a significant difference between clinicians on thin liquid ratings (p = 0.0005). Without any specific training, raters demonstrated high reliability when rating the overall amount of residue on FEES. Years of experience with FEES did not influence residue ratings, suggesting that expert ratings of overall residue amount are not unique or specialized. Rating the overall amount of residue on FEES appears to be a simple visual-perceptual task for puree and cracker boluses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Altman DG. Practical statistics for medical research. 1st ed. London: Chapman and Hall; 1991.

    Google Scholar 

  2. Bernieri F. Coordinated movement and rapport in teacher-student interactions. J Nonverbal Behav. 1988;12:120–38.

    Article  Google Scholar 

  3. Bernieri F, Resnick J, Rosenthal R. Synchrony, pseudo-synchrony, and dissynchrony: measuring the entrainments process in mother-infant dyads. J Pers Soc Psychol. 1988;54:243–53.

    Article  Google Scholar 

  4. Brunier G, Graydon J. A comparison of two methods of measuring fatigue in patients on chronic haemodialysis: visual analogue versus Likert scale. Int J Nurs Stud. 1996;33(3):338–48.

    Article  PubMed  CAS  Google Scholar 

  5. Donzelli J, Brady S, Wesling M, Craney M. Predictive value of accumulated oropharyngeal secretions for aspiration during video nasal endoscopic evaluation of the swallow. Ann Otol Rhinol Laryngol. 2003;112(5):469–75. https://doi.org/10.1177/000348940311200515.

    Article  PubMed  Google Scholar 

  6. Farneti D. Pooling score: an endoscopic model for evaluating severity of dysphagia. Acta Otorhinolaryngol Ital. 2008;28(3):135–40.

    PubMed  PubMed Central  CAS  Google Scholar 

  7. Farneti D, Fattori B, Nacci A, Mancini V, Simonelli M, Ruoppolo G, Genovese E. The pooling-score (P-score): inter- and intra-rater reliability in endoscopic assessment of the severity of dysphagia. Acta Otorhinolaryngol Ital. 2014;34(2):105–10.

    PubMed  PubMed Central  CAS  Google Scholar 

  8. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problem of two paradoxes. J Clin Epidemiol. 1990;43:543–9.

    Article  PubMed  CAS  Google Scholar 

  9. Hildebrand DK, Laing JD, Rosenthal R. Analysis of ordinal data: quantitative applications in the social sciences. Newbury Park: SAGE Publications Inc; 1977.

    Book  Google Scholar 

  10. Holmes S, Dickerson JW. Malignant disease: nutritional implications of disease and treatment. Cancer Metastasis Rev. 1987;6(3):357–81.

    Article  PubMed  CAS  Google Scholar 

  11. Hutcheson KA, Barrow MP, Barringer DA, Knott JK, Lin HY, Weber RS, Lewin JS. Dynamic imaging grade of swallowing toxicity (DIGEST): scale development and validation. Cancer. 2017;123(1):62–70. https://doi.org/10.1002/cncr.30283.

    Article  PubMed  Google Scholar 

  12. Kaneoka AS, Langmore SE, Krisciunas GP, Field K, Scheel R, McNally E, Cabral H. The boston residue and clearance scale: preliminary reliability and validity testing. Folia Phoniatr Logop. 2013;65(6):312–7. https://doi.org/10.1159/000365006.

    Article  PubMed  Google Scholar 

  13. Kelly AM, Leslie P, Beale T, Payten C, Drinnan MJ. Fibreoptic endoscopic evaluation of swallowing and videofluoroscopy: does examination type influence perception of pharyngeal residue severity? Clin Otolaryngol. 2006;31(5):425–32. https://doi.org/10.1111/j.1749-4486.2006.01292.x.

    Article  PubMed  CAS  Google Scholar 

  14. Kelly AM, Macfarlane K, Ghufoor K, Drinnan MJ, Lew-Gor S. Pharyngeal residue across the lifespan: a first look at what’s normal. Clin Otolaryngol. 2008;33(4):348–51. https://doi.org/10.1111/j.1749-4486.2008.01755.x.

    Article  PubMed  CAS  Google Scholar 

  15. Langmore SE, Schatz K, Olsen N. Fiberoptic endoscopic examination of swallowing safety: a new procedure. Dysphagia. 1988;2(4):216–9.

    Article  PubMed  CAS  Google Scholar 

  16. Lin LI. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45(1):255–68.

    Article  PubMed  CAS  Google Scholar 

  17. Lyons KD, Tickle-Degnen L. Reliability and validity of a videotape method to describe expressive behavior in persons with Parkinson’s disease. Am J Occup Ther. 2005;59(1):41–9.

    Article  PubMed  Google Scholar 

  18. Martin-Harris B, Brodsky MB, Michel Y, Castell DO, Schleicher M, Sandidge J, Blair J. MBS measurement tool for swallow impairment–MBSImp: establishing a standard. Dysphagia. 2008;23(4):392–405. https://doi.org/10.1007/s00455-008-9185-9.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Marvin S, Gustafson S, Thibeault S. Detecting aspiration and penetration using FEES With and without food dye. Dysphagia. 2016;31(4):498–504. https://doi.org/10.1007/s00455-016-9703-0.

    Article  PubMed  Google Scholar 

  20. McBride G. A proposal for strength-of-agreement criteria for Lin’s concordance correlation coefficient. Hamilton, New Zealand: National Institute of Water and Atmospheric Research Ltd. 2005. http://www.medcalc.org/download/pdf/McBride2005.pdf.

  21. Neubauer PD, Hersey DP, Leder SB. Pharyngeal residue severity rating scales based on fiberoptic endoscopic evaluation of swallowing: a systematic review. Dysphagia. 2016;31(3):352–9. https://doi.org/10.1007/s00455-015-9682-6.

    Article  PubMed  Google Scholar 

  22. Neubauer PD, Rademaker AW, Leder SB. The yale pharyngeal residue severity rating scale: an anatomically defined and image-based tool. Dysphagia. 2015;30(5):521–8. https://doi.org/10.1007/s00455-015-9631-4.

    Article  PubMed  Google Scholar 

  23. Nienstedt JC, Muller F, Niessen A, Fleischer S, Koseki JC, Flugel T, Pflug C. Narrow band imaging enhances the detection rate of penetration and aspiration in FEES. Dysphagia. 2017;32(3):443–8. https://doi.org/10.1007/s00455-017-9784-4.

    Article  PubMed  Google Scholar 

  24. Niessen A, Nienstedt J, Pflug C. Methodic background of narrow band imaging (NBI) in Dysphagia diagnostic-proposing a high sensitivity FEES. Paper presented at the Dysphagia Research Society, Barcelona. 2017.

  25. Park JM, Yong SY, Kim JH, Jung HS, Chang SJ, Kim KY, Kim H. Cutoff value of pharyngeal residue in prognosis prediction after neuromuscular electrical stimulation therapy for Dysphagia in subacute stroke patients. Ann Rehabil Med. 2014;38(5):612–9. https://doi.org/10.5535/arm.2014.38.5.612.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Pfennings L, Cohen L, van der Ploeg H. Preconditions for sensitivity in measuring change: visual analogue scales compared to rating scales in a Likert format. Psychol Rep. 1995;77(2):475–80. https://doi.org/10.2466/pr0.1995.77.2.475.

    Article  PubMed  CAS  Google Scholar 

  27. Pilz W, Baijens LW, Passos VL, Verdonschot R, Wesseling F, Roodenburg N, Kremer B. Swallowing assessment in myotonic dystrophy type 1 using fiberoptic endoscopic evaluation of swallowing (FEES). Neuromuscul Disord. 2014;24(12):1054–62. https://doi.org/10.1016/j.nmd.2014.06.002.

    Article  PubMed  Google Scholar 

  28. Pisegna JM, Kaneoka A, Leonard R, Langmore SE. Rethinking residue: determining the perceptual continuum of residue on FEES to enable better measurement. Dysphagia. 2017. https://doi.org/10.1007/s00455-017-9838-7.

    Article  PubMed  Google Scholar 

  29. Pisegna JM, and Langmore S. Measuring residue: categorical ratings versus a visual analog scale. Paper presented at the Dysphagia Research Society Annual Convention, Chicago. 2015.

  30. Pisegna JM, and Langmore S. Clinician judgments of clearing swallows are reliable without training. Paper presented at the American Speech Language and Hearing Association, Los Angeles. 2017.

  31. Pisegna JM, Langmore SE. Parameters of instrumental swallowing evaluations: describing a diagnostic dilemma. Dysphagia. 2016;31(3):462–72. https://doi.org/10.1007/s00455-016-9700-3.

    Article  PubMed  Google Scholar 

  32. Portney LG, Watkins MP. Foundations of clinical research: applications to practice. 3rd ed. Upper Saddle River: Pearson/Prentice Hall; 2015.

    Google Scholar 

  33. Rommel N, Borgers C, Van Beckevoort D, Goeleven A, Dejaeger E, Omari TI. Bolus residue scale: an easy-to-use and reliable videofluoroscopic analysis tool to score bolus residue in patients with Dysphagia. Int J Otolaryngol. 2015;2015:780197. https://doi.org/10.1155/2015/780197.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Rosenthal R. Experimenter effects in behavioral research. New York: Appleton-Century-Crofts; 1966.

    Google Scholar 

  35. Rosenthal R. Conducting judgment studies: Some methodological issues. In: The new handbook of methods in nonverbal behavior research. Oxford University Press; 2005. p. 199–234.

  36. Tickle-Degnen L, Rosenthal R. The nature of rapport and its nonverbal correlates. Psychol Inq. 1990;1(4):285–93.

    Article  Google Scholar 

  37. Tinsley H, Weiss D. Interrater reliability and agreement of subjective judgments. J Couns Psychol. 1975;22:358–76.

    Article  Google Scholar 

  38. Tohara H, Nakane A, Murata S, Mikushi S, Ouchi Y, Wakasugi Y, Uematsu H. Inter- and intra-rater reliability in fibroptic endoscopic evaluation of swallowing. J Oral Rehabil. 2010;37(12):884–91. https://doi.org/10.1111/j.1365-2842.2010.02116.x.

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgements

The lead author would like to thank Dr. Elizabeth Hoover for her input and guidance in the development of this research. We are also grateful for the clinicians, raters, patients, and researchers who contributed to this work.

Author information

Authors and Affiliations

  1. Department of Otolaryngology, Boston University School of Medicine, 820 Harrison Ave., FGH Building, 4th Floor, Boston, MA, 02118, USA

    Jessica M. Pisegna, James C. Borders & Susan E. Langmore

  2. Sargent College of Health and Rehabilitation Sciences, Boston University, 635 Commonwealth Ave, Boston, MA, 02215, USA

    Jessica M. Pisegna & Susan E. Langmore

  3. Boston Medical Center, 830 Harrison Ave., Suite 1400, Boston, MA, 02118, USA

    Jessica M. Pisegna & James C. Borders

  4. The University of Tokyo Hospital Rehabilitation Center, Tokyo, Japan

    Asako Kaneoka

  5. Department of Occupational Therapy, Boston University College of Health and Rehabilitation Sciences: Sargent College, 635 Commonwealth Avenue, Boston, MA, 02215, USA

    Wendy J. Coster

  6. University of California at Davis, Davis, CA, 95616, USA

    Rebecca Leonard

Authors
  1. Jessica M. Pisegna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James C. Borders
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Asako Kaneoka
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wendy J. Coster
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rebecca Leonard
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Susan E. Langmore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jessica M. Pisegna.

Ethics declarations

Conflict of interest

Salary and tuition support was provided to the first and last authors from the National Institute on Deafness and Other Communication Disorders of the National Institutes of Health under Award Number R01DC012584 (PI: Kumar). The Department of Speech, Language, and Hearing of Sargent College (Boston University) also provided financial support for this research via the Dudley Allen Sargent Research Fund.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards (DEEMED EXEMPT BY IRB, as it did constitute research with human subjects, but ALSO meets the requirements of a defined low-risk category that is exempt from SOME (but not all) of the requirements governing human subjects research.).

Informed Consent

The regulatory requirement for a consent form does not apply to Exempt research.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pisegna, J.M., Borders, J.C., Kaneoka, A. et al. Reliability of Untrained and Experienced Raters on FEES: Rating Overall Residue is a Simple Task. Dysphagia 33, 645–654 (2018). https://doi.org/10.1007/s00455-018-9883-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00455-018-9883-x

Keywords

Search

Navigation