Advertisement

Advances in Health Sciences Education

, Volume 13, Issue 4, pp 397–406 | Cite as

Development of a content-valid standardized orthopedic assessment tool (SOAT)

  • Mark LafaveEmail author
  • Larry Katz
  • Dale Butterwick
Original Paper

Abstract

Content validation of an instrument that measures student performance in OSCE-type practical examinations is a critical step in a tool's overall validity and reliability [Hopkins (1998), Educational and Psychological Measurement and Evaluation (8th ed.). Toronto: Allyn & Bacon]. The purpose of the paper is to outline the process employed to create a content-valid Standardized Orthopedic Assessment Tool (SOAT). Orthopedic assessment skills are employed by athletic therapists, physiotherapists and physicians. All follow very similar diagnostic algorithms and that system was used to develop the original SOAT [Cyriax (1982). Textbook of Orthopaedic Medicine, (vol. 1) Bailliere Tindall]. To validate the tool, the study followed procedures outlined by Violato, Salami, and Muiznieks (2002), Journal of Manipulative Physiological Therapeutics, 25, 111–115, and Butterwick, Paskevich, Vallevand and Lafave (2006), Journal of Allied Health: a modified Ebel procedure. An expert group of athletic therapists from across Canada were chosen to create the content valid tool. Representation from all accredited athletic therapy programs in Canada was sought. Experts participated in three stages of content validation: Stage one consisted of individuals grading tasks on difficulty (hard, medium, easy) and importance (essential, important, not important) for 8 body regions (cervical spine, lumbar spine, shoulder, elbow, wrist/hand/thumb, hip, knee and lower leg/foot/ankle) and three diagnoses for each body region (24 total). If 80% consensus was not achieved in the first stage, then in stage two, face to face discussion is meant to clarify positions and achieved consensus, if possible. The history component, the observation component, scanning exams, clearing joints above and below the lesion site and active range of motion, passive range of motion and isometric resisted testing all yielded 80% consensus in the first two stages of development. A third stage was added to this process because a new model of measurement was conceptualized during the second stage due to a lack of consensus on two orthopedic assessment categories: special testing and palpation. The new model consisted of a ``clinical reasoning" metric that tied each section of an orthopedic assessment (history; observation; scanning and clearing; examination, special testing; palpation; conclusion) together and would permit the expert rater to evaluate the student performance based on the student's rationale for tests chosen rather than the traditionally rigid checklists. At least 80% consensus was achieved with the new ``clinical reasoning" metric and the originally contentious special testing and palpation categories. The new SOAT that underwent content validation may be a hybrid between the traditional OSCE-type checklists and global scales that provide a more realistic and thus more valid depiction of orthopedic assessment clinical competence. Now that content validation has been established, the next steps include inter-rater reliability testing.

Keywords

Validity Validation Content Validation Reliability Orthopedic assessment Orthopedic evaluation Clinical competence Assessing clinical competence Ebel Procedure Validation committee Athletic therapy Physiotherapy Athletic therapist Physiotherapist Orthopedic Assessment Evaluation Checklists Global rating scales Objective structured clinical examination OSCE Performance based examinations Rating scales 

Notes

Acknowledgements

This study was completed with the support of the Canadian Athletic Therapists Association (CATA) Research Grant. Thanks to the expert validation committee for all their time and energy to complete this research. The PI would like to thank the CATA Certification Committee for partnering to bring experts to the second stage of the validation process. Finally, a special thanks to the University of Calgary Athletic Therapy Education Group for their assistance and guidance throughout this process.

References

  1. Butterwick, D. J., Paskevich, D. M., Vallevand, A. L. C., & Lafave, M. R. (2006). The development of content valid technical skill assessment instruments for athletic taping skills. Journal of Allied Health, 35, 149–157.Google Scholar
  2. Cantor, J. A. (1989). A validation of Ebel’s method for performance standard setting through its application with comparison approaches to a selected criterion-referenced test. Educational and Psychological Measurement, 49, 709–721.CrossRefGoogle Scholar
  3. Coady, D., Kay, L., & Walker, D. (2003). Regional musculoskeletal examination. Clinical Rheumatology, 9, 67–71.CrossRefGoogle Scholar
  4. Cox, K. (2000). Examining and recording clinical performance; a critique and some recommendations. Education for Health, 13, 45–52.CrossRefGoogle Scholar
  5. Cyriax, J. (1982). Textbook of orthopedic medicine. (vols. One). Tindall: Bailliere.Google Scholar
  6. Ebel, R., & Frisbie, D. (1986). Essentials of Educational Measurement. (4 ed.) Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  7. Gorter, S., Rethans, J., Scherpbier, A., van der Heijde, D., Houben, H., & van der Vleuten, C. et al. (2000). Developing case specific checklists for SP based assessments in internal medicine. Academic Medicine, 75, 1130–1137.CrossRefGoogle Scholar
  8. Harden, R., Stevenson, M., Downie, W., & Wilson, G. (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1, 447–451.Google Scholar
  9. Hodges, B. (2003). Validity and the OSCE. Medical Teacher, 25, 250–254.CrossRefGoogle Scholar
  10. Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., & Hanson, M. (1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine, 74, 1129–1134.CrossRefGoogle Scholar
  11. Hopkins, K. (1998). Educational and psychological measurement and evaluation. (8th ed.). Toronto: Allyn & Bacon.Google Scholar
  12. Hoppenfeld, S. (1976). Physical examination of the spine and extremities. Norwalk: Appleton & Lange.Google Scholar
  13. Joorabchi, B., & Devries, J. (1996). Evaluation of clinical competence; the gap between expectation and performance. Pediatrics, 97, 179–186.Google Scholar
  14. Kay, L., & Walker, D. (1998). Improving musculoskeletal clinical skills teaching. A regionwide audit and intervention study. British Medical Journal, 57, 656–659.Google Scholar
  15. Magee, D. (2002). Orthopedic physical assessment. (4th ed.). Toronto: Saunders.Google Scholar
  16. Neufeld, V., & Norman, G. (1985). Assessing clinical competence. (vols. 7). New York: Springer.Google Scholar
  17. Norman, G., Tugwell, P., Feightner, W., Muzzin, L., & Jacoby, L. (1985). Knowledge and clinical problem solving. Medical Teacher, 19, 344–356.Google Scholar
  18. Norman, G., van der Vleuten, C., & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity; issues of validity, efficiency and acceptability. Medical Education, 25, 119–126.CrossRefGoogle Scholar
  19. Regehr, G., MacRae, H., Reznick, R., & Szalay, D. (1998). Comparing the pschyometric properties of checkliss and global rating scales for assessmeing performance on an OSCE-format examination. Academic Medicine, 73(9), 993–997.CrossRefGoogle Scholar
  20. Reznick, R., Regehr, G., Yee, G., Rothman, A., Blackmore, D., & Dauphinee, D. (1998). High Stakes Examinations; What do we know about measurement? Academic Medicine, 73, S97–S99.CrossRefGoogle Scholar
  21. Shultz, S., Houglum, P., & Perrin, D. (2005). Examinations of Musculosketetal Injuries, (2 ed.) Champaign, IL: Human Kinetics.Google Scholar
  22. Van der Vleuten, C., Norman, G., & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity: issues of reliability. Medical Education, 25, 110–118.Google Scholar
  23. Van Luijk, S., & Van der Vleuten, C. (1990). A comparison of checklists and rating scales in performance based testing. In I. Hart, R. Harden, & J. Des Marchais (Eds.), Current Developments in Assessing Clinical Competence, Montreal: Can-Heal Publications.Google Scholar
  24. Violato, C., Salami, L., & Muiznieks, S. (2002). Certification Examinations for Massage Therapists: A Psychometric Analysis. Journal of Manipulative Physiological Therapeutics, 25, 111–115.CrossRefGoogle Scholar
  25. Violato, C., Marini, A., & Lee, C. (2003). A validity study of expert judgement procedures for setting cutoff scores on high stakes credentialing examinations using cluster analysis. Evaluation and the Health Professions, 26, 59–72.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, Inc. 2006

Authors and Affiliations

  1. 1.Department of Physical Education and Recreation StudiesMount Royal CollegeCalgaryCanada
  2. 2.Faculty of KinesiologyUniversity of CalgaryCalgaryCanada

Personalised recommendations