Skip to main content
SpringerLink
Log in

A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2 %) and non-technical (83.2 vs. 75.1 %) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Board of Internal Medicine. (2013). Retrieved Sept 18, 2013, from http://www.abim.org/certification/policies/imss/im.aspx.

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Barsuk, J., Ahya, S., Cohen, E., McGaghie, W., & Wayne, D. (2009). Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. American Journal of Kidney Diseases, 54, 70–76.

    Article  Google Scholar 

  • Boots, R. J., Egerton, W., McKeering, H., & Winter, H. (2009). They just don’t get enough! Variable intern experience in bedside procedural skills. Internal Medicine, 39, 222–227.

    Article  Google Scholar 

  • Brennan, R. (2001). GENOVA suite programs. Retrieved Apr 15, 2013, from http://www.education.uiowa.edu/centers/casma/computer-programs.aspx.

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychology Bulletin, 56, 81–105.

    Article  Google Scholar 

  • Card, S., Snell, L., & O’Brien, B. (2006). Are Canadian General Internal Medicine training program graduates well prepared for their future careers? BMC Medical Education, 6, 56.

    Article  Google Scholar 

  • Castorr, A. H., Thompson, K. O., Ryan, J. W., Phillips, C. Y., Prescott, P. A., & Soeken, K. L. (1990). The process of rater training for observational instruments: Implications for interrater reliability. Research in Nursing Health, 13, 311–318.

    Article  Google Scholar 

  • Cook, D., & Beckman, T. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119, 166.e7–166.e16.

    Article  Google Scholar 

  • Downing, S. M. (2003). Validity: On meaningful interpretation of assessment data. Medical Education, 37, 830–837.

    Article  Google Scholar 

  • Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38, 1006–1012.

    Article  Google Scholar 

  • Ericsson, K. A. (2008). Deliberate practice and acquisition of expert performance: A general overview. Academic Emergency Medicine, 15, 988–994.

    Article  Google Scholar 

  • Frank, J. (Ed.). (2005). The CanMEDS 2005 physician competency framework: Better standards. Better physicians. Better care. Ottawa: The Royal College of Physicians and Surgeons of Canada.

    Google Scholar 

  • Ginsburg, S., Regehr, G., Hatala, R., McNaughton, N., Frohna, A., Hodges, B., et al. (2000). Context, conflict, and resolution: A new conceptual framework for evaluating professionalism. Academic Medicine, 75, S6–S11.

    Article  Google Scholar 

  • Gorman, C. A., & Rentsch, J. R. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology, 94, 1336–1344.

    Article  Google Scholar 

  • Hicks, C. M., Gonzalez, R., Morton, M. T., Gibbons, R. V., Wigton, R. S., & Anderson, R. J. (2000). Procedural experience and comfort level in internal medicine trainees. Journal of General Internal Medicine, 15, 716–722.

    Article  Google Scholar 

  • Huang, G., Smith, C. C., Gordon, C., Feller-Kopman, D., Davis, R., Phillips, R., et al. (2006). Beyond the comfort zone: Residents assess their comfort performing inpatient medical procedures. American Journal of Medicine, 119, 71.e17–71.e24.

    Article  Google Scholar 

  • Jefferies, A., Simmons, B., Tabak, D., McIlroy, J., Lee, K., Roukema, H., et al. (2007). Using an objective structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate training. Medical Teacher, 29, 183–191.

    Article  Google Scholar 

  • Kneebone, R., Nestel, D., Yadollahi, F., Brown, R., Nolan, C., Durack, J., et al. (2006). Assessing procedural skills in context: Exploring the feasibility of an Integrated Procedural Performance Instrument (IPPI). Medical Education, 40, 1105–1114.

    Article  Google Scholar 

  • Kromann, C., Jensen, M., & Ringsted, C. (2009). The effect of testing on skills learning. Medical Education, 43, 21–27.

    Article  Google Scholar 

  • LeBlanc, V., Tabak, D., Kneebone, R., Nestel, D., MacRae, H., & Moulton, C. (2009). Psychometric properties of an integrated assessment of technical and communication skills. American Journal of Surgery, 197, 96–101.

    Article  Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–104). New York: American Council on Education and Macmillan.

    Google Scholar 

  • Moulton, C., Tabak, D., Kneebone, R., Nestel, D., MacRae, H., & LeBlanc, V. (2009). Teaching communication skills using the integrated procedural performance instrument (IPPI): A randomized controlled trial. American Journal of Surgery, 197, 113–118.

    Article  Google Scholar 

  • Naik, V., Devito, I., & Halpern, S. (2003). Cusum analysis is a useful tool to assess resident proficiency at insertion of labour epidurals. Canadian Journal of Anesthesia, 50, 694–698.

    Article  Google Scholar 

  • Norman, G., Bordage, G., Page, G., & Keane, D. (2006). How specific is case specificity? Medical Education, 40, 618–623.

    Article  Google Scholar 

  • Pugh, D., Touchie, C., Code, C., & Humphrey-Murto, S. (2010). Teaching and testing procedural skills: Survey of Canadian Internal Medicine program directors and residents. International Conference on Residency Education, Ottawa, ON. [OM Abstract 76]. Retrieved on Sept 18, 2013 from http://www.openmedicine.ca/article/view/439/353.

  • Reznick, R., Regehr, G., MacRae, H., & Martin, J. (1997). Testing technical skill via an innovative “Bench Station” Examination. The American Journal of Surgery, 173, 226–230.

    Article  Google Scholar 

  • Royal College of Physicians and Surgeons of Canada. (2011). Objectives of training in Internal Medicine. Retrieved Sept 18, 2013 from http://www.deptmedicine.utoronto.ca/Assets/DeptMed+Digital+Assets/Core+Internal+Medicine+Files/rcpscobjectives.pdf.

  • Wayne, D., Barsuk, J., O’Leary, K., Fudala, M., & McGaghie, W. (2008). Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. Journal of Hospital Medicine, 3, 48–54.

    Article  Google Scholar 

  • Wickstrom, G. C., Kolar, M. M., Keyserling, T. C., Kelley, D. K., Xie, S. X., Bognar, B. A., et al. (2000). Confidence of graduating internal medicine residents to perform ambulatory procedures. Journal of Internal Medicine, 15, 361–365.

    Google Scholar 

Download references

Acknowledgments

We would like to acknowledge Dr. John (Jack) R. Boulet, Ms. Lesley Ananny, and the staff at the Ottawa Exam Centre, Academy for Innovation in Medical Education (AIME), and University of Ottawa Skills and Simulation Centre (uOSSC) for their support and valuable advice. Funding for this Project was provided by the University of Ottawa through the Academy for Innovation in Medical Education, the Department of Medicine, and the Office of Postgraduate Residency Education. Dr. Pugh was also partly supported by the W. Dale Dauphinee Fellowship from the Medical Council of Canada.

Author information

Authors and Affiliations

  1. Department of Medicine, University of Ottawa, Ottawa, ON, Canada

    Debra Pugh, Timothy J. Wood, Susan Humphrey-Murto & Claire Touchie

  2. Departments of Medicine, Surgery and Anesthesia, University of Ottawa, Ottawa, ON, Canada

    Stanley J. Hamstra

  3. Department of Medical Education, University of Illinois at Chicago, Chicago, IL, USA

    Rachel Yudkowsky & Georges Bordage

  4. Division of General Internal Medicine, The Ottawa Hospital, General Campus, 501 Smyth Road, Box 209, Ottawa, ON, K1H 8L6, Canada

    Debra Pugh

Authors
  1. Debra Pugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stanley J. Hamstra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Timothy J. Wood
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Susan Humphrey-Murto
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Claire Touchie
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rachel Yudkowsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Georges Bordage
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Debra Pugh.

Appendix: Rating scales for non-technical skills

Appendix: Rating scales for non-technical skills

figure a
figure b

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pugh, D., Hamstra, S.J., Wood, T.J. et al. A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents. Adv in Health Sci Educ 20, 85–100 (2015). https://doi.org/10.1007/s10459-014-9512-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-014-9512-x

Keywords

Search

Navigation