Skip to main content
SpringerLink
Log in

Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

The Ottawa Surgical Competency Operating Room Evaluation (OSCORE) is an assessment tool that has gained prominence in postgraduate competency-based training programs. We undertook a systematic review and narrative synthesis to articulate the underlying validity argument in support of this tool. Although originally developed to assess readiness for independent performance of a procedure, contemporary implementation includes using the OSCORE for entrustment supervision decisions. We used systematic review methodology to search, identify, appraise and abstract relevant articles from 2005 to September 2020, across MEDLINE, EMBASE and Google Scholar databases. Nineteen original, English-language, quantitative or qualitative articles addressing the use of the OSCORE for health professionals’ assessment were included. We organized and synthesized the validity evidence according to Kane’s framework, articulating the validity argument and identifying evidence gaps. We demonstrate a reasonable validity argument for the OSCORE in surgical specialties, based on assessing surgical competence as readiness for independent performance for a given procedure, which relates to ad hoc, retrospective, entrustment supervision decisions. The scoring, generalization and extrapolation inferences are well-supported. However, there is a notable lack of implications evidence focused on the impact of the OSCORE on summative decision-making within surgical training programs. In non-surgical specialties, the interpretation/use argument for the OSCORE has not been clearly articulated. The OSCORE has been reduced to a single-item global rating scale, and there is limited validity evidence to support its use in workplace-based assessment. Widespread adoption of the OSCORE must be informed by concurrent data collection in more diverse settings and specialties.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Cook, D. A. (2015). Much ado about differences: Why expert-novice comparisons add little to the validity argument. Advances in Health Sciences Education, 20(3), 829–834.

    Article  Google Scholar 

  • Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49(6), 560–575.

    Article  Google Scholar 

  • Crossley, J., Johnson, G., Booth, J., & Wade, W. (2011). Good questions, good answers: Construct alignment improves the performance of workplace-based assessment scales. Medical Education, 45(6), 560–569.

    Article  Google Scholar 

  • Cutrer, W. B., Russell, R. G., Davidson, M., & Lomis, K. D. (2020). Assessing medical student performance of Entrustable Professional Activities: A mixed methods comparison of Co-Activity and Supervisory Scales. Medical Teacher, 42(3), 325–332.

    Article  Google Scholar 

  • Dudek, N. L., Papp, S., & Gofton, W. T. (2015). Going paperless? Issues in converting a surgical assessment tool to an electronic version. Teaching and Learning in Medicine, 27(3), 274–279.

    Article  Google Scholar 

  • Dudek, N., Gofton, W., Rekman, J., & McDougall, A. (2019). Faculty and resident perspectives on using entrustment anchors for workplace-based assessment. Journal of Graduate Medical Education, 11(3), 287–294.

    Article  Google Scholar 

  • Fitzpatrick, R., Paterson, N. R., Watterson, J., Seabrook, C., & Roberts, M. (2019). Development and implementation of a mobile version of the OSCORE assessment tool and case log for competency-based assessment in urology residency training: An initial assessment of utilization and acceptance among residents and faculty. Canadian Urological Association Journal, 13(2), 4.

    Google Scholar 

  • Gerull, W., Zihni, A., & Awad, M. (2019). Operative performance outcomes of a simulator-based robotic surgical skills curriculum. Surgical endoscopy, 1–6.

  • Gillis, M. E., Scott, S. A., Richardson, C. G., Oxner, W. M., Gauthier, L., Wilson, D. A., & Glennie, R. A. (2020). Developing and Assessing the Feasibility of Implementing a Surgical Objective Structured Clinical Skills Examination (S-OSCE). Journal of Surgical Education.

  • Gilchrist, T., Hatala, R., & Gingerich, A. (2021). A collective case study of supervision and competence judgments on the inpatient internal medicine ward. Perspectives on medical education, 10(3), 155–162.

    Article  Google Scholar 

  • Gofton, W. T., Dudek, N. L., Wood, T. J., Balaa, F., & Hamstra, S. J. (2012). The Ottawa surgical competency operating room evaluation (OSCORE): A tool to assess surgical competence. Academic Medicine, 87(10), 1401–1407.

    Article  Google Scholar 

  • Halman, S., Fu, A. Y. N., & Pugh, D. (2020). Entrustment within an objective structured clinical examination (OSCE) progress test: Bridging the gap towards competency-based medical education. Medical Teacher, 42(11), 1283–1288.

    Article  Google Scholar 

  • Hauer, K. E., Boscardin, C., Fulton, T. B., Lucey, C., Oza, S., & Teherani, A. (2015). Using a curricular vision to define entrustable professional activities for medical student assessment. Journal of general internal medicine, 30(9), 1344–1348.

    Article  Google Scholar 

  • Iobst, W. F., Sherbino, J., Cate, O. T., Richardson, D. L., Dath, D., Swing, S. R., Harris, P., Mungroo, R., Holmboe, E.S., Frank, J.S. & International CBME Collaborators. (2010). Competency-based medical education in postgraduate medical education. Medical Teacher, 32(8), 651–656.

    Article  Google Scholar 

  • Ji, S., Hwang, C., Karmakar, M., Matusko, N., Thompson-Burdine, J., Williams, A. M., ... & Sandhu, G. (2020). Association of intraoperative entrustment with clinical competency amongst general surgery residents. The American Journal of Surgery, 219(2), 245–252.

  • Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.

    Article  Google Scholar 

  • Klasen, J. M., & Lingard, L. A. (2021). The butterfly effect in clinical supervision. Perspectives on Medical Education, 10(3), 145–147.

    Article  Google Scholar 

  • Lord, J. A., Zuege, D. J., Mackay, M. P., Ordons, A. R. D., & Lockyer, J. (2019). Picking the right tool for the job: A reliability study of 4 assessment tools for Central Venous Catheter Insertion. Journal of Graduate Medical Education, 11(4), 422–429.

    Article  Google Scholar 

  • MacEwan, M. J., Dudek, N. L., Wood, T. J., & Gofton, W. T. (2016). Continued validation of the OSCORE (Ottawa Surgical Competency Operating Room Evaluation): Use in the simulated environment. Teaching and Learning in Medicine, 28(1), 72–79.

    Article  Google Scholar 

  • Martin, L., Sibbald, M., Brandt Vegas, D., Russell, D., & Govaerts, M. (2020). The impact of entrustment assessments on feedback and learning: Trainee perspectives. Medical Education, 54(4), 328–336.

    Article  Google Scholar 

  • Meholick, A. L., Jesneck, J. L., Thanawala, R. M., & Seymour, N. E. (2020). Use of a secure Web-based data management platform to track resident operative performance and program educational quality over time. Journal of Surgical Education.

  • Messick, S. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational researcher, 18(2), 5–11.

    Article  Google Scholar 

  • Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews, 4(1), 1–9.

    Article  Google Scholar 

  • Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical teacher, 29(9–10), 855–871.

  • Ode, G. E., Buck, J. S., Wally, M., Scannell, B. P., & Patt, J. C. (2019). Obstacles Affecting the Implementation of the OSCORE for Assessment of Orthopedic Surgical Skills Competency. Journal of Surgical Education, 76(3), 881–892.

    Article  Google Scholar 

  • Prudhomme, N., O'Brien, M., McConnell, M. M., Dudek, N., & Cheung, W. J. (2020). Relationship between ratings of performance in the simulated and workplace environments among emergency medicine residents. Canadian Journal of Emergency Medicine, pp 1–8.

  • Reed, D. A., Cook, D. A., Beckman, T. J., Levine, R. B., Kern, D. E., & Wright, S. M. (2007). Association between funding and quality of published medical education research. Journal of the American Medical Association, 298, 1002–1009.

    Article  Google Scholar 

  • Rekman, J., Gofton, W., Dudek, N., Gofton, T., & Hamstra, S. J. (2016). Entrustability scales: Outlining their usefulness for competency-based clinical assessment. Academic Medicine, 91(2), 186–190.

    Article  Google Scholar 

  • Royal College of Physicians and Surgeons CBD oberservations templates. (2021). Retrieved February 14, 2021, from https://www.royalcollege.ca/rcsite/cbd/assessment/wbas/cbd-assessment-templates-e

  • Saliken, D., Dudek, N., Wood, T. J., MacEwan, M., & Gofton, W. T. (2019). Comparison of the ottawa surgical competency operating room evaluation (OSCORE) to a single-item performance score. Teaching and Learning in Medicine, 31(2), 146–153.

    Article  Google Scholar 

  • Tavares, W., Kuper, A., Kulasegaram, K., & Whitehead, C. (2020). The compatibility principle: On philosophies in the assessment of clinical competence. Advances in Health Sciences Education, 25(4), 1003–1018.

    Article  Google Scholar 

  • Ten Cate, O. (2020). When I say… entrustability. Medical Education, 54(2), 103–104.

    Article  Google Scholar 

  • Ten Cate, O., Schwartz, A., & Chen, H. C. (2020). Assessing trainees and making entrustment decisions: On the nature and use of entrustment-supervision scales. Academic Medicine, 95(11), 1662–1669.

    Article  Google Scholar 

  • Thanawala, R., Jesneck, J., & Seymour, N. E. (2018). Novel educational information management platform improves the surgical skill evaluation process of surgical residents. Journal of Surgical Education, 75(6), e204–e211.

    Article  Google Scholar 

  • Thanawala, R. M., Jesneck, J. L., & Seymour, N. E. (2019). Education management platform enables delivery and comparison of multiple evaluation types. Journal of Surgical Education, 76(6), e209–e216.

    Article  Google Scholar 

  • Thoma, B., Hall, A. K., Clark, K., Meshkat, N., Cheung, W. J., Desaulniers, P., Ffrench, C., Meiwald, A., Meyers, C., Patocka, C., Beatty, L., & Chan, T. M. (2020). Evaluation of a national competency-based assessment system in emergency medicine: A CanDREAM study. Journal of Graduate Medical Education, 12(4), 425–434.

    Article  Google Scholar 

  • Van Heest, A. E., Agel, J., Ames, S. E., Asghar, F. A., Harrast, J. J., Marsh, J. L., Patt, J. C., Sterling, R. S., & Peabody, T. D. (2019). Resident surgical skills web-based evaluation: a comparison of 2 assessment tools. JBJS, 101(5), e18.

    Article  Google Scholar 

  • Weller, J. M., Castanelli, D. J., Chen, Y., & Jolly, B. (2017). Making robust assessments of specialist trainees’ workplace performance. BJA: British Journal of Anaesthesia, 118(2), 207–214.

Download references

Acknowledgements

We would like to thank Dean Giustini and Vanessa Kitchin who were instrumental in guiding our literature search.

Author information

Authors and Affiliations

  1. The University of British Columbia, Vancouver, BC, Canada

    Martha Spencer, Jonathan Sherbino & Rose Hatala

Authors
  1. Martha Spencer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan Sherbino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rose Hatala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martha Spencer.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Spencer, M., Sherbino, J. & Hatala, R. Examining the validity argument for the Ottawa Surgical Competency Operating Room Evaluation (OSCORE): a systematic review and narrative synthesis. Adv in Health Sci Educ 27, 659–689 (2022). https://doi.org/10.1007/s10459-022-10114-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-022-10114-w

Keywords

Search

Navigation