Skip to main content
SpringerLink
Log in

Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance

  • Published:
Journal of Cancer Education Aims and scope Submit manuscript

Abstract

The Accreditation Council for Graduate Medical Education’s Next Accreditation System requires training programs to demonstrate that fellows are achieving competence in medical knowledge (MK), as part of a global assessment of clinical competency. Passing American Board of Internal Medicine (ABIM) certification examinations is recognized as a metric of MK competency. This study examines several in-training MK assessment approaches and their ability to predict performance on the ABIM Hematology or Medical Oncology Certification Examinations. Results of a Hematology In-Service Examination (ISE) and an Oncology In-Training Examination (ITE), program director (PD) ratings, demographic variables, United States Medical Licensing Examination (USMLE), and ABIM Internal Medicine (IM) Certification Examination were compared. Stepwise multiple regression and logistic regression analyses evaluated these assessment approaches as predictors of performance on the Hematology or Medical Oncology Certification Examinations. Hematology ISE scores were the strongest predictor of Hematology Certification Examination scores (β = 0.41) (passing odds ratio [OR], 1.012; 95 % confidence interval [CI], 1.008–1.015), and the Oncology ITE scores were the strongest predictor of Medical Oncology Certification Examination scores (β = 0.45) (passing OR, 1.013; 95 % CI, 1.011–1.016). PD rating of MK was the weakest predictor of Medical Oncology Certification Examination scores (β = 0.07) and was not significantly predictive of Hematology Certification Examination scores. Hematology and Oncology ITEs are better predictors of certification examination performance than PD ratings of MK, reinforcing the effectiveness of ITEs for competency-based assessment of MK.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Swing SR (2007) The ACGME outcome project: retrospective and prospective. Med Teach 29:648–654

    Article  PubMed  Google Scholar 

  2. Accreditation Council for Graduate Medical Education. ACGME common program requirements. Approved September 28, 2014. http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs_07012015.pdf. Accessed October 19, 2015

  3. Nasca TJ, Philibert I, Brigham T, Flynn TC (2012) The next GME accreditation system—rationale and benefits. N Engl J Med 366(11):1051–1056

    Article  CAS  PubMed  Google Scholar 

  4. Collichio FA, Kayoumi KM, Hande KR, Hawkins RE, Hawley JL, Adelstein DJ, D’Angelo JM, Stewart JA (2009) Developing an in-training examination for fellows: the experience of the American Society of Clinical Oncology. J Clin Oncol 27(10):1706–1711

    Article  PubMed  Google Scholar 

  5. Grossman RS, Fincher R-ME, Layne RD, Seelig CB, Berkowitz LR, Levine MA (1992) Validity of the in-training examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med 7(1):63–67

    Article  CAS  PubMed  Google Scholar 

  6. Waxman H, Braunstein G, Dantzker D, Goldberg S, Lefrak S, Lichstein E, Ratzan K, Schiffman F (1994) Performance on the internal medicine second-year residency in-training examination predicts the outcome of the ABIM certifying examination. J Gen Intern Med 9(12):692–694

    Article  CAS  PubMed  Google Scholar 

  7. Babbott SF, Beasley BW, Hinchey KT, Blotzer JW, Holmboe ES (2007) The predictive validity of the internal medicine in-training examination. Am J Med 120(8):735–740

    Article  PubMed  Google Scholar 

  8. Grabovsky I, Hess BJ, Haist SA, Lipner RS, Hawley JL, Woodward S, Cary Engleberg N (2015) The relationship between performance on the infectious diseases in-training and certification examinations. Clin Infect Dis 60(5):677–683

    Article  PubMed  Google Scholar 

  9. Lohr KM, Clauser A, Hess BJ, Gelber AC, Valeriano-Marcet J, Lipner RS, Haist SA, Hawley JL, Zirkle S, Bolster MB (2015) Performance on the adult rheumatology in-training examination and relationship to outcomes on the rheumatology certification examinations. Arthritis Rheum 67(11):3082–3090

    Article  Google Scholar 

  10. McDonald FS, Zeger SL, Kolars JC (2008) Associations between United States Medical Licensing Examination (USMLE) and Internal Medicine In-Training Examination (IM-ITE) scores. J Gen Intern Med 23(7):1016–1019

    Article  PubMed  PubMed Central  Google Scholar 

  11. Kay C, Jackson JL, Frank M (2015) The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med 90(1):100–104

    Article  PubMed  Google Scholar 

  12. Kolen MJ, Brennan RL (2014) Test equating, scaling, and linking: methods and practices, 3rd edn. Springer, New York, NY

    Book  Google Scholar 

  13. Allen MJ, Yen WM (1979) Validity. In: Introduction to measurement theory, 1st edn. Wadsworth, Inc., Belmont, CA, pp 95–117

    Google Scholar 

  14. The Internal Medicine Subspecialty Milestones Project. A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Internal Medicine in collaboration with the Alliance for Academic Internal Medicine and the Association of Specialty Professors. http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/InternalMedicineSubspecialtyMilestones.pdf. July 2015. Accessed October 17, 2015

  15. Lipner RS, Hess BJ, Phillips RL Jr (2013) Specialty board certification in the United States: issues and evidence. J Contin Educ Health Prof 33(Suppl 1):S20–S35

    Article  PubMed  Google Scholar 

  16. American Society of Hematology. Resources for training program directors. Hematology-Oncology Curricular Milestones. http://www.hematology.org/Educators/Training-Directors/2720.aspx. Accessed October 10, 2015

  17. American Society of Clinical Oncology. ASCO and ASH present: the Hematology-Oncology Curricular Milestones. http://www.asco.org/sites/www.asco.org/files/ho_curricular_milestones_4.18.14_final.pdf. 2015. Accessed October 17, 2015

  18. Collichio F, Kayoumi K, Sierra L, Clayton CP, Raymond M, Muchmore EA (2015) The Next Accreditation System: a strategy for implementing new reporting standards using a hematology/oncology model. Am J Med 128(2):200–7

    Article  PubMed  Google Scholar 

  19. Holmboe ES, Wang Y, Meehan TP, Tate JP, Ho S-Y, Starkey KS, Lipner RS (2008) Association between maintenance of certification examination scores and quality of care for medicare beneficiaries. Arch Intern Med 168(13):1396–1403

    Article  PubMed  Google Scholar 

  20. Norcini JJ, Webster GD, Grosso LJ, Blank LL, Benson JA Jr (1987) Ratings of residents’ clinical competence and performance on certification examination. J Med Educ 62:457–462

    CAS  PubMed  Google Scholar 

Download references

Acknowledgments

The authors wish to thank Lyndsey R. Sierra of the American Society of Clinical Oncology for her contributions for getting this research project initiated and for facilitating our access to the data used in this study.

Author information

Authors and Affiliations

  1. Division of Hematology and Oncology, University of North Carolina-Chapel Hill, Physicians Office Building, 3rd Floor 170 Manning Drive, CB# 7305, Chapel Hill, NC, 27599, USA

    Frances A. Collichio

  2. Hess Consulting, 272 Rue du Replat, Lévis, Quebec, G7A 5E4, Canada

    Brian J. Hess

  3. Division of Hematology/Oncology, University of California-San Diego School of Medicine and Veterans Affairs San Diego Healthcare System, 9500 Gilman Drive #9111-E, La Jolla, CA, 92093, USA

    Elaine A. Muchmore

  4. American Board of Internal Medicine, 510 Walnut St, Suite 1700, Philadelphia, PA, 19106, USA

    Lauren Duhigg & Rebecca S. Lipner

  5. Test Development Services, National Board of Medical Examiners, 3750 Market St, Philadelphia, PA, 19104, USA

    Steven Haist

  6. National Board of Medical Examiners, 3750 Market St, Philadelphia, PA, 19104, USA

    Janine L. Hawley & Carol A. Morrison

  7. Education and Training, American Society of Hematology, 2021 L Street NW, Suite 900, Washington, DC, 20036, USA

    Charles P. Clayton

  8. American Society of Clinical Oncology, 2318 Mill Road, Suite 800, Alexandria, VA, 22314, USA

    Marilyn J. Raymond

  9. Education and Training, American Society of Hematology, 2021 L Street NW, Suite 900, Washington, DC, 20036, USA

    Karen M. Kayoumi

  10. Division of Hematology/Oncology, University of Michigan Health System and Veterans Affairs Ann Arbor Health System, C345 Med Inn Building/SPC 5848, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109, USA

    Scott D. Gitlin

  11. University of Michigan Health System, C345 Med Inn Building/SPC 5848, 1500 E. Medical Center Drive, Ann Arbor, MI, 48109-5848, USA

    Scott D. Gitlin

Authors
  1. Frances A. Collichio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian J. Hess
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elaine A. Muchmore
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lauren Duhigg
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rebecca S. Lipner
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Steven Haist
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Janine L. Hawley
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Carol A. Morrison
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Charles P. Clayton
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Marilyn J. Raymond
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Karen M. Kayoumi
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Scott D. Gitlin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Scott D. Gitlin.

Ethics declarations

Institutional Review Board Approval

This study was approved (determined to be “exempt” status) by IRB 00000436 of the American Institutes for Research, Washington, D.C.

Conflict of Interest

The study represents original work, and all authors meet the criteria for authorship and accept responsibility for the scientific content of the manuscript. Dr. Hess is a former employee of the American Board of Internal Medicine (ABIM) and is currently a consultant for ABIM. Ms. Duhigg and Dr. Lipner are employees of the ABIM. Dr. Haist, Dr. Morrison, and Ms. Hawley are employees of the National Board of Medical Examiners, which is contracted to develop and score both the hematology in-service examination and the medical oncology in-training examination. Mr. Clayton and Ms. Kayoumi are employees of the American Society of Hematology. Dr. Raymond was formerly an employee of the American Society of Clinical Oncology. Dr. Muchmore is a volunteer for the ACGME, by serving on the Residency Review Committee-Internal Medicine (RRC-IM). After data collection was completed for this study, Dr. Gitlin was appointed to a position on the Hematology Board of the American Board of Internal Medicine.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

(DOCX 17 kb)

Supplementary Material 2

(DOCX 144 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Collichio, F.A., Hess, B.J., Muchmore, E.A. et al. Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance. J Canc Educ 32, 647–654 (2017). https://doi.org/10.1007/s13187-016-0993-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13187-016-0993-6

Keywords

Search

Navigation