Evaluating the Utility, Reliability, and Validity of a Resident Performance Evaluation Instrument

Abstract

Objective

The primary goal of this project was to create and evaluate the utility, validity, and reliability of an instrument assessing resident performance during child and adolescent psychiatry (CAP) training. In three stages, the instrument was developed and evaluated for utility in assessing skills critical to CAP. Items on the Global Assessment Evaluation (GAE) were derived from the six core competencies identified by the American College of Graduate Medical Education (ACGME). Secondary goals included producing an instrument that could be used by other CAP training programs and describing a process that could be followed by training programs to create their own assessment instruments.

Methods

Faculty members developed a CAP resident performance evaluation instrument. In a three-stage process, faculty utilized the tool to evaluate residents during and after rotations. Statistical findings guided revisions to improve the utility and reliability of the instrument. For the final version of the GAE, intra-class correlation coefficients were calculated to assess inter-rater reliability, and Principal Components Analysis provided further insight into the dimensions of resident assessment.

Results

The final version of the GAE showed overall and construct validity by capturing significant differences among residents and matched faculty members’ overall impressions of resident performance. Intra-class correlation coefficient values for the overall score (0.945) and individual scales showed good reliability. Resident performance was not correlated with rotation site or model of care.

Conclusions

The GAE has proved a valuable instrument in tracking the progress, strengths, and weaknesses in resident performance over the course of training. Data from multiple evaluations over time provide useful information about resident performance in a way that one or few evaluations does not. This finding is consistent with the practice of semiannual reviews and the new Clinical Competency Committees, both of which are required by the ACGME.

Appendixes

Appendix 1

Example of GAE Item with Behavioral Descriptors and Levels

Item 11: Psychotherapy Primary core competency: Medical knowledge Secondary competencies: Patient care, Practice-based learning and improvement, Interpersonal and communication skills
1	Does not understand and does not implement psychotherapeutic principles
2	Discusses and is able to implement some practical principles of limited psychotherapy models. Despite knowledge, fails to follow child’s lead, appears uncomfortable during the session, or uses some language/techniques not appropriate for patient’s age or presentation.
3	Discusses and is able to implement several models of psychotherapy and practical approaches as applied to children and families. Generally follows child’s lead, uses language/techniques appropriate for patient’s age and presentation, appears comfortable with the child, and appears comfortable with the chosen model of therapy.
4	Discusses cases fluently according to many models of psychotherapy as applied to children and families. Implements psychotherapeutic interventions flexibly, based on needs of child and family. Consistently follows child’s lead, uses language/techniques appropriate for patient’s age and presentation, follows the process of the session, asks relevant questions, and is comfortable with the child’s expressed affect.
5	Knows classic and modern theories and approaches to psychotherapy. Has in-depth, detailed knowledge of theory, principles, and practice guidelines for psychotherapies. Is highly proficient in implementing appropriate psychotherapeutic interventions. N/A

Appendix 2

Outline of Items Included in Principal Components Analysis

Patient care

• 1. Gathers accurate and essential information

• 2. Gathers information from collateral sources

• 3. Mental Status Examination

• 4. Formulation

• 5. Diagnoses

• 6. Treatment plan

• 7. Rapport and therapeutic alliance

• 8. Evolution of treatment

Medical knowledge

• 9. Overall medical knowledge

• 10. Evidence-based medicine

• 11. Psychotherapy

• 12. Psychopharmacology

Practice-based learning and improvement

• 13. Practice-based learning and improvement

Interpersonal and communication skills

• 14. Supervision

• 15. Communication with patients, families, and/or translators

• 16. Communication and interaction with coworkers

• 17. Communication through Medical Record

Professionalism and ethical behavior

• 18. Professionalism

• 19. Ethical decision-making and cultural sensitivity

Systems-based practice

• 20. Systems-based practice