Validation of a Method for Assessing Resident Physicians’ Quality Improvement Proposals
Residency programs involve trainees in quality improvement (QI) projects to evaluate competency in systems-based practice and practice-based learning and improvement. Valid approaches to assess QI proposals are lacking.
We developed an instrument for assessing resident QI proposals—the Quality Improvement Proposal Assessment Tool (QIPAT-7)—and determined its validity and reliability.
QIPAT-7 content was initially obtained from a national panel of QI experts. Through an iterative process, the instrument was refined, pilot-tested, and revised.
Seven raters used the instrument to assess 45 resident QI proposals.
Principal factor analysis was used to explore the dimensionality of instrument scores. Cronbach’s alpha and intraclass correlations were calculated to determine internal consistency and interrater reliability, respectively.
QIPAT-7 items comprised a single factor (eigenvalue = 3.4) suggesting a single assessment dimension. Interrater reliability for each item (range 0.79 to 0.93) and internal consistency reliability among the items (Cronbach’s alpha = 0.87) were high.
This method for assessing resident physician QI proposals is supported by content and internal structure validity evidence. QIPAT-7 is a useful tool for assessing resident QI proposals. Future research should determine the reliability of QIPAT-7 scores in other residency and fellowship training programs. Correlations should also be made between assessment scores and criteria for QI proposal success such as implementation of QI proposals, resident scholarly productivity, and improved patient outcomes.
KEY WORDSquality improvement systems-based practice practice-based learning and improvement assessment evaluation study validation study
- 1.ACGME Outcome Project. Available at: http://www.acgme.org/outcome/comp/compFull.asp. Accessed July 29, 2006.
- 11.ACT (Achieving Competence Today) Online. Available at: http://www.actcurriculum.org/. Accessed June 27, 2006.
- 12.Langley GJ, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: Jossey-Bass; 1996.Google Scholar
- 13.Cleghorn GD, Headrick LA. The PDSA cycle at the core of learning in health professions education. Joint Comm J Qual Improv. 1996;22(3):206–12.Google Scholar
- 15.DeVellis RF. Scale Development: Theory and Applications. London: Sage Publications; 1991.Google Scholar
- 16.Gorsuch RL. Factor Analysis, Second Edition. Hillsdale: Lawrence Erlbaum Associates, Inc.; 1983.Google Scholar
- 20.Messick S. Validity. In: Linn RL, ed. Educational Measurement. 3rd ed. Phoenix, AZ: Oryx Press; 1993.Google Scholar
- 22.Standards for Educational and Psychological Testing. Washington, DC: American Education Research Association; 1999.Google Scholar