Is there a Valid and Reliable Assessment of Diagnostic Knee Arthroscopy Skill?

Abstract

Background

The Basic Arthroscopic Knee Skill Scoring System (BAKSSS) has construct validity as an objective measure of arthroscopic proficiency when used to assess the task of performing arthroscopic meniscectomies on cadaver knees. The reliability of this instrument is unknown.

Questions/purpose

We asked whether (1) a simple modification of the BAKSSS would show construct validity similar to that in the initial BAKSSS study, (2) this assessment would be reliable, and (3) this assessment could be used as a high-stakes pass or fail test.

Methods

Twenty-three orthopaedic residents performed diagnostic knee arthroscopies on cadaveric knee specimens. Their competency was assessed by three live raters using the modified BAKSSS. Interrater reliability was assessed by comparing the scores given by each rater to each subject.

Results

The modified BAKSSS showed construct validity with junior residents achieving lower scores (mean score, 20) than senior residents (mean score, 33). The modified BAKSSS had an interrater reliability of kappa = 0.685–0.852. The modified BAKSSS had a kappa = 0.543 when used as a proficiency test for diagnostic arthroscopy.

Conclusions

The modified BAKSSS is useful for assessing diagnostic knee arthroscopy proficiency. Future scoring systems should be designed to be generalizable so they can be applied to multiple procedures without the need for modification, allow for video-based assessment, and must be rigorously tested for reliability and other types of validity (eg, face validity, content validity, and criterion-related validity).

Clinical Relevance

Having a valid and reliable assessment of basic arthroscopic procedures may allow educators to more adequately evaluate individual residents and the effectiveness of various training modalities.

Appendix 1: Modified BAKSSS Global Rating Scale

Instrument handling

• 1- Repeatedly makes tentative or awkward movements with instruments

• 2-

• 3- Competent use of instruments, although occasionally appeared stiff or awkward

• 4-

• 5- Fluid moves with instruments and no awkwardness

Depth perception

• 1- Constantly overshoots target, slow to correct

• 2-

• 3- Some overshooting or missing of target

• 4-

• 5- Accurately directs instruments in the correct plane to target

Bimanual dexterity

• 1- Noticeably awkward with nondominant hand, poor coordination between hands

• 2-

• 3- Uses both hands but does not maximize interaction between hands

• 4-

• 5- Expertly uses both hands in complementary manner to provide optimum performance

Flow of operation and forward planning

• 1- Frequently stopped operating or needed to discuss next move

• 2-

• 3- Demonstrated ability for forward planning with steady progression of operative procedure

• 4-

• 5- Obviously planned course of operation with effortless flow from one move to the next

Knowledge of instruments

• 1- Frequently asked for the wrong instrument or used inappropriate instrument

• 2-

• 3- Knew the names of most instruments and used appropriate instrument for the task

• 4-

• 5- Obviously familiar with the instruments required and their names

Efficiency

• 1- Many unnecessary, inefficient movements, constantly changing focus or persisting without progress

• 2-

• 3- Slow but planned movements are reasonably organized with few unnecessary or repetitive movements

• 4-

• 5- Confident, clear economy of movement and maximum efficiency.

Knowledge of specific procedure

• 1- Deficient knowledge, needed specific instruction at most operative steps

• 2-

• 3- Knew all important aspects of the operation

• 4-

• 5- Demonstrated familiarity with all aspects of the operation

Autonomy

• 1- Unable to complete entire task, even with verbal guidance

• 2-

• 3- Able to complete task safely with moderation guidance

• 4-

• 5- Able to complete task independently without prompting

Quality

• 1- Very poor

• 2-

• 3- Competent

• 4-

• 5- Clearly superior