The MISTELS program to measure technical skill in laparoscopic surgery

Evidence for reliability
  • M. C. Vassiliou
  • G. A. Ghitulescu
  • L. S. Feldman
  • D. Stanbridge
  • K. Leffondré
  • H. H. Sigman
  • G. M. Fried
Article

Abstract

Background

The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) is a series of five tasks with an objective scoring system. The purpose of this study was to estimate the interrater and test–retest reliability of the MISTELS metrics and to assess their internal consistency.

Methods

To determine interrater reliability, two trained observers scored 10 subjects, either live or on tape. Test–retest reliability was assessed by having 12 subjects perform two tests, the second immediately following the first. Interrater and test–retest reliability were assessed using intraclass correlation coefficients. Internal consistency between tasks was estimated using Cronbach’s alpha.

Results

The interrater and test–retest reliabilities for the total scores were both excellent at 0.998 [95% confidence interval (CI), 0.985–1.00] and 0.892 (95% CI, 0.665–0.968), respectively. Cronbach’s alpha for the first assessment of the test–retest was 0.86.

Conclusions

The MISTELS metrics have excellent reliability, which exceeds the threshold level of 0.8 required for high-stakes evaluations. These findings support the use of MISTELS for evaluation in many different settings, including residency training programs.

Keywords

Laparoscopic training Simulation Education Evaluation Reliability 

Notes

Acknowledgments

This work was funded by an unrestricted educational grant from Tyco Healthcare Canada.

References

  1. 1.
    American Psychological Association (1985) Standards for educational and psychological testing. American Psychological Association, Washington, DCGoogle Scholar
  2. 2.
    Bland JM, Altman DG (1997) Cronbach’s alpha. Br Med J 314: 572Google Scholar
  3. 3.
    Cronbach L (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16: 297–302CrossRefGoogle Scholar
  4. 4.
    Derossis AM, Antoniuk M, Fried GM (1999) Evaluation of laparoscopic skills: a 2-year follow-up during residency training. Can J Surg 42: 293–296PubMedGoogle Scholar
  5. 5.
    Derossis AM, Bothwell J, Sigman HH, Fried GM (1998) The effect of practice on performance in a laparoscopic simulator. Surg Endosc 12: 1117–1120CrossRefPubMedGoogle Scholar
  6. 6.
    Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL (1998) Development of a model for training and evaluation of laparoscopic skills. Am J Surg 175: 482–487CrossRefPubMedGoogle Scholar
  7. 7.
    Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, Fried GM (2003) Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 17: 964–967CrossRefPubMedGoogle Scholar
  8. 8.
    Fried GM, Derossis AM, Bothwell J, Sigman HH (1999) Comparison of laparoscopic performance in vivo with performance measured in a laparoscopic simulator. Surg Endosc 13: 1077–1082CrossRefPubMedGoogle Scholar
  9. 9.
    Gall M, Borg W, Gall J (1996) Educational research: an introduction. Longman, White Plains, NYGoogle Scholar
  10. 10.
    Hanna GB, Drew T, Clinch P, Hunter B, Cuschieri A (1998) Computer-controlled endoscopic performance assessment system. Surg Endosc 12: 997–1000CrossRefPubMedGoogle Scholar
  11. 11.
    Reznick R, Regehr G, MacRae H, Martin J, McCulloch W (1997) Testing technical skill via an innovative “bench station” examination. Am J Surg 173: 226–230CrossRefPubMedGoogle Scholar
  12. 12.
    Rosser JC, Rosser LE, Savalgi RS (1997) Skill acquisition and assessment for laparoscopic surgery. Arch Surg 132: 200–204PubMedGoogle Scholar
  13. 13.
    Shrout P, Fleiss J (1979) Intraclass correlations: uses in assessing rater reliability. Psych Bull 86: 420–428CrossRefGoogle Scholar
  14. 14.
    Streiner D, Norman G (1996) Health measurement scales: a practical guide to their development and use. Oxford University Press, New YorkGoogle Scholar

Copyright information

© Springer Science+Business Media, Inc. 2006

Authors and Affiliations

  • M. C. Vassiliou
    • 1
  • G. A. Ghitulescu
    • 1
  • L. S. Feldman
    • 1
  • D. Stanbridge
    • 1
  • K. Leffondré
    • 2
  • H. H. Sigman
    • 1
  • G. M. Fried
    • 1
  1. 1.Steinberg–Bernstein Centre for Minimally Invasive Surgery, Department of SurgeryMcGill University Health CentreMontrealCanada
  2. 2.Department of Social and Preventive MedicineUniversity of MontréalMontrealCanada

Personalised recommendations