Does Perception of Usefulness of Arthroscopic Simulators Differ with Levels of Experience?
- 870 Downloads
Some commercial simulators are available for training basic arthroscopic skills. However, it is unclear if these simulators allow training for their intended purposes and whether the perception of usefulness relates to level of experience.
We addressed the following questions: (1) Do commercial simulators have construct (times to perform tasks) and face validity (realism), and (2) is the perception of usefulness (educational value and user-friendliness) related to level of experience?
We evaluated two commercially available virtual reality simulators (Simulators A and B) and recruited 11 and nine novices (no arthroscopies), four and four intermediates (one to 59 arthroscopies), and seven and nine experts (> 60 arthroscopies) to test the devices. To assess construct validity, we recorded the median time per experience group for each of five repetitions of one identical navigation task. To assess face validity, we used a questionnaire to judge up to three simulator characteristic tasks; the questionnaire asked about the realism, perception of educational value, and perception of user-friendliness.
We observed partial construct validity for Simulators A and B and considered face validity satisfactory for both simulators for simulating the outer appearance and human joint, but barely satisfactory for the instruments. Simulators A and B had equal educational value according to the participants. User-friendliness was judged better for Simulator B although both were graded satisfactory. The perception of usefulness did not differ with level of experience.
Our observations suggest training on either simulator is reasonable preparation for real-life arthroscopy, although there is room for improvement for both simulators.
These simulators provide training in surgical skills without compromising patient safety.
KeywordsVirtual Reality Numerical Rating Scale Face Validity Task Time Lateral Meniscus
Substantial surgical skills are required to perform an arthroscopic procedure without risking iatrogenic injury of articular cartilage and within the routine time scheduled for the operation [16, 17, 20, 23]. Learning arthroscopic skills takes considerable time and implicates an increased risk of surgical errors during the early stages of the learning curve when operating on patients [4, 17, 24]. The traditional learning model where the trainee is supervised continuously by the surgeon attempts to minimize these surgical errors. However, as the training time for acquiring arthroscopic skills is being reduced [8, 12] and demands from society for high-quality healthcare increase , initiatives have been taken to train basic skills away from the operating room [8, 12].
Training arthroscopic surgical skills preferably is performed with actual instrument handling. The theory that states skilled motor behavior relies on accurate predictive models of our body and the environment we interact with (eg, instruments) supports this approach [5, 14, 28, 33, 34]. These predictive models are stored in our central nervous system. To do a certain task, the best available predictive model is selected. A key feature in this theory is that these predictive models are tuned, updated, and learned by providing feedback from our sensory organs (vision and proprioception).
This requires medical simulators that facilitate adequate training. A broad spectrum of simulators has been described in the literature. Traditionally, cadaveric material has been used as a substitute for live patients [10, 20]. Its importance is evident; however, disadvantages are the availability of cadaveric material and preparation time. Two types of simulators have been introduced to overcome the disadvantages of cadaveric material: anatomic bench models [25, 32] and virtual reality systems [4, 15, 17, 26, 35]. As these simulator developments have reached maturity, they have become commercially available. However, it is unclear if these simulators qualify as suitable training means for arthroscopic skills.
We therefore addressed the following questions: (1) do commercial simulators have construct (times to perform tasks) and face validity (realism), and (2) is the perception of usefulness (educational value and user-friendliness) related to level of experience?
Materials and Methods
On February 18, 2009, we performed a systematic search with the Google™ and Yahoo!® search engines as these contain the largest set of Web pages [1, 7]. Combinations of search terms were used: arthroscopy, simulator, orthopaedic, models, simulation, and trainer. A complementary search was performed in classification code G09B23/28 of the patent database Esp@cenet®. Eight different physical and virtual reality arthroscopy simulators were commercially available. Companies were sent an invitation with the request to provide the simulator for 2 weeks at our institute. Two companies agreed to participate: Toltech Knee Arthroscopy Simulator (Touch of Life Technologies, Aurora, CO, USA) and InsightArthroVR® Arthroscopy Simulator (GMV, Madrid, Spain). The other companies [6, 9, 19, 21, 25, 29, 30] refrained for various reasons unrelated to financial issues.
All participants were scheduled a maximum period of 30 minutes. They had no opportunity to familiarize themselves with either simulator before the experiment. The researcher showed the selection of exercises and performance of the calibration protocol and tasks for the test.
The assessment of construct validity (time to perform a task) was based on one basic navigation task. As the simulators were unlikely to offer a navigation task that was the same, one navigation task was prescribed that can and could be performed on all simulators for comparison. With the arthroscope placed in the anterolateral portal and the probe in the anteromedial portal, nine anatomic landmarks had to be probed sequentially: medial femoral condyle, medial tibial plateau, posterior horn of the medial meniscus, midsection of the medial meniscus, ACL, lateral femoral condyle, lateral tibial plateau, posterior horn of the lateral meniscus, and midsection of the lateral meniscus . The participants were asked to repeat this navigation task up to five times in a limit of 10 minutes. The navigation task time was defined as described previously  and determined with a separate video recording of the simulator monitor in which the virtual intraarticular joint is presented. We recorded the median time per experience group for each of five repetitions of the navigation task.
Questions addressing face validity
Face validity aspect
What is your opinion of the outer appearance of this simulator?
Is it clear in which joint you will be operating?
Is it clear which portals are being used?
How realistic is the intraarticular anatomy?
How realistic is the texture of the structures?
How realistic is the color of the structures?
How realistic is the size of the structures?
How realistic is the size of the intraarticular joint space?
How realistic is the arthroscopic image?
How realistic do the instruments look?
How realistic is the motion of your instruments?
How realistic does the tissue feel when you are probing?
Questions addressing educational value and user-friendliness
Educational value I
The simulator allows training of joint inspection*
The simulator allows training of therapeutic intervention*
The simulator allows training of joint irrigation*
The variation of exercises offered by the simulator is adequate*
Difference in required skill level between exercises is adequate*
Educational value II
The simulator is a good way to prepare for a real-life arthroscopic operation*
How clear are the instructions to start an exercise on the simulator?
How clear is the presentation of your performance by the simulator?
Is it clear how you can improve your performance?
How motivating is the way the results are presented to improve your performance?
I felt the need to read a manual before operating the simulator*
The presence of normal distributions of task times was assessed by Kolmogorov-Smirnov tests. Owing to small sample sizes and skewed distributions, the task times were analyzed nonparametrically. Construct validity was determined for each simulator independently by using Kruskal-Wallis tests to calculate the overall presence of differences in task times between the three experience groups for each of the five task repetitions. The significance level was adjusted for multiple comparisons with the Bonferroni-Holm procedure (alpha = 0.05) ; when we detect significant differences we performed pair-wise comparisons between the experience groups separately using Mann-Whitney U tests. The scores of the three separate aspects of face validity of the simulators (Table 1) and User-friendliness I (Table 2) were expressed as mean summary scores of the corresponding questions. Educational Value I (Table 2) was expressed as a sum score of five dichotomous questions and ranged from 0 to 5. The mean summary scores (Face Validity and User-friendliness I) were verified for normality by Kolmogorov-Smirnov tests, expressed as mean and SD, and assessed for differences between both simulators with Student’s t tests. The ordinal scale of Educational Value I was presented as medians with ranges and analyzed using a Mann-Whitney U test. The dichotomous questions (Educational Value II and User-friendliness II) expressed as categorical yes/no answers were presented as frequencies and percentages (%) and analyzed by chi square tests or Fisher’s exact test (in case one or more cells had expected counts less than five). The significance level was adjusted for multiple comparisons with the Bonferroni-Holm procedure (alpha = 0.05) .
As arthroscopic simulators gain maturity and become commercially available, it is unclear whether they are suitable for use in training. We therefore addressed the following questions: (1) do commercial simulators have construct (times to perform tasks) and face validity (realism), and (2) is the perception of usefulness (educational value and user-friendliness) related to level of experience?
We note limitations to our study. First is the relatively small number of participants in each experience group, which could have led to nonsignificant results and the skewed distribution of task times. The groups could not be enlarged owing to logistic limitations; however, care was taken to include all experts and intermediates present at the time of testing to prevent selection bias. Other evaluation studies with simulators have experienced similar problems in recruiting participants [3, 17, 22, 32]. Second is the absence of transfer or predictive validation, which was not feasible in the time frame. Studies performed with similar arthroscopy simulators [9, 12] do show training on these systems decreases the operative learning curve. These findings are in line with the opinion of all participants that training on either simulator will be good preparation before performing real-life arthroscopy. Third, our study is limited to two arthroscopic simulators, which were, in principle, not that distinctive as they are both virtual reality systems with haptic feedback devices. This is reflected in the results. If other types of simulators would have been included, such as anatomic bench models, a wider palette of alternatives could have been described and differences would be more pronounced. Fourth, only one navigation task was used to observe construct validity. The choice of this task is in line with tasks evaluated in other studies [12, 17, 22, 32] and is indicated as an important arthroscopic skill to master before operating in the theater . Fifth, only a few tasks were used to determine face validity, educational value, and user-friendliness. These tasks were chosen carefully and reflected the way exercises are built up and feedback on performance is given by each simulator. Therefore, we assumed the participants were given a good impression of the learning environment of each simulator. Sixth, the choice of expert level was somewhat arbitrary, especially for the novice versus intermediate groups. This could have influenced the demonstration of construct validity as the experience level might have been insufficiently distinctive.
Neither simulator showed full construct validity (Fig. 4) because the task times were substantially similar for all repetitions between the novices and experts, and similar between intermediates and experts. These findings are comparable to those in the study by Srivastava et al. , who used a similar division in experience levels and found no substantial differences between the groups. They speculated the results may be influenced by the fact that experts knew what to expect and novices were very motivated. This could be true for our study. A more detailed comparison with other studies cannot be made as the criteria to qualify as expert, intermediate, or novice differ among studies [17, 26, 32], or a different acceptable significance level was chosen . It is recommended to set uniform experience levels when performing this type of study. By using the study of O’Neill et al. , a solid foundation for assigning experience levels was aimed for. Task time was chosen as an outcome measure, as it is widely used and validated in assessing surgical skills learning, it can be measured using all commercially available arthroscopy simulators, and it makes overall objective comparison possible.
Face validity was observed for both simulators, although there is room for improvement. The presence of tactile feedback in an arthroscopy simulator is considered essential to imitate clinical practice adequately and train safe manipulation [22, 36]. Intermediates and experts indicated tissue probing was unrealistic on both simulators (Fig. 5). Training skills without receiving natural feedback could lead to an offset in the internal models stored in our central nervous system. This might increase errors in the operating room. Performing realistic force feedback for cutting or shaving is another challenge to implement in these simulators . The intraarticular joint space of Simulator B was considered large. Additionally, as both simulators present virtual reality images, they leave an artificial impression. This could be improved with the latest animation techniques used in the gaming industry. These face validity results are comparable to results of other studies, in which imitation of the real-life situation generally is sufficient, but none is given a perfect score [2, 17, 18, 31, 32]. An explanation could be that simulators that do not resemble a human joint are graded more mildly as it is so obvious that they do not resemble reality, whereas simulators that come close to the real-life representation are scrutinized more thoroughly for small deviations. Educational value was perceived for both simulators by intermediates and experts. This subjective opinion is supported by Issenberg et al. , who identified a top 10 list of most important educational criteria for medical simulators. Both simulators fulfill seven of 10 criteria, including the most important ones: give feedback on performance, allow repetitive practice, and allow integration into the curriculum. Unfortunately, they do not offer training of precise portal placement, which is another important skill to be mastered before starting to operate on patients .
Overall, Simulator B was considered more user-friendly than Simulator A, although Simulator A was graded satisfactory (Fig. 5). The feedback given by Simulator B resembles the way mainstream computer games do this. For both simulators, there is room for improvement. Simulator B offers a larger variety of exercises, and is more user-friendly, whereas Simulator A showed a more distinct difference in task time between experts and novices. Teaching surgeons can embrace this type of simulator for implementation in curricula.
We thank Touch of Life Technologies and GMV for providing their simulators for use free and we also thank all participants in the evaluation.
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
- 1.Barker J, Kupersmith J. Recommended search engines. UC Berkeley Library. Available at: http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/SearchEngines.html. Accessed December 10, 2008.
- 6.DelltaTech. Simendo arthroscopy. Available at: http://www.simendo.eu. Accessed February 18, 2009.
- 7.Dogpile.com. Different engines, different results: web searchers not always finding what they’re looking for online. Available at: http://www.infospaceinc.com/files/Overlap-DifferentEnginesDifferentResults.pdf. Accessed December 10, 2008.
- 8.Farnworth LR, Lemay DE, Wooldridge T, Mabrey JD, Blaschak MJ, DeCoster TA, Wascher DC, Schenck RC Jr. A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: the financial impact of orthopaedic surgical training in the operating room. Iowa Orthop J. 2001;21:31–35.PubMedGoogle Scholar
- 11.Holm S. A simple sequentially rejective multiple test procedure. Scand J Statist. 1979;6:65–70.Google Scholar
- 15.Lu J, Chen J, Cakmak H, Maass H, Kuhnapfel U, Bretthauer G. A knee arthroscopy simulator for partial meniscectomy training. Proceedings of the 7th Asian Control Conference. Hong Kong, China; 2009:763–767.Google Scholar
- 19.Mentice. Arthroscopy. Available at: http://www.mentice.com/. Accessed February 18, 2009.
- 21.Mitsubishi Electric Research Laboratories. Knee arthroscopy simulation using volumetric knee models. Available at: http://www.merl.com/projects/kneesystem2/. Accessed February 18, 2009.
- 25.Pacific Research Laboratories, Inc. Sawbones. Available at: http://www.sawbones.com/. Accessed February 18, 2009.
- 29.Simsurgery. SEP products. Available at: http://www.simsurgery.com/web/. Accessed February 18, 2009.
- 30.Simulab Corp. Orthopedic products. Available at: http://www.simulab.com. Accessed February 18, 2009.