There is consensus that education in minimally invasive surgery should be intensified and that more objective assessment of surgeons’ skills should be introduced to ensure high-quality treatment.3 The growing need for training in advanced laparoscopic skills outside the operating room and continued advances in computer technology have led to growing interest in the development of virtual reality (VR) simulators.1,2 Another simulator system is augmented reality (AR), which refers to systems that merges computer graphics and real imagery into a single, coherent perception of an enhanced world around the user. This means that the image one sees on the simulator screen is comprised of a real video image overlaid with a graphics image. This can, for example, be used for directional explanation.

According to various studies2,3,4 force (tactile) feedback is fundamental for good laparoscopic training and results in significantly improved skills transfer to the trainee compared with training without force feedback.2 AR laparoscopic simulators have realistic force feedback because the operator practices on real objects or tissue using surgical laparoscopic instruments. Objective performance assessment is fundamental to provide formative feedback during training, though conventional box trainers lack this ability. AR laparoscopic simulators retain the benefits of a box trainer, such as the realistic haptic feedback, but additionally generate objective measures of performance, similar to VR simulators.1

This study focused on training, simulation, and skills assessment in laparoscopic surgery. Because of the importance of both force feedback and objective assessment, we compared the ProMIS (Haptica) AR laparoscopic simulator with the LapSim (Surgical Sciences) VR laparoscopic simulator. Issues such as realism, didactic value, haptic feedback, and user acceptance were validated by participants. The purpose of this study was to evaluate which training system is of more value in curricula for surgical residents in laparoscopic skills training.

We hypothesized that as a training tool for laparoscopic surgery, the ProMIS AR simulator would be better evaluated by the participants than the LapSim VR simulator.

MATERIALS AND METHODS

Subjects

In total, 90 participants took part in this study. Forty-four participants were tested during the Year Congress of the Dutch Surgical Society 2006, Veldhoven; 34 in the Catharina Hospital, Eindhoven; and and 12 at the International Student Congress of Medical Sciences (ISCOMS) 2006, Groningen, The Netherlands. Participants were allotted to 3 groups based on their clinical laparoscopic experience: experts, who performed more than 100 clinical procedures; intermediates, with less than 100 clinical procedures; and novices, who had no laparoscopic experience.

Equipment

ProMIS Augmented Reality Simulator

The ProMIS AR simulator (Haptica, Dublin, Ireland) (Fig. 1) was used in this study. A torso-shaped mannequin (737-mm long × 508-mm wide × 22-9mm deep) with a black neoprene cover contains an instrument tracking system by means of 3 cameras arranged to identify any instrument inside the simulator from 3 different angles. The left and right cameras are positioned to capture instrument motion, looking in caudal direction of the left and right sides of the mannequin, respectively. The laparoscopic camera is positioned at the pubic symphysis of the mannequin looking cephalad and provides the endoscopic image on the screen.

Figure 1.
figure 1

ProMIS Augmented Reality laparoscopic simulator (Haptica), with 5-mm endograsps and 5- and 10-mm versaport trocars (Tyco Auto Suture, New Haven, CT, USA).

Full size image

The camera tracking system captures instrument motion with Cartesian coordinates in the x, y, and z planes at the average rate of 30 (fps). The laparoscopic instruments of choice (Figs. 1 and 2) are marked with 2 rings of yellow electrical tape at the distal end of the shaft, at a fixed distance, to serve as a reference point for the camera tracking system (Fig. 2). For data collection and video imaging display, a Sony Vaio portable notebook computer is used (with a 2.80-GHz Intel Pentium 4 processor running Windows XP Home Edition).

Figure 2.
figure 2

The needle holders (Karl Storz, Tutlingen, Germany) are marked with the black-yellow tags on the shaft to enable video tracking.

Full size image

The simulator records time, instrument path length, and smoothness of movement (through changes in instrument velocity and changes in direction) for each instrument (right and left hand) during each separate task within the training module. After completion of the task, ProMIS AR provides measurements and statistics on each performed task and module. In addition, a full video and virtual playback of the trainee’s performance are saved, reviewed, and evaluated. Different trays may be placed in the mannequin for each task, such as the suturing pads for the suture and knot-tying tasks. In this study we used a 1-cm-thick suturing pad, which is used in traditional box trainers.

LapSim Virtual Reality Simulator

The LapSim VR simulator (Surgical Sciences) (Fig. 3) is a PC-based trainer for laparoscopic surgery. This simulator is uses a Dell Precision desktop computer (with a 1.5-GHz dual processor systems, Vidia Quadro2EX 32 MB graphics card, and running Windows 2000) on a 17-in., CRT monitor. The VR laparoscopic instruments (Immersion Medical, San Jose, CA, USA) consist of 2 handles with sensors using a 5-degree-of-freedom tracking system for each instrument. The image on the monitor is a computer-generated virtual reality representation of laparoscopic tasks. The software version used in this study was Basic Skills 3.0.

Figure 3.
figure 3

LapSim Virtual Reality laparoscopic simulator (Surgical Sciences).

Full size image

Parameters recorded by the simulator are time, instrument-path length, tissue damage, overall score, and a pass/fail score. These parameters are calculated by the tracking system in each of the instrument handles.

Questionnaire

Participants were asked to fill out a questionnaire that consisted of 3 parts: the 1st part referred to demographics and laparoscopic and/or simulator experience. The second part referred to realism, didactic value, haptic feedback, and usefulness of the ProMIS AR simulator; the feature “didactic value” referred to the additional didactic value of a curriculum with this simulator implemented in it. The last part referred to realism, didactic value, haptic feedback, and usefulness of the LapSim VR simulator. The latter 2 parts had to be answered on a 5-point Likert scale. The questionnaire ended with an open-ended question for general remarks on both simulators.

Protocol

The participants commenced by filling out the first part of the questionnaire. Subsequently, all participants received an introduction about both simulators by means of posters with a short verbal explanation. Information was given about the various training modules available and about the feedback provided by the simulators. Furthermore, the tasks to be performed on both simulators within the scope of this study were clarified. A demonstration video was additionally shown on the ProMIS AR and an instruction text on the LapSim VR before each task. The order in which the participants encountered the simulators was systemically altered (at random) to avoid possible carryover effects. The participants first performed the translocation task, followed by the suturing task(s) (Figs. 4, 5, and 6, 7). After completing the tasks on 1 of the simulators, the participants filled out the corresponding questionnaire. They then performed the equivalent tasks on the other simulator, followed by the last part of the questionnaire.

Figure 4.
figure 4

LapSim virtual reality simulator: “Lifting and grasping” task.

Full size image
Figure 5.
figure 5

LapSim virtual reality simulator: “Suturing” task.

Full size image
Figure 6.
figure 6

ProMIS augmented reality simulator: “Translocation” task.

Full size image
Figure 7.
figure 7

ProMIS augmented reality simulator: “Suturing” task.

Full size image

A maximum time limit of 3 minutes for the translocation task on the LapSim and 5 minutes for the remaining tasks was used, as trained surgeons are expected to be able to perform such a task easily within this time. The data (Table 1) recorded by the simulators was extracted to compare the construct validity of both simulators.

Table 1. Modules and parameters used for this study
Full size table

Data Analysis

All data were processed and analyzed using SPSS 13.0. Significant difference in opinion about these 2 simulators was calculated with the paired t-test. The significant difference in the level of skills extracted from the simulators between the 3 groups was calculated with the Kruskal–Wallis test.

RESULTS

Demographics

A total of 90 subjects participated in this study: o30 “experts”, 30 “intermediates” and 30 “novices”. The 3 groups were not homogenous in all demographic aspects. The expert and intermediate groups contained relatively more male participants (76.7% and 93.3%, respectively), whereas amongst the novices, there were more female participants (63.3%). This can be explained by the fact that in the population we tested, gender division was not equal either. Most (96.7%) participants were right handed, which was equally divided in all groups. The age difference was considerably different between groups (Table 2).

Table 2. Participant demographics
Full size table

The experts all had laparoscopic suturing experience in the clinical setting (with more than 56% >20 times), more than half of the intermediate group had some suturing experience, and none of the novices had any suturing experience (Table 3). All experts and intermediates had assisted in laparoscopic procedures before, while some novices had not even seen a laparoscopic procedure (Table 3). Most intermediates had prior experience on laparoscopic simulators such as VR systems and box trainers, as these are used in training curricula.

Table 3. Participant laparoscopic experience
Full size table

Realism and Haptics

As presented in Table 4, there was a significant difference in participant opinion with regard to features of the 2 simulator systems. Overall, the ProMIS AR scored higher on all aspects than the LapSim VR simulator, with a P-value of 0,000.

Table 4. Difference in opinion about ProMIS augmented reality and LapSim virtual reality simulators
Full size table

The expert group rated “resistance of needle and thread” on the LapSim VR as not good, with a mean of 1.57. Of all the participants, 83.4% judged the “haptic sensations” and 88.9% the “resistance and movements of the instruments” in the ProMIS AR translocations task as good to very good (4 resp. 5 on the Likert scale). These features from the LapSim VR score were rated badly to very badly in 62.2% and 67.8% of respondents, respectively (1 resp. 2 on the Likert scale) (Table 4). On the “suturing” features, the ProMIS AR was rated 2.0 points higher on average than the LapSim VR. The ProMIS AR “suturing task” was considered very realistic by 53.3% of the experts in contrast to 0% for the LapSim VR. The “haptic sensation” and “resistance of needle and thread” of this task were rated 4 and 5 in 70% and 83.3%, respectively, on the ProMIS AR and only 4.4% and 6.6% of respondents, respectively, gave a good rating for the LapSim VR simulator (Table 4). On the ProMIS AR simulator, “haptic sensations of the tissue” was regarded by all participants as less than the other features questioned, but it was still regarded as quite good (mean 3.92). The mean rating remained 2.0 point higher than the rating on the LapSim VR (mean 1.92).

Training Properties

Training properties of both simulators were also part of this study. On the didactic value, the ProMIS AR scored higher than the LapSim VR by all participants (Table 5). The “ProMIS AR simulator as a training tool for surgical residents” scored 4.67 in the expert group and 4.30 in the novice group. The LapSim VR simulator was scored 2.67 and 3.03, respectively, by the same groups for this feature. Of the experts, 66.7% consider the ProMIS AR a very good “tool for training laparoscopic skills to surgical residents,” contrary to 3.3% for the LapSim VR. On the learning properties of the simulators, 83.3% of the experts indicated that the “ProMIS AR can teach trainees the proper skills,” whereas 13.3% stated that the LapSim VR can serve this purpose.

Table 5. Difference in training properties of the ProMIS augmented reality and LapSim virtual reality simulators
Full size table

Opinion on the LapSim VR was divided amongst all participants, which is demonstrated by the wide standard deviations in Table 5. In general, the rating of LapSim VR by the novice group was higher compared with ratings by the intermediate and expert groups. This can be explained because the novice group consists mainly of interns who have not seen, assisted with, or handled a laparoscopic camera before. Therefore, they have no reference point from which to compare the laparoscopic simulators nor to judge on realism. Especially in this group, one may observe a difference between the part of the group that started the study on the LapSim VR first and the other part that started first on the ProMIS AR. But the general opinion on the open-ended question after completion of the training session was that the ProMIS AR simulator was much more realistic and a better training system than the LapSim VR simulator.

Differences in Construct Validity

The current study enabled collection of parameters recorded by both simulators, which are presented in Tables 6 and 7. To assess the parameter significance and determine skill levels, we used the Kruskal–Wallis test. In all tasks investigated in this study, we used a time limit from 300 seconds on each task of the ProMIS AR (“translocation,” “needle handling,” and “knot -tying”). On the LapSim VR, a limit was set at 180 seconds (30 seconds for each handling) for the “translocation” task and 300 seconds for the “suturing and knot-tying” task. On the ProMIS AR simulator, time was shown to be a significant parameter to determine skill level for both “translocation” and “suturing” skills (P = 0.000); on the LapSim VR, there were no significant differences found in this parameter.

Table 6. Construct validity of the ProMIS augmented reality simulator
Full size table
Table 7. Construct validity of the LapSim virtual reality simulator
Full size table

“Smoothness” was calculated on the ProMIS AR by directions and accelerations in instrument movements. From the results depicted in Table 6, one can conclude that all measured parameters on the ProMIS AR were significant to determine skill levels (P < 0.01) except for the “left path length” of the “suturing” task. On the other hand, all parameters measured on the LapSim VR not significant for establishing construct validity, except for the “tissue damage” on the “suturing” task (P = 0.009).

DISCUSSION

Simulator implementation into surgeon training curricula is of paramount importance, and it progressively gains acceptance.5,6 The impact of simulator use is strongly dependent on the correct choice of the system to be used and should be based upon the outcome of validation studies.7 Validation should provide information indicating the usefulness of the chosen system in general and should in particular indicate which system should be used in which training and/or curriculum. Data on (face) validation of the LapSim VR laparoscopic simulator and the ProMIS AR laparoscopic simulator are relatively scarce.2,5,7,8,9,10

Various devices are used to aid training in laparoscopic skills. These devices range from simple box trainers to sophisticated VR trainers. Whereas VR trainers may have some advantages, most participants of such studies feel that inanimate box trainers help more, are more interesting, and are preferred over VR trainers if only 1 trainer is allowed.11 This opinion is supported by the outcome of the current study, as all the participants favored a non-VR environment, with the real instruments used on physical objects, as in traditional box trainers. In the tasks we investigated in this study, we did not use the VR features of the ProMIS AR simulator ,which are present for some modules (e.g., orientation and diathermia), so the video images were similar to those of box trainers.

Various studies have indicated that haptic (force) feedback is very important in laparoscopic simulation and increases the effectiveness of VR training systems.12–14 But this is also one of the most controversial issues in VR laparoscopic simulators design, because it is very difficult and complicated to incorporate realistic haptic feedback into these systems.12 For AR haptics, this is not an issue because of the nature of the system. The outcome of the current study allows the conclusion that ProMIS AR has far better feedback during task performance compared with LapSim VR. Apparently ProMIS AR offers a good alternative to VR laparoscopic simulation, as it retains the benefits of VR and additionally offers more realistic haptic feedback. As shown in Tables 4 and 5, the ProMIS AR simulator rated higher than the LapSim VR simulator on realism, haptic features, and didactic value.

In this study, we encountered the Hawthorn effect,15 which is the influence of the individual performance on the opinion about the simulators. For example, on the LapSim VR simulator, when participants had to perform the “suturing and knot-tying” task, the majority of the participants (in all groups) were not able to pass the needle through the tissue and therefore could not tie knots. This annoyed most of them and was noticeable when they had to fill out the questionnaire. On the ProMIS AR, these suturing skills were tested separately, which caused less frustration.

The outcome of the current study indicates that the construct validity on the “translocation” and “suturing tasks” of the ProMIS AR laparoscopic simulator adequately distinguished between experts and novices. Other studies have shown equal results on the construct validity of this simulator.2,16 This cannot be stated for the LapSim VR laparoscopic simulator, which only shows construct validity on the “tissue damage” of the “suturing task” (Table 6). Previous studies8,10,17–19 on the construct validity of the LapSim VR show conflicting results: only a few parameters were to show construct,17,18 or only when the most (attending surgeons) and least (interns) experienced were compared.8 One study10 even concluded that novices tended to have a better performance compared with the experienced laparoscopists during their first exposure to the LapSim.

To be an effective training tool, the simulator must provide metrics that are meaningful and informative to the trainee. “Time of performance” as a sole parameter might not be the best criterion on which to grade the trainee. The primary issue is that the correct technique is used and a tight knot is made, whereas “time of performance” is a secondary issue. Therefore, it is important to evaluate other metrics recorded by the simulator. In the ProMIS AR simulator, the other parameters, such as “path length” and “smoothness”, give a better impression of the performance and are shown to have a construct validity.

The only measured parameter from the LapSim VR, with construct validity, is “tissue damage”, but this is not sufficient as a sole metric upon which to base assessment of the level of skills of the trainee. This study specifically shows that the ProMIS AR laparoscopic simulator has a better construct validity and is therefore better as a training tool concerning feedback recordings than is the LapSim VR laparoscopic simulator.

CONCLUSION

The outcome of this study allows us to conclude that the ProMIS AR laparoscopic simulator is regarded as a better tool for training surgical residents than the LapSim VR laparoscopic simulator on the limited number of tasks tested. The reason for this outcome is, in our opinion, that AR offers better realism, haptic feedback, didactic value, and construct validity than does VR, and it also gives useful feedback to determine trainee skill levels. It is recommended to implement the ProMIS AR laparoscopic simulator in the training curricula in laparoscopic skills for surgical residents. The place of this simulator in the curricula is in the beginning of the surgical residency, at the cholecystectomy level. It is important that surgical residents not only learn basic laparoscopic skills at this level but also suturing skills for when they, e.g., need to tie the cystic duct when perforations occur in the clinical setting.