Laparoscopic surgery is not as straightforward as open surgery and requires a range of additional psychomotor and visual-spatial skills. The surgeon has to become proficient in dealing with the counterintuitive manipulation of the instruments, the two-dimensional (2D) representation of the three-dimensional (3D) operating field, and a considerable loss of haptic feedback [1, 2]. Currently, expertise in laparoscopy is still mainly assessed on the basis of the number and type of clinical laparoscopic procedures performed (clinically based expertise) [3, 4]. It is tacitly assumed that a surgeon who is proficient in laparoscopic tissue manipulation and can perform complex tasks like laparoscopic suturing will also be proficient in tasks commonly rated lower in complexity, such as translocation and tissue manipulation. Navigation with a 30° angled laparoscope is considered to be an even easier task. Therefore, the least experienced person of the surgical team often has the assignment to control the laparoscope during a procedure. However, it is important to realise that the nature of laparoscopic tasks such as tissue manipulation and navigation with an angled laparoscope differ considerably and that the required eye–hand coordination partly relies on different visual–spatial and psychomotor abilities. Hence, the interaction with the various features of laparoscopy interfaces (handling of instruments and information) may be difficult to compare and rank in terms of complexity.

Virtual reality (VR) simulators are becoming a popular tool for training basic laparoscopy skills. In addition, they can fulfil the growing need for objective proficiency assessment and provide an effective alternative for clinical training [3–5]. The overall potential, general value, and construct validity of VR simulators have been proven in multiple studies [6–8]. Most studies involved either tasks related to tissue manipulation or tasks related to navigation with an angled laparoscope [2, 9–12]. Only a limited number of studies incorporated both laparoscopic tissue manipulation and navigation with an angled laparoscope [13–15]. The majority of the studies investigated the realism or value of a VR trainer, focussing predominantly on the performances of novices. Little is known about the relation between the performances on fundamentally different laparoscopic tasks, such as bimanual tissue manipulation and angled laparoscope navigation, and the influence of experience. The main objective of this study is to fill this gap by investigating the relation between the performances in these tasks by novice and experienced laparoscopists. The camera navigation (CN) task with a 30° angled laparoscope and the place arrow task (PA) of the SimSurgery SEP VR simulator (SimSurgery AS, Oslo, Norway) were used as representative tasks. Prior to investigating this relation however, we established the face and construct validity of these two tasks on the SimSurgery SEP. (The terminology of the European Association for Endoscopic Surgery (EAES) consensus guidelines for validation of virtual reality surgical simulators is followed [6]).

Materials and methods

Sixty-six participants took part in this study either at the Annual Congress of the Dutch Surgical Society 2007 or at the Catharina Hospital Eindhoven, The Netherlands. The test environments were equivalent: a separate room in which the participants could perform the tasks on the simulator. The participants were allotted to one of two groups based on their indicated clinical laparoscopic experience (Fig. 1). Participants who had not performed any clinical laparoscopic procedures were defined as novices; their medical knowledge and experience were at least at the level of a Dutch medical intern. Participants who had performed more than 50 clinical laparoscopic procedures, and who were familiar with using a 30° angled laparoscope, were defined as experienced.

Fig. 1
figure 1

Overview of the study protocol

Full size image

Simulator

This study focused on the SimSurgery SEP simulation software (SimSurgery AS, Oslo, Norway), which includes a range of tasks in a VR environment to train different laparoscopy skills. The software provides learning objectives, instructions, and a demonstration video before each task. The tasks included in this study were the camera navigation (CN) task with a 30° angled laparoscope and the place arrow (PA) task, which represents a bimanual tissue manipulation task (Fig. 2). The software was used on two different hardware platforms: the SimPack surgical interface (SimSurgery AS, Oslo, Norway), and a Xitact/Mentice platform consisting of two Xitact IHP manipulators (Mentice AB/Xitact SA, Morges, Switzerland). The software produced the same data in both hardware–software combinations and provided numerical data and graphical presentation of the scores after the performance of each task (Table 1).

Fig. 2
figure 2

Screenshots of the CN task (left) and the PA task (right)

Full size image
Table 1 Performance data
Full size table

A preliminary analysis revealed several extraordinary results for the dropped arrows and the closed entries scores for the PA task on the Xitact/Mentice platform, in comparison with the scores on the SimPack platform. Further investigation revealed that these extreme scores could only be explained by differences in the technical characteristics of the hardware and the hardware–software interaction between the SimSurgery SEP simulation software and the Xitact/Mentice platform. Therefore, the scores for dropped arrows and closed entries were excluded from further data analysis for the simulator system with the Xitact/Mentice hardware platform.

Protocol

First, the participants filled out the first part of the questionnaire on demographics and laparoscopy experience (Fig. 1). Next, they received an introduction to the simulator and explanation of the tasks following a standardised procedure. During the introduction it was clearly stated that the researchers were not affiliated with the manufacturer of the simulator and that all data would be analysed anonymously. All participants performed each task twice on one of the hardware platforms. The tasks and type of hardware platform were presented to the participants in random order. Only the scores of the second runs were used to assess the performances. Finally, the participants filled out the remaining part of the questionnaire, in which they were asked to rate the realism, didactic value of the simulator on five-point scales, plus the difference between the perceived and anticipated level of difficulty of the tasks.

Data analysis and statistics

SPSS 13.0 software (SPSS Inc., Chicago, USA) was used for statistical analysis of the simulator performance data and questionnaire data. A p ≤ 0.05 was considered statistically significant.

Results

Twenty-nine participants performed the tasks on the SimPack (11 novices, 18 experienced), and 37 participants performed the tasks on the Xitact/Mentice platform (22 novices, 15 experienced) (Table 1).

Validity of simulator tasks

The opinion of the participants was not affected by the hardware platform that the participants used (Mann–Whitney U test, two-tailed). Both the novices (referent group, N = 33) and the experienced participants (expert group, N = 33) rated the tasks and representation of the behaviour of the laparoscope and graspers as realistic (Table 2). SimSurgery SEP was rated as a realistic and valuable didactic tool by both groups. However, there are some differences in the ratings between the groups, and the level of agreement within the groups. In both groups the participants stated that the CN task was more difficult than expected, while the PA task was not rated as being more difficult than expected. These ratings correlated with some of the performance scores for these tasks (Spearman’s correlation, two-tailed), such as in the CN task with the total tip trajectory in both groups (novices: r s = 0.435; experienced: r s = 0.447) and the number of targets lost out of view in the novice group (r s = 0.464). The ratings for the difference between anticipated and perceived level of difficulty for the PA task correlated in both groups with the time to accomplish this task (novices: r s = 0.404; experienced: r s = 0.428) and the number of lost arrows (novices: r s = 0.535; experienced: r s = 0.362).

Table 2 The rated realism and value of SimSurgery SEP (five-point scale)
Full size table

Preliminary analysis of the performance data of both setups showed that the type of hardware platform did affect some performance scores significantly (Mann–Whitney U test, two-tailed). Therefore, the performance data was assessed for both hardware platforms separately (Table 1). For the SimPack platform, the Mann–Whitney U test (one-tailed) showed a significant difference between the scores of the novices and the experienced participants on the total time to accomplish both tasks and the total tip trajectories. On the Xitact/Mentice platform, the experienced participants performed both tasks in significantly less time than the novices, with significantly shorter tip trajectories. Additionally, the experienced participants also lost significantly fewer targets out of view during the CN task and lost fewer arrows during the PA task.

Correlations within and between tasks

On the SimPack platform, for both groups jointly, the total time to accomplish the tasks correlated with the total tip trajectory of the same task (Figs. 3 and 4). The time to accomplish the CN task correlated with the time to accomplish the PA task (Fig. 5). The tip trajectories of the CN and the PA task also correlated (Fig. 6). The error scores on lost targets out of view and lost arrows correlated with the total tip trajectory of the according tasks (CN: r s = 0.722; PA: r s = 0.375). Within the novice group, the time to accomplish the tasks and the total tip trajectories correlated. In the experienced group they only correlated for the CN task. The tip trajectory of the CN task correlated with the tip trajectory of the PA task within both groups. The error scores on lost targets out of view correlated with the total tip trajectory of the CN task (novices: r s = 0.788; experienced: r s = 0.639). However, the scores on number of lost arrows correlated with the total tip trajectory of the PA task only in the experienced group (r s = 0.573). On the Xitact/Mentice platform, for both groups jointly, the time to accomplish the tasks correlated with the total tip trajectory of the same task (Figs. 3 and 4). The time to accomplish the CN task correlated with the time to accomplish the PA task (Fig. 5), and so did the tip trajectories of the CN and PA tasks (Fig. 6). The error scores on lost targets out of view and lost arrows correlated with the total tip trajectory of the according tasks (CN: r s = 0.757; PA: r s = 0.671). Within the novice group, the time to accomplish the task and the total tip trajectory correlated within the CN task and within the PA task. The time to accomplish the tasks and the total tip trajectory of the same task were also correlated in the experienced group. However, only the tip trajectories scores of the experienced group for the CN and PA tasks correlated significantly with each other. Within both groups, the error scores on lost targets out of view correlated with the total tip trajectory of the CN task (novices: r s = 0.905; experienced: r s = 0.624). And for the novices, the scores on number of lost arrows correlated with the total tip trajectory of the PA task (r s = 0.554).

Fig. 3
figure 3

Scatter plots of the tip trajectory as a function of the time

Full size image
Fig. 4
figure 4

Scatter plots of the tip trajectory as a function of the time

Full size image
Fig. 5
figure 5

Scatter plots of the time to accomplish the two tasks

Full size image
Fig. 6
figure 6

Scatter plots of the tip trajectory in the two tasks

Full size image

Discussion

This study shows that the SimSurgery SEP is a valid and valuable tool to assess skills in bimanual tissue manipulation and navigation with a 30° angled laparoscope, enabling differentiation between novice and experienced laparoscopists on both the SimPack platform and the IHP manipulators of Xitact/Mentice. Face validity was established for the camera navigation and place arrow tasks. However, it is important to realise that the ratings given by the participants could be affected by social–psychological effects. Although VR simulation of laparoscopic tasks is no longer a novelty in the field, the opinion of especially the novices could still be influenced by the novelty of this particular system (Table 2). The combination of the subject expectancy effect and attribution theory most likely influenced the ratings on the realism and didactic value as well. These effects are well known within the field of product usability assessment [16]. The subject expectancy effect is a cognitive bias that occurs when a participant expects a given result, which could unconsciously influence the outcome of the experiment. The attribution theory relates to the reasoning people use to explain their behaviour with something else; how they attribute causes to events and how their cognitive perception affects their reasoning [16]. Most novices were probably not expecting to get excellent scores yet, or did not have any idea what scores to expect. Therefore, they most likely attributed any experience of difficulties with performing the task well to themselves. The experienced participants could have been expecting an excellent score for both tasks. Therefore, they might have attributed any disappointing performance scores predominantly to the simulator, and rated the properties of the simulator accordingly. The correlation of ratings with some of the performance scores for the tasks could partly be explained by the presence of these effects. Several experienced participants made an additional remark, stating that the abstract visual environment could have affected their performance negatively, as they are used to have anatomical landmarks as reference points when manipulating the laparoscope. Stefanidis et al. also discussed this aspect in relation to participant’s frustration, but concluded that the difference in their study was too small to be of practical significance [12].

Comparison of the performance scores within and between the two tasks revealed that there is an obvious trend between the scores in terms of time to accomplish the PA or CN task and the total tip trajectory for the same task, in general and within the novices and experienced groups (Figs. 3 and 4). When comparing the performances of both tasks it is more difficult to discover such a trend within the groups, as the scores are more scattered (Figs. 5 and 6). The scores for the total tip trajectory in both tasks correlated in the experienced group for both platforms, while the scores for the time to accomplish both tasks did not correlate. This could imply that overall experience in handling laparoscopic instruments, such as dealing with the fulcrum effect, does result in better general coordinated aiming or a smoother motion with laparoscopic instruments, and thus a shorter total tip trajectory.

The differences between the performances by the novices and the experienced participants were more distinctive for the PA task than for the CN task. Together with the overall complexity and inconsistency of the relation between the performance scores on both tasks, this supports our assumption that the eye–hand coordination and interaction with laparoscopy interfaces during different tasks deviate considerably, and involve different psychomotor abilities. None of the previously published studies on the performance of angled laparoscope navigation and bimanual tissue manipulation on VR simulators investigated the relation between performances of these tasks. It appears that the general assumption that clinically based expertise in laparoscopic tissue manipulation entails skilfulness in angled laparoscope navigation persisted in all these studies, including those focussing only on camera navigation [9–11].

VR simulators could play an important role in fulfilling the desire for objective proficiency assessment and in accomplishing a shift towards criterion-based training [3–5]. Imperative prior to such a shift however, is a better understanding of how to define proficiency for laparoscopy in general, and for the broad range of activities that laparoscopy includes in particular. The proficiency thresholds for different tasks could also be dissimilar. Proficiency assessment and training should match the characteristics of each specific type of activity or task, in particular when the eye–hand coordination and interaction with the interface in these tasks are fundamentally different.

Limitations

There are some technical limitations to this study, as mentioned in the “Materials and Methods” sections. Due to a communication issue between the SimSurgery SEP software and the Xitact/Mentice hardware, the scores for the number of dropped arrows and closed entries were unreliable and had to be excluded. This situation was previously unidentified by the manufacturers. Following our findings, SimSurgery has adjusted the software to interpret the hardware output better. Overall, the tasks were performed in slightly less time and with a shorter tip trajectory on the Xitact/Mentice platform. This could originate from the fact that the Xitact/Mentice IHP manipulators leave the instruments less freedom of movement than the cannulae in the operating surface of the SimPack.

Conclusions

Within the camera navigation and the place arrow tasks, the performance scores for time to accomplish the task and the total tip trajectory generally correlated significantly. Between these tasks however, a correlation was not always found. This suggests that the general assumption that clinically based expertise in tissue manipulations entails skilfulness in navigation with an angled laparoscope is not completely true. Training and assessment of basic laparoscopic skills should focus on both of these tasks independently. More research is needed to better identify which skills are minimally required for fundamentally different laparoscopic tasks, and at what proficiency level. The physical and cognitive aspects of the interaction with the interface by different proficiency levels also need to be studied further to ensure a good match between proficiency assessment and training in the virtual setting and performance in the clinical setting.