The virtual reality simulator dV-Trainer® is a valid assessment tool for robotic surgical skills
Exponential development of minimally invasive techniques, such as robotic-assisted devices, raises the question of how to assess robotic surgery skills. Early development of virtual simulators has provided efficient tools for laparoscopic skills certification based on objective scoring, high availability, and lower cost. However, similar evaluation is lacking for robotic training. The purpose of this study was to assess several criteria, such as reliability, face, content, construct, and concurrent validity of a new virtual robotic surgery simulator.
This prospective study was conducted from December 2009 to April 2010 using three simulators dV-Trainers® (MIMIC Technologies®) and one Da Vinci S® (Intuitive Surgical®). Seventy-five subjects, divided into five groups according to their initial surgical training, were evaluated based on five representative exercises of robotic specific skills: 3D perception, clutching, visual force feedback, EndoWrist® manipulation, and camera control. Analysis was extracted from (1) questionnaires (realism and interest), (2) automatically generated data from simulators, and (3) subjective scoring by two experts of depersonalized videos of similar exercises with robot.
Face and content validity were generally considered high (77 %). Five levels of ability were clearly identified by the simulator (ANOVA; p = 0.0024). There was a strong correlation between automatic data from dV-Trainer and subjective evaluation with robot (r = 0.822). Reliability of scoring was high (r = 0.851). The most relevant criteria were time and economy of motion. The most relevant exercises were Pick and Place and Ring and Rail.
The dV-Trainer® simulator proves to be a valid tool to assess basic skills of robotic surgery.
KeywordsdV-Trainer Surgical education Da Vinci robot Reliability and validity Robotic surgery Simulation
The Da Vinci® robot (Intuitive Surgical, Sunnyvale, CA, USA) is a tool that has been implemented in more than 1,600 operating rooms throughout the world in many different fields (e.g., urology, general surgery, gynecology, heart and thoracic surgery, head and neck surgery). The skills required for robotic surgery are different for laparoscopic surgery or open surgery: clutching, lack of force feedback, Endowrist® manipulation, camera control, 3D-vision. This involves highly specialized training for surgeons and residents. However, teaching such minimally invasive procedures in the operating room according to the Halsted model (with two attending surgeons) is not the most appropriate, because the operator is the only one present at the console and because the learning curve on real patients has financial and medicolegal implications .
From this standpoint, it is important to remember the difficulties encountered when laparoscopic surgery was first introduced . This has in part led to the development of simulation tools, from basic training boxes to complex virtual reality simulators [3, 4, 5]. Since 2009, the American College of Surgeons (ACS) and the Society of Gastrointestinal Endoscopic Surgeons (SAGES) have required all residents to obtain Fundamentals of Laparoscopic Surgery certification (FLS), a license for laparoscopy. Before being recognized as standard, these exercises were extensively studied over a 7 year period [6, 7, 8] to determine whether they met the requirements of large-scale assessments : ease of use, low cost, reliability, accuracy, validity of skills assessment, and correlation with future surgical performance .
Currently, even though recommendations from the Minimally Invasive Robotic Association (MIRA) and SAGES in 2007 encourage the rapid implementation of such a curriculum, there is no equivalent of FLS in robotic surgery. It will be necessary to develop similar tools for training in robotic surgery. Working with the actual robot on anatomical samples, animal models, or inanimate models is costly in terms of equipment and mobilizing the robot (estimated cost: $500/h). Robotic surgery simulators could offer a more economical training alternative. Three simulators, the Ross® by the Roswell Park Cancer Institute (Buffalo, NY), the dV-Trainer® by MIMIC Technologies (Seattle, WA), and the Da Vinci skill simulator® by Intuitive Surgical, which is the Da Vinci Si® console with dV-Trainer® software, are currently available. Before implementation of those new tools in robotic curricula, objective validation is required.
The purpose of our study was to validate the dV-Trainer® as an assessment tool for specific skills in robotic surgery. The first part was dedicated to test face validity (degree of resemblance between the actual robot and the simulator), content validity (interest of the simulator for a training program), and construct validity (degree to which the results on the simulator reflect the actual skill of the subject). These validities have already been proven in other studies on a previous version of the dV-Trainer. The second part of the study tested for the first time reliability (reproducibility of scoring of the subjects when performing the same task twice) and concurrent validity (equivalence between an assessment on the simulator and an assessment on actual Da Vinci®).
Materials and methods
The dV-Trainer® (MIMIC Technologies®) is a robotic surgery simulator consisting of a console that reproduces the look and feel of the Da Vinci system workspace, foot pedals, master controls, and hardware platform with surgical simulation software (M-Sim®, Beta version 184.108.40.2062). It offers a range of training exercises in a virtual 3D environment. M-Sim® software includes a scoring utility with seven criteria: time, economy of motion, drops, instrument collisions, excessive instrument force, instruments out of view, and master workspace range. A total percentage score representing a combination of these criteria is automatically generated by a computerized algorithm created by the manufacturer.
Surgeons, residents, medical students, engineers, and nurses involved in a course in our training center from December 2009 to April 2010 and robotic experts giving these courses were invited to participate in a prospective, institutional review-board study. The participants were prospectively categorized in five groups according to their robotic surgery experience: group 1 (>100 cases), group 2 (10–40 cases), group 3 (no complete case; >4 h at the console), group 4 (no experience in robotic; surgeon or resident), group 5 (no experience in robotic; no experience in surgery).
3D perception: Pick and Place consists in placing red, blue, or yellow objects in corresponding coloured boxes.
Clutching: Peg Board consists in grasping rings on a vertical stand with the left hand and then passing them to the right hand before placing them on a peg.
Visual force feedback: Ring and Rail consists in moving a ring along a twisted metal rod without applying excessive force to either the ring or the rail.
Endowrist manipulation (dexterity when working with one or more instruments): Match Board consists in placing nine numbers and letters in specific squares on a board.
Camera control: Camera Targeting consists in focusing the camera on different blue spheres spread across a broad pelvic cavity.
After finishing the protocol, each subject completed a demographic questionnaire, a face validity questionnaire with two questions (Was this exercise realistic? What are the advantages and drawbacks of the robot and the simulator?), and a content validity questionnaire with two questions (Was this exercise interesting for basic skills learning? Did you prefer simulator or actual robot for basic skills learning?).
Scores on simulator session were exported from the M-Sim® software. Scores on actual Da Vinci® were given by two experts based on de-identified videos, one of the endoscopic view and another of the participant’s arms. They used a scoring system developed in our institution based on six criteria (time, fluidity, excessive force, instrument use, camera use, ergonomy). Validity of this scoring system was tested in a previous unpublished study; interobserver reliability was high (r = 0.802).
Face validity. Subjects without previous robotic experience were excluded from this analysis.
Content validity. Subjects without previous robotic experience were excluded from this analysis.
Construct validity. It was tested using ANOVA  with a threshold of p < 0.05 and Student’s t test with a threshold of p < 0.05.
Reliability. Test-retest reliability  was assessed at the end of the 4 h dV-Trainer session on two consecutive series of exercises using Pearson’s coefficient.
Concurrent validity (equivalence between an assessment on the simulator and an assessment on actual Da Vinci®) was tested using Pearson’s coefficient  to compare, for each subject, the automatically generated score on the simulator and the score of the real exercise on Da Vinci given by the experts. Statistics were produced using Microsoft Excel 2007 (Microsoft Office®).
Experience in laparoscopy (year)
Experience in robotic surgery
1 = Experts
14.2 ± 5.3
264 ± 164 cases
48.2 ± 5.8
2 = Intermediates
6.5 ± 6.9
21 ± 12 cases
43.3 ± 6.4
3 = Beginners
2.6 ± 3.4
0 cases; 6.4 ± 2.0 h
31.3 ± 5.2
4 = Surgeons and residents
3.3 ± 4.9
0 cases; 0.22 ± 0.45 h
34 ± 8.4
5 = Nurses and medical students
0 cases; 0 h
29.7 ± 6.8
- (I-a)Face Validity: The realism of the exercises was considered high or very high by most of the subjects 67.6 % (range, 48.6–81.1 %). Match Board and Camera Targeting were the most realistic exercises (Table 2). Responses are summarized in Table 3 for the question, “What are the advantages and drawbacks of the dV-Trainer in learning robotic surgery?”Table 2
Very high realism
Pick and Place (one-hand basic manipulation)
Peg Board (clutching)
20 (54 %)
Ring and Rail (visual force feedback)
Match Board (two-hand complex manipulation)
Camera Targeting (camera moving)
Very high interest
Pick and Place (one-hand basic manipulation)
Peg Board (clutching)
Ring and Rail (visual force feedback)
Match Board (two-hand complex manipulation)
Camera targeting (camera moving)
14 (38)Table 3
Advantages and disadvantages of the dV-Trainer®
Basic skills learning (clutching; camera)
Fragility and bugs
Less mobility; difficult rotations
Evaluation of skills
Feeling like Da Vinci
Feeling different from Da Vinci
More difficult than actual robot
No risk to broken instrument or Da Vinci
No fine manipulation
Exercises sometimes too difficult
Da Vinci Surgical System
More fluidity and precision
Better 3D vision
Easier to use
Time for installation
No force feedback
Need for materials/animals
Limited training program
Content Validity: The interest of the exercises was considered high or very high by most of the subjects 76.2 % (range, 45.9–91.9 %). Match Board and Camera Targeting were considered the most interesting exercises (Table 2). Most subjects cited the dV-Trainer as the best tool for basic skills learning: 48.6 % answered “simulator”; 16.2 % answered “robot”; 32.4 % answered “both”; and 2.8 % did not answer.
- (I-c)Construct validity: The global scores were strongly correlated with previous experience in robotic surgery. Conversely, the standard deviation within each group diminished with experience. The scores were 56 % ± 11.7, 59.4 % ± 11.4, 62.6 % ± 9.3, 66.1 % ± 8.9, 77.3 % ± 8.2, respectively, for groups 5, 4, 3, 2, and 1 (Fig. 2). Single factor analysis of variance revealed a significant difference between the five groups (ANOVA, p = 0.0024). Robotic surgeons (groups 1 and 2) outperformed subjects with no experience (groups 3, 4, and 5; t test, p = 0.00092). Analyses by exercises and by criteria confirmed this result, except for force and instruments out of view (Table 4).Table 4
Reliability 5–6 (Pearson coefficient)
Reliability 6–7 (Pearson coefficient)
Construct validity 5 groups (ANOVA)
Construct validity 2 groups (t Test)
Concurrent validity (Pearson coefficient)
Compared to Da Vinci S
Series of five exercises
Series of 5 exercises
Pick and Place
Pick and Place
Ring and Rail
Ring and Rail
Economy of motion
Excessive instrument Force
Instrument out of view
- (II-a)Reliability: Analysis of the learning curve (Fig. 3) revealed a plateau after six series of exercises. Reliability was analyzed between the fifth and sixth attempts (Pearson; r = 0.851) and between the sixth and seventh attempts (Pearson; r = 0.847). The same calculation was performed for each of the five exercises and for each of the seven scoring criteria. Results are summarized in Table 4.
Concurrent validity: The overall scores attributed by experts on the robot were strongly correlated with scores automatically generated by the dV-Trainer (Pearson, r = 0.822). An analysis by exercise and criterion, conducted by matching the seven simulator criteria with the six criteria of our robotic scoring system, confirmed this result, but only for the “Pick and Place” and the “Ring and Rail” exercises and the criteria time and economy of motion (Table 4).
Robotic surgery simulators are economical training tools that could offer standardized and objective skills assessments. The principles of evaluating surgical simulators are well established. Common benchmarks on which simulators are judged include reliability, as well as face, content, construct, concurrent, and predictive validities (correlation between results on simulator and future results in operating room).
Our study is the largest to date in terms of the number of exercises tested (n = 5), exercises performed (n = 1,164), participants (n = 75), and levels of skill distinguished (n = 5). The first part has confirmed on a large scale and on the new version of the dV-Trainer®, the face, content, and construct validity, which was the case in previous studies by Kenney et al. , Sethi et al. , and Lendvay et al. . The dV-Trainer is realistic, useful for training, and able to distinguish accurately five levels of robotic skills from novices to experts.
The second part demonstrated for the first time on a robotic simulator the reliability of skill assessment and concurrent validity. So, dV-Trainer® skills assessment can replace an assessment by expert in a robotic surgery dry lab. Lerner’s study  already proved the equivalence of progress between a group trained on simulator and a group trained on actual robot but did not evaluate assessment equivalence.
A detailed analysis of the exercises (Table 4) found that two of them, “Pick and Place” and “Ring and Rail,” were simple and highly relevant, offering good reliability as well as construct and concurrent validity. The Camera Targeting exercise was relevant, with good reliability and construct validity, but lacked concurrent validity. This could be explained by the difficulty of modelling this camera control exercise in a dry lab. The Peg Board and Match Board exercises—the more difficult ones—were less relevant due to lower reliability. This could be explained by significant variations of criteria for instrument collisions, force, and drops. Fifty percent of the overall score is determined by these parameters, which leads to significant variations in the global score. This could be corrected by pooling the results of those two exercises.
A detailed analysis of criteria (Table 4) identified two highly relevant parameters: time and economy of motion. Three criteria (drops, collisions, and master workspace) showed a trend toward significance. The “instrument out of view” criterion was discriminating for groups 2, 3, 4, and 5 (in training), but group 1 (experienced experts) obtained very poor results. The poor results of the experts could be explained by their experience, which allowed them to continue the exercise safely with instruments out of view, whereas in beginners instrument out of view often means lost control of instruments. Excessive force was identified as not statistically significant, because most of the participants have 100 % score for this criterion.
These encouraging initial results should be qualified, because only five of the 30 exercises available on the dV-Trainer® were tested. Moreover, predictive validity  was not studied, because this would have required a longer period of time. Finally, the dV-Trainer does not allow simulations of stitches or dissection and is limited to basic exercises. For the moment, this implies the use of a robot for advanced training . It is highly possible that in the future the development of surgical simulation modules will allow them to practice more extended training with completely simulated surgical cases.
The results of this study position the dV-Trainer® as a good candidate for a large-scale skills certification program similar to the FLS (Fundamentals of Laparoscopic Surgery). The Ross® [21, 22] and Da Vinci skills simulator®  have been validated by only few studies.
Other more exhaustive, and ideally multicentric, studies of all the available exercises would be necessary to select the most relevant and to assess predictive validity to know the impact of simulation training on human procedures. A comparative study of the three simulators also would be useful. Thanks to rigorous methodology , they could define the role of those new tools in skills certification and a multimodal proficiency-based curriculum.
The dV-Trainer® simulator is a reliable tool in the field of robotic surgery that meets the quality requirements of skills certification. It will undoubtedly be a useful training and assessment tool in the field of robotic surgery.
The authors thank Ecole de Chirurgie de Nancy and its staff, CRAN (Centre de Recherche en Automatisme de Nancy) and its staff, Conseil Régional de Lorraine, Communauté Urbaine du Grand Nancy, and Association des Chefs de Service du CHU de Nancy. This work was supported by Conseil Régional de Lorraine, Communauté Urbaine du Grand Nancy, and Association des Chefs de Service du CHU de Nancy.
Cyril Perrenot, Dr. Perez, Dr. Tran, Jehl Jean-Philippe, Dr. Felblinger, Dr. Bresler, and Dr. Hubert have no conflict of interest or financial ties to disclose.