Computer-enhanced laparoscopic training system (CELTS): bridging the gap

  • N. Stylopoulos
  • S. Cotin
  • S.K. Maithel
  • M. Ottensmeyer
  • P.G. Jackson
  • R.S. Bardsley
  • P.F. Neumann
  • D.W. Rattner
  • S.L. Dawson
Original article

Abstract

Background

There is a large and growing gap between the need for better surgical training methodologies and the systems currently available for such training. In an effort to bridge this gap and overcome the disadvantages of the training simulators now in use, we developed the Computer-Enhanced Laparoscopic Training System (CELTS).

Methods

CELTS is a computer-based system capable of tracking the motion of laparoscopic instruments and providing feedback about performance in real time. CELTS consists of a mechanical interface, a customizable set of tasks, and an Internet-based software interface. The special cognitive and psychomotor skills a laparoscopic surgeon should master were explicitly defined and transformed into quantitative metrics based on kinematics analysis theory. A single global standardized and task-independent scoring system utilizing a z-score statistic was developed. Validation exercises were performed.

Results

The scoring system clearly revealed a gap between experts and trainees, irrespective of the task performed; none of the trainees obtained a score above the threshold that distinguishes the two groups. Moreover, CELTS provided educational feedback by identifying the key factors that contributed to the overall score. Among the defined metrics, depth perception, smoothness of motion, instrument orientation, and the outcome of the task are major indicators of performance and key parameters that distinguish experts from trainees. Time and path length alone, which are the most commonly used metrics in currently available systems, are not considered good indicators of performance.

Conclusion

CELTS is a novel and standardized skills trainer that combines the advantages of computer simulation with the features of the traditional and popular training boxes. CELTS can easily be used with a wide array of tasks and ensures comparability across different training conditions. This report further shows that a set of appropriate and clinically relevant performance metrics can be defined and a standardized scoring system can be designed.

Keywords

Simulation Laparoscopic skills training Education Virtual reality Laparoscopic surgery 

Minimally invasive surgery is a technically demanding discipline requiring unique skills that are not necessary for conventional open surgery. These skills have traditionally been acquired under the apprenticeship model in animal and human subjects. Recent efforts to develop standardized structured training programs in minimally invasive surgery have generally involved the use of training boxes or computer-based virtual reality simulations. However, none of these trainers has been widely accepted and officially integrated into a surgical training curriculum or any other sanctioned training course. Among the impediments to simulator acceptance by organized medicine are the lack of realism and the lack of appropriate performance assessment methodologies. Thus, it is clear that there is a large and growing gap between the need for better training methodologies and the available training systems.

In an effort to bridge this gap and overcome the disadvantages of the currently available training methods, we developed the Computer-Enhanced Laparoscopic Training System (CELTS), as a step toward a more realistic, clinically relevant, and standardized skills trainer. In this report, we describe CELTS and discuss the motivation for developing such a system.

Materials and methods

CELTS consists of a mechanical interface, a customizable set of tasks, a standardized performance assessment methodology, and an Internet-based software interface.

Mechanical interface

The system is capable of tracking the motion of two laparoscopic instruments while the trainee performs a variety of surgical training tasks. A Virtual Laparoscopic Interface (VLI) (Immersion., San Jose, CA, USA) was modified; this enabled us to use real laparoscopic instruments. The use of real laparoscopic instruments with this training system provides a simple solution to the human–instrument interactions discrepancy encountered with other laparoscopic simulators (Fig. 1). Visual feedback is provided with a movable laparoscopic camera connected to a light source (Telecam SL NTSC/Xenon 175; Karl Storz Endoscopy-America, Culver City, CA, USA), thus replicating the same visual feedback used in minimally invasive surgical procedures.
Figure 1 A,B

Mechanical interface of the Computer-Enhanced Laparoscopic Training System (CELTS). C A Virtual Laparoscopic Interface (VLI) has been modified to enable the use of real laparoscopic instruments. The instruments are connected to the new assembly by a ‘spring-clip’ mechanism, in which a cavity is created in the main tubular shaft, and D each shaft coupling has a cantilever-spring—mounted ‘plug’ that is fitted into the cavity.

The tasks

When using CELTS, the operator may choose to complete (a) a task already specifically designed for CELTS, (b) a task described in established training programs [3, 8, 10], or (c) a novel task. Because of the system architecture, specific new metrics are not required for each new training task; the tasks and standardized performance measurement system are independent of each other. This is a particular strength of the approach we have chosen. In Fig. 2, three tasks that have been used in CELTS are presented. A basic instructional guide (CD-ROM–based tutorial) describing each task has also been prepared. The first task requires the operator to move a small triangular peg with a loop at the top from one stenciled mat to another using a suture needle as a hook. This tests the subtleties of needle loading and positioning and requires appropriate pronation and supination to drive the needle through the loop. The second task requires the operator to place a running suture through three preordained sets of black marks in a foam pad, thus simulating the closure of a laceration. This task tests subtleties of needle positioning and movement through tissue, as well as the appropriate use of both hands. The third task requires the operator to tie a laparoscopic knot on a pad, thus testing this specific skill.
Figure 2 A

Three tasks that have been used to teach the subtleties of delicate laparoscopic skills: moving a peg, running-suture, and knot tying. B Front view of the CELTS system. The interface device, the railed locking and alignment mechanism, and two of the task trays are shown.

Using synthetic models for each of these tasks provides accurate deformation and force feedback during manipulation, resolving the tissue–instrument force feedback problem associated with virtual reality. For each training task, the system uses a railed locking and alignment mechanism to secure a common task tray to the base (Fig. 2). Once it is locked in place, the operator can proceed with the training exercise without dislodging the task tray from the camera’s field of view. Task trays can be easily and quickly changed. This system provides task designers with a model within which to develop new tasks, as well as creating a common scale among all tasks tested by CELTS.

Performance assessment methodology

CELTS is the first laparoscopic trainer to incorporate a standardized and task-independent scoring system for performance assessment. This concept has been described in a previous report [2]. To define a set of quantitative performance metrics that are independent of the training task, we studied the manner by which expert surgeons evaluate a trainee’s performance in the operating room. This evaluation process relies on subjective observation of instrument handling and the overall outcome of the task. Using this limited information, the expert surgeon qualitatively characterizes the performance of the novice on each of the key parameters that are required for efficient laparoscopic manipulations, including compact spatial distribution of the tip of the instrument, smooth motion, good depth perception, instrument orientation, and ambidexterity. Time to complete the task, as well as overall outcome, is also considered. Using kinematics analysis theory, CELTS transforms this subjective and qualitative evaluation into quantitative metrics, specifically measuring the following five kinematic parameters: time to complete the task, depth perception, path length of the instruments, motion smoothness, and response (instrument) orientation (Fig. 3). These are calculated as cost functions; thus, a lower value indicates a better performance.
Figure 3

Definitions of the five kinematic parameters and the computation of the final score. All the variables are time-dependent. [x(t),y(t),z(t)]T is the three-dimensional position of the tip of the instrument, and θ(t) is the rotation of the instrument about its axis.

To establish an expert performance baseline database for each of the three tasks, a panel of surgeons who are considered to be experts in laparoscopy completed each of the tasks in repetition. Thus, utilizing a z-score statistic (Fig. 3), any subsequent performance by a trainee is compared to an expert performance and assigned a standardized overall score from 0 to 100.

The software interface

A software interface was developed for data processing. Raw data consists of time-stamped values of the position and orientation of each of the two laparoscopic instruments. The raw data are filtered and the performance metrics and standardized score are computed, thus providing immediate feedback as to which skills need improvement. The user interface is implemented using the computer programming languages C++, Fast Light Toolkit (FLTK), Java Server Pages (JSP), and OpenGL. It offers real time and a playback display of the path of the laparoscopic instrument’s tip while completing the task. Each score report also includes a visual display of these paths taken by the experts as compared to the trainee (Fig. 4).
Figure 4 A

CELTS provides feedback to the user immediately after the completion of a training session through a Web-based interface. The novice can compare his or her performance to the performance of the experts; thus, he or she learns which skills need to be improved. B Screenshot of our user interface displaying the path of the expert (left) and the novice (right) after completion of a task. The trajectory is color-coded based on velocity. A compact path is characteristic of an expert’s performance.

A database interface was added to maintain each user’s profile information and record all vital information of their task performance. The database system uses a popular public-domain package called MySQL [7]. With this type of database system, a separate database server process is started on the local machine (or on a remote machine). The CELTS system can establish a secure connection to the database server and then make queries to add or manipulate any records within the approved database. The database contains the trainee’s unique identification number, demographic data, and expertise level. Each time a trainee uses the system, one new record is added to that individual’s database. This new record includes the user identification number, session date and time, task number identifying which task was tested, complete raw tracking measurements, computed metric parameters, and the overall score.

A Web server is created on the system that runs the main application. Thus, the database information is conveniently accessible for review through a secure Web interface (i.e., a dedicated Web site has been designed to give access to the most important parameters available in the database). Once the database has been populated with records at the end of a training session, the trainee and/or instructor can immediately retrieve current or past records from a Web page.

Survey administration

We administered a survey to a panel of 30 expert surgeons attending the 8th annual meeting of the Society of American Gastrointestinal Endoscopic Surgeons (SAGES) as a means of exploring surgeons’ requirements for an “ideal” laparoscopic skills trainer. The experts were also asked to rate the importance of various metrics in assessing performance.

Validation studies

To validate our system, we initiated a two-phase study. The aim of the study was to evaluate the ability of CELTS to discriminate between experts and nonexperts performing the same task. For the initial experiment, we asked three expert surgeons, none of whom were included in the initial expert database, to perform each of the previously listed three tasks repeated 10 times. All of the trials were scored by CELTS. After completing the trials, the surgeons were asked to rate their own performance as “perfect” or “satisfactory”; this enabled us to determine if the scores reported by CELTS correlated with the experts’ subjective evaluation of their own performance. For the second phase, we also asked a group of five novices to perform the same set of tasks. Again, each trial was scored by CELTS. Expert and novice scores were compared to assess if CELTS was able to reliably distinguish between the two.

Results

Survey administration

The results of the survey are shown in Fig. 5. Most of the experts surveyed agreed on the importance of skills training and the need for a standardized performance assessment methodology. The results of the survey confirmed our assumption that the medical community is not satisfied with the currently available virtual reality simulators. Training boxes are considered the best means of practicing surgical skills outside the operating room. As shown in Fig. 5D, the metric parameters that are used in currently available simulators (time and path length) received the lowest score of importance among the surgeons in this survey. In contrast, the metric parameters used in the CELTS system were ranked most important in assessing task performance.
Figure 5

Responses to four of the items in our survey. A Should skills training be officially integrated in residency programs and medical curricula? B Using a scale of 0–100, rate the available training modalities. C Is a standardized performance assessment methodology necessary? D Using a scale of the 0–100, rate the importance of the parameters shown in assessing task performance.

Validation studies

All three experts and five novices completed the tasks. With regard to the expert performances, there was a perfect correlation between the experts’ subjective evaluation of the performance and the scores assigned by CELTS. In addition to an overall score that discriminated between “perfect” and “satisfactory” performances, the system provided additional feedback by clearly identifying and quantifying the key individual metric parameters that contributed to the overall score. In Fig. 6, the results for the first task (movement of a small triangular peg) are presented. The “satisfactory” expert performances were characterized by significantly higher values (i.e., worse scores) compared to “perfect” performances in four of the five parameters (time, path length, depth perception, and instrument orientation). There was no statistically significant difference in motion smoothness between the two types of expert performances. As also shown in Fig. 6, the overall scores of the novices, as well as all five of their individual parameter scores, were significantly lower than those of the “satisfactory” performances of expert surgeons. Thus, CELTS reliably discriminated between expert and novice performances. Additionally, a visual depiction of instrument paths showed that a compact spatial distribution of the tip of the laparoscopic instrument is characteristic of an expert performance (Fig. 4).
Figure 6

The five metric parameters do not only discriminate experts (EE) from novices (NO), but they are also very sensitive indicators of performance when the overall performance is judged by the expert to be only “satisfactory” (ES). PL, path length; DP, depth perception; SM; smoothness of motion; IO instrument orientation.

Discussion

Our goal was to develop an advanced educational and clinically relevant training system after considering surgeons’ requirements for an “ideal” laparoscopic skills trainer. CELTS is a laparoscopic skills trainer that uses real instruments, a full-color video display, software-based task independent metrics, and a standardized performance assessment methodology. It is a novel computer-enhanced training system that aims to bridge the gap between currently available simulators and needed training methodologies.

The need for better training methodologies has recently been highlighted by many medical organizations with an educational focus. The American Board of Medical Specialties (ABMS) and the Accreditation Council on Graduate Medical Education (ACGME) have initiated a joint outcomes project to identify and quantify the factors that constitute “medical competence” and promote the development of appropriate training models that would improve medical performance and skills acquisition [1]. The Institute of Medicine, in its landmark report “To Err Is Human,” explicitly recommended (Recommendation 8.1) that “hospitals and medical training facilities should adopt proven methods of training such as simulation” [5]. Additionally, the conclusion of the Surgical Simulation Conference, held in Boston in April 2001 and sponsored by the American College of Surgeons, was that “simulators should be used well into the future to teach, refine and test surgical skills” [4]. There is no doubt that, especially in laparoscopic surgery, skills training optimizes the learning experience in the operating room, which is a limited and expensive resource, by increasing the trainee’s familiarity and level of confidence with the fundamentals of laparoscopy.

Although the importance of training has been well established, there is currently no consensus on the best and most effective training methods. Animal models, which are considered the most effective training modality, are not the most realistic and thus are not the preferred method of training, as shown by our survey. The use of training boxes in which rudimentary tools and objects simulate anatomical structures remains the most popular modality. Our survey also confirmed that the medical community is not satisfied with the currently available virtual reality simulators. Computer simulation emerged as a promising tool that might provide new solutions to the limitations of the current training systems. Computer simulation can revolutionize medical education through the quantification of performance and the standardization of training regimens. Computer-assisted simulators can quantify a variety of parameters, such as instrument motion, applied forces, instrument orientation, and dexterity, all of which cannot be measured with non–computer-based training systems. With proper assessment and validation, such systems can provide both initial and ongoing assessment of an operator’s skill. Additionally, a computerized trainer can provide either terminal (post task completion) or concurrent (real-time) feedback during training episodes, thus enhancing skills acquisition. According to Dr. David Leach, the executive director of ACGME, “What we measure we tend to improve” [6]. The implicit challenge in Dr. Leach’s comment is for us to make measurements that are relevant to those skills that require improvement. In the field of surgical simulation, standard measurements have not yet been agreed upon. Currently, most simulators measure time and path length while performing a particular task, but these measures are not considered sufficient indicators of performance, according to our panel of experts.

Until recently, there was a tendency to view performance assessment and metrics in simplistic terms. The non–computer-based laparoscopic training boxes and the first computer-based trainers used only outcome measures to evaluate performance and learning. However, effective metrics should not only provide outcome information, but should also evaluate the key factors that affect performance. Currently available training systems lack a standardized performance assessment methodology, which is an essential component of a successful educational tool. CELTS is the first laparoscopic skills trainer that incorporates a standardized set of five metrics, each of which measures a specific skill that should be mastered by the laparoscopic surgeon in training. A trainee’s performance is compared to the performance of an expert surgeon. After each training session, CELTS reports not only the trainee’s evaluation, but also the scores of expert surgeons performing the same task. Thus, the feedback system of CELTS serves as a virtual instructor, eliminating the need for the physical presence of an instructor during each training session. Additionally, the flexible Web-based interface provides both instructors and trainees with remote access, further facilitating the educational process.

Another major issue with the simulators is the requisite level of realism. Surgeons believe that the ideal trainer is one that reproduces real operating conditions and teaches tangible operative skills. However, current virtual reality systems cannot provide “real-world” authenticity. Although it has been shown that practicing on simple abstract tasks can lead to skills acquisition [9], surgeons historically have never used abstract tasks for their training. This may explain in part why the currently available computer-based skills trainers are not completely accepted by the surgical community. It is clear that without an objective, standardized, and clinically meaningful feedback system, the simplistic and abstract tasks used in the majority of available training systems are not sufficient to learn the subtleties of delicate laparoscopic tasks and manipulation, such as suturing.

There are other fundamental issues that cannot be ignored. The most important of these are force feedback and visual feedback. While force feedback is diminished in laparoscopic manipulations, surgeons adapt to this inherent disadvantage by developing clever psychological adaptation mechanisms and special perceptual and motor skills. Conscious-inhibition (gentleness) is considered one of the major adaptation mechanisms. Conscious-inhibition means that surgeons use visual feedback cues to sense applied force, despite a lack of actual force feedback. We have called this adaptive transformation “visual haptics.” Using “visual haptics,” a surgeon is able to appropriately modify the amount of mechanical force applied to tissues, predominantly based on the input of visual cues. The visual cues are primarily tissue deformations. For example, a surgeon may not be able to feel a structure that is stretched when retracted, but he or she may sense the retraction by noting subtle indicators such as color change, alteration of contour, and adjacent tissue integrity on the monitor. Although force feedback is a requirement for the ideal trainer, the introduction of force feedback in computer-based learning systems is difficult and requires the knowledge of two elements: instrument–tissue interaction (computation of forces that are applied during surgical manipulations) and human–instrument interaction (design and development of an interface). These are active research areas, and efficient and cost-effective solutions remain to be found. However, the importance of realistic visual feedback that depicts tissue deformations accurately cannot be overstated. The creation of virtual deformable objects is a cumbersome process that requires developing a mathematical model and a knowledge of the object’s behavior with different types of manipulation. Given the need for accurate visual feedback and the limitations of current technology, we believe that the simplest and most cost-effective solution is using real laparoscopic instruments and cameras as well as synthetic task pads, as we have done in CELTS.

In conclusion, we have developed a novel computer-enhanced laparoscopic skills trainer that combines the advantages of computer simulation with those of the traditional and popular training boxes. We also defined a set of appropriate and clinically relevant performance metrics and created a standardized scoring system that compares a trainee’s performance to that of an expert. The initial proof of concept studies have demonstrated the validity of this novel approach and further studies are in progress.

References

  1. 1.
    Accreditation Council for Graduate Medical Education (ACGME) Outcome Project. Available online at: http://www.acgme.org/outcome/
  2. 2.
    Cotin, S, Stylopoulos, N, Ottensmeyer, M, Neumann, P, Rattner, D, Dawson, S 2002

    Metrics for laparoscopic skills trainers: the weakest link!

    Dohi, TKikinis, R eds. Lecture notes in computer science; vol 2488.Springer-VerlagBerlin3543
    Google Scholar
  3. 3.
    Derossis, AM, Fried, GM, Abrahamowicz, M, Sigman, HH, Barkun, JS, Meakins, JL 1998Development of a model for training and evaluation of laparoscopic skills.Am J Surg175482487CrossRefPubMedGoogle Scholar
  4. 4.
    Healy GB, Shore A, Meglan D, Russel M, Satava R (2002) The use of simulators in surgical education. Final report to the Board of Regents of the American College of Surgeons. Working Conference, Boston College, Chestnut Hill, MA, USA, 19–21 April 2002Google Scholar
  5. 5.
    Kohn, LTCorrigan, JMDonaldson, MF eds. 1999To err is human: building a safer health system.Institute of Medicine, National Academy PressWashington (DC)Google Scholar
  6. 6.
    Metrics for Objective Assessment of Surgical Skills Workshop, Scottsdale, AZ, USA, 9–10 July 2001. Final report. Available online at: http://www.tatrc.org
  7. 7.
    MySQL database. Available online at: http://www.mysql.com
  8. 8.
    Rosser, JC, Rosser, LE, Savalgi, RS 1997Skill acquisition and assessment for laparoscopic surgery.Arch Surg132200204PubMedGoogle Scholar
  9. 9.
    Seymour, NE, Gallagher, AG, Roman, SA, O’Brien, MK, Bansal, VK, Andersen, DK, Satava, RM 2002Virtual reality training improves operating room performance: results of a randomized double-blinded studyAnn Surg236458464CrossRefPubMedGoogle Scholar
  10. 10.
    United States Surgical Corporation (USSC) Surgical Skills Training Center2000Report on the analysis and validation of the FLS examination.SAGES: Fundamentals of Laparoscopic Surgery (FLS) Project. Society of American Gastrointestinal Endoscopic SurgeonsSanta Monica (CA)120Google Scholar

Copyright information

© Springer-Verlag 2004

Authors and Affiliations

  • N. Stylopoulos
    • 1
    • 2
    • 3
  • S. Cotin
    • 2
    • 3
  • S.K. Maithel
    • 1
    • 3
  • M. Ottensmeyer
    • 2
  • P.G. Jackson
    • 1
  • R.S. Bardsley
    • 2
  • P.F. Neumann
    • 2
  • D.W. Rattner
    • 1
    • 3
  • S.L. Dawson
    • 2
    • 3
  1. 1.Department of SurgeryMassachusetts General HospitalBostonUSA
  2. 2.The Simulation GroupCenter for Integration of Medicine and Innovative Technology (CIMIT)USA
  3. 3.Department of Surgery, Harvard Medical SchoolHarvard University, 25 Shattuck Street, BostonUSA

Personalised recommendations