1 Aims

The aim of this research was to investigate the extent to which virtual reality simulators equipped with haptic devices, providing feedback in the form of 3D vision, sense of touch and sound, can enhance undergraduate students’ psychomotor skills and tutors’ assessment of these skills.

2 Background

According to the theory of skill acquisition developed by Ackerman (1988), during the skill learning process, learners go through three phases of psychomotor development: the Cognitive phase, Associative phase and Autonomous phase. By measuring different psychomotor skills, it should be possible to distinguish these phases of skill acquisition between a novice and an expert practitioner. These assessment techniques could help to shape the formative and summative assessment methods by identifying the skills acquisition stage of the trainees during their learning process. The study reported here was conducted in the dental clinical environment but would be equally applicable to any discipline in which psychomotor skills form an important component of students’ learning.

In many university healthcare undergraduate settings, clinical skills progression is traditionally monitored by tutor observation and assessed by looking at the end result of the pre-clinical task, such as the removal of decayed tissue from a tooth or the administering of an injection into a mannequin (de Peralta et al. 2017). With large student cohorts (e.g. more than 50) and few tutors (2–3) present during their traditional phantom head simulator training sessions, such assessment techniques can rarely measure the actual procedure followed by every student to achieve the desired outcome. The tutors are mainly only able to see the final outcome of every student because there are too many to observe at the same time. Furthermore the assessment ratings can be subjective to the experience and opinion of the tutors rather than based on specific reproducible performance data (Shahriari-Rad 2012).

According to Ackerman (1988), based on the work of Fitts and Posner (1967), the three phases of psychomotor developments can be measured accurately by specific psychometric tests. These psychometric tests can be used as a baseline measurement to monitor students’ progress during their clinical skills’ training programmes. Furthermore, previous studies showed that the use of virtual reality simulators could provide a better psychomotor learning achievement through the specific pre-clinical task-based courses (Grantcharov 2006; Botden et al. 2007; Suksudaj 2010; Urbankova and Engebretson 2011; Tregunna et al. 2012).

Previous research into the interactions between cognitive psychology, educational technology and computing in recent years has considered many forms of digital learning systems (e.g. see Ifenthaler et al. 2015). However, even though there is a growing use of haptic (sense of touch) technologies within many undergraduate healthcare programmes (e.g. see de Boer et al. 2015; San Diego et al. 2012; Quinn et al. 2003), there have been few large scale quantitative studies published to date of the relationship between students’ psychomotor skills and haptic learning environments which are becoming widely used in the health sciences in higher education (Bakr et al. 2014; Urbankova and Engebretson 2011; Wang et al. 2015). Consequently, there has been a need for further investigations into the relationship between students’ psychomotor skills and their clinical skills development, which is the focus of the study reported in this paper.

3 Introduction

The research study reported in this paper involved the fourth study (Study 4) of four consecutive student cohort studies conducted in 2007/2008, 2008/2009, 2009/2010 and 2010/2011 of over 520 dental undergraduate first year students in total at King’s College London (San Diego et al. 2012; Shahriari-Rad 2014), followed by the 2nd year of the 2010/2011 entry cohort. The Bachelor of Dental Surgery undergraduate programme to become a qualified dentist in the UK is 5 years. This 2-year study involved 140 students who started their undergraduate programme in October 2010 and continued into their second year in October 2011. All students in Year 1 worked in both the traditional clinical skills (Phantom-head) laboratory and the haptically simulated virtual reality systems (HapTEL) laboratory at the beginning of their first year of the BDS Programme.

Pre- and post-psychometric tests and clinical examination results evaluated in the previous studies showed significant psychomotor skills’ development of the students especially in the area of spatial awareness within a 3-months period (one term) of pre-clinical training (Shahriari-Rad 2014). The study, reported in this paper, complements the previous studies by investigating the students’ skill acquisition over an 18-month period measuring the specific stages of students’ psychomotor skills enhanced by practising those skills using the virtual reality dental simulator.

4 Methods

4.1 Participants

In order to obtain their Bachelor of Dental Surgery Degree (BDS) undergraduate students need to complete a 5-year full-time undergraduate course. The participants in this 2 year study consisted of a single year cohort of 140 (2010/11) dental undergraduates at King’s College London and in the second year, the same cohort augmented by 20 additional students joining the course (See Table 2). They were studied over two consecutive years and assessed for their clinical skills and knowledge using both psychometric and traditional tests shown in Table 2. The additional students were not assessed for their psychomotor skills, only through their traditional exams.

During academic year 2010/11—pre- and post (after the learning experience) psychometric skills were measured.

During academic year 2011/12—objective structured clinical examination (OSCE) and Clinical Skills Examination (CSE) of all students were conducted.

4.2 Educational Setting for the Study

The Educational setting for the students being studied consisted of two laboratories: the traditional Phantom Head Laboratory (see Fig. 1) used in most undergraduate university dental schools and the hapTEL virtual dental work station laboratory (see Fig. 2). Extensive dental skills’ training is conducted over 18 months in the traditional Phantom-head laboratory (PHL) and for this research we focused on the students’ first year’s use. This initial focus of their learning in the PHL during the period of this research was on caries removal and cavity preparation and other basic clinical skills.

Fig. 1
figure 1

Tutor instructing students in the Phantom Head Laboratory

Full size image
Fig. 2
figure 2

Tutor instructing students in the hapTEL virtual laboratory

Full size image

In 2009, the first virtual reality simulator laboratory was created comprising of 12 dental haptic stations as part of the hapTELFootnote 1 project to investigate the different aspects of using virtual reality simulators compared to the traditional Phantom-head systems. The ethical approval to carry out this study as part of the hapTEL project was granted from King’s College London Ethics Committee before the start of the project.

The research phase presented in this paper involved investigating 140 students learning with haptic simulators in the first term of their course, alternating with working in the PHL alternative weeks. All students for this study were spread across 12 tutor groups and used the hapTEL VRS systems as well as the conventional Phantom-head system in turn (every Wednesday; students attended 12 Wednesdays of training). Initially, all students attended a lecture in the Phantom-head laboratory from 9:00 till 10:30 a.m. After the lecture, 24 students working in pairs, used the hapTEL laboratory while the rest stayed in the Phantom-head laboratory for clinical skills training.

The pairs of students used the hapTEL system for 90 min; one as the clinical operator and the partner as the ‘dental nurse”, swapping over the roles of operator and nurse half way through the session. Students working in the Phantom Head laboratory were provided with plastic realistic teeth on which they had to operate to remove decayed tissue from firstly a ‘floating tooth” holding it in their hand and then a lower tooth screwed into an artificial jaw with a larger area of caries (decayed tissue) which they had to remove, retaining as much healthy tooth as possible.

Students using the hapTEL system had similar tasks involving four stages: (1) Trial task: a virtual tooth without any carious lesion, (2) Task 1: a virtual ‘floating’ tooth with shallow carious lesion, (3) Task 2: a virtual tooth in a jaw with slightly deeper carious lesion and (4) Task 3: a virtual tooth in a jaw with a cruciate-form carious lesion. These images were displayed in 3D for the learners (using special 3D glasses). The task involved removing artificial decayed tooth material represented in dark brown coloured tetrahedrons whilst carefully avoiding removing the healthy parts of the tooth, i.e. enamel (in off-white coloured tetrahedrons) and healthy dentine (in ivory coloured tetrahedrons), around the boundaries of the decayed part; and also avoiding hitting and removing the pulp (in red coloured tetrahedrons).

In March 2012, 160 BDS2 students (including 140 from the previous year) attended their clinical skills examination (see Table 2), which consisted of a series of traditional practical tests and oral tests held towards the end of the second term of their second year.

4.3 Assessment Methods

The assessment methods for this study involved both the traditional methods used throughout the undergraduate Dental programme and the additional psychometric tests acquired or developed by the hapTEL project (see San Diego et al. 2014; Shahriari-Rad 2014) to measure the students’ psychomotor skills and spatial perceptions.

4.3.1 Traditional Assessments

These consisted of formative assessments during the first year and summative assessments of students’ clinical skills after 18 months from the beginning of the course towards the end of the second term in the second year.

4.3.1.1 Formative Assessment of Students’ Clinical Skills

This was provided by tutor observation and individual feedback to students. The overall performance was judged by the tutors on the cavity preparation work completed by each student on artificial or discarded natural teeth at the end of the training sessions.

4.3.1.2 Traditional Summative Tests

These consisted of two examinations of all students in the second year of their five year programme as indicated in Table 2 below. The CSE examinations comprise of two parts: practical examination and oral interview. Two examiners awarded a grade for each student and the final mark was then the average of these two grades. The maximum achievable score for these traditional tests was 100%:

  1. 1.

    The clinical skills examination (CSE) which included a practical component (which is also called License to Cut). Table 2 shows the number of students (N = 160) who took this exam.

  2. 2.

    The oral examination (OER). Oral part of License to Cut examination which assesses the student’s knowledge of clinical skills and oral health.

4.3.2 Psychometric Tests

The series of five psychometric and spatial reasoning tests used by the hapTEL project summarized in Table 1 were used for this study. For fuller details of these tests see San Diego et al. (2012) and Shahriari-Rad (2014, p 69–80). They consisted of the following:

Table 1 Psychometric tasks and scores
Full size table
  1. 1.

    Spatial reasoning test (SRT) developed by ARCO to measure spatial relation and depth perception. This test, which involves identifying three different correct views of a 3D shape drawn on paper (or displayed on a screen for the online version), measures students’ abilities to comprehend static or stationary objects and their ability to picture the way an object will look after it has been moved to another position.

  2. 2.

    Fine motor skills (FMS) Peg-board test developed by Morrisby involves picking up small bolts (pins) with one’s fingers, putting on washers and nuts then inserting each one into a hole within a fixed specified time, to measure the students’ abilities to carry out precise movement of their hands and their hand-eye-finger coordination.

  3. 3.

    Fine motor skills (FMS) Tweezers Test also developed by Morrisby is similar to the one above but involves the students having to pick up bolts with tweezers within a fixed specified time, which is more difficult and it measures very precise movement of their hands and their hand-eye-finger coordination.

  4. 4.

    Gross motor skills (GMS); the Minnesota Manual Dexterity Test involved each student picking up 60 discs, one at a time, with one hand and turning it over with their leading hand and placing it back into the hole. This measured the students’ abilities to handle tools such as drills and syringes which need good gross motor movements.

The grading variables for each of these five tests varied according to the specific tasks given below and shown in Table 1.

Test 1 (SRT) Scores were allocated for the 29 questions; the maximum achievable score was 29.

Tests 2 and 3 (FMS with and without tweezers). The number of operations (pins inserted) completed within 2.5 min; the maximum achievable score was 29 bolts (pins).

Test 4 (GMS). The number of drafts which could be rotated and then placed in a hole was 60. The score was the total time taken to complete this task therefore the shorter the time the better the achievement.

Test 5 (BDS). This test requires students to replicate a 2-D shape using 3D blocks that have red, white, and half-red and half-white sides to form specific patterns within a limited time (see Fig. 3). Students’ scores are calculated from nine patterns starting from (Pattern 6) to the most difficult one (Pattern 14). The highest obtainable score is 68.

Fig. 3
figure 3

Block design test measuring spatial awareness

Full size image

4.4 Application of the Tests

The psychometric tests were administered by the research team in October (Pre-test) and December (Post-test) 2010 as shown in Table 2. The duration of every test was accurately timed for every student by trained test administrators. The total time taken to test all 140 students with all 5 tests administered by 12 testers took approximately two days (14 h).

Table 2 Tests and number of participants
Full size table

The total number of students taking each psychometric pre and post-test in Year 1 varied between 125 and 132 because of absences due to illness.

The total number of students taking the traditional tests in Year 2 was 160 which included the 20 additional students joining the course in Year 2.

5 Results

The total number of test results which were included in the analysis reported in this paper were confined to the 124 students who took all seven tests between October 2010 and December 2010 (Pre and post-psychomotor tests) and 160 including the additional students in March 2012 (traditional).

For the statistical analysis of the results, IBM SPSS Statistics 20 software was used to compute the Spearman’s rho (r) for the purpose of finding the correlation between the test variables since the distribution of the data for the tests were not normal. Table 3 below shows a sample of 10 students’ results for six of the scores excluding the FMS (Tweezer) test results.

Table 3 Sample of 10 students scores in clinical skills examination and psychometric pre-tests
Full size table

As can be seen from Table 3 and the other results tables below, the students’ scores varied significantly: e.g. OER score (50–73), GMS score (183–264). Table 4 shows the paired samples statistics for pre- and post- psychometric tests showing the comparison between the mean values of pre and post-psychometric scores using T test statistical analysis. The mean values of the scores of SRT (Fingers), SRT (Tweezers), GMS and BDT tests show students’ improvement in these tests. Only the SRT (Fingers) test score showed that mainly students did not improve in this test after 3 months, the fingers pre-test mean being 25.13 and the post-test mean being 24.55.

Table 4 Paired samples statistics for pre and post psychometric tests
Full size table

The mean value for the pre-GMS-October 2010 test was 198.86 s and for the post-GMS-December 2010 test was 191.49 s. This means that the mean value of GMS’s change was 3.7%. In other words, the students in the post-GMS-Dec2010 test finished the task 3.7% faster than in the pre-GMS-Oct2010 test.

The mean value for pre-BDT-Oct2010 test was 50.67 and for post-BDT-Dec2010 test was 57.15 (see Table 4). This shows an increase of 11.3% in this test. This indicates that using the hapTEL VRS and the conventional Phantom-head systems improved their spatial awareness, hand-eye-finger coordination and depth perception within the period of one term. The results show that there was a general improvement in the FMS (Tweezers) test. The mean value for this test changed from 16.10 to 16.55 (2.8% improvement).

In summary, the mean value comparison between pre- and post-psychometric tests was found to be as follows:

  • 8% increase in SRT scores;

  • 2.3% decrease in the FMS (Fingers) test score;

  • 3.0% increase in the FMS (Tweezer) test scores;

  • 3.7% faster in GMS scores;

  • Approximately 12.8% increase in BDT scores.

Pre and post-GMS and BDT and the FMS (Tweezers) tests showed a strong correlation between each other with high level of confidence (p < 0.01). Pre and post FMS (fingers) showed a low association at a significant level p < 0.05.

Table 5 shows the correlation results between the pre-psychometric scores of the students and their CSE results and Table 6, the results between the post-psychometric scores and the CSE results. These included the additional 20 students who joined in the second year.

Table 5 Correlations between pre-psychometric scores (2010 and CSE results 2012)
Full size table
Table 6 Correlations between post-psychometric scores (2010) and CSE results (2012)
Full size table

Although the results show a statistically strong correlation between pre and post-psychometric scores there is only one significant correlation between the clinical skills’ traditional test results and the gross motor skills (GMS) results. The non-significance of the relationship between the psychometric tests and the OER and PER results however is not unsurprising since these traditional exams assess a much wider range of dental skills than students’ spatial reasoning, fine and gross motor skills and there were 20 additional students results not included in the earlier psychometric tests. However, the increase shown in pre and post-SRT scores (7.99%) is an indication of an improvement in the students’ spatial relation ability and 3D perceptions, which could be due to the training, and practice in both the hapTEL and Phantom-head laboratories.

6 Discussion

In this study all 140 students in Year 1 used both the hapTEL VRS system and the Phantom-head system as part of their first term of the dental programme. The use of both traditional and virtual reality simulation models of training resulted in an improvement in certain psychomotor skills which are essential for students’ clinical practice. These included an improvement in visio-spatial abilities, perceptual awareness, gross and fine motor skills which are fundamental skills required for all health care students. Assessing clinical skills during training, as mentioned above, can sometimes be subjective. These psychometric test results support the earlier study by Moorthy et al. (2003) who argued that without objective feedback, there is difficulty in improving performance, and therefore more use should be made of validated methods such as checklists, rating scales and dexterity analysis systems such as psychometric tests.

The results of this study showed a strong association between pre- and post-test scores of GMS and between pre- and post test scores of the BDT spatial reasoning tests. This shows that the use of both hapTEL and Phantom-head simulator systems improved certain psychomotor skills, spatial awareness and 3D/depth perception over a 3-month period which supports the previous findings by Suksudaj (2010) and Urbankova and Engebretson (2011), that the use of virtual reality simulators can provide a better psychomotor learning through the specific dental programme.

However, there was insignificant correlation between the psychometric results and the traditional clinical skills examination results. This implies that the latter do not reveal the improvement in specific clinical skills which are fundamental to dental skills acquisition.

This study provided an opportunity to measure the psychomotor development of students before and after use of both traditional and advanced haptic systems. According to the theory of skill acquisition (Fitts and Posner 1967; Ackerman 1988) discussed in the background to the study above, the improvements shown in the students’ psychomotor skills means that throughout the practice, these not only progressed from the Cognitive stage to the Associative stage but also they started getting to the Autonomous stage at which the students would usually perform the task without engaging too much of the brain’s cognitive activities. However, further analysis of all the previous years’ evidence and recent new student data (Cox et al. 2015) needs to be completed in order to find out which of these psychometric tests can measure the progression through these stages most reliably.

This study however had several limitations. The numbers of students who took the traditional exams in the 2nd year included those 20 who had not taken the pre- and post-psychometric tests in October 2010. Further analysis of the traditional exams content needs to be conducted in order to single out the questions and tasks direclty relevant to manual dexterity and spatial reasoning skills. These results are currently being analysed and will be reported in a later paper.

7 Conclusions

The combined use of both hapTEL VRS and the conventional Phantom-head systems (although not in the same order for all students) improved spatial awareness, hand-eye-finger coordination and 3D/depth perception within the period of one term. Using psychometric tests can reveal specific skill development amongst students not identified by traditional assessment methods.

These results confirmed the consistency of skill improvement through the three phases of skill acquisition (Ackerman 1988; Fitts and Posner 1967) although more analysis is needed of the specific types of tests which reliably measured these skill phases. The result of this research could therefore inform the development of formative and summative dental clinical skills’ assessment to measure and monitor the student’s psychomotor training with more regular and instant feedback in an objective way using computers along with the traditional Phantom-head mannequin.