Background

E-learning, defined as ‘the application of telecommunications and electronic devices which enable students and learners to receive instruction from some distant location’, has evolved greatly due to the advent of the internet [1]. Today, it is impossible to imagine life without electronic learning, consulting and sharing information. Major advantages of electronic learning are global availability, relatively low cost, options to (unlimited) extensions and regular updates and option to refer to and to link to current textbooks. Case-based learning in an e-learning environment has been proven to be an effective method to practice clinical reasoning [2]. Moreover, current technology allows construction of portals providing interaction between tutor and student/fellow at regional but also (inter)national/global level.

In an effort to combine instruction and formative assessment (i.e. assessment with feedback during the learning process to improve learning) as well as competency-based medical education (CBME), the ESPE launched an initiative to develop an interactive e-learning portal for paediatric endocrinology [3–5]. The ESPE e-learning portal (http://www.espe-elearning.org) can be viewed on a computer and also on a mobile device, such as tablet or smartphone. It provides a rich source of information on various aspects of paediatric endocrinology such as differences of sex development (DSD).

DSD comprise a wide array of congenital conditions in which the development of one or more of the gonads, reproductive structures and genitalia is atypical [6]. Many individuals are recognized in the newborn period when variation in genital development is noted. Later presentations may occur in children or adolescents because of delayed or incomplete pubertal development. Chapters within the e-learning portal concisely describe physiology, pathophysiology and practical approaches to investigation and management. In addition, real-life cases are presented, and the medical student, resident or fellow is invited to solve the diagnostic and management problems in a step-wise and interactive manner [3]. Access to the portal is provided through an automated login procedure via password protection and is global and unlimited.

The most important feature of the portal is the integration of learning, instruction and assessment in the management of patients with paediatric endocrine disorders [3]. According to the CanMEDS framework, ‘competence’ is defined as ‘the minimally expected standard for an individual ready to start independent practice’ [7–9]. Several competencies are highly relevant for optimal patient-centred care. In addition to developing skills as ‘medical expert’, the competencies of ‘communicator’ and ‘collaborator’ are crucial, because informing parents and patients with DSD in a sensitive and appropriate manner is crucial [10, 11].

The further development of learning and competency assessment tools is necessary to extend the impact of the e-learning portal. Assessment of learners fulfils two distinct goals: formative assessment is defined as assessment during the learning process with direct feedback (assessment for learning), whereas summative assessment is a test or exam with a score at the end of a learning process (assessment of learning) [12]. So far, there is little experience in the integration and evaluation of formative assessment in online learning in the education for medical specialists [13].

Aims

This study was intended to evaluate the role of e-learning in the formative assessment of the medical expert competency. We have chosen to focus on a group of fellows and experts representing one specialty, paediatric endocrinology.

An additional objective was to evaluate the development of communication skills in informing patients and parents about diagnosis and management. Special attention was paid to evaluate the value of personalized feedback and the feedback by different experts. A qualitative analysis of the communication skills such as wording and empathy will be discussed in more detail in a separate paper.

The third aim was to assess religious, regional and cultural aspects relevant to management of DSD by performing the study on a global level. An analysis of the presence of these aspects in the communication by fellows will also be part of a separate study.

Study Outline

Senior paediatric endocrinologists (experts) were approached and asked to provide names and e-mail addresses of paediatricians undergoing subspecialty training in paediatric endocrinology (referred to hereafter as fellows) in their respective departments who agreed to being approached to participate in the assessment study. The fellows were invited to login into a special e-learning course within the ESPE e-learning portal, where they had access to the general content of the system, including chapters on DSD.

Access was provided to two interactive cases on the subject of DSD prepared as e-learning exercises in English. The two clinical cases are within the expertise of paediatric endocrinologists, and it is likely these cases would be referred to these specialists. The study also addresses the role of the paediatric endocrinologist in an interdisciplinary DSD team.

Case 1, entitled ‘A newborn with ambiguous genitalia’, focused on the stepwise approach to investigation and management of a newborn referred by a midwife because of atypical genital development due to congenital adrenal hyperplasia. Case 2, entitled ‘Ranya: an adolescent girl with delayed menarche’, addressed the approach to a 17-year-old girl with delayed menarche as a result of 46, XY gonadal dysgenesis. The format of these cases includes a stepwise clinical approach to the assessment of a patient with DSD, followed by multiple-choice (MC) questions testing medical knowledge (medical expert competency). The correct answer to each MC question is provided to the learner directly after a correct answer and if incorrect after the second attempt to answer it. At crucial stages of the problem-solving process, three open-ended questions (i.e. free text response) were included for each case, to which the fellow was invited to describe how (s)he would handle the situation, with particular focus on how the fellow would inform the parents and/or patient of the key information regarding diagnosis, implications and management. The open questions from both cases are presented in Table 1.

Table 1 The three open questions presented to the fellows for each case
Full size table

The replies to the MC questions were stored centrally for further analysis, and the replies to the open-ended questions were submitted to two to three experts from various parts of the world (excluding the expert at the institution where the fellow was employed); at least one of the experts was chosen to be from the same geographic region as the fellow. The experts were invited to reply with a brief formative commentary to the fellows’ responses. They were free to use an example answer prepared by the study team (see supplementary Table 1). The fellows were asked to acknowledge and reply to the comments of the experts. All these procedures were handled anonymously through the e-learning portal.

To evaluate the responses of the fellows as objectively as possible, several key items were identified a priori by the study team as critical elements of an adequate response. The set of response elements (case 1, question 1 N = 7; case 1, question 2 N = 8; case 2, question 1 N = 6) prepared by the study team was submitted to the group of experts with the request to indicate at least five key items that, in their opinion, should be mentioned in the answer by the fellow. From their responses, a set of key items was derived, which was used to evaluate the answers of the fellow by scoring the number of key items mentioned in their response.

Finally, the fellows were invited to complete a brief web-based questionnaire regarding feasibility/receptivity and value of personalized feedback related to competence development.

Statistical Methods

The time fellows were logged in to the portal was registered automatically. Time spent was analysed by taking the median, as the distribution was skewed. For the calculation of the median, outliers with a z-value greater than 2.68 were identified and eliminated. To compare the time spent in different parts of the world, the time spent was represented in a box-and-whisker plot for both cases. Whiskers represent minimum and maximum values; boxes represent first and third quartiles and the median; outliers with an absolute z-value greater than 2.68 were excluded (1 for case 1 and 2 for case 2). Apart from the registration of the time logged in to the portal, fellows were asked in the web-based questionnaire to indicate the time range they spent on both cases and how much time they spent on answering the questions and discussion with the expert for each separate case. The median time range was selected as being most illustrative.

For the analysis of the percentage of correct answers for multiple-choice questions, the percentage of correct answers on the first try was calculated. For the analysis of the open questions, the number of key items that was discussed correctly in the reply was scored. A percentage was calculated of number of key items mentioned versus the total number of key items.

Results

Fifty-four experts (paediatric endocrinologists) from various parts of the world (Africa, Asia, Australia, Europe, North America, South America) were invited to participate; 41 agreed and 32 ultimately contributed. Of the 77 fellows proposed by the experts, 68 fellows participated from the following countries: Africa (5), Asia (12), Australia (4), Europe (27), North America (10) and South America (10). The global distribution of participating experts and fellows is provided in Table 2.

Table 2 Distribution of the experts and fellows by continent
Full size table

Sixty-seven of the participating fellows answered at least one of the questions for case 1; 62 fellows answered all MC questions for case 1. The median time they were logged into the portal for case 1 was 35 min, standard deviation 26 min. Sixty of the participating fellows answered at least 1 question for case 2; 58 completed all MC questions for case 2. The median time they were logged into the portal was 21 min, standard deviation 14 min. Fifty fellows answered all the open questions and MC questions for both cases. Time spent logged into the e-learning portal evaluating the cases and answering questions online is presented in Fig. 1. The time registered by the portal system may not be a full representation of the time spent by the fellows as they may have worked additionally offline on the cases.

Fig. 1
figure 1

a, b The time (in minutes) spent at the e-learning portal by fellows studying and responding to questions for case 1 (a) and for case 2 (b) is shown in box-and-whisker plots according to the continent in which the fellows worked. Whiskers represent minimum and maximum values; boxes represent first and third quartiles and the median; outliers with an absolute z-value greater than 2.68 were excluded (1 for case 1 and 2 for case 2)

Full size image

MC Questions

The percentage of correct answers on the first attempt is given for each MC question for case 1 and case 2 (Fig. 2). Notably, fewer than 50 % of fellows achieved the correct answer on the first attempt for 16 of the 20 questions (case 1 and 2 combined); particularly low scores were observed for the questions relating to medical history (case 1: questions 1, 3, 6), differential diagnosis (case 2: questions 3, 5) and management (case 1: questions 11,13,14).

Fig. 2
figure 2

a, b Percentage fellows who gave the correct answer to the multiple-choice question on the first attempt. Each question is listed by its number on the horizontal axis, a for case 1 and b for case 2. Note: Case 1, question15, had no correct response due to a technical error

Full size image

Open Questions and Key Items

Sixteen experts responded to the request to identify at least five key elements needed for an adequate response to the open questions. Table 3 provides an overview of the key items and the number of experts naming these response elements preidentified by the study team for each open question. While the great majority of experts noted the importance of having a diagnostic plan for both cases, only about half identified sex assignment for the newborn with congenital adrenal hyperplasia or discussion of normal and variant sex development as key items, and fewer than 40 % indicated that follow-up was an important facet of the response for the young woman with gonadal dysgenesis.

Table 3 Key items mentioned by 16 experts and 50 fellows for questions 1 and 2 of case 1 and question 1 of case 2
Full size table

Fifty fellows responded to the open questions for case 1 and 45 fellows responded to the open questions for case 2. The number of key items mentioned in the answers of the fellows to questions 1 and 2 of case 1 and question 1 of case 2 was scored by three members of the study team, resulting in a total score per open question. Table 3 presents the number and percentage of individual fellows naming these key items. Like the experts, the great majority of fellows identified the importance of a diagnostic plan, but few mentioned the concept of follow up. Table 4 presents the mean number of key items identified by fellows, which was less than 50 % of the key elements for case 1, questions 1 and 2 and just over 50 % for case 2, question 3. Finally, Fig. 3 gives an overview of the percentage of correct answers for both MC and open questions per continent.

Table 4 Mean (SD) number of key items mentioned by fellows compared to total number of key items
Full size table
Fig. 3
figure 3

Percentage of correct answers on multiple choice and total score (percentage of maximum score) on open questions for each case, by continent

Full size image

Personalized Feedback by Multiple Experts

The fellows received personalized feedback to the open questions from two to three experts each, one being from the country or region of the fellow and one or two from another country or continent. The experts were free to use a set of response elements prepared by the study team (see supplementary Table 1). The comments by the experts varied in terms of extent of the reply and personal input (“in my experience……..”), but no regional differences could be established (data not shown).

Subsequently, the fellows were asked to acknowledge the feedback of the experts. A summary is given of the type of acknowledgement provided by 40 fellows (Table 5).

Table 5 Categorization of the acknowledgement of 40 fellows to the feedback provided by the experts
Full size table

Examples of positive responses were as follows: ‘I will definitely use some of the above examples and phrases in clinical practice’ and ‘Feedback is very relevant. It gives a few points which I did not think about (like religion, informing family) and also gives useful information about the different things you have to say to the parents. The text with the “basic” explanation is very useful and gives a structure for starting the conversation (very practical)’; ‘Nice to have good comment and to hear that the expert will say more or less the same’; ‘The suggested text about the mechanism of CAH is amazing, very didactical and clear and very able tool to explain parents about this disease. The comparison with the big river and smaller creeks is very efficient to make parents understand what happens in this disease. Thank you so much for your positive feedback’.

There was only one negative reaction, stating ‘All in all. unfortunately not very useful, relevant or helpful comments.’

It was notable that 19 % of the fellows mentioned difficulties in understanding the assignment requested in question 3 for both cases, in which the fellow was asked to write a letter to the patient at the age of 17 years to explain her condition. Apparently, this question was not sufficiently clear, as most of the fellows wrote a transition letter to a hospital-based endocrinologist to arrange follow-up of care.

Fellows’ Evaluation

Twenty of 68 participating fellows answered our web-based online demographic survey. Most fellows were in training, 4 participants had longer experience (5, 7, 9 and 12 years). Global distribution was as follows: Africa (2), Asia (5), Australia (2), Europe (6), North America (4) and South America (1). Time spent was evaluated by 6 time ranges (0–15; 15–30; 30–60; 60–90; 90–120 min or other). The median time range for reading both cases and answering the questions was 90–120 min (selected by 8 fellows). Four fellows indicated that they spent much more time, varying from 180 min to 2 days. The median time range for studying case 1, including the answering and discussion with the expert, was between 30 and 60 min and between 60 and 90 min. Four fellows indicated that they needed much more time, up to 1 day. The median time range for studying case 2 was also between 30 and 60 min and between 60 and 90 min, with 3 fellows indicating that they needed much more time. Notably, these reported times are more than double the mean times spent working on the cases online, as registered by the e-learning portal, indicating that fellows undertook additional offline research and study.

Finally, the participating fellows were asked whether they thought it was useful to receive feedback from the experts. All 20 fellows answered positively. For example, one fellow stated ‘Yes. When all experts provided similar feedback, it really sent a message that this feedback was important. It was also useful to receive varying feedback from people with different experiences. Some experts provided very detailed answers which will be helpful in my approach to patients in future’. Another interesting response was ‘It is very useful to receive feedback from various experts because each expert has his one way in dealing with the problem. And, when different experts give their opinion, you can use a little bit from every answer. Also, some experts find that certain part of the explanation has to be better while some find it good, because of the difference in personal interest’.

Cultural, Religious and Regional Aspects

Experts were asked to indicate cultural, religious and regional items of relevance in the answers to the open questions. Twelve of 32 experts replied to this request and mentioned these aspects summarized in Table 6.

Table 6 Regional, cultural and religious aspects of DSD management reported by 12 experts
Full size table

The most frequently mentioned items were fertility and the role of the family. In some cultures, the girl will visit the doctor only with the mother without involvement of the father. In other cultures, the family will demand that the patient is not addressed directly—at least not until she is 18 years old. The possibility for delayed sex registration differs per country. Sometimes, sex assignment can be postponed, but this may be unacceptable to families. Several experts expressed a strong preference for male sex assignment for instance in 46, XX newborns with congenital adrenal hyperplasia with severe virilization. Some experts mentioned that possibilities for follow-up are limited for socioeconomic reasons or due to lack of confidence in the healthcare system.

Discussion

In this exploratory study, we evaluated the applicability and feasibility at a global level of the ESPE e-learning portal in assessing junior paediatricians (fellows) during their subspecialty paediatric endocrinology training, regarding knowledge and skills for investigation and management of children and adolescents with DSD. In general, learning in practice is regarded as the most powerful way of learning. However, even in major referral centres, the number of patients with DSD is small, thus exposure to the clinical management of newly referred patients is very limited. Therefore, there is also a strong need for alternative learning methods to train the necessary competencies for a medical specialist as described by the CanMEDS framework [8, 9]. E-learning is increasingly being used to train competencies such as ‘scholar’ and medical expert [7]. The responses to the MC questions provided some insight into the medical expert competency. Although not formally validated, these questions refer to current textbook knowledge. The baseline knowledge with respect to relevant history, differential diagnosis and management of the participating fellows was notably limited judging on the score of the MC questions. During this study, knowledge is gathered by formative questions in the e-learning and by feedback provided by the experts. Concluding from the acknowledgements from the fellows and from the online questionnaire amongst the fellows, this leads to extended skill of medical expert. This argues in favour of the integration of formative learning by e-learning in the training for paediatric endocrinologist.

To our knowledge, the use of e-learning to train other competencies such as communicator and collaborator is not common practice. This study shows, however, that e-learning with personal feedback offers possibilities to train these competencies. In principal, the portal system allows direct interaction between fellow and tutor/expert. As the emphasis is on formative assessment, the tutor is in a position to correct any misunderstanding of the assignment or to request further details. The management of DSD requires an interdisciplinary approach, and in this study, we focused on the role of the paediatric endocrinologist in the team. It is of note that many key items in the open questions considered by the experts to be an essential part of the set of response elements were not mentioned by the fellows. This can be explained by the low exposure to such complex cases in practice due to the limited number of patients, highlighting the potential role for and value of e-learning.

The results of our study support the use of e-learning for formative assessment, providing direct feedback (assessment for learning). The anonymity between the fellow and faculty may have a positive aspect. Fellows may be anxious or uncomfortable discussing patients with their mentors and other senior faculty members. This study allowed them to write their personal thoughts and opinions without the anxiety of presenting to a respected faculty mentor. Indeed, the personalized comments by the experts proved to be very formative, as the fellows indicated that they learned a great deal from the personalized feedback. They were confronted with certain aspects that they were not aware of, such as religious and cultural aspects and addressing sensitive issues. Moreover, it was gratifying to note that most fellows in the subsequent acknowledgement expressed their appreciation. When feedback is provided in a summative context, for instance as a test or exam with a pass/fail score (assessment of learning), it is not always used effectively by learners, which pleads for a formative context [12, 14].

The time spent to study the assigned clinical cases did vary. Fellows from South America, Africa and Asia took more time for both cases than did fellows from other continents, presumably because of the cases and questions were presented in English, which would not be the native language of those fellows.

This study aimed at demonstrating the applicability of e-learning at a global level and identifying regional and cultural differences in medical practice. This was done by an inventory under the participating experts by identifying relevant cultural, religious and regional items. On the other hand, replies of the fellows were analysed per continent, which showed no great and significant differences. In the inventory amongst experts, the most frequently mentioned item was fertility and the role of the family. In some cultures, fertility is extremely important, as it affects ‘marriageability’ for women. Egg cell donation may not be allowed and adoption might be uncommon [15–17]. A religious official may play a crucial role in management in terms of decision making and with respect to openness on informing family and friends [16, 17]. Options regarding genital surgery involving gonadectomy, hysterectomy or fertility preservation procedures may be subject to religious recommendations [18]. In some countries, management of DSD is seriously hampered by lack of access to essential medications such as hydrocortisone and to specialized (paediatric) urosurgical expertise [15, 19]. Furthermore, recommendations for genetic counselling and transition of care to an adult endocrinologist or gynaecologist are lacking. Several experts mentioned the association with transgenderism or homosexuality, which in some cultural/religious background is condemned [16, 17]. It is of interest to note that the fellows in their first reply to the open questions did not mention cultural, religious or regional aspects. However, some acknowledged in their second reply to the expert that religious considerations are important. From the acknowledgements given by the fellows, we conclude that they achieved new insights on these aspects for future use.

As online and blended learning has become common place educational strategy in higher education, educators need to reconceptualise fundamental issues of teaching, learning and assessment in non-traditional spaces. Effective online formative assessment can lead to valuable learning experiences through formative feedback and enhanced learner engagement [13].

Standards for the formative effects of e-learning have been developed aimed at medical professionals who have graduated from formal training and residency or used by medical professionals as part of continuous learning to maintain their competency and develop new knowledge [20, 21]. So far, there is little experience in the integration and evaluation of formative assessment in online learning in the education for medical specialists. We have conducted an exploratory study to evaluate the feasibility of the integration of online formative feedback in the post-doc training of paediatric endocrinologists. From the lessons learned, future studies can be developed for other medical specialties.

As a limitation of the study, it is acknowledged that English is not the native language of many participating fellows, which may have influenced their understanding of the questions and formulation of their replies. It is remarkable that fellows and experts worldwide showed great enthusiasm and have put much time and effort. Fellows indicate that this way of learning and, in specific, the personalized feedback received by multiple experts, is very valuable. They were appreciative of the fact that experts from different continents responded to their replies to the open questions. Although preformatted reply text was supplied by the study team, the participating experts indicated that their careful reply to the fellows’ answers to the open questions was very time consuming.

A future development could be to assemble electronic meetings to allow direct personalized feedback via live discussion. We also foresee that a regional or national use of the e-learning portal could involve direct interaction between fellow (student) and expert (tutor).

In a separate study, the approach and the wording chosen by the fellow informing parents of a newborn with DSD and an adolescent girl with DSD will be assessed in detail.

Conclusion

Conversely, despite a worldwide movement in medical education to shift away from standardized instruction, learning and assessment toward individualization, here, we present a study designed to incorporate multiple continents and nations. However, this method preserves the individualisation of learning and offers direct, qualitative and formative input from ‘teachers’. Our method acknowledges that both the study of medical science and guidelines for clinical management are increasingly international. Also, internet-based information exchange means that parents and patients share professionals’ awareness of international medical care standards.

Fellows and experts highly value the integration of formative assessment as part of online learning in the training of clinical competencies (such as medical expert, communicator and collaborator). The results of this study indicate that at fellow level, there can be limited knowledge with regard to the management of newborns and adolescents with DSD. E-learning with formative feedback by experts is not only appreciated by fellow subspecialists but also may have an important role in their clinical training.

An appreciation of regional, religious and cultural aspects of DSD care was highlighted as an important factor in the management of DSD as exemplified in this e-learning. These aspects were not initially mentioned by the fellows. However, by the acknowledgements, we conclude that due to the feedback of the experts, fellows achieved new insights and will use this in future communication to patients and parents.

In conclusion, in this exploratory study, we demonstrated a global application of the ESPE e-learning portal. We found it useful and valued for the training and assessment of junior paediatricians during their subspecialty training in paediatric endocrinology regarding knowledge and skills for investigation and management of children and adolescents with DSD. We also foresee possibilities for the training of other medical specialties.