Medical Science Educator

, Volume 26, Issue 4, pp 679–689 | Cite as

Global Application of Assessment of Competencies of Paediatric Endocrinology Fellows in the Management of Differences of Sex Development (DSD) Using the ESPE Portal

  • Laura Kranenburg
  • Sam T. H. Reerds
  • Martine Cools
  • Julie Alderson
  • Miriam Muscarella
  • Kalinka Grijpink
  • Charmian A. Quigley
  • Stenvert L. S. Drop
Open Access
Original Research



Using written submission with formative feedback as a teaching method, the European Society for Paediatric Endocrinology (ESPE) developed an interactive e-learning portal for predoctoral and postdoctoral education and training: ( The aim of the study was to evaluate the role of e-learning in the assessment of competencies of paediatric endocrinology fellows. Competencies included as medical knowledge communication skills and knowing what information to give to patients and parents with a difference of sex development about diagnosis and management.

Materials and Method

Sixty-eight fellows and 32 experts from various countries participated in the study. Fellows answered online multiple-choice and open questions relating to two clinical interactive problem solving cases (case 1: a new-born with congenital adrenal hyperplasia; case 2: a 17-year-old girl with gonadal dysgenesis). Experts provided anonymous feedback online, using responses guidance (standard questions for eliciting a uniform consideration of competence).


Fifty percent of fellows answered 16 out of 20 questions correctly on their first attempt at questions about the clinical case scenarios. However, experts judged that only 50 % of the necessary elements of questions 1 and 2 (case 1) and question 2 (case 2) were given in these correct answers. The personalized formative feedback to the open questions provided by the two to three experts ‘marking’ each submission was highly valued by participating fellows. The most frequently cited items relating to cultural, religious and regional aspects of differences of sex development (DSD) management were relating fertility and role of the family. These were listed by 12/32 experts. Fellows did not mention them in their first reply to the open questions but acknowledged their importance in their second reply.


The ESPE e-learning portal, offering direct interaction between fellow and trainer, is highly valued by both parties, as helpful in developing medical competencies and is therefore a feasible tool for distance learning.


On line learning E-learning Subspecialty training DSD 


E-learning, defined as ‘the application of telecommunications and electronic devices which enable students and learners to receive instruction from some distant location’, has evolved greatly due to the advent of the internet [1]. Today, it is impossible to imagine life without electronic learning, consulting and sharing information. Major advantages of electronic learning are global availability, relatively low cost, options to (unlimited) extensions and regular updates and option to refer to and to link to current textbooks. Case-based learning in an e-learning environment has been proven to be an effective method to practice clinical reasoning [2]. Moreover, current technology allows construction of portals providing interaction between tutor and student/fellow at regional but also (inter)national/global level.

In an effort to combine instruction and formative assessment (i.e. assessment with feedback during the learning process to improve learning) as well as competency-based medical education (CBME), the ESPE launched an initiative to develop an interactive e-learning portal for paediatric endocrinology [3, 4, 5]. The ESPE e-learning portal ( can be viewed on a computer and also on a mobile device, such as tablet or smartphone. It provides a rich source of information on various aspects of paediatric endocrinology such as differences of sex development (DSD).

DSD comprise a wide array of congenital conditions in which the development of one or more of the gonads, reproductive structures and genitalia is atypical [6]. Many individuals are recognized in the newborn period when variation in genital development is noted. Later presentations may occur in children or adolescents because of delayed or incomplete pubertal development. Chapters within the e-learning portal concisely describe physiology, pathophysiology and practical approaches to investigation and management. In addition, real-life cases are presented, and the medical student, resident or fellow is invited to solve the diagnostic and management problems in a step-wise and interactive manner [3]. Access to the portal is provided through an automated login procedure via password protection and is global and unlimited.

The most important feature of the portal is the integration of learning, instruction and assessment in the management of patients with paediatric endocrine disorders [3]. According to the CanMEDS framework, ‘competence’ is defined as ‘the minimally expected standard for an individual ready to start independent practice’ [7, 8, 9]. Several competencies are highly relevant for optimal patient-centred care. In addition to developing skills as ‘medical expert’, the competencies of ‘communicator’ and ‘collaborator’ are crucial, because informing parents and patients with DSD in a sensitive and appropriate manner is crucial [10, 11].

The further development of learning and competency assessment tools is necessary to extend the impact of the e-learning portal. Assessment of learners fulfils two distinct goals: formative assessment is defined as assessment during the learning process with direct feedback (assessment for learning), whereas summative assessment is a test or exam with a score at the end of a learning process (assessment of learning) [12]. So far, there is little experience in the integration and evaluation of formative assessment in online learning in the education for medical specialists [13].


This study was intended to evaluate the role of e-learning in the formative assessment of the medical expert competency. We have chosen to focus on a group of fellows and experts representing one specialty, paediatric endocrinology.

An additional objective was to evaluate the development of communication skills in informing patients and parents about diagnosis and management. Special attention was paid to evaluate the value of personalized feedback and the feedback by different experts. A qualitative analysis of the communication skills such as wording and empathy will be discussed in more detail in a separate paper.

The third aim was to assess religious, regional and cultural aspects relevant to management of DSD by performing the study on a global level. An analysis of the presence of these aspects in the communication by fellows will also be part of a separate study.

Study Outline

Senior paediatric endocrinologists (experts) were approached and asked to provide names and e-mail addresses of paediatricians undergoing subspecialty training in paediatric endocrinology (referred to hereafter as fellows) in their respective departments who agreed to being approached to participate in the assessment study. The fellows were invited to login into a special e-learning course within the ESPE e-learning portal, where they had access to the general content of the system, including chapters on DSD.

Access was provided to two interactive cases on the subject of DSD prepared as e-learning exercises in English. The two clinical cases are within the expertise of paediatric endocrinologists, and it is likely these cases would be referred to these specialists. The study also addresses the role of the paediatric endocrinologist in an interdisciplinary DSD team.

Case 1, entitled ‘A newborn with ambiguous genitalia’, focused on the stepwise approach to investigation and management of a newborn referred by a midwife because of atypical genital development due to congenital adrenal hyperplasia. Case 2, entitled ‘Ranya: an adolescent girl with delayed menarche’, addressed the approach to a 17-year-old girl with delayed menarche as a result of 46, XY gonadal dysgenesis. The format of these cases includes a stepwise clinical approach to the assessment of a patient with DSD, followed by multiple-choice (MC) questions testing medical knowledge (medical expert competency). The correct answer to each MC question is provided to the learner directly after a correct answer and if incorrect after the second attempt to answer it. At crucial stages of the problem-solving process, three open-ended questions (i.e. free text response) were included for each case, to which the fellow was invited to describe how (s)he would handle the situation, with particular focus on how the fellow would inform the parents and/or patient of the key information regarding diagnosis, implications and management. The open questions from both cases are presented in Table 1.
Table 1

The three open questions presented to the fellows for each case



Question text

1. A newborn with ambiguous genitalia


You are the paediatric endocrinology fellow in charge; you have obtained the medical history information and you have examined the baby. What do you tell the parents at this stage?


You have just made the diagnosis of congenital adrenal hyperplasia (CAH), most likely as a result of 21α-hydroxylase deficiency. Give a detailed account of how you will inform the parents of the diagnosis.


Many years later, when Eve is 17 years of age, you prepare a letter at the time of transition of her medical care to an internist-endocrinologist and gynaecologist. You provide her with a written summary of her medical history.

Specifically you explain once more in understandable terms:

a. what kind of condition she has

b. what happened before she was born

c. why she had surgery

d. why she is being referred to an internist-endocrinologist / gynaecologist for ongoing care

e. whether she is required to take medication all her life

f. whether she will be able to bear children in the typical way and if so, the risk is that one of her children will also be affected

Prepare this letter.

2. Ranya, an adolescent girl with delayed menarche


You have made the diagnosis 46,XY gonadal dysgenesis.

Give a detailed account of how you will arrange the sharing of information to Ranya and her parents.

Consider the following elements:

the setting; allotted time; key elements to be discussed regarding the diagnosis; relevant therapeutic issues; long-term implications; follow up; other considerations


Give a detailed account of how you will inform Ranya and her parents of the increased risk for development of a gonadal germ cell tumour and the therapeutic options.

Consider the following elements: relative risk; time frame; therapeutic procedures and timing; follow up; other considerations.


Ranya is now 19 years of age. Hormone replacement with oestrogen and progesterone is resulting in regular withdrawal bleedings. A laparoscopic bilateral gonadectomy was performed at age 18, revealing the presence of a streak gonad on the left side and a streak with gonadoblastoma on the right side. The excision of the gonadoblastoma was complete and no further treatment was indicated.

Some months after surgery, Ranya sends you an email message stating that she has a boyfriend and she asks you to summarize in understandable terms what kind of condition she has.

Specifically, she asks you to explain:

a. what kind of condition do I have?

b. what is the cause?

c. why did I have to undergo surgery?.

d. Do I have to take medication all my life? Are there limitations to be expected regarding my general health?

f. Other considerations

The replies to the MC questions were stored centrally for further analysis, and the replies to the open-ended questions were submitted to two to three experts from various parts of the world (excluding the expert at the institution where the fellow was employed); at least one of the experts was chosen to be from the same geographic region as the fellow. The experts were invited to reply with a brief formative commentary to the fellows’ responses. They were free to use an example answer prepared by the study team (see supplementary Table 1). The fellows were asked to acknowledge and reply to the comments of the experts. All these procedures were handled anonymously through the e-learning portal.

To evaluate the responses of the fellows as objectively as possible, several key items were identified a priori by the study team as critical elements of an adequate response. The set of response elements (case 1, question 1 N = 7; case 1, question 2 N = 8; case 2, question 1 N = 6) prepared by the study team was submitted to the group of experts with the request to indicate at least five key items that, in their opinion, should be mentioned in the answer by the fellow. From their responses, a set of key items was derived, which was used to evaluate the answers of the fellow by scoring the number of key items mentioned in their response.

Finally, the fellows were invited to complete a brief web-based questionnaire regarding feasibility/receptivity and value of personalized feedback related to competence development.

Statistical Methods

The time fellows were logged in to the portal was registered automatically. Time spent was analysed by taking the median, as the distribution was skewed. For the calculation of the median, outliers with a z-value greater than 2.68 were identified and eliminated. To compare the time spent in different parts of the world, the time spent was represented in a box-and-whisker plot for both cases. Whiskers represent minimum and maximum values; boxes represent first and third quartiles and the median; outliers with an absolute z-value greater than 2.68 were excluded (1 for case 1 and 2 for case 2). Apart from the registration of the time logged in to the portal, fellows were asked in the web-based questionnaire to indicate the time range they spent on both cases and how much time they spent on answering the questions and discussion with the expert for each separate case. The median time range was selected as being most illustrative.

For the analysis of the percentage of correct answers for multiple-choice questions, the percentage of correct answers on the first try was calculated. For the analysis of the open questions, the number of key items that was discussed correctly in the reply was scored. A percentage was calculated of number of key items mentioned versus the total number of key items.


Fifty-four experts (paediatric endocrinologists) from various parts of the world (Africa, Asia, Australia, Europe, North America, South America) were invited to participate; 41 agreed and 32 ultimately contributed. Of the 77 fellows proposed by the experts, 68 fellows participated from the following countries: Africa (5), Asia (12), Australia (4), Europe (27), North America (10) and South America (10). The global distribution of participating experts and fellows is provided in Table 2.
Table 2

Distribution of the experts and fellows by continent






North America

South America

Experts (N = 32)







Fellows (N = 68)







Sixty-seven of the participating fellows answered at least one of the questions for case 1; 62 fellows answered all MC questions for case 1. The median time they were logged into the portal for case 1 was 35 min, standard deviation 26 min. Sixty of the participating fellows answered at least 1 question for case 2; 58 completed all MC questions for case 2. The median time they were logged into the portal was 21 min, standard deviation 14 min. Fifty fellows answered all the open questions and MC questions for both cases. Time spent logged into the e-learning portal evaluating the cases and answering questions online is presented in Fig. 1. The time registered by the portal system may not be a full representation of the time spent by the fellows as they may have worked additionally offline on the cases.
Fig. 1

a, b The time (in minutes) spent at the e-learning portal by fellows studying and responding to questions for case 1 (a) and for case 2 (b) is shown in box-and-whisker plots according to the continent in which the fellows worked. Whiskers represent minimum and maximum values; boxes represent first and third quartiles and the median; outliers with an absolute z-value greater than 2.68 were excluded (1 for case 1 and 2 for case 2)

MC Questions

The percentage of correct answers on the first attempt is given for each MC question for case 1 and case 2 (Fig. 2). Notably, fewer than 50 % of fellows achieved the correct answer on the first attempt for 16 of the 20 questions (case 1 and 2 combined); particularly low scores were observed for the questions relating to medical history (case 1: questions 1, 3, 6), differential diagnosis (case 2: questions 3, 5) and management (case 1: questions 11,13,14).
Fig. 2

a, b Percentage fellows who gave the correct answer to the multiple-choice question on the first attempt. Each question is listed by its number on the horizontal axis, a for case 1 and b for case 2. Note: Case 1, question15, had no correct response due to a technical error

Open Questions and Key Items

Sixteen experts responded to the request to identify at least five key elements needed for an adequate response to the open questions. Table 3 provides an overview of the key items and the number of experts naming these response elements preidentified by the study team for each open question. While the great majority of experts noted the importance of having a diagnostic plan for both cases, only about half identified sex assignment for the newborn with congenital adrenal hyperplasia or discussion of normal and variant sex development as key items, and fewer than 40 % indicated that follow-up was an important facet of the response for the young woman with gonadal dysgenesis.
Table 3

Key items mentioned by 16 experts and 50 fellows for questions 1 and 2 of case 1 and question 1 of case 2


Key items identified a priori by study team

Number of experts (% of 16 experts)

Number of fellows (% of total 50 [case 1] and 45 [case 2])

Case 1 question 1

1. Diagnostic plan

13 (81)

40 (80)

2. Setting

11 (69)

36 (72)

3. Reassurance

11 (69)

26 (52)

4. Informing family and friends

11 (69)

13 (26)

5. Sex assignment

9 (56)

6 (12)

6. Explanation of normal and variant sexual development

8 (50)

11 (22)

7. DSD team and follow up

6 (38)

10 (20)

Case 1 question 2

1. Diagnostic plan

13 (81)

45 (90)

2. Therapeutic options

11 (69)

34 (68)

3. Follow up

9 (56)

9 (18)

4. Long term implications

9 (56)

8 (16)

5. Option of surgery

9 (56)

23 (46)

Case 2 question 1

1. Explanation of diagnosis

15 (94)

35 (77)

2. Long term implications

12 (75)

15 (33)

3. Therapeutic options

13 (81)

28 (62)

4. Setting

10 (63)

36 (80)

5. Gonadal tumour risk

11 (69)

20 (44)

6. Follow up

6 (38)

14 (31)

The total number and percentage of experts and fellows naming the item is given

Fifty fellows responded to the open questions for case 1 and 45 fellows responded to the open questions for case 2. The number of key items mentioned in the answers of the fellows to questions 1 and 2 of case 1 and question 1 of case 2 was scored by three members of the study team, resulting in a total score per open question. Table 3 presents the number and percentage of individual fellows naming these key items. Like the experts, the great majority of fellows identified the importance of a diagnostic plan, but few mentioned the concept of follow up. Table 4 presents the mean number of key items identified by fellows, which was less than 50 % of the key elements for case 1, questions 1 and 2 and just over 50 % for case 2, question 3. Finally, Fig. 3 gives an overview of the percentage of correct answers for both MC and open questions per continent.
Table 4

Mean (SD) number of key items mentioned by fellows compared to total number of key items


Total number of key items

Mean (SD) number of key items mentioned

Percentage correct

Case 1, question 1


2.8 (1.4)

40 %

Case 1, question 2


2.3 (1.2)

46 %

Case 2, question 1


3.4 (1.5)

57 %

Fig. 3

Percentage of correct answers on multiple choice and total score (percentage of maximum score) on open questions for each case, by continent

Personalized Feedback by Multiple Experts

The fellows received personalized feedback to the open questions from two to three experts each, one being from the country or region of the fellow and one or two from another country or continent. The experts were free to use a set of response elements prepared by the study team (see supplementary Table 1). The comments by the experts varied in terms of extent of the reply and personal input (“in my experience……..”), but no regional differences could be established (data not shown).

Subsequently, the fellows were asked to acknowledge the feedback of the experts. A summary is given of the type of acknowledgement provided by 40 fellows (Table 5).
Table 5

Categorization of the acknowledgement of 40 fellows to the feedback provided by the experts

Type of acknowledgement

Case 1 question 1

Case 1 question 2

Case 1 question 3

Case 2 question 1

Case 2 question 2

Case 2 question 3

Total number of acknowledgements







A = thank you for suggestions; it is useful

54 %

43 %

38 %

52 %

53 %

56 %

A/B = thanks/ little exchange of opinion

6 %

5 %

4 %

2 %

3 %

2 %

B = exchange of opinion regarding content

2 %

5 %

6 %

4 %

9 %

3 %

C = will follow your suggestion in the future

19 %

26 %

13 %

21 %

12 %

13 %

D = I agree with your suggestions

9 %

16 %

8 %

13 %

11 %

9 %

H = it is nice that the expert agrees with me

0 %

0 %

1 %

1 %

0 %

1 %

E = need (first) feedback from my superior/chief resident

1 %

0 %

0 %

0 %

0 %

0 %

F = had difficulty understanding the question

2 %

1 %

19 %

2 %

1 %

5 %

G = not useful/helpful/ relevant comments

1 %

0 %

0 %

2 %

1 %

2 %

I = No acknowledgement

6 %

4 %

11 %

3 %

10 %

9 %

Note that the total number of acknowledgements is greater than the total number of fellows because fellows responded to feedback from two to three experts per question

Examples of positive responses were as follows: ‘I will definitely use some of the above examples and phrases in clinical practice’ and ‘Feedback is very relevant. It gives a few points which I did not think about (like religion, informing family) and also gives useful information about the different things you have to say to the parents. The text with the “basic” explanation is very useful and gives a structure for starting the conversation (very practical)’; ‘Nice to have good comment and to hear that the expert will say more or less the same’; ‘The suggested text about the mechanism of CAH is amazing, very didactical and clear and very able tool to explain parents about this disease. The comparison with the big river and smaller creeks is very efficient to make parents understand what happens in this disease. Thank you so much for your positive feedback’.

There was only one negative reaction, stating ‘All in all. unfortunately not very useful, relevant or helpful comments.’

It was notable that 19 % of the fellows mentioned difficulties in understanding the assignment requested in question 3 for both cases, in which the fellow was asked to write a letter to the patient at the age of 17 years to explain her condition. Apparently, this question was not sufficiently clear, as most of the fellows wrote a transition letter to a hospital-based endocrinologist to arrange follow-up of care.

Fellows’ Evaluation

Twenty of 68 participating fellows answered our web-based online demographic survey. Most fellows were in training, 4 participants had longer experience (5, 7, 9 and 12 years). Global distribution was as follows: Africa (2), Asia (5), Australia (2), Europe (6), North America (4) and South America (1). Time spent was evaluated by 6 time ranges (0–15; 15–30; 30–60; 60–90; 90–120 min or other). The median time range for reading both cases and answering the questions was 90–120 min (selected by 8 fellows). Four fellows indicated that they spent much more time, varying from 180 min to 2 days. The median time range for studying case 1, including the answering and discussion with the expert, was between 30 and 60 min and between 60 and 90 min. Four fellows indicated that they needed much more time, up to 1 day. The median time range for studying case 2 was also between 30 and 60 min and between 60 and 90 min, with 3 fellows indicating that they needed much more time. Notably, these reported times are more than double the mean times spent working on the cases online, as registered by the e-learning portal, indicating that fellows undertook additional offline research and study.

Finally, the participating fellows were asked whether they thought it was useful to receive feedback from the experts. All 20 fellows answered positively. For example, one fellow stated ‘Yes. When all experts provided similar feedback, it really sent a message that this feedback was important. It was also useful to receive varying feedback from people with different experiences. Some experts provided very detailed answers which will be helpful in my approach to patients in future’. Another interesting response was ‘It is very useful to receive feedback from various experts because each expert has his one way in dealing with the problem. And, when different experts give their opinion, you can use a little bit from every answer. Also, some experts find that certain part of the explanation has to be better while some find it good, because of the difference in personal interest’.

Cultural, Religious and Regional Aspects

Experts were asked to indicate cultural, religious and regional items of relevance in the answers to the open questions. Twelve of 32 experts replied to this request and mentioned these aspects summarized in Table 6.
Table 6

Regional, cultural and religious aspects of DSD management reported by 12 experts

Regional, cultural, religious aspects

Number of experts (of total experts)


8 (12)

Role of family

6 (12)

Role of religious official

6 (12

Genital surgery

5 (12)

Sex assignment

6 (12)

Hormonal replacement

4 (12)

Role of karyotype

4 (12)

Sex registration

4 (12)

Follow up

4 (12)

Association with transgender

3 (12)

The most frequently mentioned items were fertility and the role of the family. In some cultures, the girl will visit the doctor only with the mother without involvement of the father. In other cultures, the family will demand that the patient is not addressed directly—at least not until she is 18 years old. The possibility for delayed sex registration differs per country. Sometimes, sex assignment can be postponed, but this may be unacceptable to families. Several experts expressed a strong preference for male sex assignment for instance in 46, XX newborns with congenital adrenal hyperplasia with severe virilization. Some experts mentioned that possibilities for follow-up are limited for socioeconomic reasons or due to lack of confidence in the healthcare system.


In this exploratory study, we evaluated the applicability and feasibility at a global level of the ESPE e-learning portal in assessing junior paediatricians (fellows) during their subspecialty paediatric endocrinology training, regarding knowledge and skills for investigation and management of children and adolescents with DSD. In general, learning in practice is regarded as the most powerful way of learning. However, even in major referral centres, the number of patients with DSD is small, thus exposure to the clinical management of newly referred patients is very limited. Therefore, there is also a strong need for alternative learning methods to train the necessary competencies for a medical specialist as described by the CanMEDS framework [8, 9]. E-learning is increasingly being used to train competencies such as ‘scholar’ and medical expert [7]. The responses to the MC questions provided some insight into the medical expert competency. Although not formally validated, these questions refer to current textbook knowledge. The baseline knowledge with respect to relevant history, differential diagnosis and management of the participating fellows was notably limited judging on the score of the MC questions. During this study, knowledge is gathered by formative questions in the e-learning and by feedback provided by the experts. Concluding from the acknowledgements from the fellows and from the online questionnaire amongst the fellows, this leads to extended skill of medical expert. This argues in favour of the integration of formative learning by e-learning in the training for paediatric endocrinologist.

To our knowledge, the use of e-learning to train other competencies such as communicator and collaborator is not common practice. This study shows, however, that e-learning with personal feedback offers possibilities to train these competencies. In principal, the portal system allows direct interaction between fellow and tutor/expert. As the emphasis is on formative assessment, the tutor is in a position to correct any misunderstanding of the assignment or to request further details. The management of DSD requires an interdisciplinary approach, and in this study, we focused on the role of the paediatric endocrinologist in the team. It is of note that many key items in the open questions considered by the experts to be an essential part of the set of response elements were not mentioned by the fellows. This can be explained by the low exposure to such complex cases in practice due to the limited number of patients, highlighting the potential role for and value of e-learning.

The results of our study support the use of e-learning for formative assessment, providing direct feedback (assessment for learning). The anonymity between the fellow and faculty may have a positive aspect. Fellows may be anxious or uncomfortable discussing patients with their mentors and other senior faculty members. This study allowed them to write their personal thoughts and opinions without the anxiety of presenting to a respected faculty mentor. Indeed, the personalized comments by the experts proved to be very formative, as the fellows indicated that they learned a great deal from the personalized feedback. They were confronted with certain aspects that they were not aware of, such as religious and cultural aspects and addressing sensitive issues. Moreover, it was gratifying to note that most fellows in the subsequent acknowledgement expressed their appreciation. When feedback is provided in a summative context, for instance as a test or exam with a pass/fail score (assessment of learning), it is not always used effectively by learners, which pleads for a formative context [12, 14].

The time spent to study the assigned clinical cases did vary. Fellows from South America, Africa and Asia took more time for both cases than did fellows from other continents, presumably because of the cases and questions were presented in English, which would not be the native language of those fellows.

This study aimed at demonstrating the applicability of e-learning at a global level and identifying regional and cultural differences in medical practice. This was done by an inventory under the participating experts by identifying relevant cultural, religious and regional items. On the other hand, replies of the fellows were analysed per continent, which showed no great and significant differences. In the inventory amongst experts, the most frequently mentioned item was fertility and the role of the family. In some cultures, fertility is extremely important, as it affects ‘marriageability’ for women. Egg cell donation may not be allowed and adoption might be uncommon [15, 16, 17]. A religious official may play a crucial role in management in terms of decision making and with respect to openness on informing family and friends [16, 17]. Options regarding genital surgery involving gonadectomy, hysterectomy or fertility preservation procedures may be subject to religious recommendations [18]. In some countries, management of DSD is seriously hampered by lack of access to essential medications such as hydrocortisone and to specialized (paediatric) urosurgical expertise [15, 19]. Furthermore, recommendations for genetic counselling and transition of care to an adult endocrinologist or gynaecologist are lacking. Several experts mentioned the association with transgenderism or homosexuality, which in some cultural/religious background is condemned [16, 17]. It is of interest to note that the fellows in their first reply to the open questions did not mention cultural, religious or regional aspects. However, some acknowledged in their second reply to the expert that religious considerations are important. From the acknowledgements given by the fellows, we conclude that they achieved new insights on these aspects for future use.

As online and blended learning has become common place educational strategy in higher education, educators need to reconceptualise fundamental issues of teaching, learning and assessment in non-traditional spaces. Effective online formative assessment can lead to valuable learning experiences through formative feedback and enhanced learner engagement [13].

Standards for the formative effects of e-learning have been developed aimed at medical professionals who have graduated from formal training and residency or used by medical professionals as part of continuous learning to maintain their competency and develop new knowledge [20, 21]. So far, there is little experience in the integration and evaluation of formative assessment in online learning in the education for medical specialists. We have conducted an exploratory study to evaluate the feasibility of the integration of online formative feedback in the post-doc training of paediatric endocrinologists. From the lessons learned, future studies can be developed for other medical specialties.

As a limitation of the study, it is acknowledged that English is not the native language of many participating fellows, which may have influenced their understanding of the questions and formulation of their replies. It is remarkable that fellows and experts worldwide showed great enthusiasm and have put much time and effort. Fellows indicate that this way of learning and, in specific, the personalized feedback received by multiple experts, is very valuable. They were appreciative of the fact that experts from different continents responded to their replies to the open questions. Although preformatted reply text was supplied by the study team, the participating experts indicated that their careful reply to the fellows’ answers to the open questions was very time consuming.

A future development could be to assemble electronic meetings to allow direct personalized feedback via live discussion. We also foresee that a regional or national use of the e-learning portal could involve direct interaction between fellow (student) and expert (tutor).

In a separate study, the approach and the wording chosen by the fellow informing parents of a newborn with DSD and an adolescent girl with DSD will be assessed in detail.


Conversely, despite a worldwide movement in medical education to shift away from standardized instruction, learning and assessment toward individualization, here, we present a study designed to incorporate multiple continents and nations. However, this method preserves the individualisation of learning and offers direct, qualitative and formative input from ‘teachers’. Our method acknowledges that both the study of medical science and guidelines for clinical management are increasingly international. Also, internet-based information exchange means that parents and patients share professionals’ awareness of international medical care standards.

Fellows and experts highly value the integration of formative assessment as part of online learning in the training of clinical competencies (such as medical expert, communicator and collaborator). The results of this study indicate that at fellow level, there can be limited knowledge with regard to the management of newborns and adolescents with DSD. E-learning with formative feedback by experts is not only appreciated by fellow subspecialists but also may have an important role in their clinical training.

An appreciation of regional, religious and cultural aspects of DSD care was highlighted as an important factor in the management of DSD as exemplified in this e-learning. These aspects were not initially mentioned by the fellows. However, by the acknowledgements, we conclude that due to the feedback of the experts, fellows achieved new insights and will use this in future communication to patients and parents.

In conclusion, in this exploratory study, we demonstrated a global application of the ESPE e-learning portal. We found it useful and valued for the training and assessment of junior paediatricians during their subspecialty training in paediatric endocrinology regarding knowledge and skills for investigation and management of children and adolescents with DSD. We also foresee possibilities for the training of other medical specialties.



The following are in alphabetical order senior experts who contributed actively to the study and are considered coauthors of this article: Shereen Abdelgaffar (Department of Paediatrics, Cairo University, Cairo, Egypt), Amir Babiker (Department of Paediatrics, King Saud University medical City, Riadh, Saudi Arabia), Willie Bakker-van Waarde (Department of Paediatrics, Beatrix Children’s Hospital, Groningen University Medical Centre, Groningen, the Netherlands), Antonio Balsamo (Department of Paediatrics, University of Bologna, Bologna, Italy), Raja Brauner (Fondation Ophtalmologique Adolphe de Rothschild and Université Paris Descartes, Paris, France), Jean Pierre Chanoine (Department of Paediatrics, British Columbia Children’s Hospital,Vancouver, Canada), Steven Chernausek (Department of Paediatrics, Childern’s Medical Research Institute, Oklahoma University, Oklahoma City, USA) Asma Deeb (Department of Paediatrics, Mafraq Hospital, Abu Dhabi, UAE), Patricia Fechner (Department of Paediatrics, Seattle Children’s Hospital, Seattle, USA), Alina German ( Department of Paediatrics. Technion University Medical Centre, Haifa, Israel), Evelien Gevers (Department of Paediatrics, Royal London Hospital, Barts Health NHS Trust, Queen Mary University, London, UK), Olaf Hiort (Department of Paediatrics, University of Lubeck, Lubeck, Germany), Paul Martin Holterhus, (Department of Paediatrics, University of Kiel, Kiel, Germany), Lourdes Ibanez (,Department of Paediatrics, Hospital Sant Joan de Déu, University of Barcelona, Barcelona, Spain),Violeta Iotova (Department of Paediatrics, University of Varna, Varna, Bulgaria), Anders Juul ( Department of Paediatrics, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark), Anna Lauber-Biason ( Department of Medicine, University of Fribourg, Fribourg, Switzerland), Mohamad Maghnie (Department of Paediatrics, Children’s Hospital Giannina Gaslini, Genova, Italy), Alejandro Martinez-Aguayo ( Department of Paediatrics, Pontificia Universidad Católica de Chile, Santiago de Chile, Chilli), Berenice B. Mendonca (Department of Paediatrics, Medical School, University of São Paulo, Sao Paulo, Brazil), Kristen Neville ( Department of Paediatric Endocrinology, Sydney Children’s Hospital, Randwick, NSW, Australia), Anna Nordenstrom ( Department of Paediatrics, Karolinka Institutet, Stockholm, Sweden), Wilma Oostdijk (,Department of Paediatrics, Leiden University Medical Center, Leiden, the Netherlands), Rodolfo Rey (Department of Paediatrics, Hospital de Niños Ricardo Gutiérrez, Buenos Aires, Argentina), Stefan Riedl ( Department of Paediatrics, St Anna Children’s Hospital, Medical University of Vienna, Vienna, Austria), Meilan Rutter (Department of Paediatrics, Cincinnati Children’s Hospital Medical Centre, Cincinnati, USA), Nalini Shah (Department of Paediatrics, Seth G S Medical College and KEM Hospital, Mumbai, India), Shubha Srinivasan (Department of Paediatrics, Children’s Hospital at Westmead, Sydney, Australia ), Selma Witchel (Department of Paediatrics, Children’s Hospital of Pittsburgh, University of Pittsburgh, Pittsburgh, USA) and Xiaoping Luo (Department of Paediatrics, Tongji Hospital, Wuhan, China).

The authors gratefully acknowledge an educational research grant from Eli Lilly and Company, Indianapolis, IN, USA (Grant ID 100218713).

This article is based upon work from COST Action BM1303 DSDnet: A systematic elucidation of differences of sex development, supported by COST (European Cooperation in Science and Technology).

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflicts of interest.

Supplementary material

40670_2016_333_MOESM1_ESM.docx (39 kb)
ESM 1 (DOCX 38 kb)


  1. 1.
    Sleator RD. The evolution of eLearning background, blends and blackboard. Sci Prog. 2010;93(Pt 3):319–34.CrossRefGoogle Scholar
  2. 2.
    Wu B, Wang M, Johnson JM, Grotzer TA. Improving the learning of clinical reasoning through computer-based cognitive representation. Med Educ Online. 2014;19:25940.Google Scholar
  3. 3.
    Grijpink-van den Biggelaar K, Drop SL, Schuwirth L. Development of an e-learning portal for pediatric endocrinology: educational considerations. Horm Res Paediatr. 2010;73(3):223–30.CrossRefGoogle Scholar
  4. 4.
    Kranenburg-van Koppen LJC, Grijpink-van den Biggelaar K, Drop SL. An interactive E-learning portal in pediatric endocrinology: practical experience. Pharmacy. 2013;1:160–71.CrossRefGoogle Scholar
  5. 5.
    Drop SL, Kranenburg L, Reerds S, Grijpink K, Iotova I. The ESPE elearning webportal : a global tool for the instruction and formative assessment of pediatric endocrinology. Scripta Scientifica Medica. 2014;46:11–8.CrossRefGoogle Scholar
  6. 6.
    Lee PA, Nordenström A, Houk CP, Ahmed SF, Auchus R, Baratz A, Baratz Dalke K, Liao LM, Lin-Su K, Looijenga 3rd LH, Mazur T, Meyer-Bahlburg HF, Mouriquand P, Quigley CA, Sandberg DE, Vilain E, Witchel S, Global DSD Update Consortium. Global disorders of sex development update since 2006: perceptions, approach and care. Horm Res Paediatr. 2016;85(3):158–80.CrossRefGoogle Scholar
  7. 7.
    Busari JO, Stammen LA, Gennissen LM, Moonen RM. Evaluating medical residents as managers of care: a critical appraisal of assessment methods. Adv Med EducPract. 2014;5:27–37.Google Scholar
  8. 8.
    Van der Lee N, Fokkema JP, Westerman M, Driessen EW, van der Vleuten CP, Scherpbier AJ, Scheele F. The CanMEDS framework: relevant but not quite the whole story. Med Teach. 2013;35(11):949–55.CrossRefGoogle Scholar
  9. 9.
    Yip HK, Smales RJ. Review of competency-based education in dentistry. Br Dent J. 2000;189(6):324–6.CrossRefGoogle Scholar
  10. 10.
    Nordenström A, Thyen U. Improving the communication of healthcare professionals with affected children and adolescents. Endocr Dev. 2014;27:113–27.Google Scholar
  11. 11.
    Muscarella M, Kranenburg-van Koppen L, Grijpink-van den Biggelaar K, Drop SL. Global application of disorders of sex development-related electronic resources: e-learning, e-consultation and e-information sharing. Endocr Dev. 2014;27:268–83.Google Scholar
  12. 12.
    Schuwirth L, Ash J. Assessing tomorrow’s learners: in competency-based education only a radically different holistic method of assessment will work. Six things we could forget. Med Teach. 2013;35(7):555–9.CrossRefGoogle Scholar
  13. 13.
    Gikandi JW, Morrowa D, Davis NE. Online formative assessment in higher education: a review of the literature. Computers and education. 2011;57:2333–51.CrossRefGoogle Scholar
  14. 14.
    Harrison CJ, Könings KD, Schuwirth L, Wass V, van der Vleuten C. Barriers to the uptake and use of feedback in the context of summative assessment. Adv Health Sci Educ Theory Pract. 2015;20(1):229–45.CrossRefGoogle Scholar
  15. 15.
    Raza J, Mazen I. Achieving diagnostic certainty in resource-limited settings. Endocr Dev. 2014;27:257–67.Google Scholar
  16. 16.
    Ediati A, Juniarto AZ, Birnie E, Drop SL, Faradz SM, Dessens AB. Gender development in Indonesian children, adolescents, and adults with disorders of sex development. Arch Sex Behav. 2015;44(5):1339–61.CrossRefGoogle Scholar
  17. 17.
    Ediati A, Faradz SM, Juniarto AZ, van der Ende J, Drop SL, Dessens AB. Emotional and behavioral problems in late-identified Indonesian patients with disorders of sex development. J Psychosom Res. 2015;79(1):76–84.CrossRefGoogle Scholar
  18. 18.
    Al Herbish AS, Al Jurayyan NA, Abo Bakr AM, Abdullah MA, Al Husain M, Al Rabeah AA, Patel PJ, Jawad A, Al Samarrai AI. Sex reassignment: a challenging problem—current medical and Islamic guidelines. Ann Saudi Med. 1996;16(1):12–5.Google Scholar
  19. 19.
    Juniarto Z, van der Zwan YG, Santosa A, Ariani MD, Eggers S, Hersmus R, Themmen AP, Bruggenwirth HT, Wolffenbuttel KP, Sinclair A, White SJ, Looijenga LH, de Jong FH, Faradz SM, Drop SL. Hormonal evaluation in relation to phenotype and genotype in 286 patients with a disorder of sex development from Indonesia. Clin Endocrinol (Oxf). 2016.Google Scholar
  20. 20.
    Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(5):909–22.CrossRefGoogle Scholar
  21. 21.
    De Leeuw RA, Westerman M, Nelson E, Ket JCF, Scheele F. Quality specifications in postgraduate medical e-learning: an integrative literature review leading to a postgraduate medical e-learning model. BMC Med Educ. 2016;16:168.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2016

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Laura Kranenburg
    • 1
    • 2
  • Sam T. H. Reerds
    • 2
  • Martine Cools
    • 3
  • Julie Alderson
    • 4
  • Miriam Muscarella
    • 5
  • Kalinka Grijpink
    • 6
  • Charmian A. Quigley
    • 7
  • Stenvert L. S. Drop
    • 2
  1. 1.Department of RheumatologyMaasstad ZiekenhuisRotterdamthe Netherlands
  2. 2.Department of Pediatrics, Division of Pediatric EndocrinologySophia Children’s Hospital/ErasmusMCRotterdamthe Netherlands
  3. 3.Department of Pediatrics, Division of Pediatric EndocrinologyGhent University, Ghent University HospitalGhentBelgium
  4. 4.University Hospitals Bristol NHS Foundation TrustBristolUK
  5. 5.International DSD Steering Committee, USSF School of MedicineSan FranciscoUSA
  6. 6.Division Education and Student Support, Faculty EEMSDelft University of TechnologyDelftthe Netherlands
  7. 7.Division of Pediatric Endocrinology and DiabetologyIndiana University Medical CenterIndianapolisUSA

Personalised recommendations