Development and Evaluation of an Online Risk Communication Teaching Program for Medical Students

The objective of this study was to develop, pilot, and evaluate an online program for teaching clinical risk communication skills to medical students. A new teaching program, “Risk Talk Online,” was developed and piloted at Tufts University School of Medicine (TUSM) and Washington University School of Medicine (WUSM) in 2014–2015. The program’s content was based on “Risk Talk,” a risk communication curriculum previously developed at TUSM, and consisted of didactic and observational instruction using videorecordings of physician-patient discussions illustrating key elements of effective risk communication. The program was implemented among students at the beginning of the second medical school year at TUSM and the end of the first medical school year at WUSM. The program’s feasibility, acceptability, and efficacy in improving subjective and objective risk communication competence were assessed. A total of 204 TUSM and 115 WUSM students completed the program. Average time of completion was approximately 40 min, and students rated the program’s educational value very highly. Subjective (self-reported) competence in risk communication significantly improved (P < 0.001), as did objective risk communication competence (P < 0.001) assessed by students’ performance on a skills-oriented knowledge assessment administered before and after completion of the program. A new online risk communication teaching program was feasible, acceptable, and efficacious in improving subjective and objective risk communication competence among first- and second-year students at two medical schools. More research is needed to refine the program and determine optimal strategies for integrating risk communication training in medical education.


Introduction
Risk information-information about the probability of future outcomes-is a cornerstone of evidence-based medicine (EBM) and a primary form of evidence in health care. Risk communication, the open two-way exchange of information and opinion about risk [1], is a prerequisite for shared decision making (SDM) and an integral task in numerous important clinical endeavors, ranging from informed consent to advance care planning. Yet effective risk communication is extremely challenging due to several well-documented problems: innumeracy, psychological biases that limit comprehension of risk information, and inadequate risk communication skills among clinicians. Formal training in risk communication has thus been an increasingly recognized need in medical education; an international expert working group recently identified risk communication as a core competency and central focus for efforts to train clinicians in SDM [2]. In recent years, a growing number of institutions have begun to develop and implement risk communication training programs at various levels of medical education [3][4][5][6]. However, these programs have

Study Population and Setting
The study population consisted of medical students at two US medical schools: Tufts University School of Medicine (TUSM) and Washington School of Medicine (WUSM). TUSM students participated at the beginning of their second medical school year, and WUSM students participated at the end of their first year. Students at both schools receive instruction in EBM, clinical epidemiology, and biostatistics during the first and second medical school years. At TUSM, students complete classes in Epidemiology/Biostatistics over a 3month period at the beginning of the first year, a 6-week course on Evidence-Based Medicine at the end of the first year, and small-group Problem-Based Learning sessions devoted to teaching skills in searching and evaluating evidence. At WUSM, students receive instruction in EBM, Clinical Epidemiology, and Biostatistics through a longitudinal Practice of Medicine Course spanning the first and second years, and additional coursework throughout all 4 years of the MD curriculum. In the WUSM preclinical curriculum, these topics are introduced and reinforced through a mix of lecture-based and small-group (case-based) instruction.

Program Development
The content of the new program was derived from our workshop-based BRisk Talk^curriculum, the development and evaluation of which have been previously reported [6]. Integrating elements of a curriculum originally developed at SGUL and further adapted at TUSM, Risk Talk focused on helping students understand theoretical principles and best practices of risk communication, based on evidence from the risk communication literature. The new program, Risk Talk Online, was designed to preserve these aims and to deliver essential content of the original Risk Talk curriculum through didactic and observational components. Representative screenshots of the program are shown in Appendix 1.
The didactic component consisted of a narrated slide presentation that began by defining and contextualizing risk communication as a key activity within the broader process of shared decision making. The presentation then discussed the challenge of effective risk communication in terms of two major problems: (1) innumeracy and (2) cognitive biases and heuristics. It gave examples of these problems, drawing from relevant literature [7,8] and employing interactive exercises demonstrating common pitfalls in interpreting risk information. The presentation then discussed practical strategies for addressing the challenges of effective risk communication, summarized the essential steps of risk communication (Table 1), and reviewed essential steps in effective risk communication, operationalized using verbal Bscripts^-exemplary language to facilitate learning of communication skills (Table 2). Developed in our original Risk Talk curriculum [6], these scripts focused on (1) improving conceptual understanding of risk [9]; (2) increasing the evaluability, or interpretability, of risk information (e.g., using common denominators and rounding to make risk estimates more interpretable); and (3) reducing cognitive biases in the interpretation of risk information (e.g., using multiple frames-both positive and negative-and emphasizing absolute versus relative risk reduction in describing benefits and harms of treatment).
The observational component of the program consisted of four brief (2-4 min) high-quality videorecordings of an exemplary discussion about statin treatment for cardiovascular disease prevention, conducted by an experienced physician with a standardized patient and employing the Risk Talk Bscriptsî ntroduced earlier in the program. The videorecordings modeled different essential steps of risk communication: (1) setting the stage, (2) explaining the magnitude of risk, (3) explaining the reference class and acknowledging uncertainty, and (4) recapping and summarizing the information. The videos were filmed by a professional video production company and edited by the research team.
The Risk Talk Online program utilized the interactive elearn software program Articulate® and was designed to run automatically while also enabling users to view the program at their own pace, and to stop, restart, and search and advance to different parts of the program as they desired. The program also included electronic links to two downloadable reference materials which students were encouraged to use as reference materials for future learning and workshop participation: (1) a handout summarizing the Risk Talk scripts and essential steps of risk communication, and (2) a brief review paper on clinical risk communication [10]. The entire program took 31.5 min to complete, without any user-initiated pauses.

Program Implementation
Risk Talk Online was implemented in different ways at each study site. At TUSM, the program was administered in September 2014 as a mandatory preparatory exercise prior to a second-year workshop on advanced communication skills. During this workshop, students rotated through separate teaching sessions on risk communication, motivational interviewing, and dealing with Bdifficult^patients. The 70min risk communication workshop consisted of a 10-min

Assess preferences for information
BWould you like me to go into more detail about the pros and cons of the treatment choices, including numbers and statistics?2 . Explain the magnitude of risks, using multiple frames • Both percentages and natural frequencies BBased on who you are, medical experts estimate that your risk of developing colon cancer in your lifetime is 10 %.B This is the same as saying that 10 out of 100 people like you will develop colon cancer during their lifetimes.•

Both negative and positive frames
BMedical experts estimate that about 10 out of 100 people like you will develop colon cancer during their lifetimes.B You can also look at this in another way: 90 out of 100 people like you [with your characteristics] will stay free of colon cancer.•

Emphasize absolute risk reduction
BStudies show that taking DRUG X lowers people's risk of a heart attack by 25 %, or about 1/4. But to see if it is worth taking DRUG X, we need to count the number of people who are actually helped by DRUG X…B The studies showed that DRUG X lowered the total number of people having heart attacks from 4 out of 100 to 3 out of 100.^So DRUG X helped about 1 out of every 100 people who took it.3 . Explain meaning of the reference class BOne way to think about this is to imagine a group of [100] people like you, who have the following things in common: [sex, age, family history of cancer… etc.]. Of this group of [100] similar people, 10 will develop colon cancer in their lifetime.4 . Acknowledge uncertainty • Meaning of chance/inability to predict single events BWe can't predict the future of any one person. Estimates of risk, of the chances, only tell us how many people in some group are likely to get cancer. They can't tell us who will get the disease or not.B Even if the risk of a complication from surgery is 10 out of 100, we don't know whether you will be one of the unlucky 10 who will suffer the complication, or the lucky 90 who will not.• General scientific uncertainty BEstimates of risk, of the chance of something happening, are not perfect because science is not perfect. We don't know everything we need to know to predict the future.M ed.Sci.Educ. (2016) 26:557-567 recap and introduction, followed by three 20-min dyadic casebased role-play sessions, in which each student Bdoctor^took 15 min conveying risk information to a fellow student Bpatient,^and then received 5 min of group feedback. One week before the scheduled workshop, students were emailed the URL and instructions on how to access the online program.
At WUSM, the program was administered in March 2015 as part of the Scientific Methods sub-section of the first-year Practice of Medicine course. All first-year WUSM students were required to complete the Risk Talk Online program in preparation for a 90-min workshop session on risk communication. This session included didactic instruction in risk communication and a faculty-led demonstration with an SP. One week before the workshop session, students received an email with a link and directions on how to complete the program and the associated pre-and post-assessments.
A baseline and post-program survey, containing the measures described below, were administered before and after program completion using the HIPAA-compliant web survey platforms Qualtrics® (at TUSM) and DatStat® (at WUSM). At TUSM, survey completion was mandatory, although students could voluntarily opt out of having their responses used for research; at WUSM, all student data was deidentified prior to analysis and thus no voluntary opt-out was required. The study protocol was exempted from Institutional Review Board (IRB) review at Tufts University School of Medicine and Washington University School of Medicine.

Program Evaluation and Measures
Evaluation focused on the program's feasibility, acceptability, and efficacy. Feasibility was assessed by measuring (1) program completion rate and (2) self-reported program completion time. Students were also asked whether they completed the program in a single sitting. Acceptability was assessed using a three-item measure of students' perceptions of the program's value, adapted from our prior study (Appendix 2-1).
Efficacy was assessed by measuring students' subjective (self-reported) and objective risk communication competence before and after completing Risk Talk Online. Subjective risk competence was assessed using our previously developed three-item measure of confidence in risk communication skills [6] (Appendix 2-2). This measure ascertained students' perceived level of confidence in their understanding of medical statistics, ability to effectively communicate medical statistics to patients and/or families, and ability to help patients and/or families understand information about probabilities or risks. Objective risk communication competence was assessed by a newly developed four-item measure (Appendix 2-3) consisting of four case vignettes testing students' knowledge of practical communication strategies for accomplishing key tasks emphasized in our curriculum: (1) framing risk reduction in absolute versus relative terms, (2) explaining the reference class, (3) conveying uncertainty in risk information, and (4) simplifying risk information to facilitate understanding. These items were written to be practical in orientation-i.e., to assess knowledge of not only theoretical concepts, but of preferred approaches (language and communication strategies) to convey risk information in clinical encounters.

Data Analysis
Descriptive statistics for all variables were computed and within-subjects, pre-post changes in subjective and objective risk communication competence were assessed using pairedsample t tests. The proportions of correct responses to individual objective communication competence items were also computed, and pre-post changes were compared using McNemar's test. The critical level of significance was set at P <0.05. Efficacy Within-subject pre-post differences in subjective and objective risk communication competence are shown in Table 3. A programming error prevented matching baseline and post-program surveys to unique individuals in the TUSM cohort; we therefore applied probabilistic matching based on survey date, time, and IP addresses, which enabled successful survey matching for 143/204 (70 %) of TUSM students. Students at both sites demonstrated significant improvement on all measures after program completion.

Results
Performance on individual objective risk communication items are shown in Table 4. The proportion of correct responses was lowest for the item on framing risk reduction in absolute versus relative terms, intermediate for the items on explaining the reference class and simplifying risk information, and highest for the item on communicating uncertainty in risk estimates. Pre-post changes were significant for each item, although largest for the item on the reference class and smallest for the item on explaining uncertainty.

Discussion
In this study a new risk communication training program, Risk Talk Online, was developed, implemented at two US medical schools, and found to be feasible, acceptable, and efficacious. The program was completed in approximately 40 min and in a single sitting by over 90 % of students, who rated its value very highly. The program produced significant improvements in subjective and objective risk communication competence. Together, these findings provide strong proof-of-principle evidence supporting the program's value and generalizability.
Importantly, Risk Talk Online was designed and implemented not as a stand-alone intervention but as part of a more comprehensive Bflipped-classroom^training program integrating additional didactic instruction and experiential skills practice in classroom settings. Our findings support the need for such additional instruction. Despite significant improvement in students' objective risk communication competence after they completed the online program, only a minority (<20 %) were able to correctly answer all four knowledge questions. Students struggled with particular items including the need to frame risk reduction in absolute (versus relative) terms. Although this concept received great emphasis in the teaching program, our data suggest the need for more reinforcement as well as refinement of the program's content. It should be noted, furthermore, that the knowledge measure used to assess objective risk communication competence was new, and further work to refine the measure and assess its reliability and validity is an important next step for future research.
More research is also needed to determine whether online training in risk communication improves actual communication skills, which were not assessed in the current study. We do have convincing evidence-from our previous OSCE-based experimental study of the Risk Talk curriculum on which the content and approach of the current e-learn program is based-that our overall teaching intervention improves communication skills [6]. It stands to reason, furthermore, that had students in the current study also received experiential training prior to their competence assessments, even greater improvements might have been observed. Nevertheless, whether the Risk Talk Online program-either alone or in combination with workshop-based experiential training-results in similar improvement is a key question for future research.
Broader integration and dissemination of risk communication training in medical education ultimately requires generalizable strategies for overcoming curricular time and resource barriers. We believe that e-learning programs such as Risk Talk Online provide one promising strategy for accomplishing this goal. Similar Bflipped classroom^programs have been successfully employed in the teaching of other clinical skills and may facilitate dissemination of risk communicating training at other levels of medical education, including not only undergraduate but graduate and postgraduate training. For all types and levels of learners, e-learning programs for teaching risk communication skills would ideally be deployed as one component of a more comprehensive curriculum, but they could also serve as stand-alone programs if curricular time and resources were limited. We believe risk communication  is such a critical advanced communication skill that even some basic training is far preferable to none at all. Several limitations qualify our study findings and call for further research. Although the Risk Talk Online program was well received, its usability was not evaluated. Formal usability testing and refinement of the program are a necessary focus for future work. The durability of students' knowledge improvements, and their relationship to patient care outcomes, also remain to be shown. Further research is also needed to assess the feasibility and effectiveness of implementing Risk Talk Online at different points in the medical school curriculum or for different levels of learners. The optimal strategies for integrating risk communication teaching with other related components of the medical school curriculum-e.g., instruction in evidence-based medicine and advanced communication skills-also require further investigation. Finally, more work is needed to explore how risk communication teaching might be incorporated in graduate and continuing medical education.
Despite these limitations, the current study provides promising evidence on the educational value of an online clinical risk communication teaching program. We believe that Risk Talk Online represents an important first step in the development of feasible and effective strategies for teaching risk communication skills, and in disseminating this teaching in medical education. The current study provides proof-of-principle evidence of the program's value, and paves the way for further work to refine the program and improve physicians' competence in communicating risk information to patients.

Conclusions
Risk Talk Online, a new online risk communication teaching program, is feasible, acceptable, and efficacious in improving subjective and objective risk communication competence among first-and second-year medical students. More research is needed to refine the program and determine optimal strategies for integrating risk communication training in medical education. reduction in risk-from an average baseline risk of 4 % to a reduced risk of 3 %.
Scenario 2: Mr. Smith is 62 years old, and very interested in taking MAGICSTATIN after seeing a television advertisement showing the MAGIC trial data. His total and LDL cholesterol have remained elevated despite his diligent attempts at diet and exercise over the past 3 years. He also has mild hypertension which has been well controlled on a diuretic. Which of the following is the most important factor that Mr. Smith needs to consider in deciding about whether MAGICSTATIN is the right treatment for him? THE CORRECT ANSWER IS B: In deciding whether a drug is appropriate for a particular patient, the most important consideration is whether the patient matches the Breference class^-the patient population in which the drug has been evaluated. The more closely that Mr. Smith's cardiac risk factors match those of the patients for whom MAGICSTATIN has been shown to be effective, the more confidence Mr. Smith can have that he will derive the same benefit from the drug.
Scenario 3: Your patient Mrs. Jones comes to your office to ask your help in interpreting some information she received online. She used an established, well-validated Brisk calculator^produced by the National Cancer Institute, which computed a B24.3 % risk^of developing breast cancer in her lifetime, compared to an average risk of 10.8 %. She is worried and asks, BWhat does this number mean: should I be concerned?^Which of the following is the best response to Mrs. Jones?
A. The number means you are definitely at higher than average risk for breast cancer. There's reason to be concerned, but it's impossible to know whether you or any one person will develop cancer or not. B. The number means you are definitely at higher than average risk for breast cancer. There's no reason to be concerned, however, because it's impossible to know whether you or any one person will develop cancer or not. C. The number means you may be at higher than average risk for breast cancer. There's reason to be concerned, but it's impossible to know whether you or any one person will develop cancer or not.
D. The number means you may be at higher than average risk for breast cancer. There's no reason to be concerned, however, because it's impossible to know whether you or any one person will develop cancer or not.

THE CORRECT ANSWER IS C:
This response appropriately conveys the fundamental uncertainty in all risk estimates, while also acknowledging the value of risk estimates. No matter how strong our risk evidence is, we can never definitely determine a single person's risk for a future event. It is also impossible to know whether any given person-regardless of their estimated risk-will experience a future outcome such as cancer. Yet in spite of these uncertainties, evidence-based risk evidence, such as the estimates produced by statistical models, provides essential guidance for health care decisions, and should be taken seriously rather than dismissed.
Scenario 4: J.S. is a 37-year-old woman who is 16 weeks pregnant and considering prenatal screening. The best available evidence suggests that her baby boy has a 1 in 122 risk of a chromosomal abnormality at delivery, and a 1 in 242 risk of Down syndrome. She is trying to decide whether to undergo amniocentesis for further testing, which carries a 1 in 200 risk of pregnancy loss. Which of the following is the best way of explaining these risks to J.S.?
A. Your child's risk of a chromosomal abnormality is about 1 %, his risk of Down syndrome is a little less than 0.5 %, and the risk of pregnancy loss as a result of amniocentesis is about 0.5 %. B. Your child's risk of a chromosomal abnormality is about twice as high as his risk of Down syndrome or a pregnancy loss as a result of amniocentesis. C. Your child's risk of a chromosomal abnormality is 1 in 122, his risk of Down syndrome is 1 in 242, and the risk of pregnancy loss as a result of amniocentesis is 1 in 200. D. Your child's risk of a chromosomal abnormality is about 1 in 120, his risk of Down syndrome is about 1 in 240, and the risk of pregnancy loss as a result of amniocentesis is about 1 in 200.

THE CORRECT ANSWER IS A:
This response provides the quantitative information that patients need to simultaneously compare three risks, while also simplifying the information and minimizing the amount of arithmetic and computation that patients themselves need to perform. It avoids the B1-in-x^format which is confusing to interpret, converts unusual fractions such as B1/122^into percentages, and then uses rounding so that patients can compare multiple risks more easily.
*Correct answers in italics, provided to respondents after module completion Open Access This article is distributed under the terms of the Creative Comm ons Attribution 4.0 International License (http:// creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.