Fostering medical students’ lifelong learning skills with a dashboard, coaching and learning planning

Introduction To develop lifelong learning skills, students need feedback, access to performance data, and coaching. A new medical curriculum incorporated infrastructural supports based on self-regulated learning theory and the Master Adaptive Learner framework to engage students in reflection and learning planning. This study examines students’ experience with a performance dashboard, longitudinal coaching, and structured time for goal-setting. Methods Focus groups with first-year medical students explored performance dashboard usage, coaching and learning planning. We analyzed findings using thematic analysis. Results informed development of a 29-item survey rated strongly disagree (1) to strongly agree (5) to investigate experience with the dashboard, coaching and learning goals program. The survey was distributed to one first-year medical student class. We performed descriptive statistics and factor analysis. Results In three focus groups with 21 participants, students endorsed using the dashboard to access performance information but had trouble interpreting and integrating information. They valued coaches as sources of advice but varied in their perceptions of the value of discussing learning planning. Of 152 students, 114 (75%) completed the survey. Exploratory factor analysis yielded 5 factors explaining 57% of the variance: learning goals development (α = 0.88; mean 3.25 (standard deviation 0.91)), dashboard usage (α = 0.82; 3.36 (0.64)), coaching (α = 0.71; 3.72 (0.64)), employment of learning strategies (α = 0.81; 3.67 (0.79)), and reflection (α = 0.63; 3.68 (0.64)). Discussion The student performance dashboard provides efficient feedback access, yet students’ use of this information to guide learning is variable. These results can inform other programs seeking to foster lifelong learning skills. Electronic supplementary material The online version of this article (10.1007/s40037-018-0449-2) contains supplementary material, which is available to authorized users.


Introduction
Lifelong learning behaviours essential for physician practice entail personal development through continuous acquisition of knowledge and skills, with focus on learning, adapting and discovering [1][2][3]. To guide students in cultivating habits of continuous improvement for integration into practice, educators must encourage practice, reflection and reinforcement [4]. This approach to learning can improve performance and enhance a learning climate [5,6]. However, medical school curricula do not typically provide early learners with the support and opportunity to engage in new approaches to learning, assess the impact of these approaches, and reattempt under the guidance of experienced mentors.
The Master Adaptive Learner conceptual framework describes how students can engage in reflection on performance to promote habits of continuous learning and selfimprovement [7]. The Master Adaptive Learner purposefully plans for learning, uses intentional learning strategies, self-assesses progress, and adjusts in repeated cycles of learning and adaptation [7]. These steps align with the tenets of self-regulated learning theory [8]. Provision of timely, accurate information about a learner's own performance enables 'informed self-assessment,' which is more accurate than self-assessment unguided by feedback [9].
Critical to student success as a Master Adaptive Learner is adequate infrastructural support to operationalize four steps: planning, learning, assessing and adjusting [7]. Educators must provide learners with information, resources and support for adequate reflection for meaningful selfimprovement. Students' reflections on their performance, including strengths and areas for improvement, should be informed by comprehensive qualitative and quantitative data [9][10][11]. As such, educational dashboards are now being adopted by some training programs to make performance data readily available [12]. To undergo self-improvement via dashboard review, students actively interpret information through steps that align with the Master Adaptive Learner: awareness and visualization of data; self-reflection and data interpretation; sense-making to construct goals and learning plans; and adjusting performance via behavioural change [13]. Engaging in informed self-reflection with a longitudinal coach enhances students' ability to create useful learning goals for continuous improvement [14].
Informed by the Master Adaptive Learner framework, a new medical school curriculum integrated three targeted infrastructural supports: an individual performance dashboard, coaching relationship and opportunities to reflect on performance and set learning goals. This study explores how students engage in self-regulated learning behaviours with these infrastructural supports and how they value each support. Findings may inform educators about engaging early medical learners in the behaviours of lifelong learners.

Design
This mixed methods study uses an exploratory sequential design [15]. Mixed methods are appropriate when data collected with one method inform additional data collection with another method, thereby expanding the range of information [16]. Qualitative focus groups revealed medical students' descriptions and perceptions about the various infrastructural supports designed to facilitate self-regulated learning behaviours and informed design of a quan-titative survey of students' engagement in these learning behaviours. The survey purpose was to quantify students' priorities and values related to multiple infrastructural supports to guide the school on further interventions including faculty development.

Setting
The University of California, San Francisco, (UCSF) School of Medicine is an urban, public research-intensive institution in California, USA. In fall 2016, the School launched its new Bridges curriculum, which emphasizes health systems science, inquiry, and longitudinal foundational and clinical sciences learning. The School's competencies and milestones guide curriculum and assessment [17] (https://meded.ucsf.edu/md-program/current-students/ curriculum/md-competencies) as part of programmatic assessment [18]. Three infrastructural supports promote lifelong learning skills in the new curriculum: Dashboard: An electronic learner performance dashboard displays quantitative and qualitative data from student performance in various formative and summative assessments [12]. Informed by general and targeted needs assessments by the school's educators and a literature review, the dashboard centralizes and displays all assessment and performance data in a timely manner with performance metrics for easy interpretation with benchmark data and standards [19,20]. Student dashboard access is limited to the student, coach, deans, and staff who maintain the dashboard.
Coach: Each student has a faculty coach for guidance and support throughout medical school; coaches support approximately six students each in two medical school cohorts. Students meet with their coaches four times in the first phase (18 months) of the curriculum for individual performance reviews and goal setting, and weekly for clinical skills learning. Coaches receive training to use the dashboard and undergo simulation training on development of learning goals and mentorship through standardized student encounters.
Reflection and goal-setting: Students are introduced to the cycle of self-regulated learning early in the curriculum. They learn that their dashboard is a tool to use both independently and with their coach to monitor their progress and create SMART (Specific, Measurable, Attainable, Result-Based, and Time-Bound) learning goals [21]. Assessment-Reflection-Coaching-Health (ARCH) weeks provide structured times for students to reflect and meet with coaches. Students and coaches are advised to gauge students' performance on the school's milestones relative to expectations and to peers, and to view students' prior individual learning goals in the dashboard.
The UCSF Institutional Review Board approved the study. The work was carried out in accordance with the Declaration of Helsinki.

Phase 1: Qualitative
Subjects and sampling: We used convenience sampling for focus groups. First-year students were invited through the class listserve to participate. All respondents to a first email who were able to attend a scheduled focus group were included.
Data collection: In spring 2017, two trained moderators, a fourth-year medical student (AQ) and research assistant (PY), led three 50-minute focus groups. Students provided written informed consent and completed a five-item demographic survey. Focus group questions explored students' understanding of the purpose of the electronic dashboard and how and why they used it, their use of other sources of performance information, and their coach interactions to review performance and create learning plans (see Appendix 1 in the Online Electronic Supplementary Material). All focus groups were audiotaped, professionally transcribed, and de-identified.
Analysis: Data analysis occurred concurrently with data collection for early analysis to iteratively inform subsequent data collection. Three investigators (a fourth-year medical student, research assistant, and faculty member (AQ, PY, KEH)) analyzed focus group data using thematic analysis. As data were collected, these three investigators read each transcript independently to identify key themes. Through iterative discussion, they refined the themes into a codebook. Using the constant comparative method, they compared findings within and across transcripts [22]. Two investigators independently coded the transcripts and reconciled discrepancies through discussion. These three investigators reviewed the coded data to identify relationships among codes and synthesize information into larger themes. Data collection continued until sufficient information was obtained about the infrastructural supports to inform survey design. Investigators used Dedoose software, version 6.1.18 (SocioCultural Research Consultants, LLC, Los Angeles, California) to code, organize and retrieve coded data.
Investigators considered reflexivity throughout data collection and analysis [23] by repeatedly sharing with one another their perspectives based on their experience and role in the school and how this influenced their reactions to the data.

Phase 2: Quantitative
Subjects and sampling: All 152 finishing first-year students received an email invitation for the survey through Qualtrics (Provo, UT) in April 2017. Non-responders received up to two email reminders.
Data collection: Informed by focus group results and the Master Adaptive Learner conceptual framework [7], three investigators (KEH, NI, CB) developed draft survey items following procedures for survey design [24]. Items targeted students' experience with and perceptions of the utility of new curricular infrastructures designed to facilitate selfregulated learning behaviours. Focus groups and the Master Adaptive Learner framework prompted item generation addressing dashboard use alone and in coaching encounters, discussing performance, informed-self assessment, reflection, and goal-setting. Investigators initially drafted 31 items that each mapped to a phase of the Master Adaptive Learner framework. A14-member health professions education panel of clinicians, statisticians, and medical students reviewed draft survey items. Based on their feedback, two irrelevant or redundant items were removed. The final 29 items addressed the four phases of the Master Adaptive Learner framework: Planning (5), Learning (7), Assessing (12), and Adjusting (5). Respondents rated their level of agreement for each item on a 5-point Likert-type scale (1 = strongly disagree, 5 = strongly agree).
Analysis: Responses were de-identified before analysis. We calculated descriptive statistics for all 29 items. Exploratory factor analysis with varimax rotation identified latent variables characterizing students' perceptions of the value of the resources and activities related to the Master Adaptive Learner framework. Although latent variables may not be readily apparent, they can drive responses to survey items [25]. Factors with eigen values greater than 1 were retained. Items with factor loadings of 0.4 were considered significant for this sample size [26]. Cronbach's α, mean and standard deviation (SD) were calculated for each factor. We compared composite means by conducting t-tests with Bonferroni correction (0.05/10 = 0.005) for multiple comparisons to avoid finding significant results by chance. We used SPSS for Windows Version 24.0 for quantitative analyses (IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp.)
Focus group results addressed three themes: information access, information interpretation, and coaching for change.
Information access: Most students described using the dashboard primarily to view score reports from written examinations and other performance reports that were automatically uploaded. One student explained: 'I think of it as the assessment help, and for any type of assessment I can find out how I did there.' (Group 2) While students in each group initially described obtaining score reports in the dashboard, each group then characterized other purposes of the dashboard, including capturing individual and longitudinal performance information. One student noted: 'It's one of the few sites where I feel like it's personal to me.' (Group 3)

Information interpretation:
Students had difficulty interpreting all information housed in the dashboard independently. Viewing information about their performance seemed to provide them some information while also raising many questions. One student observed: 'It's still a little bit vague, but that's the information we have on how we're doing.' (Group 3) They desired more feedback, more longitudinal performance views, transparency in grading, and individualized comments on all assessments. Many students shared technical questions about particular dashboard features or suggested features that could be added. Many wanted more detailed information about their performance particularly relative to peers. The dashboard raised concerns for some about how the school might use performance information in ways they might not know to compare them with peers and school expectations.
Coaching for change: Students praised their coaches highly as supports and mentors, yet expressed mixed opinions of their experience discussing learning planning with coaches. For some students, the process of setting learning goals for posting in the dashboard and discussing with the coach seemed productive. Coach discussions enhanced students' perceptions of their ability to interpret their own progress: In required progress meetings with coaches, some students viewed the dashboard together with coaches and found this step enlightening. One student appreciated that the coach advised: 'See, by itself, this may be not the most useful thing in the world, but over time you can see a trend.' (Group 1) Another student described the coach providing clarification about information in the dashboard: 'With her eye, she can also make sense of the information and tell me so this is how it can be helpful for you.' (Group 1). Students perceived learning plan development with coaches to be more effective when goals were meaningful to the student rather than seeming to be dictated by data in the dashboard. They appreciated longitudinal coach relationships for individualized guidance and reassurance. They shared tentative thoughts that they might be developing a habit of mind through reflecting and articulating goals. One student characterized progress meetings: 'By giving us this time almost off they're forcing us to zoom out and step back and say, hey, wait, what am I working towards?' (Group 1).
However, others reported that coaches did not steer them to use the dashboard together; those students trusted that their coaches had checked the dashboard in advance to ensure performance met expectations. Students valued agenda flexibility to address issues most salient to the student when meeting with coaches. Students reported that coaching meetings focused on varying topics including general academic support, career guidance, and students' personal wellbeing: Some students were still considering how to engage in the process of using performance review to identify and implement goals. One student explained:

I don't always go into meetings with my coach with clear hopes or goals, but actually for my last session I had no ideas, and she was really helpful in talking to me and figuring out where to improve.' (Group 3)
Many students similarly described reviewing their performance without having previously engaged in planning as part of a cyclic process.
Factor analysis yielded five factors explaining 57% of the variance characterizing students' perception of the vari-ous Master Adaptive Learner behaviours and infrastructure components. Cronbach's alpha for the five factors ranged from 0.70 to 0.88. Factor 1 represented items related to learning goals development (6 items, factor mean 3.25, SD 0.91). Factor 2 addressed dashboard usage (9 items, mean 3.36, SD 0.64). The third factor addressed employment of learning strategies (6 items, mean 3.72, SD 0.64). The fourth factor focused on coaching (5 items, mean 3.67, SD 0.79). The fifth factor addressed reflection (5 items, mean 3.68, SD 0.64). Students' perception of learning strategies (p < 0.000), coaching (p < 0.000), and reflection (p < 0.000) were all significantly higher compared with dashboard usage and learning goals development. Three individual items with the most positive responses (mean >4) pertained to students' endorsement of their own skills in self-directed and lifelong learning, and incorporation of feedback.

Discussion
Guided by the Master Adaptive Learner framework, the new curriculum focused on three targeted infrastructural supports: individual performance dashboard, coaching relationship, and opportunities for reflection and goal-setting. This mixed methods study illustrates how students vary in their understanding of connections between performance information, reflection and working with coaches to enact personal change. Their survey results showed mild endorsement of the value of infrastructural supports for self-regulated learning behaviours, with more positive perceptions of their own learning strategies, coaching for improvement and their own reflection than for learning goals development and dashboard usage. Findings suggest that students seem to need more training, coaching, support or experience to develop all Master Adaptive Learner skills. Similarly, a recent study of resident learning goals showed that residents struggled with lifelong learning behaviours and required heavy faculty support to use learning goals [27].
Despite experiencing SMART goals and a performance dashboard within the curriculum structure, most students did not report optimally utilizing these to engage in new or different approaches to their learning. Development of lifelong learning skills requires practice with specific coaching [4]. Improved orientation to students could enhance their understanding of the cyclical process of self-regulated learning as described in the Master Adaptive Learner framework and the potential utility of infrastructure supports to actualize lifelong learning skills [28]. As our participants progress into clerkships, the perceived value of information about their workplace performance could become a more prominent driver for continuous enhancement [29]. Our students' perceptions of themselves as skilled lifelong learners contradicts their reported engagement with the tools and ac-tivities of lifelong learners. Other literature corroborates the limitations of self-assessment of one's skills [30]. Our participants' emphasis on the value of coaching reinforces this essential ingredient for enhancing the usefulness of assessment activities for promoting learning [31,32].
Focus group participants valued dashboard information as evidence of their achievements which may suggest a performance rather than a mastery perspective [33]. Students sought dashboard content for comparison with peers and assessment results rather than to guide self-improvement. While reinforcement of successful performance can feel validating and potentially enhance performance [5], identification of areas needing growth is essential for physicians to develop competence. Student's selfefficacy-perceptions of capability to achieve goals-is fostered by enacting adaptive, self-regulated learning behaviours that motivate attempts at new learning activities [34]. Accurate self-assessment achieved through guided feedback focuses self-regulated learning appropriately on tasks for growth [35]. Increased emphasis on formative assessment may facilitate culture change towards mastery, a more adaptive learning approach. This study has limitations. Findings from this single institution study may not generalize to other institutions or later years in medical school. Focus group volunteers may not have represented the whole class. We did not collect data about survey nonrespondents, although we did have a high survey response rate. We did not measure learning strategies or include performance measures to determine the effects of particular strategies on learning goal accomplishment. This study demonstrates that early medical students view themselves as skilled with learning strategies yet use a performance dashboard primarily to confirm performance achievements and find lower value in setting learning goals. Additional practice and support may enable students to link pieces of the Master Adaptive Learner cycle to instil habits of lifelong learning.