Peer-assisted debriefing of multisource feedback: an exploratory qualitative study
The Manitoba Physician Achievement Review (MPAR) is a 360-degree feedback assessment that physicians undergo every 7 years to retain licensure. Deliberate reflection on feedback has been demonstrated to encourage practice change. The MPAR Reflection Exercise (RE), a peer-assisted debriefing tool, was developed whereby the physician selects a peer with whom to review and reflect on feedback, committing to change. This qualitative study explores how physicians who had undergone the MPAR used the RE, what areas of change are identified and committed to, and what they perceived as the role of reflection in the MPAR process.
The MPAR RE was piloted out to a cohort of MPAR-reviewed physicians. Thematic analysis was conducted on completed exercises (n = 61). Semi-structured interviews were conducted with individuals (n = 6) who completed the MPAR RE until saturation was reached.
Physicians reviewed feedback with a range of peers, including colleagues, staff, and spouses. Many physicians were surprised by feedback, both positive and negative, but interviewees found the RE useful in processing feedback. Areas where physicians committed to change were diverse, covering all CanMEDS roles. Most physicians identified themselves as being successful in implementing change, though time, habit, and structures were cited as barriers.
Peer-assisted debriefing can assist reflection of multisource feedback. It is easy to implement, is not resource-intensive, and feedback implies that it is effective at promoting change. Participants, with the aid of peers, identified areas for change, developed approaches for change, and largely thought themselves successful at implementing changes. Areas of change included all seven CanMEDS roles.
KeywordsMultisource feedback Reflection Peer-assisted debriefing Commitment to change CanMEDS
Continuing professional development
Manitoba Physician Achievement Review
Physician performance assessments, particularly multisource feedback (MSF) approaches such as 360-degree assessments, are now common requirements for demonstrating continued competence and maintaining certification to practice . In Manitoba, these reviews have been mandatory since 2011; physicians must complete them every 7 years to remain registered to practice in the province. 360-degree assessments are tools whereby physicians are reviewed by colleagues, co-workers, and patients about their performance, as well as completing a self-assessment questionnaire. In Manitoba, the Manitoba Practice Achievement Review (MPAR) is used to collect this feedback, which is then collated into a report that is returned to the physician under review.
Physician attributes assessed in the Manitoba Physician Achievement Review
Psychosocial management of patients
Information for patients
Assessments such as the MPAR are still relatively new in the province, and while they are well-validated tools, literature on how physicians use this feedback demonstrates that dedicated reflection is required for performance data to change and improve practice, and that physicians will generally not make any changes based on feedback if left to their own devices [3, 4, 5]. Indeed, physicians who have been assessed via MSF approaches have identified that having another individual help them reflect on feedback would increase the likelihood of using that feedback [3, 6, 7]. While quantitative data is necessary for rating physicians against average scores, qualitative feedback acquired by reflection, engaging physicians as partners in the MSF process rather than just recipients, is useful for a physician to translate scores into practice change  .The studies that have acknowledged the necessity of peer-assisted reflection have generally turned to full peer mentoring or coaching, time-intensive and costly approaches that require trained mentors or coaches who will read a physician’s entire 360-degree report and interview the physician to create a personal development plan .
While this approach may be ideal for encouraging practice change, it may not be an appropriate approach for the 80% of physicians in the province who are deemed competent to continue practice after they successfully complete the MPAR; though improvements are encouraged and expected, these successful physicians should not be overburdened with extensive follow-up. To encourage reflection and practice change, the MPAR Reflection Exercise (RE) was developed. This exercise is a peer-assisted debriefing tool, where the physician under review selects a peer, someone familiar with the physician’s practice, with whom to review their MPAR report, after which they fill out the RE, committing to change. Documented reflection and commitment-to-change has been demonstrated to encourage actual practice changes [9, 10].
Peer-assisted learning, where individuals who know each other, and who are not trained educators, are engaged in educational processes, has been demonstrated to allow for more forthcoming communication between the ‘student’ and the peer . Debriefing, a term most often used in crisis management and simulation learning, is appropriate in this context, insofar as it refers to critical reflection with the purpose of resulting in practice change, while having the added benefit of providing an opportunity for building morale and calmly addressing negative or traumatic issues, such as unexpectedly receiving poor feedback . Thus, the collective term ‘peer-assisted debriefing’ implies a productive, formative, casual learning experience. The RE developed by the Continuing Professional Development (CPD) – Medicine Office at the University of Manitoba for peer-assisted debriefing of the MPAR feedback directs and focuses reflection, requiring less time to complete than a full post-assessment interview process. Debriefing can be done with any peer chosen by the physician, and does not require a dedicated, trained mentor to come in and do a full interview.
Each year, approximately 200 Manitoba physicians are required to complete the MPAR process. The MPAR RE form is included with the final report that is mailed to physicians following review. Completion of this exercise is eligible for CPD credits from the appropriate crediting body.
The RE has been piloted out to two cohorts of physicians who have completed the MPAR. Completed exercises were examined using thematic analysis to understand how the forms are being used, what areas of feedback are focused on and what changes are committed to . We also examined whom physicians were choosing as their assisting peers and how or whether the changes they identified were carried out.
For more in depth feedback on the debriefing and reflection process, semi-structured telephone interviews were also conducted with individuals who completed the RE. The aim of this interview was to allow participants to share their thoughts on the peer-assisted debriefing process and tool, as well as the MPAR process as a whole.
The coding structure was developed both inductively and deductively; coding of the REs was organized at the top level by the section of the form, and then inductively coded within each section, while interview codes were also organized into deductive topic areas before being coded inductively. SM completed the primary coding of the REs. A research assistant verified the coding structure and completed the primary coding of the interview transcripts, which was then verified by SM. JF then verified the coding of both sets of data in discussion with SM.
This study was approved by the Health Research Ethics Board at the University of Manitoba. Consent was not sought for secondary analysis of completed RE forms; interview participants provided written informed consent.
Forms were collected for 2 years, from late 2012 until late 2014, of which 61 (74% male, 26% female; 54% family medicine, 41% other specialties, 5% unspecified) were completed and returned for CPD credits. If approximately 200 Manitoba physicians are reviewed each year, 61 forms over 2 years represents a 15% response rate. Multiple invitations were sent to all 61 of these eligible individuals to participate in the interview process; six interviews were completed. Although the research team had hoped for more interview participants, saturation was reached by the fourth interview. Nonetheless, the generalizability of the interview results must be considered cautiously. All interviewees were male, with 33% in family medicine.
We had anticipated that most physicians would choose a physician colleague to debrief with; while approximately 60% did choose a fellow physician, 40% chose a different peer. Often this was another co-worker such as a nurse or business manager, which many stated was because they felt this was a person that would be impacted by the changes that needed to be made.
“I chose a peer who is somebody that I have known a long time, that I went through medical school with, whose judgement I trust greatly, who I have known from the past just to have good insights into looking at things like this. And somebody that I was comfortable with, and you know, I felt comfortable in sharing this information with him.”
Committing to change
Reflecting on progress
Two months after committing to change, participants returned to their REs to reflect on the progress they had made. Questions in this section of the tool focused on rating success of change, barriers to change, and any future plans with regard to the areas of change they had committed to. Most participants (73%) felt they were successful in at least some of their chosen areas of change, although many had future plans to improve further. For instance, one physician identified staff communication as an area for improvement; it was a poorly scored area which the physician attributed to poor management of personal stress. In order to address this change, the physician decreased their patient load to reduce stress, and integrated staff meetings into their practice. This process helped the physician identify a further area for change, namely that the practice was understaffed and that there was a need to hire additional staff. Common changes reported included implementation of electronic medical record (EMR) systems to address issues with documentation and scheduling time for urgent same-day appointments and patient calls. One area of change that was often cited as still being in need of improvement was both attending and providing CPD. Time, habit, and structural issues, such as the inability to change physical office space, were commonly cited barriers to changes. Reflecting on the impact that the MPAR process had on their practice, physicians tended to report feeling more confident in the areas in which they had scored well. While there were a small minority of physicians who did not find value in the process, for most, the MPAR merely reinforced that they were overall doing well, with only minor practice changes needed.
Role of reflection exercise
In interviews, we asked participants to discuss the role of the RE and the peer in facilitating reflection. The peers chosen by the interviewees included physician colleagues, nurses and an MPAR representative. All interviewees had identified area of change including, for instance, improving documentation, improving timeliness of referral responses, providing CPD to colleagues, changing scheduling procedures, and implementing electronic medical records. All interviewees reported at least some success in implementing changes, though some identified areas of further change as they tried to break old habits or make changes with other staff members (e.g., improving staff communication with patients). Interview data were organized into themes of: comparing perspectives; structure for reflection; and approachability and vulnerability.
One physician stated how it made him think from the patient’s point of view, making him think of the consequences for patients when he made practice changes.
“But just the opportunity to be able to talk to somebody, and if there was anything, and then sometimes that perspective is like, oh, I hadn't actually thought of it that way, or, you know, pointing out, well as much as you thought that score was a little lower, it actually fits with what everybody else thought you did too, so. So that objective eye.”
Structure for reflection
“I can flip though the report, but having the reflection tool actually gave me a framework to go back to it and say - so it did help. Because it gave me a little more of a systematic approach to the report, rather than just kind of flipping through the pages and comparing where did the flag go up, where did the flag go down? I can kind of analyze it a little more specifically.”
Approachability and vulnerability
Even those who did not feel any changes were necessary found the reflection a useful exercise to complete. Reflecting gave physicians a chance to really consider how they felt about the feedback they had received, and most were very reassured by this feedback about how they practiced.
“...you start with the college, it is a little bit intimidating, and it is at least it was to me. You know there is a certain amount of vulnerability where you ask people to let you know what you think about the work you do, and about the way you approach your work, and you are kind of leaving yourself open to criticism or comment. And so it was interesting to get another person's perspective about feedback that I got. I think that the reflection exercise forces you to look at the information that you are provided a little more carefully and closely than you would otherwise. I could see having received the report, thumbed through it, and thought “Gee I think it is mostly pretty good and be happy with that. But the reflection exercise forces you to look at it a little more closely.””
This study represents an important contribution to understanding whether physicians use performance assessment feedback to improve their practice. Dedicated reflection has been demonstrated to be required for performance data to change and improve practice, as physicians will generally not naturally make changes based on feedback alone. Peer-assisted debriefing can facilitate this reflection, and is similar to other peer-based reflection processes observed in other contexts . It is easy to implement, is not resource-intensive, and qualitative feedback implies that it is effective at encouraging change. Debriefing and guided reflection has been shown to be both effective at promoting change and perceived as a valuable exercise among students following simulation activities . Although the context of this study is quite different from studies among students, the evidence indicates that further research in this area is warranted. It is unclear as to whether specific attributes of the physician’s practice type or of the peer selected for the debriefing (such as the difference between a physician and non-physician colleague) impacts on the process or its outcomes.
Participants, with the aid of their peers, identified areas for change, developed an intervention to make changes, and by and large, felt successful at implementing those changes. Reflecting on the process, participants found a more objective perspective helpful, with the exercise providing structure to work through the report of their assessment with a peer without the pressure of the regulatory college, with the incentive of CPD credits. The authors acknowledge that this data is limited insofar as it is a self-selecting sample, using self-reported data with participation incentivized by CPD credits. While this study relies on physician self-reports of change, many did report tangible changes, such as changes to scheduling systems and the implementation of EMRs. Peer-assisted debriefing is likely more than sufficient at encouraging change for the 80% of physicians that are left to process reports completely independently following a multisource feedback review. Moreover, there is some evidence that commitment to change statements can predict actual change in practice .
Despite concerns that 360 assessments focus heavily on Communicator, Collaborator, and Professional domains, areas of change identified by participants covered all seven CanMEDS Roles, including Medical Expert, Leader, Scholar, and Health Advocate. It is as yet unclear whether this was based on the report alone, discussions and feedback from the peer, simply the reflection process, or some combination thereof. Further study is warranted into how physicians identified areas for change and, objectively how effective they were in implementing change following peer-assisted debriefing and guided reflection.
The majority of physicians who completed the peer-assisted debriefing, including reflection and commitment to change, identified areas for change in their practice. These areas covered all the roles a physician plays, as defined by the CanMEDS framework. Interviews with physicians who had undergone the complete process discussed three major themes: comparing perspectives, structure for reflection, and approachability and vulnerability. Though further research is warranted into the nature of the peer relationship and the implementation of changes, the peer-assisted debriefing via a reflection exercise was a means to support physicians in interpreting and making sense of feedback, providing an opportunity for physicians to reflect and process the results of their 360 assessment to plan for continuous improvement.
The authors would like to acknowledge the Office of Continuing Competency and Assessment (previously Division of Continuing Professional Development) for their role in the administration of the MPAR reflection exercise CPD activity.
Availability of data and materials
The datasets generated in the current study are not publicly available due to individual privacy issues.
All authors made substantive intellectual contributions to the study, collaboratively designing, analyzing, and interpreting the research, and writing and revising the manuscript. All authors have given approval for this manuscript to be published.
Ethics approval and consent to participate
This study was approved by the Health Research Ethics Board at the University of Manitoba (H2013:419). Consent was not sought for secondary analysis of completed RE forms; interview participants provided written informed consent.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 2.The College of Physicians and Surgeons of Manitoba. About MPAR. Manitoba Physician Achievement Review. http://mpar.ca/welcome/about-mpar. Accessed 15 Feb, 2015.
- 8.Sargeant J, Lockyer J, Mann K, Holmboe E, Silver I, Arson H, Driessen E, MacLeod T, Yen W, Ross K, Power M. Facilitated reflective performance feedback: developing an evidence- and theory-based model that builds relationship, explores reactions and content, and coaches for performance change (R2C2). Acad Med. 2015;90(12):1698–706.CrossRefGoogle Scholar
- 11.Patel PD, Kischnick DB, Bickel SG, Ziegler CH, Miller KH. Evaluating the utility of peer-assisted learning in pediatrics. Med. Sci Educ. 2011;21(4):210–3.Google Scholar
- 14.Frank JR, Snell L, Sherbino J, editors. CanMEDS 2015 physician competency framework. Ottawa: Royal College of Physicians and Surgeons of Canada; 2015.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.