Skip to main content
SpringerLink
Log in

A review of multi-source feedback focusing on psychometrics, pitfalls and some possible solutions

  • Review Paper
  • Published:
SN Social Sciences Aims and scope Submit manuscript

Abstract

With the re-emergence of competency-based frameworks in professional education, multisource feedback (MSF) has become a common method for assessing various competencies, including communication, professionalism, and aspects of team-based performance. A wide variety of publications over the past 50 years or more in the business and health literature would seem to support the use of MSF at least for quality improvement (QI) purposes. However, our own experience with using MSF in physicians has been quite mixed, with some physicians embracing the experience and making improvements and others the complete opposite. We decided to review the existing literature on MSF to try to identify key aspects of successful MSF programs. This paper presents a structured critique of the literature on the use of MSF in physician populations. The findings were surprising as key assumptions around the validity and reliability of MSF were not consistently met, key lessons from earlier research were not carried over to present day programs and perhaps most concerning was a lack of evidence for MSF producing meaningful sustained behavior change. From these findings we suggest some key areas of potential improvement in MSF programs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data availability

This manuscript has no associated data.

References

  • Accreditation Council for Graduate Medical Education. (2006). Introduction to competency-based residency education. https://216.92.22.76/discus/messages/21/MODULE__1_Facilitator_Manual-424.pdf

  • Ao P, Kahlon S, Daniels V, Goldstein C (2014) Development of a multi-source feedback tool in the assessment of intrinsic CanMEDS roles for senior internal medicine residents. In JGME online, https://www.jgme.org/userimages/ContentEditor/1412610797951/2014.ICRE.Abstracts.with.Cover.pdf

  • Archer J, Norcini J, Southgate L, Heard S, Davies H (2005) mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? Adv Health Sci Educ 13:191–192

    Google Scholar 

  • Archer J, McGraw M, Davies H (2010a) Assuring validity of multi-source feedback in a national programme. Arch Dis Child 95:330–335

    Article  Google Scholar 

  • Archer J, McGraw M, Davies H (2010b) Assuring validity of multi-source feedback in a national programme. PMJ 86:526–531

    Google Scholar 

  • Ashworth N, Kain NA, Jess E et al (2020) Survey of physician attitudes to using multisource feedback for competence assessment in Alberta. BMJ Open. https://doi.org/10.1136/bmjopen-2020-037610

    Article  Google Scholar 

  • American Board of Medical Specialties & Accreditation Council for Graduate Medical Education. (2000). Toolbox of assessment methods. https://uit.no/Content/333597/5%20Evaluationtoolbox.pdf

  • Bracken D, Timmreck C, Church A (2001) The handbook of multi-source feedback. Jossey-Bass Inc, San Francisco

    Google Scholar 

  • Bramley T (2005) A rank-ordering method for equating test by expert judgment. JAM 6:202–223

    Google Scholar 

  • Brinkman WB, Geraghty SR, Lanphear BP, Khoury JC, Gonzalez del Ray JA, Dewitt TG, Britto MT (2007) Effect of multi-source feedback on resident communication skills and professionalism: a randomized controlled trial. Arch Pediatr Adolesc Med 161:44–49

    Article  Google Scholar 

  • Brown A, Maydeu-Olivares A (2011) Item response modeling of forced-choice questionnaires. Educ Psychol Meas 71:460–502

    Article  Google Scholar 

  • Brutus S (2009) Words versus numbers: a theoretical exploration of giving and receiving narrative comments in performance appraisal. Hum Resour Manag Rev 2:144–157

    Google Scholar 

  • Campbell J, Wright C (2012) GMC multi-source feedback questionnaires interpreting and handling multi-source feedback results: Guidance for appraisers. https://www.gmcuk.org//media/documents/guidance-for-appraisers---pms-45189197.pdf

  • Campbell J, Richards SH, Dickens A, Greco M, Narayanan A, Bearley S (2008) Assessing the professional performance of UK doctors: an evaluation of the utility of the general medical council patient and colleague questionnaires. Qual Saf Health Care 17:187–193

    Article  Google Scholar 

  • Campbell J, Narayanan A, Burford B, Greco M (2010) Validation of a multi-source feedback took for use in general practice. Educ Prim Care 21:165–179

    Article  Google Scholar 

  • Carraccio C, Wolfsthal S, Englander R, Ferentz K, Martin C (2002) Shifting paradigms: from Flexner to competencies. Acad Med 77:361–367

    Article  Google Scholar 

  • Crossley JG (2015) Development and validation of a trustworthy multi-source feedback instrument to support nurse appraisals. J Contin Educ Health Prof 35:91–98

    Article  Google Scholar 

  • Davies HA, Archer JC (2005) Multi-source feedback using sheffield peer review assessment tool (SPRAT)—development and practical aspects. Clin Teach 2:77–81

    Article  Google Scholar 

  • De Champlain AF, Gotzmann A, Qin S (2016) Assessing the reliability of your performance assessment scores: some considerations in selecting an appropriate framework. JGME 8:504–506

    Article  Google Scholar 

  • Donnon T, Al Ansari A, Al Alawi A, Violato C (2014) The reliability, validity, and feasibility of multi-source feedback physician assessment: a systematic review. Acad Med 89:511–516

    Article  Google Scholar 

  • Eva KW, Regehr G (2013) Effective feedback for maintenance of competence: from data delivery to trusting dialogues. CMAJ 185:463–464

    Article  Google Scholar 

  • Ferguson J, Wakeling J, Bowie P (2014) Factors influencing the effectiveness of multi-source feedback in improving the professional practice of medical doctors: a systematic review. BMC Med Educ 14:1–12

    Article  Google Scholar 

  • Frank JR, Snell L, Sherbino J (2015) CanMEDS 2015 physician competency framework. Royal College of Physicians and Surgeons of Canada https://canmeds.royalcollege.ca/uploads/en/framework/CanMEDS%202015%20Framework_EN_Reduced.pdf

  • Garra G, Wackett A, Thode H (2011) Feasibility and reliability of a multi-source feedback tool for emergency medicine residents. JGME 3:356–359

    Article  Google Scholar 

  • General Medical Council. (2011). The good medical practice framework for appraisal and revalidation. (London: General Medical Council). https://www.gmc-uk.org/-/media/documents/The_Good_medical_practice_framework_for_appraisal_and_revalidation___DC5707.pdf_56235089.pdf

  • Hall W, Violato C, Lewkonia R, Lockyer J, Fidler H, Toews J, Moores D et al (1999) Assessment of physician performance in Alberta—the physician achievement review. CMAJ 161:52–57

    Google Scholar 

  • Ilgen D, Davis C (2000) Bearing bad news: reactions to negative performance feedback. Appl Psychol 49:550–565

    Article  Google Scholar 

  • Kane M (1992) An argument-based approach to validation. Psychol Bull 112:527–535

    Article  Google Scholar 

  • Kane M (2004) Certification testing as illustration of argument-based validation. Measurement 2:135–170

    Google Scholar 

  • Kluger AN, DeNisi A (1996) The effects of feedback interventions on performance: a historical review, meta-analysis, and a preliminary feedback. Psychol Bull 119:254–284

    Article  Google Scholar 

  • Kumar D (2005) Performance appraisal: the importance of rater training. J Kuala Lumpur RMPC 4:1–17

    Google Scholar 

  • Lee KL, Tsai SL, Chiu YT, Ho MJ (2016) Can student self-ratings be compared with peer ratings? A study of measurement invariance of multi-source feedback. Adv Health Sci Educ 21:401–413

    Article  Google Scholar 

  • Lipner RS, Blank LL, Leas BF, Fortna GS (2002) The value of patient and peer ratings in recertification. Acad Med 77:s64–s66

    Article  Google Scholar 

  • Lockyer J (2003) Multi-source feedback in the assessment of physician competencies. J Contin Educ Health Prof 23:4–12

    Article  Google Scholar 

  • Lockyer J (2013) Multi-source feedback: can it meet criteria for good assessment? J Contin Educ Health Prof 33:89–98

    Article  Google Scholar 

  • Lockyer J, Violato C, Fidler HM (2003) Likelihood of change: a study assessing surgeon use of multi-source feedback data. Teach Learn Med 15:168–174

    Article  Google Scholar 

  • Lockyer J, Violato C, Fidler HM (2007) What multi-source feedback factors influence physician self-assessments? A five-year longitudinal study. Acad Med 82:s77–s80

    Article  Google Scholar 

  • McCarthy AM, Garavan TN (2001) 360° feedback process: performance, improvement and employee career development. J Eur Indus Train 25:5–32

    Article  Google Scholar 

  • McGaghie WC, Sajid AW, Miller GE, Telder TV, Lipson L et al. (1978) Competency-based curriculum development in medical education: an introduction/William C. McGaghie ... [et al.]; with the assistance of Laurette Lipson. World Health Organization. https://apps.who.int/iris/handle/10665/39703

  • McGill DA, van der Vleuten CPM, Clarke MJ (2011) Supervisor assessment of clinical and professional competence of medical trainees: a reliability study using workplace data and a focused analytical literature review. Adv Health Sci Educ 16:405–425

    Article  Google Scholar 

  • Messick S (1989) Validity. In: Linn RL (ed) Educational measurement, 3rd edn. Macmillan, New York, pp 13–104

    Google Scholar 

  • Miller A, Archer J (2010) Impact of workplace-based assessment on doctors’ education and performance: a systematic review. BMJ 341:c5064

    Article  Google Scholar 

  • Moonen-van Loon JMW, Overeem K, Donkers HHLM, van der Vleuten CPM, Driessen EW (2013) Composite reliability of a workplace-based assessment toolbox for postgraduate medical education. Adv Health Sci Educ 18:1087–1102

    Article  Google Scholar 

  • Moonen-van Loon JMW, Overeem K, Govaerts MJB, Verhoeven BH, van der Vleuten CPM, Driessen E (2015) The reliability of multi-source feedback in competency-based assessment programs: the effects of multiple occasions and assessor groups. Acad Med 90:1093–1099

    Article  Google Scholar 

  • Morse JM (2010) Simultaneous and sequential qualitative mixed method designs. Qual Inq 16:483–491

    Article  Google Scholar 

  • Moses J, Hollenbeck GO, Sorcher M (1993) Other people’s expectations. Hum Resour Manag 32:283–297

    Article  Google Scholar 

  • Overeem K, Wollersheim HC, Driessen E, Lombarts K, Van De Ven G, Grol R, Arah O (2009) Doctors’ perceptions of why 360-degree feedback does (not) work: a qualitative study. Med Educ 43:12–25

    Article  Google Scholar 

  • Overeem K, Lombarts MJ, Arah OA, Klazinga NS, Grol RP, Wollersheim HC (2010) Three methods of multi-source feedback compared: a plea for narrative comments and coworkers’ perspectives. Med Teach 32:141–147

    Article  Google Scholar 

  • Overeem K, Wollersheim HC, Onyebuchi AA, Cruijsberg J, Grol R, Lombarts K (2012) Evaluation of physicians’ professional performance: an iterative development and validation study of multi-source feedback instruments. BMC Health Serv Res 12:1–11

    Article  Google Scholar 

  • Peiperl MA (2001) Getting 360 degree feedback right. Harv Bus Rev 79:142–177

    Google Scholar 

  • Rosti RT, Shipper F (1998) A study of the impact of training in a management development program based on 360 feedback. J Manag Psychol 13:77–89

    Article  Google Scholar 

  • Saedon H, Salleh S, Balakrishnan A, Imray C, Saedon M (2012) The role of feedback in improving the effectiveness of workplace-based assessments: a systematic review. BMC Med Educ 12:25

    Article  Google Scholar 

  • Sargeant J, Mann K, Ferrier S (2005) Exploring family physicians’ reactions to multi-source feedback: perceptions of credibility and usefulness. Med Educ 39:497–504

    Article  Google Scholar 

  • Sargeant J, McNaughton E, Mercer S, Murphy D, Sullivan P, Bruce D (2011) Providing feedback: exploring a model (emotion, content, outcomes) for facilitating multi-source feedback. Med Teach 33:744–749

    Article  Google Scholar 

  • Schwartz A, Carraccio C, Hicks P et al (2011) Assessment in graduate medical education: a primer for pediatric program directors. American Board of Pediatrics. https://www.abp.org/sites/abp/files/pdf/primer.pdf

  • Sherbino J, Frank JR, Flynn L, Snell LL (2011) “Intrinsic Roles” rather than “armour”: renaming the “non-medical expert roles” of the CanMEDS framework to match their intent. Adv Health Sci Educ 16:695–697

    Article  Google Scholar 

  • Smither JW, London M, Reilly RR (2005) Does performance improve following multi-source feedback? A theoretical model, meta-analysis, and review of empirical findings. Pers Psychol 58:33–66

    Article  Google Scholar 

  • Thurston LL (1959) The measurement values. University of Chicago Press, Chicago

    Google Scholar 

  • Torres A, Greenacre M (2002) Dual scaling and correspondence analysis of preferences, paired comparisons and ratings. Int J Res Market 16:401–405

    Article  Google Scholar 

  • Violato C, Lockyer J, Fidler H (2003) Multi-source feedback: a methods of assessing surgical practice. BMJ 326:546–548

    Article  Google Scholar 

  • Violato C, Lockyer J, Fidler H (2008) Changes in performance: a 5 year-longitudinal study of participants in a multi-source feedback programme. Med Educ 42:1007–1013

    Article  Google Scholar 

  • Vivekananda-Schmidt P, MacKillop L, Crossley J, Wade W (2013) Do assessor comments on a multi-source feedback instrument provide learner-centered feedback? Med Educ 47:1080–1088

    Article  Google Scholar 

  • Wood L, Hassell A, Whitehouse A, Bullock A, Wall D (2006) A literature review of multi-source feedback systems within and without health services, leading to 10 tips for their successful design. Med Teach 28:e185-191

    Article  Google Scholar 

  • Wright C, Richards S, Hill JJ, Roberts MJ, Norman GR, Greco M, Taylor MR, Campbell JL (2012) Multi-source feedback in evaluating the performance of doctors: the example of the UK General Medicine Council patient and college questionnaires. Acad Med 87:1668–1678

    Article  Google Scholar 

  • Zhao Y, Zhang X, Chang Q, Sun B (2013) Psychometric characteristics of the 360 feedback scales in professionalism and interpersonal and communication skills of assessment of surgery residents China. J Surg Educ 70:628–635

    Article  Google Scholar 

Download references

Funding

None.

Author information

Authors and Affiliations

  1. Department of Medicine, University of Alberta, Edmonton, Canada

    Nigel Ashworth & Nicole Kain

  2. Medical Council of Canada, Ottawa, Canada

    André F. De Champlain PhD

  3. Research & Evaluation Unit, College of Physicians & Surgeons of Alberta, Edmonton, Canada

    Nigel Ashworth & Nicole Kain

Authors
  1. Nigel Ashworth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. André F. De Champlain PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicole Kain
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ADC had initial responsibility for the psychometric evidence section; NK the historical and qualitative methodology sections; NA the limitations and pitfalls; and ADC, NK, NA the guide to improvements. NA wrote the final document with editing and additions from ADC and NK.

Corresponding author

Correspondence to Nigel Ashworth.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ashworth, N., De Champlain, A.F. & Kain, N. A review of multi-source feedback focusing on psychometrics, pitfalls and some possible solutions. SN Soc Sci 1, 24 (2021). https://doi.org/10.1007/s43545-020-00033-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s43545-020-00033-1

Keywords

Search

Navigation