Factors affecting perceived credibility of assessment in medical education: A scoping review

Abstract

Assessment is more educationally effective when learners engage with assessment processes and perceive the feedback received as credible. With the goal of optimizing the educational value of assessment in medical education, we mapped the primary literature to identify factors that may affect a learner’s perceptions of the credibility of assessment and assessment-generated feedback (i.e., scores or narrative comments). For this scoping review, search strategies were developed and executed in five databases. Eligible articles were primary research studies with medical learners (i.e., medical students to post-graduate fellows) as the focal population, discussed assessment of individual learners, and reported on perceived credibility in the context of assessment or assessment-generated feedback. We identified 4705 articles published between 2000 and November 16, 2020. Abstracts were screened by two reviewers; disagreements were adjudicated by a third reviewer. Full-text review resulted in 80 articles included in this synthesis. We identified three sets of intertwined factors that affect learners’ perceived credibility of assessment and assessment-generated feedback: (i) elements of an assessment process, (ii) learners’ level of training, and (iii) context of medical education. Medical learners make judgments regarding the credibility of assessments and assessment-generated feedback, which are influenced by a variety of individual, process, and contextual factors. Judgments of credibility appear to influence what information will or will not be used to improve later performance. For assessment to be educationally valuable, design and use of assessment-generated feedback should consider how learners interpret, use, or discount assessment-generated feedback.

Appendices

Appendix

Supplemental digital Appendix 1: Search strategy for OVID medline

#	Search Statement
1	((Evaluator* or feedback or Assessment or Assess or assessed or judgements or assignment or assignments or Exam or exams or Examination* or Questionnaire or Questions or Testing or test or tests or Tester or Evaluation or Evaluate or Rating or Scales or Appraisal or Score or Scores or Grades) adj2 (credibility or receptivity or Helpful or counterproductive or Credible or Constructive or Justifiable or defensible or Relevant or Reasonable or legitimate or Influence or productive or trust or trustworthy or trustworthiness or merit or value or acceptable or appropriate or applicable or fair or fairness)).ab,kf,ti
2	*Feedback/ or formative feedback/ or *feedback, psychological/
3	(Perception* or perceived).ab,hw,kf,ti
4	Perception/
5	3 or 4
6	2 and 5
7	1 or 6
8	Education, medical/ or education, medical, graduate/ or education, medical, undergraduate/ or "internship and residency"/
9	Students, Medical/
10	Schools, Medical/
11	*Clinical Competence/
12	"Clerkship*".ab,kf,ti
13	"Undergraduate medic*".ab,kf,ti
14	"Graduate medic*".ab,kf,ti
15	((Resident or residents or residency) adj3 (medicine or school or education)).ab,kf,ti
16	((Intern or Interns or Internship) adj3 (medicine or school or education)).ab,kf,ti
17	Medical students.ab,kf,ti
18	Medical schools.ab,kf,ti
19	(Post?graduate* adj2 medic*).ab,kf,ti
20	"House officer*".ab,kf,ti
21	"Registrar*".ab,kf,ti
22	foundation year.ab,kf,ti
23	"Junior doctor*".ab,kf,ti
24	8 or 9 or 10 or 11 or 12 or 13 or 14 or 15 or 16 or 17 or 18 or 19 or 20 or 21 or 22 or 23
25	Nursing home.ab,ti
26	24 not 25
27	N.sb
28	26 not 27
29	7 and 28
30	Lmit 29 to (english or french)
31	Limit 30 to yr = "2000 -Current"

Supplemental digital Appendix 2: References of all studies included in this scoping review

Citation
Arnold L, Shue CK, Kritt B, Ginsburg S, Stern DT. Medical students' views on peer assessment of professionalism. Journal of General Internal Medicine. 2005;20(9), 819–824
Areemit RS, Cooper CM, Wirasorn K, Paopongsawan P, Panthongviriyakul C, & Ramani S. (2020). Hierarchy, "Kreng Jai" and feedback: A grounded theory study exploring perspectives of clinical faculty and medical students in Thailand. Teaching & Learning in Medicine, 1–10
Barsoumian AE, and Yun HC. (2018). Augmenting fellow education through spaced multiple-choice questions. Military Medicine, 183(1), e122-126
Beaulieu AM, Kim BS, Topor DR, Dickey CC. (2019). Seeing is believing: An exploration of what residents value when they receive feedback. Academic Psychiatry, 43(5), 507–511
Bello RJ, Sarmiento S, Meyer ML, Rosson GD, Cooney DC, Lifchez SD, & Cooney CM. (2018). Understanding surgical resident and fellow perspectives on their operative performance feedback needs: A qualitative study. Journal of Surgical Education, 75(6), 1498–1503
Bing-You R, Hayes V, Palka T, Ford M, & Trowbridge R. (2018). The art (and artifice) of seeking feedback: Clerkship students' approaches to asking for feedback. Academic Medicine, 93(8), 1218–1226
Bleasel J, Burgess A, Weeks R, Haq I. Feedback using an ePortfolio for medicine long cases: Quality not quantity. BioMed Central Medical Education. 2016;16(1), 278
Bogetz AL, Orlov N, Blankenburg R, Bhavaraju V, McQueen A, & Rassbach C. (2018). How residents learn from patient feedback: A multi-institutional qualitative study of pediatrics residents' perspectives. Journal of Graduate Medical Education, 10(2), 176–184
Bowen L, Marshall M, Murdoch-Eaton D. (2017). Medical student perceptions of feedback and feedback behaviors within the context of the "Educational Alliance". Academic Medicine, 92(9), 1303–1312
Braund H, Dalgarno N, McEwen L, Egan R, Reid MA, & Baxter S. (2019). Involving ophthalmology departmental stakeholders in developing workplace-based assessment tools. Canadian Journal of Ophthalmology, 54(5), 590–600
Brits H, Bezuidenhout J, van der Merwe LJ, Joubert G.  (2020). Students’ voices: Assessment in undergraduate clinical medicine. Pan African Medical Journal, 36, 1–9
Brown JM, Lowe K, Fillingham J, Murphy PN, Bamforth M, Shaw NJB. An investigation into the use of multi-source feedback (MSF) as a work-based assessment tool. Medical teacher. 2014;36(11), 997–1004
Burgess A, Mellis C. Receiving feedback from peers: Medical students' perceptions. The Clinical Teacher. 2015;12(3), 203–207
Castonguay V, Lavoie P, Karazivan P, Morris J, Gagnon R. (2019). Perceptions of emergency medicine residents of multisource feedback: Different, relevant, and useful information. Annals of Emergency Medicine, 74(5), 660–669
Chaffinch CN, Poynter SE, Turner T, Schwartz A, APPD-LEARN-NBME Pediatrics Milestones Assessment Group. Learner feedback on the pediatrics milestones assessment project . Academic Pediatrics. 2013;13(4), e6-e7
Cho SP, Parry D, Wade W. Lessons learnt from a pilot of assessment for learning. Clinical Medicine. 2014;14(6):577–84
Craig S, Braitberg G, Nicolas C, White G, Egerton-Warburton D. Assessment and feedback in emergency medicine training: Views of Australasian emergency trainees. Emergency Medicine Australasia. 2010;22(6), 537–547
Curran VR, Deacon D, Schulz H, Stringer K, Stone CN, Duggan N, & Coombs-Thorne H. (2018). Evaluation of the characteristics of a workplace assessment form to assess Entrustable Professional Activities (EPAs) in an undergraduate surgery core clerkship. Journal of Surgical Education, 75(5), 1211–1222
Curran VR, Butler R, Duke P, Eaton WH, Moffatt SM, Sherman GP, Pottle M. Evaluation of the usefulness of simulated clinical examination in family-medicine residency program. Medical Teacher. 2007;29(4), 406–407
Dijksterhuis MGK, Schuwirth LWT, Braat DDM, Teunissen PW, Scheele F. A qualitative study on trainees' and supervisors' perceptions of assessment for learning in postgraduate medical education. Medical Teacher. 2013;35(8), e1396-1402
Duijn CCMA, Welink LS, Moki M, Ten Cate OTJ, Kremer WDJ, Bok, HGJ. Am I ready for it? Students' perceptions of meaningful feedback on entrustable professional activities. Perspectives on Medical Education. 2017
Eady K, and Moreau KA. (2018). Using parent feedback: A qualitative study of residents' and physician-educators' perspectives. Perspectives on Medical Education, 7(1), 33–39
Feller K, and Berendonk C. (2020). Identity matters—perceptions of inter-professional feedback in the context of workplace-based assessment in Diabetology training: A qualitative study. BMC Medical Education, 20(1), 33-41
Foley T, McLoughlin L, Walsh EK, Leggett P, O'Reilly M, Owens M, & Jennings AA. (2018). The candidate perspective of the clinical competency test (CCT) of the MICGP examination: A mixed-methods study. British Journal of General Practice Open, 2(3), bjgpopen18X101605
Gaunt A, Patel A, Rusius V, Royle TJ, Markham DH, & Pawlikowska T. (2017). 'Playing the game': How do surgical trainees seek feedback using workplace-based assessment? Medical Education, 51(9), 953–962
Given K, Hannigan A, McGrath D. (2016). Red, yellow and green: What does it mean? How the progress test informs and supports student progress. Medical Teacher, 38(10), 1025–1032
Green A, Miller E, Krupat E, White A, Taylor WC, Hirsh DA, Wilson RP, Betancourt JR. Designing and implementing a cultural competence OSCE: Lessons learned from interviews with medical students. Ethnicity & Disease. 2007;17(2), 344–350
Gulbas L, Guerin W, Ryder HF. Does what we write matter? Determining the features of high- and low-quality summative written comments of students on the internal medicine clerkship using pile-sort and consensus analysis: A mixed-methods study. BioMed Central Medical Education. 2016;16, 145-153
Hagiwara Y, Ross J, Lee S, Sanchez-Reilly S. (2017). Tough conversations: Development of a curriculum for medical students to lead family meetings. American Journal of Hospice and Palliative Medicine, 34(10), 907–911
Harrison CJ, Konings KD, Dannefer E, Schuwirth L, Wass V, Van der Vleuten C. Factors influencing students' receptivity to formative feedback emerging from different assessment cultures. Perspectives of Medical Education. 2016;5(5), 276–284
Harrison CJ, Molyneux AJ, Blackwell S, Wass VJ. How we give personalised audio feedback after summative OSCEs. Medical Teacher. 2015;37(4), 323–326
Heeneman S, Oudkerk Pool A, Schuwirth LWT, van der Vleuten CPM, Driessen E. The impact of programmatic assessment on student learning: Theory versus practice. Medical Education. 2015;49(5), 487–498
Ingram JR, Anderson EJ, Pugsley L. Difficulty giving feedback on underperformance undermines the educational value of multi-source feedback. Medical Teacher. 2013;35(10), 838–846
Jawaid M, Masood Z, Jaleel F. Students' perception of surgical objective structured clinical examination (OSCE) at Dow University of Health Sciences. Journal of Postgraduate Medical Institute. 2014;28(1), 19–23
Jefferies A, Simmons B, Ng E, Skidmore M. Assessment of multiple physician competencies in postgraduate training: Utility of the structured oral examination. Advances in Health Sciences Education. 2011;16(5), 569–577
Johnson G, Barrett J, Jones M, Parry D, Wade W. Feedback from educational supervisors and trainees on the implementation of curricula and the assessment system for core medical training. Clinical Medicine. 2008;8(5), 484–489
Junod Perron N, Louis-Simonet M, Cerutti B, Pfarrwaller E, Sommer J, Nendaz M. The quality of feedback during formative OSCEs depends on the tutors' profile. BioMed Central Medical Education. 2016;16(1), 293-301
Kalet AL, Sanger J, Chase J, Keller A, Schwartz MD, Fishman ML, Garfall AL, Kitay A. Promoting professionalism through an online professional development portfolio: Successes, joys, and frustrations. Academic Medicine. 2007;82(11), 1065–1072
Khairy GA. Feasibility and acceptability of Objective Structured Clinical Examination (OSCE) for a large number of candidates: Experience at a university hospital. Journal of Family and Community Medicine. 2004;11(2), 75–78
Khorashad AK, Salari S, Baharvahdat H, Hejazi S, Lari SM, Salari M, Mazloomi M, Lari SM. The assessment of undergraduate medical students’ satisfaction levels with the Objective Structured Clinical Examination. Iranian Red Crescent Medical Journal. 2014;16(8), e13088-13092
Kim H, Bar Ad V, McAna J, Dicker AP. Evaluating the quality, clinical relevance, and resident perception of the radiation oncology in-training examination: A national survey. Practical Radiation Oncology. 2016;6(1), 44–49
Korszun A, Winterburn PJ, S, H, Tapper-Jones L, Houston H. Assessment of professional attitude and conduct in medical undergraduates. Medical Teacher. 2005;27(8), 704–8
Labaf A, Eftekhar H, Majlesi F, Anvari P, Sheybaee-Moghaddam F, Jan D, Jamali A. Students' concerns about the pre-internship objective structured clinical examination in medical education. Education for Health. 2014;27(2), 188–192
LaDonna KA, Hatala R, Lingard L, Voyer S, Watling C. (2017). Staging a performance: Learners' perceptions about direct observation during residency. Medical Education, 51(5), 498–510
Lefroy J, Hawarden A, Gay SP, McKinley RK, Cleland J. Grades in formative workplace-based assessment: A study of what works for whom and why. Medical Education. 2015;49(3), 307–320
Levine JC, Geva T, Brown DW. Competency testing for pediatric cardiology fellows learning transthoracic echocardiography: Implementation, fellow experience, and lessons learned. Pediatric Cardiology. 2015;36(8), 1700–1711
MacNeil K, Cuncic C, Voyer S, Butler D, Hatala R. (2020). Necessary but Not Sufficient: Identifying Conditions for Effective Feedback during Internal Medicine Residents' Clinical Education. Advances in Health Sciences Education, 25(3), 641–654
McKavanagh P, Smyth A, Carragher A. Hospital consultants and workplace-based assessments: How foundation doctors view these educational interactions? Postgraduate Medical Journal. 2012;88(1037), 119–124
McLaughlin K, Coderre S, Woloschuk W, Mandin H. Does blueprint publication affect students' perception of validity of the evaluation process? Advances in Health Sciences Education. 2005;10(1), 15–22
McLay R, Rodenhauser P, Anderson D, Stanton M, Markert RJ. Simulating a full-length psychiatric interview with a complex patient: An OSCE for medical students. Academic Psychiatry. 2002;26, 162–167
Mohanaruban A, Flanders L, Rees H. (2018). Case-based discussion: perceptions of feedback. The Clinical Teacher, 15(2), 126–131
Moreau KA, Eady K, Jabbour M. (2019). Exploring residents' reactions to and use of parent feedback in a pediatric emergency department: A grounded theory study. Medical Teacher, 41(2), 207–214
Mukhtar O, Griffin J, Naheed S, Bryant C. (2018). The Multiple Consultant Report: New assessment; same old problems. The Clinical Teacher, 15(1), 34–37
Murdoch-Eaton D, Sargeant J. Maturational differences in undergraduate medical students' perceptions about feedback. Medical Education. 2012;46(7), 711–721
Nesbitt A, Baird F, Canning B, Griffin A, Sturrock A. Student perception of workplace-based assessment. The Clinical Teacher. 2013;10(6), 399–404
Nikendei C, Kraus B, Lauber H, Schrauth M, Weyrich P, Zipfel S, Junger J, Briem S. An innovative model for teaching complex clinical procedures: Integration of standardised patients into ward round training for final year students. Medical Teacher. 2007;29(2), 246–252
Olsson J, Ekblad S, Bertilson BC, Toth-Pal E. (2018). Swedish adaptation of the General Medical Council's multisource feedback questionnaires: A qualitative study. International Journal of Medical Education, 9, 161–169
Papinczak T, Young L, Groves M. Peer assessment in problem-based learning: A qualitative study. Advances in Health Sciences Education. 2007;12(2), 169–186
Perera J, Lee N, Win K, Perera J, Wijesuriya L. Formative feedback to students: the mismatch between faculty perceptions and student expectations. Medical Teacher. 2008;30(4), 395–399
Pierre RB, Wierenga A, Barton M, Branday JM, Christie CDC. Student evaluation of an OSCE in paediatrics at the University of the West Indies, Jamaica. BioMed Central Medical Education. 2004;4(22), 1-7
Preston R, Gratani M, Owens K, Roche P, Zimanyi M, & Malau-Aduli B. (2020). Exploring the impact of assessment on medical students' learning. Assessment & Evaluation in Higher Education, 45(1), 109–124
Ramani S, Konings KD, Ginsburg S, van der Vleuten CPM. (2020). Relationships as the backbone of feedback: Exploring preceptor and resident perceptions of their behaviors during feedback conversations. Academic Medicine, 95(7), 1073–1081
Rees C, Sheard C, McPherson A. Communication skills assessment: The perceptions of medical students at the University of Nottingham. Medical Education. 2002;36(9), 868–878
Ricci M, St-Onge C, Xiao J, Young M. (2018). Students as stakeholders in assessment: how students perceive the value of an assessment. Perspectives on Medical Education, 7(6), 352–361
Ringsted C, Henriksen AH, Skaarup AM, Van der Vleuten CP. Educational impact of in-training assessment (ITA) in postgraduate medical education: A qualitative study of an ITA programme in actual practice. Medical Education. 2004;38(7), 767–777
Ryan A, McColl GJ, O'Brien R, Chiavaroli N, Judd T, Finch S, & Swanson D. (2017). Tensions in post-examination feedback: information for learning versus potential for harm. Medical Education, 51(9), 963–973
Sabey A, Harris M. Training in hospitals: What do GP specialist trainees think of workplacebased assessments? Education for Primary Care. 2011;22(2), 90–99
Sargeant J, Eva K, Mann K, Lockyer J, Armson H, Van Der Vleuten C, Dornan T, Holmboe E, Chesluk B, Loney E. Conditions influencing informed self-assessment and use of feedback. Medical Education. 2011;45, 40–1
Schut S, Driessen E, van Tarwijk J, van der Vleuten C, Heeneman S. (2018). Stakes in the eye of the beholder: an international study of learners' perceptions within programmatic assessment. Medical Education, 52(6), 654–663
Shafi R, Irshad K, Iqbal M. Competency-based integrated practical examinations: Bringing relevance to basic science laboratory examinations. Medical Teacher. 2010;32(10), e443-447
Sharma S, Sharma V, Sharma M, Awasthi B, Chaudhary S. Formative assessment in postgraduate medical education—Perceptions of students and teachers. International Journal of Applied & Basic Medical Research. 2015;5, S66-70
Suhoyo Y, Van Hell EA, Kerdijk W, Emilia O, Schonrock-Adema J, Kuks JBM, Cohen-Schotanus J. Influence of feedback characteristics on perceived learning value of feedback in clerkships: Does culture matter? BioMed Central Medical Education. 2017;17(1), 69-74
Vishwakarma K, Sharma M, Matreja P, Giri V. Introducing objective structured practical examination as a method of learning and evaluation for undergraduate pharmacology. Indian Journal of Pharmacology. 2016;48(7), S47-S51
Wade L, Harrison CJ, Mattick K, Ricketts C, et al. Student perceptions of the progress test in two settings and the implications for test deployment. Advances in Health Sciences Education. 2012;17(4), 573–83
Watling CJ, Kenyon CF, Zibrowski EM, Schulz V, Goldszmidt MA, Singh I, et al. Rules of engagement: Residents' perceptions of the in-training evaluation process. Academic Medicine. 2008;83(10), S97-100
Weller JM, Jones A, Merry AF, Jolly B, Saunders D. Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: Implications for implementation. British Journal of Anaesthesia. 2009;103(4), 524–30
Wiener-Ogilvie S, Begg D. Formative assessment of GP trainees' clinical skills. Education for Primary Care. 2012;23(2), 101–6
Yielder J, Wearn A, Chen Y, Henning MA, Weller J, Lillis S, Mogol V, & Bagg W. (2017). A qualitative exploration of student perceptions of the impact of progress tests on learning and emotional wellbeing. BMC Medical Education, 17(1), 148

Supplemental digital Appendix 3: Number of publications included, presented per year across time

Supplemental digital Appendix 4: Journals of the included studies

Journal (Ntotal = 49)	N (%)
Medical Education	14 (14.3)
Medical Teacher	14 (14.3)
BMC Medical Education	7 (7.1)
Academic Medicine	5 (5.1)
Advances in Health Sciences Education	5 (5.1)
The Clinical Teacher	4 (4.1)
Education for Primary Care	3 (3.0)
Academic Psychiatry	2 (2.0)
Assessment and Evaluation in Higher Education	2 (2.0)
Clinical Medicine	2 (2.0)
Journal of Pakistani Medical Association	2 (2.0)
Journal of Clinical and Diagnostic Research	2 (2.0)
Perspectives of Medical Education	2 (2.0)
Advances in Physiology Education	1 (1.0)
American Journal of Hospice and Palliative Medicine	1 (1.0)
Anesthesia Pain and Intensive Care	1 (1.0)
Annals of Emergency Medicine	1 (1.0)
British Medical Journal	1 (1.0)
British Journal of Anaesthesia	1 (1.0)
British Journal of General Practice	1 (1.0)
British Journal of Hospital Medicine	1 (1.0)
Canadian Ophthalmological Society	1 (1.0)
Education for Health	1 (1.0)
Emergency Medicine Australasia	1 (1.0)
Ethiopian Medical Journal	1 (1.0)
Ethnicity and Disease	1 (1.0)
Indian Journal of Pharmacology	1 (1.0)
International Journal of Applied and Basic Medical Research	1 (1.0)
International Journal of Medical Education	1 (1.0)
Iran Red Crescent Medical Journal	1 (1.0)
Journal of Medicine Science	1 (1.0)
Journal of Family and Community Medicine	1 (1.0)
Journal of General Internal Medicine	1 (1.0)
Journal of Graduate Medical Education	1 (1.0)
Journal of Postgraduate Medical Institute	1 (1.0)
Military Medicine	1 (1.0)
Pan African Medical Journal	1 (1.0)
Pediatric Cariology	1 (1.0)
Postgraduate Medical Journal	1 (1.0)
Practical Radiation Oncology	1 (1.0)
Sao Paulo Medical Journal	1 (1.0)
Teaching and Learning in Medicine	1 (1.0)

Supplemental digital Appendix 5: Terminology used to refer to the concept of credibility

Term	N (%)
Useful	23 (23)
Fair	17 (17)
Valuable	10 (10)
Helpful	9 (9.1)
Satisfaction	5 (5.1)
Appropriate	4 (4.1)
Valid	4 (4.1)
Acceptable	3 (3.1)
Relevant	3 (3.1)
Adequate	2 (2.0)
Authentic	1 (1.1)
Beneficial	1 (1.1)
Comprehensive	1 (1.1)
Constructive	1 (1.1)
Effective	1 (1.1)
Meaningful	1 (1.1)
Objective	1 (1.1)
Positive	1 (1.1)
Preferred	1 (1.1)
Realistic	1 (1.1)
Reliable	1 (1.1)
Supportive	1 (1.1)
Sufficient	1 (1.1)
Unbiased	1 (1.1)

Supplemental digital Appendix 6: Overview of themes and additional supporting quotations

Factor 1	Elements of an assessment process A. Assessor or feedback provider i. Trusting relationship with assessor “You look for assessors that you know are knowledgeable and where you get something out of it, a good dialogue or really learn something. Not just marks on a sheet of paper” (Trainee 12, p. 773, as reported in (Ringsted et al., 2004)) ii. Perceived interested in long-term trainee progress “There should be a dialogue between my supervisor and myself about my performance on the activity” (Trainee SIU, p. 260 from (Duijn et al., 2017)) iii. Lack of experience/training with assessment “Trainees commented on a lack of engagement and misunderstanding around the use of WBA by some trainers:” (p. 956, (Gaunt et al., 2017)) iv. Respect “You can notice which supervisors are really teaching-minded: they tend to do teach the teacher courses, prepare themselves and give structured feedback.” (participant 5-C from (Dijksterhuis et al., 2013)) B. Procedures of assessment i. Standardization of assessment approach “Performance relative to your peers is very important … it gives you something to sort of work at … that’s actually quite a powerful motivator” (Unidentified trainee, p. 527, (Weller et al., 2009)) ii. Clear purpose “When the learner understood its purpose, he or she would buy into it and, consequently, the element would become meaningful to learning” (p. 495, (Heeneman et al., 2015)) “…their [structured learning events] role is unclear. Trainers or trainees don’t seem to [sic] able to clearly define what constitutes SLEs…” (Unidentified trainee, p. 580, (Cho et al., 2014)) iii. Clinical relevance “Make the primary exams more clinically relevant as they seem very IRRELEVANT when you are sitting them and would rather be learning about information and procedures you need in your daily practice” (Provisional trainee, p.543, (Craig et al., 2010)) iv. Timing “There is not enough exposure to issues where professionalism comes up during first and second year to warrant that much amount of reflection…I honestly think it’s a good thing to want to address the issue of professionalism; but that the portfolio is just not a good way to do so. It doesn’t actually help assess us based on our professional behavior; that will mostly come from being on the wards in third and fourth year” (Unidentified trainee, p. 1071, (Kalet et al., 2007)) “The timing of the [structured clinical exam] at the beginning of PGY1 training was also described by some residents as a helpful refresher because they hadn’t ‘been around patients for a number of months’ since completing their undergraduate medical education” (p. 407, (Curran et al., 2007)) C. Perceived quality of assessment scores “The main problem is the numerical marking. There is no consistency between doctors, some give all 10 s, others refuse to give more than a 6. I think they should be changed so the only grades are fail, pass, clear pass” (Unidentified medical student, p. 402, (Nesbitt et al., 2013)) “Being honest, you do select people that you get on with. If I’d had a problem with somebody I wouldn’t give them a form and whether that makes them valid … well it doesn’t make them valid does it because that person’s opinion might be quite important as part of the process”(Trainee 5, p. 1000, (Brown et al., 2014)) D. Format of assessment scores “You get this load of numbers comparing each class and you know adjusted and corrected and all this, I don’t fully understand it.” (P3, p. 1028, (Given et al., 2016)) E. Consequences of suboptimal performance “The only thing is, if you are being assessed with the purpose to stimulate learning and the result of the assessment is without consequences, the impact will be disappointing” (Trainee 6-C, p. e1399, (Dijksterhuis et al., 2013)) “… Maybe a student would realize that someone’s actually paying attention to their behavior... that affects their grade and that would be the motivation for them” (Trainee 3-A, p. 821, (Arnold et al., 2005)) “It’s nice to know what you did wrong so you can do better the next time… Tell us what we’re doing wrong, just about anything…cause there must always be room for improvement… ‘Do feel free to be harsh!’” (Trainee B5, p. 4, (Bleasel et al., 2016))
Factor 2	Learners’ level of training “…if I’m doing something and if someone gives me positive feedback that makes me try harder and motivates me more, if someone gives me negative feedback I sort of get downhearted” (Trainee A1c, p.718, (Murdoch-Eaton & Sargeant, 2012)) “It means very little to me to always get these ‘great job, great job, great job’ versus someone who is trying to find ways to help me get better” (Trainee FG2, p. 281, (Harrison et al., 2016)) “…[a peer would be]…probably not as reliable …especially if I didn’t know them well, because you don’t want to be harsh and you don’t want to upset them…” (Trainee A1h, p. 718, (Murdoch-Eaton & Sargeant, 2012)) “I valued what they [peers] said as much, if not a bit more sometimes than what the other person [GP], because they knew exactly what I was going, you know, you’re going through the same things, so for them to say something which is good, you know, it was quite good, you had to say something positive first or whatever and then, but then even like the things they said you could improve on you actually took on board” (Medical student 5, p. 872, (Rees et al., 2002)) “…if you didn’t know it you could ask your partner and maybe lHearn it together …I think it reinforces self-reflection or like peer reflection and then if you both are in touch with your tutor… it reinforces… if you have a problem and talk about it aloud then it seems to make more sense just to get somebody receptive to it and they give you a way to think it through…” (Trainee A5a, p. 718, (Murdoch-Eaton & Sargeant, 2012))
Factor 3	Context of medical education i. Safe learning environment “Because there is that level of trust within the group, I don’t mind my peers knowing that I might not know the answer to something. Because I don’t feel they would judge me by the fact I don’t know the answer to something that comes up as part of this” (Trainee M2, p. 1217, (Sargeant et al., 2011)) ii. Consistency of assessment-generated feedback “It is no problem to get some advice of a supervisor on a patient problem, however, usually I get a very directive answer, without him seeing the patient, while I really would like to get some structured feedback after being observed with the patient” (Trainee 3-A, p. e1399, (Dijksterhuis et al., 2013)) “Feedback is more focused now, it’s better i.e. towards clinical things and being a doctor rather than in previous years where it was more general and theoretical” (Trainee 852, p. 718, (Murdoch-Eaton & Sargeant, 2012)) “What feedback? Consistent over several hospitals” (Unidentified trainee, p. 542, (Craig et al., 2010)).