The Use of Observational Diaries in In-Training Evaluation: Student Perceptions Article Received: 24 May 2004 Accepted: 10 January 2005 DOI:
Cite this article as: Govaerts, M.J., van der Vleuten, C.P., Schuwirth, L.W. et al. Adv Health Sci Educ Theory Pract (2005) 10: 171. doi:10.1007/s10459-005-0398-5 Abstract Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential. Key words: clerkship assessment in-training-evaluation performance assessment student perceptions References Brinko, K.T. 1993 The practice of giving feedback to improve teaching Journal of Higher Education 64 574 593 Google Scholar Cleveland, J.N., Murphy, K.R. 1992 Analyzing performance appraisal as goal-directed behaviour Ferris, G. Rowland, K. eds. Research in Personnel and Human Resources Management JAI Press Greenwich, CT 121 185 Google Scholar Cushing, A. 2002 Assessment of non-cognitive factors. Norman, G.R. Vleuten, C.P.M. Newble, D.I. eds. International Handbook of Research in Medical Education Kluwer Academic Publishers Dordrecht 711 755 Google Scholar Ende, J. 1983 Feedback in clinical medical education JAMA 250 777 781 CrossRef PubMed Google Scholar Eraut, M. 1994Developing Professional Knowledge and Competence Falmer Press London Google Scholar Eva, K.W., Cunnington, J.P.W., Reiter, H.I., Keane, D.R., Norman, G.R. 2004 How can I know what I don't know? Poor self assessment in a well-defined domain Advances in Health Sciences Education 9 211 224 CrossRef PubMed Google Scholar Falchikov, N., Boud, D. 1989 Student self-assessment in higher education: A meta-analysis Review of Educational Research 59 395 430 Google Scholar Feletti, G., Cameron, D., Dawson-Saunders, B., Groselliers, J-P., Farmer, E., McAvoy, P. 1994 In-training assessment Newble, D. Jolly, B. Wakeford, R. eds. The Certification and Recertification of Doctors Cambridge University Press Cambridge 151 166 Google Scholar Gielen, S., Dochy, F., Dierick, S. 2003 Evaluating the consequential validity of new modes of assessment: The influence of assessment on learning, including pre-, post-, and true assessment effects Segers, M. Dochy, F. Cascallar, E. eds. Optimising New Modes of Assessment: In Search of Qualities and Standards Kluwer Academic Publishers Dordrecht 37 54 Google Scholar Gordon, M.J. 1991 A review of the validity and accuracy of self-assessments in health professions training Academic Medicine 66 762 769 PubMed Google Scholar Gordon, M.J. 1992 Self-assessment programmes and their implications for health professions training Academic Medicine 67 672 679 PubMed Google Scholar Govaerts, M.J.B., Vleuten, C.P.M., Schuwirth, L.W.T. 2002 Optimising the reproducibility of a performance-based test in midwifery Advances in Health Sciences Education 7 133 145 CrossRef PubMed Google Scholar Gray, J.D. 1996 Global rating scales in residency education Academic Medicine 21 S55 63 Google Scholar Handfield-Jones, R.S., Mann, K.V., Challis, M.E., Hobma, S.O., Klass, D.J., McManus, I.C., Paget, N.S., et al. 2002 Linking assessment to learning: A new route to quality assurance in medical practice Medical Education 36 949 958 CrossRef PubMed Google Scholar Hays, R., Wellard, R. 1998 In-training assessment in postgraduate training for general practice Medical Education 32 507 513 CrossRef PubMed Google Scholar Hays, R.B., Jolly, B.C., Caldon, L.J.M., McCrorie, P., McAvoy, P.A., McManus, I.C., et al. 2002 Is insight important? Measuring capacity to change performance Medical Education 36 965 971 CrossRef PubMed Google Scholar Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., Hanson, M. 1999 OSCE checklists do not capture increasing levels of expertise Academic Medicine 74 1129 1134 PubMed Google Scholar Imseis, H.M., Galvin, S.L. 2004 Faculty and resident preference for two different forms of lecture evaluation American Journal of Obstetrics and Gynecology 191 1815 1821 CrossRef PubMed Google Scholar Irby, D.M. 1995 Teaching and learning in ambulatory care settings: A thematic review of the literature Academic Medicine 70 898 931 PubMed Google Scholar Kilminster, S.M., Jolly, B.C. 2000 Effective supervision in clinical practice settings: A literature review Medical Education 34 827 840 CrossRef PubMed Google Scholar McIlroy, J.H., Hodges, B., McNaughton, N., Regehr, G. 2002 The effect of candidates' perceptions of the evaluation method on reliability of checklist and global rating scores in an objective structured clinical examination Academic Medicine 77 725 728 PubMed Google Scholar Miller, G.E. 1990 The assessment of clinical skills/competence/performance Academic Medicine 65 S63 67 PubMed Google Scholar
Morgan, D. (1997).
Focus Groups as Qualitative Research. Thousand Oaks etc: Sage Publications. Norman, G.R., Vleuten, C.P.M., Graaf, E. 1991 Pitfalls in the pursuit of objectivity: Issues of validity, efficiency, and acceptability Medical Education 25 119 126 PubMed Google Scholar Regehr, G., MacRae, H., Reznick, R.K., Szalay, D. 1998 Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination Academic Medicine 73 993 997 PubMed Google Scholar Rolfe, I., McPherson, J. 1995 Formative assessment: How am I doing? The Lancet 345 837 839 CrossRef Google Scholar Rothman, A.I., Blackmore, D., Daupinee, W.D., Reznick, R. 1997 The use of global ratings in OSCE station scores Advances in Health Sciences Education 1 215 219 CrossRef Google Scholar Sommers, P.S., Muller, J.H., Saba, G.W., Draisin, J.A., Shore, W.B. 1994 Reflections-on-action: Medical students' accounts of their implicit beliefs and strategies in the context of one-to-one clinical teaching Academic Medicine 69 S84 6 PubMed Google Scholar Spike, N., Alexander, H., Elliott, S., Hazlett, C., Kilminster, S., Prideaux, D., Roberts, T. 2000 In-training assessment – its potential in enhancing clinical teaching Medical Education 34 858 61 CrossRef PubMed Google Scholar Stewart, J., O’Halloran, C., Barton, J.R., Singleton, S.J., Harrigan, P., Spencer, J. 2000 Clarifying the concepts of confidence and competence to produce appropriate self-evaluation measurement scales Medical Education 34 903 909 CrossRef PubMed Google Scholar Toohey, S., Ryan, G., Hughes, C. 1996 Assessing the practicum Assessment and Evaluation in Higher Education 21 215 227 Google Scholar Topping, K. 2003 Self and peer assessment in school and university: Reliability, validity and utility Segers, M. Dochy, F. Cascallar, E. eds. Optimising New Modes of Assessment: In Search of Qualities and Standards Kluwer Academic Publishers Dordrecht 55 89 Google Scholar Turnbull, J., Barneveld, C. 2002 Assessment of clinical performance: In-training evaluation Norman, G.R. Vleuten, C.P.M. Newble, D.I. eds. International Handbook of Research in Medical Education Kluwer Academic Publishers Dordrecht 793 810 Google Scholar Van Luijk, S.J., Van der Vleuten, C.P.M., Schelven, R.M. 1990 The relation between content and psychometric characteristics in performance-based testing Bender, W. Hiemstra, R.J. Scherpbier, A.J.J.A. Zwierstra, R.P. eds. Teaching and Assessing Clinical Competence Boekwerk Publications Groningen 202 207 Google Scholar Vleuten, C.P.M., Norman, G.R., Graaf, E. 1991 Pitfalls in the pursuit of objectivity: Issues of reliability Medical Education 25 110 118 PubMed Google Scholar Vleuten, C.P.M., Scherpbier, A.J.J.A., Dolmans, D.H.J.M., Schuwirth, L.W.T., Verwijnen, G.M., Wolfhagen, H.A.P. 2000 Clerkship assessment assessed Medical Teacher 22 592 600 CrossRef Google Scholar