Abstract
Background
The widespread implementation of longitudinal assessment (LA) to document trainees’ progression to independent practice rests more on speculative rather than evidence-based benefits. We aimed to document stakeholders’ knowledge of- and attitudes towards LA, and identify how the supports and barriers can help or hinder the uptake and sustainable use of LA.
Methods
We interviewed representatives from four stakeholder groups involved in LA. The interview protocols were based on the Theoretical Domains Framework (TDF), which contains a total of 14 behaviour change determinants. Two team members coded the interviews deductively to the TDF, with a third resolving differences in coding. The qualitative data analysis was completed with iterative consultations and discussions with team members until consensus was achieved. Saliency analysis was used to identify dominant domains.
Results
Forty-one individuals participated in the study. Three dominant domains were identified. Participants perceive that LA has more positive than negative consequences and requires substantial ressources. All the elements and characteristics of LA are present in our data, with differences between stakeholders.
Conclusion
Going forward, we could develop and implement tailored and theory driven interventions to promote a shared understanding of LA, and maintain potential positive outcomes while reducing negative ones. Furthermore, ressources to support LA implementation need to be addressed to facilitate its uptake.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Atkins, L., Francis, J., Islam, R., O’Connor, D., Patey, A., Ivers, N., et al. (2017). A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implementation Science, 12(1), 1–18.
Baay, S., Hemmelgarn, B., Tam-Tham, H., Finlay, J., Elliott, M. J., Straus, S., et al. (2019). Understanding adults with chronic kidney disease and their caregivers’ self-management experiences: a qualitative study using the Theoretical Domains Framework. Canadian journal of kidney health and disease, 6
Birken, S. A., Powell, B. J., Shea, C. M., Haines, E. R., Kirk, M. A., Leeman, J., et al. (2017). Criteria for selecting implementation science theories and frameworks: Results from an international survey. Implementation Science, 12(1), 1–9.
Boet, S., Patey, A. M., Baron, J. S., Mohamed, K., Pigford, A. E., Bryson, G. L., et al. (2017). Factors that influence effective perioperative temperature management by anesthesiologists: A qualitative study using the Theoretical Domains Framework. Canadian Journal of Anesthesia, 64(6), 581–596. https://doi.org/10.1007/s12630-017-0845-9
Boud, D. (2015). Feedback: Ensuring that it leads to enhanced learning. Clinical Teacher, 12(1), 3–7. https://doi.org/10.1111/tct.12345
Bramley, A. L., & McKenna, L. (2021). Entrustable professional activities in entry-level health professional education: A scoping review. Medical Education, 55(9), 1011–1032. https://doi.org/10.1111/medu.14539
Briz-Ponce, L., & García-Peñalvo, F. J. (2015). An empirical assessment of a technology acceptance model for apps in medical education. Journal of Medical Systems, 39(11), 176. https://doi.org/10.1007/s10916-015-0352-x
Briz-Ponce, L., Pereira, A., Carvalho, L., Juanes-Méndez, J. A., & García-Peñalvo, F. J. (2017). Learning with mobile technologies – Students’ behavior. Computers in Human Behavior, 72, 612–620. https://doi.org/10.1016/j.chb.2016.05.027
Buetow, S. (2010). Thematic analysis and its reconceptualization as ‘Saliency Analysis.’ Journal of Health Services Research & Policy, 15(2), 123–125. https://doi.org/10.1258/jhsrp.2009.009081
Canadian Institutes of Health Research. (2012). Guide to knowledge translation planning at CIHR: integrated and end-of-grant approaches. Canadian Institutes of Health Research.
Cane, J., O’Connor, D., & Michie, S. (2012). Validation of the theoretical domains framework for use in behaviour change and implementation research. Implementation Science, 7(1), 37. https://doi.org/10.1186/1748-5908-7-37
Carr, D., Howells, A., Chang, M., Hirji, N., & English, A. (2009). An integrated approach to stakeholder engagement. Healthcare Quarterly, 12, 62–70. https://doi.org/10.12927/HCQ.2009.20754
Cheung, W. J., Patey, A. M., Frank, J. R., Mackay, M., & Boet, S. (2019). Barriers and enablers to direct observation of trainees’ Clinical performance: A qualitative study using the theoretical domains framework. Academic Medicine, 94(1), 101–114. https://doi.org/10.1097/ACM.0000000000002396
Cianciolo, A. T., & Regehr, G. (2019). Learning theory and educational intervention: Producing meaningful evidence of impact through layered analysis. Academic Medicine, 94(6), 789–794. https://doi.org/10.1097/ACM.0000000000002591
Conforti, L. N., Yaghmour, N. A., Hamstra, S. J., Holmboe, E. S., Kennedy, B., Liu, J. J., et al. (2018). The effect and use of milestones in the assessment of neurological surgery residents and residency programs. Journal of Surgical Education, 75(1), 147–155. https://doi.org/10.1016/j.jsurg.2017.06.001
Cook, D. A., Kuper, A., Hatala, R., & Ginsburg, S. (2016). When assessment data are words: Validity evidence for qualitative educational assessments. Academic Medicine, 91(10), 1359–1369. https://doi.org/10.1097/ACM.0000000000001175
Cowdell, F., & Dyson, J. (2019). How is the theoretical domains framework applied to developing health behaviour interventions? A systematic search and narrative synthesis. BMC Public Health, 19(1), 1–10.
Curran, J. A., Brehaut, J., Patey, A. M., Osmond, M., Stiell, I., & Grimshaw, J. M. (2013). Understanding the Canadian adult CT head rule trial: Use of the theoretical domains framework for process evaluation. Implementation Science, 8, 25. https://doi.org/10.1186/1748-5908-8-25
De Buck, E., Hannes, K., Cargo, M., Van Remoortel, H., Vande Veegaete, A., Mosler, H. J., Govender, T., Vandekerckhove, P., Young, T., et al. (2018). Engagement of stakeholders in the development of a Theory of Change for handwashing and sanitation behaviour change. International Journal of Environmental Health Research, 28(1), 8–22. https://doi.org/10.1080/09603123.2017.1415306
Dey, I. (2007). Grounded Theory. In C. Seale, G. Gobo, J. F. Gubrium, & D. Silverman (Eds.), Qualitative Research Practice (pp. 80–94). SAGE Publications.
Drolet, B. C., Marwaha, J. S., Wasey, A., & Pallant, A. (2017). Program director perceptions of the general surgery milestones project. Journal of Surgical Education, 74(5), 769–772. https://doi.org/10.1016/j.jsurg.2017.02.012
Francis, J. J., O’Connor, D., & Curran, J. (2012). Theories of behaviour change synthesised into a set of theoretical groupings: Introducing a thematic series on the theoretical domains framework. Implementation Science, 7(1), 35. https://doi.org/10.1186/1748-5908-7-35
Francis, J. J., Stockton, C., Eccles, M. P., Johnston, M., Cuthbertson, B. H., Grimshaw, J. M., et al. (2009a). Evidence-based selection of theories for designing behaviour change interventions: Using methods based on theoretical construct domains to understand clinicians’ blood transfusion behaviour. British Journal of Health Psychology, 14(Pt 4), 625–646. https://doi.org/10.1348/135910708X397025
Francis, J. J., Tinmouth, A., Stanworth, S. J., Grimshaw, J. M., Johnston, M., Hyde, C., et al. (2009b). Using theories of behaviour to understand transfusion prescribing in three clinical contexts in two countries: Development work for an implementation trial. Implementation Science : IS, 4, 70. https://doi.org/10.1186/1748-5908-4-70
French, S. D., Green, S. E., O’Connor, D. A., McKenzie, J. E., Francis, J. J., Michie, S., et al. (2012). Developing theory-informed behaviour change interventions to implement evidence into practice: A systematic approach using the Theoretical Domains Framework. Implementation Science, 7(1), 38. https://doi.org/10.1186/1748-5908-7-38
Friedman, K. A., Raimo, J., Spielmann, K., & Chaudhry, S. (2016). Resident dashboards: Helping your clinical competency committee visualize trainees’ key performance indicators. Medical Education Online, 21(1), 29838. https://doi.org/10.3402/meo.v21.29838
Gofton, W., Dudek, N., Barton, G., & Bhanji, F. (2017). Work based assessment implementation guide: formative tips for medical teaching practice. The Royal College of Physicians and Surgeons of Canada, 1–12. http://www.royalcollege.ca/rcsite/documents/cbd/work-based-assessment-practical-implications-implementation-guide-e.pdf
Goldman, R. H., Tuomala, R. E., Bengtson, J. M., & Stagg, A. R. (2017). How Effective are new milestones assessments at demonstrating Resident Growth? 1 Year of Data. Journal of Surgical Education, 74(1), 68–73. https://doi.org/10.1016/j.jsurg.2016.06.009
Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., & Robinson, N. (2006). Lost in knowledge translation: Time for a map? Journal of Continuing Education in the Health Professions, 26(1), 13–24. https://doi.org/10.1002/chp.47
Green, M. L., Aagaard, E. M., Caverzagie, K. J., Chick, D. A., Holmboe, E., Kane, G., et al. (2009). Charting the road to competence: Developmental milestones for internal medicine residency training. Journal of Graduate Medical Education, 1(1), 5. https://doi.org/10.4300/01.01.0003
Griffiths, J., Dalgarno, N., Schultz, K., Han, H., & van Melle, E. (2019). Competency-Based Medical Education implementation: Are we transforming the culture of assessment? Medical Teacher, 41(7), 811–818. https://doi.org/10.1080/0142159X.2019.1584276
Grol, R. P. T. M., Bosch, M. C., Hulscher, M. E. J. L., Eccles, M. P., & Wensing, M. (2007). Planning and studying improvement in patient care: The use of theoretical perspectives. The Milbank Quarterly, 85(1), 93–138. https://doi.org/10.1111/j.1468-0009.2007.00478.x
Grol, R., & Wensing, M. (2004). What Drives Change? Barriers to and Incentives for Achieving Evidence-Based Practice. The Medical Journal of Australia, 180, S57-60. https://doi.org/10.5694/j.1326-5377.2004.tb05948.x
Hallsworth, K., Dombrowski, S. U., McPherson, S., Anstee, Q. M., & Avery, L. (2020). Using the theoretical domains framework to identify barriers and enabling factors to implementation of guidance for the diagnosis and management of nonalcoholic fatty liver disease: A qualitative study. Translational Behavioral Medicine, 10(4), 1016–1030. https://doi.org/10.1093/tbm/ibz080
Harmon, D. J. (2015). User Acceptance of a Novel Anatomical Sciences Mobile App for Medical Education - An Extension of the Technology Acceptance Model. The Ohio State University.
Hart, J., & Sutcliffe, A. (2019). Is it all about the Apps or the Device?: User experience and technology acceptance among iPad users. International Journal of Human-Computer Studies, 130, 93–112. https://doi.org/10.1016/j.ijhcs.2019.05.002
Hicks, P. J., Margolis, M., Poynter, S. E., Chaffinch, C., Tenney-Soeiro, R., Turner, T. L., et al. (2016). The pediatrics milestones assessment pilot: Development of workplace-based assessment content, instruments, and processes. Academic Medicine, 91(5), 701–709. https://doi.org/10.1097/ACM.0000000000001057
Holmboe, E. S. (2015). Realizing the promise of competency-based medical education. Academic Medicine, 90(4), 411–413. https://doi.org/10.1097/ACM.0000000000000515
Holmboe, E. S., Yamazaki, K., Nasca, T. J., & Hamstra, S. J. (2020). Using longitudinal milestones data and learning analytics to facilitate the professional development of residents: Early lessons from three specialties. Academic Medicine, 95(1), 97. https://doi.org/10.1097/ACM.0000000000002899
Islam, R., Tinmouth, A. T., Francis, J. J., Brehaut, J. C., Born, J., Stockton, C., et al. (2012). A cross-country comparison of intensive care physicians’ beliefs about their transfusion behaviour: A qualitative study using the theoretical domains framework. Implementation Science, 7, 93. https://doi.org/10.1186/1748-5908-7-93
Lajoie, S. P. (2003). Transitions and trajectories for studies of expertise. Educational Researcher, 32(8), 21–25.
Luborsky, M. R., & Rubinstein, R. L. (1995). Sampling in qualitative research: Rationale, Issues, and Methods. Research on Aging, 17(1), 89–113. https://doi.org/10.1177/0164027595171005
Marceau, M., Gallagher, F., Young, M., & St-Onge, C. (2018). Validity as a social imperative for assessment in health professions education: A concept analysis. Medical Education, 52(6), 641–653. https://doi.org/10.1111/medu.13574
Michie, S., Johnston, M., Abraham, C., Lawton, R., Parker, D., & Walker, A. (2005). Making psychological theory useful for implementing evidence based practice: A consensus approach. BMJ Quality & Safety, 14(1), 26–33. https://doi.org/10.1136/qshc.2004.011155
Michie, S., Johnston, M., Francis, J., Hardeman, W., & Eccles, M. (2008). From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques. Applied Psychology, 57(4), 660–680. https://doi.org/10.1111/j.1464-0597.2008.00341.x
Nasca, T. J., Philibert, I., Brigham, T., & Flynn, T. C. (2012). The Next GME Accreditation System — Rationale and Benefits. New England Journal of Medicine, 366(11), 1051–1056. https://doi.org/10.1056/NEJMsr1200117
Nelson, J. (2017). Using conceptual depth criteria: Addressing the challenge of reaching saturation in qualitative research. Qualitative Research, 17(5), 554–570. https://doi.org/10.1177/1468794116679873
Nikou, S. A., & Economides, A. A. (2017). Mobile-based assessment: Investigating the factors that influence behavioral intention to use. Computers & Education, 109, 56–73. https://doi.org/10.1016/j.compedu.2017.02.005
Nikou, S. A., & Economides, A. A. (2018). Mobile-based assessment: A literature review of publications in major referred journals from 2009 to 2018. Computers & Education, 125, 101–119. https://doi.org/10.1016/j.compedu.2018.06.006
Nikou, S. A., & Economides, A. A. (2019). Factors that influence behavioral intention to use mobile-based assessment: A STEM teachers’ perspective. British Journal of Educational Technology, 50(2), 587–600. https://doi.org/10.1111/bjet.12609
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10(1), 53. https://doi.org/10.1186/s13012-015-0242-0
Norton, T. C., Rodriguez, D. C., & Willems, S. (2019). Applying the Theoretical Domains Framework to understand knowledge broker decisions in selecting evidence for knowledge translation in low-and middle-income countries. Health Research Policy and Systems, 17(1), 1–15. https://doi.org/10.1186/s12961-019-0463-9
O’Rourke, T., Higuchi, K. S., & Hogg, W. (2016). Stakeholder participation in system change: A New Conceptual Model. Worldviews on Evidence-Based Nursing, 13(4), 261–269. https://doi.org/10.1111/wvn.12165
Onyura, B., Légaré, F., Baker, L., Reeves, S., Rosenfield, J., Kitto, S., et al. (2015). Affordances of knowledge translation in medical education: A qualitative exploration of empirical knowledge use among medical educators. Academic Medicine, 90(4), 518–524. https://doi.org/10.1097/ACM.0000000000000590
Patey, A. M., Islam, R., Francis, J. J., Bryson, G. L., Grimshaw, J. M., & the Canada PRIME Plus Team. (2012). Anesthesiologists’ and surgeons’ perceptions about routine pre-operative testing in low-risk patients: Application of the Theoretical Domains Framework (TDF) to identify factors that influence physicians’ decisions to order pre-operative tests. Implementation Science, 7, 52. https://doi.org/10.1186/1748-5908-7-52
Patton, M. Q. (2002). Qualitative Interviewing. In Qualitative Research and Evaluation Methods (Vol. 3rd, pp. 339–428). Thousand Oak: SAGE.
Peabody, M. R., O’Neill, T. R., & Peterson, L. E. (2017). Examining the Functioning and Reliability of the Family Medicine Milestones. Journal of Graduate Medical Education, 9(1), 46–53. https://doi.org/10.4300/JGME-D-16-00172.1
Peters, H., Holzhausen, Y., Boscardin, C., Ten Cate, O., & Chen, H. C. (2017). Twelve tips for the implementation of EPAs for assessment and entrustment decisions. Medical Teacher, 39(8), 802–807. https://doi.org/10.1080/0142159X.2017.1331031
Salbach, N. M., Jaglal, S. B., Korner-Bitensky, N., Rappolt, S., & Davis, D. (2007). Practitioner and Organizational Barriers to Evidence-based Practice of Physical Therapists for People With Stroke. Physical Therapy, 87(10), 1284–1303. https://doi.org/10.2522/ptj.20070040
Scurlock-Evans, L., Upton, P., & Upton, D. (2014). Evidence-Based Practice in physiotherapy: A systematic review of barriers, enablers and interventions. Physiotherapy, 100(3), 208–219. https://doi.org/10.1016/j.physio.2014.03.001
St-Onge, C., Vachon Lachiver, É., Langevin, S., Boileau, É., Bernier, F., & Thomas, A. (2020). Lessons from the implementation of Developmental Progress Assessment: A Scoping Review. Medical Education, 54(10), 878–887. https://doi.org/10.1111/medu.14136
St-Onge, C., Young, M., Eva, K. W., & Hodges, B. (2017). Validity: One word with a plurality of meanings. Advances in Health Sciences Education, 22(4), 853–867. https://doi.org/10.1007/s10459-016-9716-3
Swanson, D. B., & Case, S. M. (1997). Assessment in basic science instruction: Directions for practice and research. Advances in Health Sciences Education, 2, 71–84.
Swing, S. R., Cowley, D. S., & Bentman, A. (2014). Assessing resident performance on the psychiatry milestones. Academic Psychiatry: The Journal of the American Association of Directors of Psychiatric Residency Training and the Association for Academic Psychiatry, 38(3), 294–302. https://doi.org/10.1007/s40596-014-0114-y
Ten Cate, O. (2005). Entrustability of professional activities and competency-bases training. Medical Education, 39(12), 1176–1177. https://doi.org/10.1111/j.1365-2929.2005.02341.x
Terzis, V., & Economides, A. A. (2011). The acceptance and use of computer based assessment. Computers & Education, 56(4), 1032–1044. https://doi.org/10.1016/j.compedu.2010.11.017
Thomas, A., & Bussières, A. (2021). Leveraging knowledge translation and implementation science in the pursuit of evidence informed health professions education. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-020-10021-y
Thomas, A., & Ellaway, R. H. (2021). Rethinking implementation science for health professions education: A manifesto for change. Perspectives on Medical Education, 10(6), 362–368.
Thomas, A., Gruppen, D., & L., van der Vleuten, C., Chilingaryan, G., Amari, F., & Steinert, Y. (2019). Use of evidence in health professions education: Attitudes, practices, barriers and supports. Medical Teacher, 41(9), 1012–1022. https://doi.org/10.1097/ACM.0000000000001057
Touchie, C., & ten Cate, O. (2016). The promise, perils, problems and progress of competency-based medical education. Medical Education, 50(1), 93–100. https://doi.org/10.1111/medu.12839
Varpio, L., Bell, R., Hollingworth, G., Jalali, A., Haidet, P., Levine, R., & Regehr, G. (2012). Is transferring an educational innovation actually a process of transformation? Advances in Health Sciences Education, 17(3), 357–367.
Webb, T., Joseph, J., Yardley, L., & Michie, S. (2010). Using the Internet to Promote Health Behavior Change. Journal of Medical Internet Research. https://doi.org/10.2196/jmir.1376
Acknowledgements
The authors would like to thank their participants in this study.
The study was funded through the Royal College of Physicians and Surgeons of Canada 2016 Strategic Request for Proposals: Mission-aligned Research in Competency-based Medical Education.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
St-Onge, C., Boileau, E., Langevin, S. et al. Stakeholders’ perception on the implementation of Developmental Progress Assessment: using the Theoretical Domains Framework to document behavioral determinants. Adv in Health Sci Educ 27, 735–759 (2022). https://doi.org/10.1007/s10459-022-10119-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-022-10119-5