Abstract
Health sciences students face many challenges in regard to clinical practical learning. A better understanding of student learning is required to address student needs in this crucial phase. The theory of self-regulated learning provides a comprehensive view of learning and could serve as a basis for further research. There are instruments to assess self-regulated learning in preclinical academic learning. However, there are no such instruments for workplace learning. The aim of the present study is to provide a comprehensive inventory from which researchers can select those scales that are relevant to their research questions in the investigation of workplace learning. Hence, the aim is to develop and validate a set of scales to assess undergraduates’ workplace learning in health sciences education in four areas (cognition, motivation, emotion, and context) on two levels (the learning process level and the metalevel). Study 1 is a qualitative multimethod study to identify indicators and develop items. It integrates the perspectives of students, teachers, and researchers and includes six steps: literature review, interviews, synthesis, item development, expert review, and cognitive pretesting. This study yields a set of scales for each area on both levels. Study 2 is a quantitative study to assess the psychometric properties. The results show acceptable values in terms of unidimensionality, reliability and validity for each of the 31 scales. The newly developed Workplace Learning Inventory is comprehensive; the scales are relevant to workplace learning and short enough that their administration is feasible in the workplace setting. The rigorous process of questionnaire development contributes to the validity of scales. By providing the Workplace Learning Inventory, we hope to encourage research on workplace learning in health sciences education from an educational psychology perspective.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
Introduction
Many students struggle in transitioning from preclinical academic learning to clinical practical learning in health sciences education, and some continue to struggle even after they have become familiar with the workplace environment (Atherley et al., 2019; Godefrooij et al., 2010; Teo et al., 2011; Westerman & Teunissen, 2013; White, 2007). To better address students’ needs in this crucial phase, a clearer understanding of different aspects of student learning is needed. A comprehensive understanding of learning is a key point since learning should not only result in high achievement. Additionally, student motivation and well-being are considered important (Dai & Sternberg, 2004; Fares et al., 2016; Frajerman et al., 2019).
Research on learning in health sciences education is seen as interdisciplinary but is predominantly informed by the health research domain (Albert et al., 2020). Health sciences education research should be informed by other disciplines, one of which is psychology and, more specifically, educational psychology. Educational psychology research has resulted in a comprehensive understanding of learning, known as self-regulated learning (SRL) (Schunk & Greene, 2018). SRL includes multiple components, such as cognition, motivation, emotion, and the perception of the learning environment as well as the metalevel of learning, considered in terms of metacognition and regulation of motivation and emotion (Ben-Eliyahu, 2019; Ben-Eliyahu & Bernacki, 2015; Panadero, 2017; Pintrich, 2004; Wolters, 2003).
In the last decade, this comprehensive view of learning has been adopted in health sciences education (Artino et al., 2015; Cleary et al., 2013; Hayat et al., 2020; van Houten-Schat et al., 2018), and there are several studies on SRL in medical education. While in educational psychology research SRL is viewed as a multifaceted construct (Pintrich, 2004) and with differentiated underlying mechanisms (Panadero, 2017), health sciences education research on SRL seems not to have adopted this differentiated view (van Houten-Schat et al., 2018). Recognizing the multifaceted nature of SRL could help to understand the underlying mechanisms of student learning in health sciences education. Furthermore, existing studies in health sciences education research mainly address learning in the preclinical academic setting and are often based on qualitative or cross-sectional quantitative methods (van Houten-Schat et al., 2018). However, not only learning in the preclinical academic setting (abbreviated as academic learning; e.g., Biwer et al., 2023) but also undergraduate learning in the practical clinical setting (abbreviated as workplace learning; e.g., Sagasser et al., 2017) is of great interest, including the transition from academic learning to workplace learning (Westerman & Teunissen, 2013).
Academic learning focuses on individuals learning of theoretical foundations in a learning environment that students can create to a large extend by themselves according to their needs. It also focuses on individuals learning of specific motor skills or social skills in a highly structured environment provided by a teacher within the framework of a propaedeutic course. In contrast, workplace learning focuses on individuals learning in a complex learning environment. The workplace can be a clinic, a clinical practice or a company. In line with self-determination theory (Deci & Ryan, 2012), students require supportive conditions for psychological growth. Hence, in such settings undergraduates do not only need to experience and attain competency but also to develop role autonomy, join the community of practice and interact with patients (Cruess et al., 2018; Morris & Behrens, 2013b).
From an educational psychology perspective, there are only a few studies about workplace learning in health sciences education, and there is a lack of recognition of the multifaceted nature of SRL as well as a lack of quantitative multivariate and prospective longitudinal studies of workplace learning in health sciences education (van Houten-Schat et al., 2018).
A prerequisite for such studies is the availability of appropriate instruments for assessing SRL in health sciences education, such as questionnaires. Established questionnaires on learning in higher education often (1) focus on academic learning (e.g. Pintrich et al., 1993), (2) address single components of learning and/or (3) are characterized by long scales (Duffy et al., 2018; Strand et al., 2013; Wolters & Benzon, 2013). To analyze and assess workplace learning, instruments are needed that (1) focus on workplace learning, (2) recognize the multifaceted nature of SRL and address the multiple components and aspects of learning and (3) provide different short scales to be feasible in longitudinal studies. The aim of the present study is to provide a comprehensive inventory from which researchers can select those scales that are relevant to their research questions in the investigation of underlying mechanisms in workplace learning.
A component-based conceptual framework for workplace learning
Students face cognitive, motivational, and emotional challenges when transferring from academic to workplace learning. Educational psychology research provides different theoretical frameworks for integrating such different components of learning (Dai & Sternberg, 2004; Slavin, 2018). We refer to the theory of SRL because of its broad view of learning and its relevance for academic success while considering motivation and affect. According to Pintrich (2004), the ideal self-regulated learner sets goals and is able to regulate cognition, motivation/affect, behavior and context to achieve a goal. SRL models can be divided into more component-based models, such as Pintrich's conceptual framework for assessing motivation and SRL (Pintrich, 2004) or Boekaerts’s six component model of SRL (Boekaerts, 1996), and more process-based models, such as Zimmerman’s cyclical phases model (Zimmerman, 2008). Because of the comprehensive view of workplace learning and because component-based models emphasize the diversity of aspects that are relevant to learning, the foundation of our study is a component-based conceptual framework for assessing workplace learning drawing on Pintrich’s differentiation between areas of regulation and Boekaerts’s differentiation of levels.
Following Pintrich (2004), we propose four different areas of SRL: cognition, motivation, emotion and context. Cognition (including cognitive and metacognitive aspects) and motivation are the core areas of SRL that can be found in many SRL models (Panadero, 2017), such as in Pintrich’s model or in Boekaerts’s six component model of SRL. In addition, emotion is a relevant component of SRL models (Efklides, 2011; Panadero, 2017). Emotion has also become an increasingly important topic in recent years within SRL theory (Ben-Eliyahu, 2019) and in educational psychology research more generally (Pekrun, 2006). In health sciences education research, well-being, a concept related to emotion, is an important topic (Duffy et al., 2018; Fares et al., 2016; Frajerman et al., 2019). Therefore, in contrast to Pintrich, who combines the aspects of motivation and affect into one area, we integrate emotion as a separate component in our model. Finally, not only is context included in Pintrich’s framework, but the importance of context in terms of the learning environment has also been pointed out in health sciences education research (Berkhout et al., 2016; van Houten-Schat et al., 2018). Based on Pintrich’s SRL model, context is not seen as objective frameworks to which students are exposed. Rather, student take an active role. It is about how students interpret the context and about their ability to change the interpretation or, if possible, the context itself to reach their learning goals. We dropped the area behavior because, as also Pintrich (2000) pointed out, it overlaps with the area cognition.
We propose two different levels based on Nelson and Narens (1990) and, more specifically for SRL, on Boekaerts (1996) and Wirth et al. (2020): the learning-process level and the metalevel.Footnote 1At the learning process level, students are in the middle of the learning process, consciously or unconsciously using cognitive strategies, experiencing different levels and aspects of motivation and emotions, and perceiving and interpreting the learning environment. At the metalevel, students step out of the learning process for a moment and reflect on their learning. The learning process level is similar to Boekaerts’s cognitive strategy-use-level and motivational beliefs as well as to Wirth and colleagues’ learning strategy layer. The metalevel is similar to Boekaerts’s goal level or Wirth and colleagues’ metacognitive layer. The metalevel is included in many SRL models but often refers only to the regulation of cognition, known as metacognition. Boekaerts suggests different levels of cognition and motivation, while Wirth and colleagues’ layers solely address the cognitive area.
We extend our understanding of learning as both the learning process level and the metalevel address emotion and context in addition to cognition and motivation. We combine the four areas and two levels, which results in eight components (see Fig. 1). At the learning process level, (1) cognition refers to cognitive learning strategies such as rehearsal, organization or elaboration strategies (Weinstein et al., 2011); (2) motivation refers to motivational aspects as described by various motivation theories, such as expectancy-value theory (Eccles & Wigfield, 2020) and achievement goal theory (Urdan & Kaplan, 2020); (3) academic emotion refers to positive and negative emotions such as pride, enjoyment, frustration or anxiety as described, for example, in the control-value theory of achievement emotions (Pekrun, 2006) and in the Medical Emotion Scale (Duffy et al., 2018); and (4) context refers to the perception of the learning environment, including the physical and social environment (Strand et al., 2013).
Eight workplace learning components are distinguished. The inner circle illustrates the learning process level components: The ideal learner implements appropriate learning strategies (cognition), is motivated (motivation), feels well (emotion) and perceives a supportive environment (context). The outer circle illustrates the metalevel components: The ideal learner steps out of the learning process and regulates cognition, motivation, emotion, and context
At the metalevel, we refer to the components by using the terms ‘cognition metalevel’, ‘motivation metalevel’, ‘emotion metalevel’ and ‘context metalevel’. Following Pintrich (2004), we assume that the learner consciously or unconsciously anticipates, plans, monitors, adapts, evaluates and reacts not only in terms of cognition but also in terms of motivation, emotion and context. This assumption is also supported by research on motivation regulation and emotion regulation (Ben-Eliyahu, 2019; Wolters, 2003). There is evidence that cognitive, motivational and emotional processes on the metalevel are not distinct but share certain regulatory mechanisms (Kim et al., 2020; they did not consider metacontext).
In contrast to component-based models, process-based models emphasize the different phases of SRL. Zimmerman’s cyclical three-phase model (2008), which is often used in medical education (van Houten-Schat et al., 2018), differentiates among the forethought, performance and reflection phases. Therefore, the ideal learner analyses a task (e.g., planning) and addresses motivation and beliefs (e.g., self-efficacy) in the forethought phase. He or she monitors cognition, emotion and effort as well as task and environmental demands in the performance phase before assessing the achievement (e.g., strategic review) and reacting to it (e.g., rewards/sanctions) in the reflection phase.
Measurement instruments
There is a wide range of methods for collecting data on the different components of learning, such as questionnaires, interviews, think-aloud techniques, learning diaries, or observations (Roth et al., 2016; Schunk & Greene, 2018; Wirth & Leutner, 2008). Self-report questionnaires are predominantly used to assess SRL in higher education (Roth et al., 2016). They allow for the assessment of core facets of learning that are not easily observable. They are also easier to administer than other methods, such as interviews or think-aloud protocols, especially in multivariate longitudinal studies. At the same time, the validity of the data produced by self-report questionnaires has been questioned (Wolters & Won, 2018), and there have been calls for the careful development of questionnaires (Gehlbach & Brinkworth, 2011).
For an overview of instruments and related scales in the field of SRL, primarily for the academic setting, see Roth et al. (2016). There are also self-report questionnaires specific to the metalevel components (Wolters, 1998), to emotion regulation in general (Burić et al., 2016; Loch et al., 2011) as well as to the workplace learning of health science students for emotion (Duffy et al., 2018) and for the learning environment (Roff & McAleer, 2017).
The established questionnaires mentioned above are not feasible in multivariate longitudinal studies of undergraduates’ workplace learning. They often address the academic setting and/or include long scales to cover a wide range of facets and to facilitate high validity. In multivariate longitudinal studies, long scales run the risk of overburdening participants (Hoerger, 2010). In addition, most instruments assess trait rather than state aspects of learning and may not be appropriate for examining change over time in longitudinal studies. Finally, when using questionnaires from different fields, such as motivation research or emotion research in multivariate studies, researchers often face the problem of construct contamination. This means that, for example, an emotion questionnaire includes as well items which address motivational aspects. In conclusion, there is a lack of self-report instruments that follow a more efficient approach with shorter scales, a focus on tracking changes over time, and distinct scales which would be appropriate for multivariate longitudinal psychological studies of workplace learning in undergraduate health sciences education.
In developing new scales with few items, different types of validity need to be thoroughly investigated. According to the American Educational Research Association et al. (2014) and Wolters and Won (2018), evidence of validity should be based on (1) content, (2) response processes, (3) internal structure, (4) relationships with other variables and (5) consequences of testing.
To address and ensure all types of validity, triangulation of methods is necessary when developing a self-report instrument. Gehlbach and Brinkworth (2011) recommend seven steps, from literature review to a pilot test of psychometric quality (see Fig. 2 for Step 1 to 6). Steps 1 to 6 can be summarized as the qualitative part of questionnaire development, addressing validity based on content and response processes. Step 7 is the quantitative part of scale development, addressing validity based on internal structure and relationships with other variables.
Method and the timeline of Study 1. The method is based on Gehlbach and Brinkworth (2011)
Aim
The aim of the present study is to develop an instrument to assess different aspects of undergraduates’ workplace learning in health science education. Our aim is to provide a comprehensive inventory from which researchers can select those scales that are relevant to their research question, rather than using all the scales in one study. Each SRL-component should be represented by several indicators (scales), but each scale should contain only a few items to be applicable and reasonable in multivariate longitudinal studies.
We consider a medium degree of situational specificity (Roth et al., 2016) to be appropriate for our purpose. First, the inventory should be specific to a practical clinical setting (as opposed to an academic setting) of health sciences education but not to a specific field or profession. Second, the scales should assess a week of workplace learning but not specific days or situations. The inventory should be designed to capture changes over time; therefore, on the continuum between a state and trait measure, it should be more of a state measure (Geiser et al., 2017; Pekrun et al., 2018).
We follow Gehlbach and Brinkworth’s (2011) seven steps and conduct two studies. Study 1, a qualitative multimethod study, includes Steps 1 to 6 to develop the indicators and items. In Study 2, a quantitative study, we examine the psychometric properties of the scales.
Study 1
The aim of Study 1 is to identify relevant indicators for each component of our model and to develop scales and items for undergraduate learning in the clinical practice setting of health sciences education. The inventory addresses four components at the learning process level, namely, cognition, motivation, emotion, and context, and four components at the learning metalevel, namely, the cognition metalevel, motivation metalevel, emotion metalevel, and context metalevel.
Method
Figure 2 shows the steps and timeline of Study 1. To increase the trustworthiness of the process, each step was discussed by a multidisciplinary heterogeneous project team including SRL and health sciences education researchers, clinical teachers, and students. The project team met bi-weekly to ensure continuous discussion and decision making.
To identify relevant indicators and develop items, we considered the entire learning process of students throughout the day, from before they arrive in the workplace to their time in the workplace and after they leave. In the following sections, we describe our general process for developing indicators and items.
In Step 1, we identified relevant indicators for each component from the literature and from existing measures. We derived the indicators from the two most widely used SRL questionnaires in higher education (Roth et al., 2016). We added indicators from measures that were designed for the practical clinical setting (but not indicators that were too specific, such as those regarding surgeries) and that were specific to undergraduates (for emotion, we found only one relevant questionnaire; for context, the decision was based on the list provided by Roff and McAleer (2017). For metalevel motivation and emotion, we derived indicators from the most widely used questionnaires.
In Step 2, we identified indicators relevant to workplace learning for each component based on stakeholder statements. We conducted semistructured interviews with 6 students and 6 clinical educators (abbreviated as teachers) from German-speaking countries and with 6 researchers in the field of SRL and/or in health sciences education from different parts of Europe, Asia and North America. To ensure heterogeneity of perspectives, students and teachers were selected from six different health sciences institutions in three different countries based on recommendations from the respective offices of the vice-rectors for teaching and learning. All persons participated voluntarily, and only those who gave written consent participated. The interviews were conducted online and lasted approximately one to two hours per participant. The interview guideline started with explaining the topic, clarifying terms, and introductory questions. Participants were asked to describe helpful and detrimental aspects regarding emotion, motivation, learning strategies and perception of the context of workplace learning before, during and after undergraduates’ presence in workplace. Each theme was shortly introduced, followed by the question (E.g., ‘In the phase before learning at the workplace, which attitudes or beliefs are beneficial for motivation, and which are a hindrance?’). Finally, the meaning of the metalevel was explained (based on Pintrich, 2004), followed by the questions (E.g., ‘Is there anything here that is particularly important for successful learning? If so, what exactly?). The interviews were recorded and transcribed. The statements were categorized according to Mayring’s process flow of content structuring (2014). First, we defined the object of analysis. Second, we developed a theoretical-driven categorization system including definitions of categories. Third, we revised the categorization system as one worked through the material. Fourth, we coded the material. Finally, we reduced and summarized the extracted statements in each category. To enhance trustworthiness, the summaries were discussed and revised where necessary together with the whole project team over the course of a two-day project meeting.
In Step 3, we synthesized the list of indicators. Some indicators from Step 1 and Step 2 overlapped; in case of discrepancies in the categorization of indicators, we used the indicators from the interviews.
In Step 4, a scale with preliminary items was developed in German for each indicator. To take the students’ perspective into account, the formulation of the items was strongly based on the students’ statements derived from the interviews in Step 2.
In Step 5, the expert review, structured feedback on the preliminary items of the questionnaire was provided by experts who have relevant scientific publications in the field of SRL and/or in health sciences education. Nine researchers from German-speaking countries participated. The researchers were asked to review the indicators and the preliminary items for clarity, relevance and representation. Open-ended comments on each item were encouraged. The indicators and items were revised on the basis of the means of the relevance and clarity scores, the sums of the representativeness scores and the open-ended comments.
In Step 6, we carried out cognitive pretesting of the items. We interviewed potential respondents (students) to determine how they understood and responded to each item (Karabenick et al., 2007; Willis, 2015). We invited all students who currently were enrolled in courses in which they were learning in the clinical practical setting (approximately 350). In these courses, students rotate between different working environments and areas of veterinary medicine (e.g., anesthesia, surgery, reproduction medicine, imaging techniques, etc.). Approximately 20 students agreed to take part in the cognitive pretest and 14 students showed up. Seven students were in their 9th semester, and seven students were in their 11th semester. The students participated voluntarily, and only students who gave written consent participated. After an introduction, the students were asked to complete the questionnaire. We used reminded retrospective verbal probing (Willis, 2015) for each component: After each section (with items relating to one component), the students were asked to explain their cognitive process in answering the items. The interviewer took notes, and the interviews were recorded for documentation purposes. The interviews lasted approximately two hours, including one break. The indicators and items were revised based on the students’ comments.
Further information on the process of developing the indicators and items of each component can be found in the supplementary material.
Results
The process of developing self-report measures for undergraduates’ workplace learning in health sciences education resulted in a comprehensive inventory. It includes several indicators for each of the four components at the learning process level in terms of cognition, motivation, emotion, and context and for each of the four components at the metalevel of learning in terms of cognition metalevel, motivation metalevel, emotion metalevel and context metalevel. The inventory comprises 31 indicators (= scales) and 159 items in total. Table 1 shows the indicators for the eight components, including definitions as well as the number of items per indicator and item examples.
Study 2
The purpose of this study is to examine the psychometric properties of the scales developed in Study 1. Three aspects are examined in detail: (1) we examine whether the scales are unidimensional to provide evidence of validity based on internal structure; (2) we analyze the reliability of the scales; and (3) we examine whether the scales relate to other variables as theoretically expected by assessing the nomological network to provide evidence of validity based on relations with other variables, i.e., for convergent validity.
Method
Participants
The results should be representative of a heterogeneous group of health science students in terms of cognition, motivation, emotion, and learning environment. We therefore decided to make an effort to reach the vast majority of a relevant cohort of students at one institution and thus obtain data from a heterogeneous group in terms of cognition, motivation, emotion, and learning environment, rather than send a questionnaire to different institutions and risk a biased sample by obtaining data from mostly motivated high achievers who feel good about their learning. Since the number of students from one institution only was insufficient for data analysis, we invited students from a second institution to participate. The target sample size was N = 200 based on a common rule of thumb for the minimum sample size when conducting confirmatory factor analysis (see Kline, 2016).
At Institution 1, the questionnaire was administered to the entire group of 200 students enrolled in a course in which students learn for the first time in the clinical practical setting over a relatively long period. This course is usually attended in the 9th semester. Students rotate between different work placements; thus, data were collected in heterogeneous workplace settings. Thirteen students did not give consent to their data being used for research purposes. Eleven participants had to be excluded from further analysis due to a high proportion of missing values (> 50%), resulting in a sample size of n = 176 at Institution 1.
At Institution 2, the questionnaire was sent via email to students in their practical year (usually in the 9th and 10th semesters) in the winter semester of 2021/2022 (n ≈ 260). Students rotate between different work placements; thus, data were collected in heterogeneous workplace settings. The questionnaire was opened 91 times, but there were 38 responses, in which more than 50% of the items were completed. All 38 participants gave consent to their data being used for research purposes. Thus, combining both samples, the total sample size was N = 214 (78% female, 21% male, 1% diverse; age: 21 to 41 years; M = 24.79, SD = 2.74).
Measures
The newly developed inventory for workplace learning in health sciences education included 31 scales measuring eight components, namely, cognition, motivation, emotion, and context at the learning process level as well as the cognition metalevel, motivation metalevel, emotion metalevel and context metalevel (see Table 1). All scales were administered using a five-point Likert scale (1 = does not apply at all, 2 = does not apply, 3 = partly applies, 4 = applies, 5 = fully applies); for the ‘control’ scales at the metalevel, 6 = This case did not occur was also included). The ‘negative emotion’ and ‘positive emotion’ scales are special cases. The items were not newly developed but derived from the MES (Duffy et al., 2018). The response format established by Duffy et al. (2018) was used: 1 = not at all; 2 = a little; 3 = moderately; 4 = fairly; 5 = very much. Established measures were used to assess the nomological network. See Table 2 for details.
Procedure
At Institution 1, the questionnaires were completed as part of the course and supported the course learning goal of “reflecting on one’s own learning and practice”. Data collection was spread over a week (either from 6 to 10th December 2021 or from 13 to 17th December 2021) using the online survey tool unipark© (EFS Survey, 2022). Most of the established scales were more trait-like measures and were presented at the beginning of the week while most of the newly developed scales were presented at the end of the week.
At Institution 2, the rectorate invited all students currently in their practical year. Students received a link to the survey that comprised all questionnaires. They were allowed to pause and continue filling in the questionnaire later between 6 and 17th December 2021 using the online survey tool unipark© (EFS Survey, 2022).
Data analysis
To assess unidimensionality, confirmatory factor analysis (CFA) was used: a one-factor model based on all items of the scale was specified for each scale, using the software Mplus 8.6 (Muthen & Muthen, 1998–2017). Full information maximum likelihood method was used to deal with missing data (Enders, 2022). Model fit was assessed using fit indices based on conventional cut-off criteria for an acceptable model fit, i.e., CFI and TLI ≥ 0.90 and RMSEA and SRMR ≤ 0.08. In the case of poor model fit, residual covariances resulting from similarities in item meaning were specified (Bandalos, 2021). In addition, standardized factor loadings were used to identify and exclude items of low psychometric quality to further improve model fit. The ‘negative emotion’ and ‘positive emotion’ scales are special cases. By performing a CFA, we aimed to identify the most relevant emotions and to provide a short version of these MES scales within this questionnaire. We also did not analyze the ‘effort’ and ‘attention control’ scales because the items were reformulated with very small changes from the established scales on academic learning (Klingsieck, 2018).
To assess reliability, McDonald's composite reliability coefficient ω (1970) was calculated for each scale. Acceptable reliability is indicated by ω ≥ 0.70. To assess the nomological network and thus to investigate whether our newly developed scales were related to the established scales as theoretically expected, we used correlations.
Results
Unidimensionality and reliability
After a total of four items were excluded, the CFA model fit was acceptable and indicated the unidimensionality of all scales. Exceptions included the ‘positive emotion’ and ‘negative emotion’ scales, which are special cases. The aim was to provide a short version of these established scales. Based on the results of the interviews in Step 2 of Study 1 in combination with the factor loadings, we excluded five out of nine items of the ‘positive emotion’ scale and six of eleven items of the ‘negative emotion’ scale. The CFA of the short scales showed acceptable model fit, indicating the unidimensionality of the two scales. The omega values of all scales were within the acceptable range, indicating acceptable reliability. See Table 3 on CFA/reliability details. The final questionnaire with all scales and items can be found at the end of the document in Table 5.
Nomological network
The nomological network was analyzed by assessing the relationship between the newly developed scales and the corresponding established scales. Please see Table 4 for the respective correlation coefficients.
Discussion
In the current study, we developed an inventory for assessing undergraduates’ workplace learning in health sciences education. To ensure validity, a thorough multimethod approach was undertaken involving students, teachers, SRL researchers and health sciences researchers in the field (Gehlbach & Brinkworth, 2011). We conducted two studies, with Study 1 representing the qualitative part of the development process and Study 2 representing the quantitative analysis of the psychometric properties of the scales. The studies yielded a comprehensive set of 31 scales addressing four different areas, namely, cognition, motivation, emotion, and context, at two different levels, namely, the learning process level and the metalevel, resulting in eight components. Each component is represented by several short scales so that the administration of the scales is feasible in the practice setting. In the following, the results are discussed separately for each component, starting with learning process level components and continuing with metalevel components.
Learning process level
At the learning process level, we included the cognition, motivation, emotion, and context components. At this level, students use cognitive learning strategies, experience different aspects and levels of motivation and emotion, and perceive and interpret the workplace context.
Cognition
The cognition component refers to learning strategies with a focus on workplace learning, i.e., learning and practicing professional medical activities. The ideal student anticipates the day as far as possible and acquires knowledge by preparing himself or herself and by planning the medical activities ahead. In the workplace, he or she acquires knowledge and skills by paying attention, rehearsing and elaborating. While in the workplace, the ideal student reviews whether he or she understands the medical procedures and clarifies unclear points. After being in the workplace, the ideal student consolidates his or her knowledge and reflects on his or her professional medical performance. The mentioned strategies are divided into cognitive learning strategies, and proximal metacognitive learning strategies, (see Table 5) and represent the whole learning process of a learning day: before, during and after students’ presence in the clinical practice setting. Psychometric analysis indicated the unidimensionality and acceptable reliability of all scales.
Cognitive learning strategies for workplace learning are different from those for academic learning (Klingsieck, 2018; Pintrich et al., 1993; Weinstein et al., 2010): First, students use cognitive learning strategies not only in the performance phase but also in the preparation and reflection phases. On closer inspection, learning strategies before and after students’ presence in the workplace can be further differentiated (e.g. into rehearsal, elaboration and organization). We decided against further differentiation because it seems more important to measure whether students prepare and consolidate and less how they do this exactly.
Second, proximal metacognitive learning strategies are a newly introduced set of scales specific to workplace learning. In Step 2 and Step 6 the students reported that they learned by planning, reviewing and reflecting on concrete professional medical activities (e.g., monitoring whether they were following the correct medical procedure to take a blood sample) and that these strategies were more important to them than planning, monitoring or reflecting on the learning process at the cognition metalevel (e.g., monitoring the cognitive learning strategies they used to achieve a learning goal).
Whereas the assessment of the nomological network revealed plausible associations between the newly developed cognitive learning strategies scales and established scales, no association between the newly developed proximal metacognitive learning strategies and the established scales were found. An exception was the newly developed ‘reviewing’ scale, which correlated positively with the established ‘regulation’ scale (Klingsieck, 2018). These findings suggest that proximal cognitive learning strategies can be seen as a distinct category of learning strategies specific to the workplace setting, but further research on the nomological network is recommended.
The development of the indicators and scales for the component cognition was a nonlinear process due to divergent feedback from researchers and students. Their views differed not so much in terms of the wording of the items but in terms of the structure of the indicators. Therefore, the list of indicators changed with each step. It is hoped that the inventory now provides a useful set of scales covering the whole cognitive learning process of a student for one day, before, during and after his or her presence in the clinical practice setting. However, the discrepancies in feedback from the researchers and students suggest the need for further research from an educational psychology perspective on learning strategies for the workplace setting.
Motivation
The motivation component refers to the initiation and maintenance of goal-directed activity. It consists of seven scales representing stakeholders’ perspectives on relevant motivational aspects of workplace learning (see Table 5). Psychometric analysis revealed the unidimensionality and acceptable reliability of the scales.
The expectancy-value theory (Eccles & Wigfield, 2020), was shown to be relevant not only to academic learning (Pintrich et al., 1993), but also to workplace learning (‘expectancy of success’ and ‘situational interest’). The results of the nomological network assessment were as expected (Kunter et al., 2002).
Also, achievement goal theory (Urdan & Kaplan, 2020) is relevant to both academic and workplace settings. The ‘performance goal approach’ scale in workplace learning needs careful interpretation because it was positively related not only to the ‘performance goal approach’ scale but also to the ‘performance goal avoidance’ scale in academic learning (Schwarzer & Jerusalem, 1999). The scales representing the avoidance component were deleted in the Workplace Learning Inventory due to the risk of biased responses and the already long list of motivational indicators. However, in Step 2 some interviewees reported that avoiding failure when performing medical activities in front of others was also a relevant motivational aspect. Further research is needed to explore achievement goal theory in the context of workplace learning, especially since achievement goal theory has been further developed in recent years (Urdan & Kaplan, 2020).
Effort and attention control have been added to the abovementioned motivational aspects based on expert review and cognitive pretesting. The nomological network analysis showed results as expected. (Boerner et al., 2005). The ‘proactive attitude’ scale addresses a new motivational aspect specific to workplace learning (if someone is willing to take action). Contrary to expectations, ‘proactive attitude’ was not associated with any of the motivational aspects of academic learning. ‘Proactive attitude’ seems to be a distinct indicator in the workplace setting, and further research on the nomological network is needed.
Emotion
The emotion component is defined ‘within the broader concept of affect, but differs from other affective phenomena, such as mood, in that emotions are more intense, have a clearer object-focus, a more salient cause, and are typically experienced for a shorter duration’ (Duffy et al., 2018). The emotion component comprises two scales, ‘positive emotions’ and ‘negative emotions. Psychometric analysis showed the unidimensionality and acceptable reliability of the scales. We did not assess the nomological network, as the scales are short versions of the established MES scales.
In the interpretation of emotions in workplace learning, it is important to remember that the terms ‘positive’ and ‘negative’ describe the quality of single emotions but not their effect on achievement. Both positive and negative emotions can help or hinder a learning process. For example, the positive emotion of curiosity can be a motivator, but high levels of curiosity can also lead to getting lost in details. A high level of the negative emotion of frustration can be demotivating, but a low level of frustration can be a motivator to do better next time and lead to higher achievement.
Context
The context component focuses on concrete contextual aspects that are relevant, i.e., helpful or detrimental, to undergraduate workplace learning. In our newly developed questionnaire, the context component is represented by the ‘organizational framework conditions’, ‘supervisory quality’, ‘staff support’, ‘peer support’ and ‘equal treatment’ scales. Psychometric analysis revealed the unidimensionality and acceptable reliability of the scales.
The relationships in the nomological network were as expected, with two exceptions: We did not expected the newly developed ‘peer support’ to be associated with the established ‘perception of teacher’ scale. A possible explanation might be that the teacher shapes the learning environment (e.g., classroom structure; Ames, 1992; Bergsmann et al., 2013) and class climate (Allodi, 2010). Additionally, ‘equal treatment’ was not associated with the established scales and further research on the nomological network of this scale is needed.
The newly developed context scales differ from established scales in that they are distinct from the scales addressing cognition, motivation, and emotion at both levels, i.e., the scale and item levels. This is important to avoid construct contamination. Some established learning environment questionnaires use a holistic definition of the learning environment and include cognitive, motivational or emotional aspects of the learning environment (AlHaqwi et al., 2014; Roff, 2005).
Furthermore, the interviews in Step 2 of Study 1 revealed the important role of peers and staff alongside other factors such as supervisory quality, organizational framework conditions, and equal treatment: Students learn not only from the teacher/supervisor but also from peers and other health professionals at the workplace. This is in line with studies on coregulation in SRL (Bransen et al., 2020) and community of practice (Cruess et al., 2018). Peers and staff also address the need for social relatedness. Social relatedness is an important determinant of personal growth according to self-determination theory (Deci & Ryan, 2012). Feeling accepted and supported by people in the workplace is relevant to students and their learning. Therefore, we decided not to integrate peers and staff into a more general ‘atmosphere’ scale or ‘framework conditions’ scale but to provide separate scales.
Metalevel
The cognition metalevel, motivation metalevel, emotion metalevel and context metalevel components regulate the respective aspects of the learning process. At this level, students are no longer at the learning process level and instead reflect on their learning process from a meta-perspective. For each of the four components, we included the ‘monitoring’ and ‘control’ scales. The psychometric analysis revealed the unidimensionality and acceptable reliability of the scales. The results regarding the different components on the metalevel are discussed together, as they have some similarities due to equivalent scales.
The inclusion of only two scales for the metalevel components contrasts with the theoretical perspective in educational psychology research on the academic setting, especially for the metalevel of cognition. Metacognition is a well-established and well-researched concept (e.g., see the various questionnaires or scales for the academic setting; Boerner et al., 2005; Edwards et al., 2014; Klingsieck, 2018; Pintrich et al., 1993) that encompasses different aspects. Pintrich (2004), for example, distinguishes among anticipation, planning, monitoring, control, evaluation, and reaction for each area. Contrary to the theoretical perspective, the students reported in the interviews in Step 2 and the cognitive pretesting in Step 6 of Study 1 that they did not think about their learning strategies, motivation, emotion and context in such a differentiated way, although they reported that thinking about their own learning behavior was crucial.
Furthermore, for the cognition metalevel, the cognitive pretesting of the cognition metalevel items revealed that the students thought about regulating concrete medical activities instead of regulating their learning behavior. These results can be interpreted in the context of the discussion about conscious and unconscious self-regulation of learning (Wirth et al., 2020). It is assumed that students regulate their learning unconsciously (i.e., anticipate, plan, monitor, control, evaluate and react) except in situations where they are faced with difficulties or challenging tasks (Flavell, 1979; Wirth et al., 2020). The decision to use equivalent scales for each component on the metalevel is supported by the findings of Kim and colleagues (Kim et al., 2020), who found that the cognition, motivation and emotion metalevels share regulatory mechanisms. Further research is needed to investigate whether the cognition, motivation, emotion, and context metalevels in workplace learning also share regulatory mechanisms.
The results of the nomological network for the metalevel components are complex. In the interpretation of the nomological network for the motivation and emotion metalevels, the different measurement foci must be taken into account. The newly developed scales focus on the question of whether students regulate motivation, emotion and the perception of context in contrast to established scales focusing on the how. We first highlight the most important results for the respective ‘monitoring’ scale and then for the ‘control’ scale.
For the cognition metalevel, the results are as expected. For the motivation metalevel, the newly developed ‘monitoring’ scale was positively associated with established scales (Schwinger et al., 2007) that are more relevant to the current situation and time e.g., increasing situational interest (Schraw & Lehman, 2001), but not with strategies that are relevant at a later time, e.g., a good grade. For the emotion metalevel, the newly developed ‘monitoring’ scale was positively associated with the established ‘rumination’ and ‘catastrophization’ scales but also with the ‘refocusing on planning’ scale (Garnefski et al., 2001; Loch et al., 2011). The association with rumination is consistent with theoretical considerations, as rumination refers to thinking about emotions (Loch et al., 2011), although monitoring does not necessarily involve rumination in the sense of becoming stuck. The interpretation of the association with catastrophizing and refocusing on planning is more complex. This could indicate that students who monitor their emotions use detrimental strategies to deal with negative emotions in addition to the helpful strategy of refocusing on planning. It could also indicate a process of dealing with emotions that begins with detrimental strategies such as rumination and catastrophizing before refocusing on planning. For context metalevel established questionnaires were missing.
Regarding the ‘control’ scales, the assessment of the nomological network showed no association with the established scales. A possible explanation for this result is the different level of scale-specificity. While the newly developed scales are on a more general level, the established scales are on a more specific level.
Strengths and limitations
To ensure the identification of indicators relevant to workplace learning and to address different types of validity, we combined qualitative and quantitative methods and included participants with different perspectives according to Gehlbach and Brinkworth’s seven steps (2011). To enhance trustworthiness, the questionnaire was developed by a multidisciplinary team that included members with different perspectives (Patton, 1999).
Our study also has some limitations because each component is a separate field of research and could be studied separately and in more depth. For example, for the area of context, the interpersonal aspects of learning between the learner and the faculty are less emphasized in the Workplace Learning Inventory (Cruess et al., 2018; Deci & Ryan, 2012; Morris & Behrens, 2013a; Roff & McAleer, 2017). Furthermore, the relationship of the newly developed scales within their nomological networks needs further attention in future studies. In the absence of established questionnaires for assessing workplace learning, we used established questionnaires for the academic setting. While it can be assumed that there is a relationship between the learning components of the academic setting and the workplace setting, this needs further investigation. Another limitation of our study is that the participants for the psychometric analysis came from only two institutions, both targeting the same health profession. We assume that the questionnaire is appropriate for different health professions because (a) the items are not specific to one health profession or field; (b) the students were in heterogeneous workplace settings; and (c) the scales and items were developed by integrating the perspectives from students, teachers and researchers from different institutions and health professions. However, results should be validated using samples from other health professions to find out, whether the items measure the same in related disciplines.
Scientific and practical implications
With regard to scientific implications, we highlight three needs that our study addresses. They have been articulated by the scientific community in relation to health sciences education. First, Albert et al. (2020) showed the need for interdisciplinarity in research on health sciences education. We address this need by integrating the educational psychology perspective on workplace learning. This is also in line with the tradition regarding research on workplace learning, where interdisciplinarity is highly valued (Hager, 2013). Second, van Houten-Schat et al. (2018) indicated the need to “unravel the sub-processes of SRL that are relevant to the clinical context in order to contribute to more elaborate SRL frameworks for this specific context” (p. 1014). They also determined the need for more quantitative studies. We address these needs by providing scales for the eight most relevant components of SRL in the workplace context which researcher can then select from based on the specific SRL model and research question. This is also in line with the call for a more holistic perspective in educational psychology research connecting different components of learning (Pekrun, 2006; Richardson et al., 2012). Third, researchers (Ciere et al., 2015; Schmitz & Perels, 2011; Schmitz et al., 2011) highlighted the potential of quantitative diary methods in studying learning in the healthcare setting. We address this issue by providing short scales and by formulating items that, viewed on a trait-state continuum (Geiser et al., 2017), address the state aspect of learning more.
With regard to practical implications, a better understanding of workplace learning can help address several problems in the practical part of health sciences education, two of which we highlight. First, a better understanding of the transition from academic learning to workplace learning addresses the problem of students struggling during the transition period (Atherley et al., 2019; Godefrooij et al., 2010; Teo et al., 2011). Students often perceive transition situations in health sciences education to be challenging and stressful (Teunissen & Westerman, 2011; Westerman & Teunissen, 2013). A better understanding of workplace learning can serve as a basis for intervention or further improvement of the curriculum. Second, a better understanding of workplace learning can address the problem of low well-being among health science students and professionals. This is especially important, as distress, depression and anxiety are severe issues (Dyrbye et al., 2006; Hope & Henderson, 2014). A better understanding of students’ workplace learning can help to identify unfavorable trends not only in student achievement but also in students’ well-being and serve as a basis for developing preventive measures.
Conclusion
The newly developed Workplace Learning Inventory is the first to address undergraduates’ workplace learning from an educational psychology research perspective. It is very comprehensive, as it addresses four different areas at two different levels, resulting in eight components of learning. Each component is addressed by several indicators and scales. The newly developed scales are short so that their administration is feasible in the workplace setting and they do not overlap and can therefore be combined in multivariate studies.
By providing the Workplace Learning Inventory, we hope to encourage multivariate studies of undergraduate workplace learning. Future studies can use the inventory for comprehensive investigations of undergraduate workplace learning in a cross-sectional or short-term longitudinal study by implementing a broad range of scales and for more detailed investigations of specific aspects in long-term longitudinal studies by selecting the respective scales. Such studies could contribute to a better understanding of workplace learning, its development over time and the associations between SRL components and other concepts relevant to workplace learning, such as stress or empathy.
Notes
Both, Boekaerts and Wirth and colleagues propose a third level, the domain-specific knowledge level or the content layer, respectively. We propose a model for workplace learning assessment specific to health sciences education, but not specific to a domain such as human medicine, veterinary medicine, nursing, physiotherapy or similar domains. As we are proposing a more general model for workplace learning assessment in health sciences education, we exclude this domain-specific layer from our model.
References
Albert, M., Rowland, P., Friesen, F., & Laberge, S. (2020). Interdisciplinarity in medical education research: Myth and reality. Advances in Health Sciences Education: Theory and Practice, 25(5), 1243–1253. https://doi.org/10.1007/s10459-020-09977-8
AlHaqwi, A. I., Kuntze, J., & van der Molen, H. T. (2014). Development of the clinical learning evaluation questionnaire for undergraduate clinical education: Factor structure, validity, and reliability study. BMC Medical Education, 14, 44. https://doi.org/10.1186/1472-6920-14-44
Allodi, M. W. (2010). The meaning of social climate of learning environments: Some reasons why we do not care enough about it. Learning Environments Research, 13(2), 89–104. https://doi.org/10.1007/s10984-010-9072-9
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84(3), 261–271. https://doi.org/10.1037/0022-0663.84.3.261
Artino, A. R., Brydges, R., & Gruppen, L. D. (2015). Self-regulated learning in healthcare profession education: Theoretical perspectives and research methods. In J. Cleland (Ed.), Researching medical education (Vol. 137, pp. 155–166). Wiley. https://doi.org/10.1002/9781118838983.ch14
Atherley, A. E., Dolmans, D., Hu, W., Hegazi, I., Alexander, S., & Teunissen, P. W. (2019). Beyond the struggles: A scoping review on the transition to undergraduate clinical training. Medical Education, 53(6), 559–570. https://doi.org/10.1111/medu.13883
Bandalos, D. L. (2021). Item meaning and order as causes of correlated residuals in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 28(6), 903–913. https://doi.org/10.1080/10705511.2021.1916395
Ben-Eliyahu, A. (2019). Academic emotional learning: A critical component of self-regulated learning in the emotional learning cycle. Educational Psychologist, 54(2), 84–105. https://doi.org/10.1080/00461520.2019.1582345
Ben-Eliyahu, A., & Bernacki, M. L. (2015). Addressing complexities in self-regulated learning: A focus on contextual factors, contingencies, and dynamic relations. Metacognition and Learning, 10(1), 1–13. https://doi.org/10.1007/s11409-015-9134-6
Bergsmann, E. M., Lüftenegger, M., Jöstl, G., Schober, B., & Spiel, C. (2013). The role of classroom structure in fostering students’ school functioning: A comprehensive and application-oriented approach. Learning and Individual Differences, 26, 131–138. https://doi.org/10.1016/j.lindif.2013.05.005
Bergsmann, E., Schultes, M.-T., Winter, P., Schober, B., & Spiel, C. (2015). Evaluation of competence-based teaching in higher education: From theory to practice. Evaluation and Program Planning, 52, 1–9. https://doi.org/10.1016/j.evalprogplan.2015.03.001
Berkhout, J. J., Helmich, E., Teunissen, P. W., & Tuenissen, P. W. (2016). The complex relationship between student, context and learning outcomes. Medical Education, 50(2), 164–166. https://doi.org/10.1111/medu.12950
Berkhout, J. J., Helmich, E., Teunissen, P. W., van der Vleuten, C. P. M., & Jaarsma, A. D. C. (2017). How clinical medical students perceive others to influence their self-regulated learning. Medical Education, 51(3), 269–279. https://doi.org/10.1111/medu.13131
Biwer, F., de Bruin, A., & Persky, A. (2023). Study smart-impact of a learning strategy training on students’ study behavior and academic performance. Advances in Health Sciences Education: Theory and Practice, 28(1), 147–167. https://doi.org/10.1007/s10459-022-10149-z
Boekaerts, M. (1996). Self-regulated learning at the junction of cognition and motivation. European Psychologist, 1(2), 100–112. https://doi.org/10.1027/1016-9040.1.2.100
Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students. Learning and Instruction, 7(2), 161–186. https://doi.org/10.1016/S0959-4752(96)00015-1
Boerner, S., Seeber, G., Keller, H., & Beinborn, P. (2005). Lernstrategien und Lernerfolg im Studium [Lerning strategies and academic success in higher education]. Zeitschrift Für Entwicklungspsychologie Und Pädagogische Psychologie, 37(1), 17–26. https://doi.org/10.1026/0049-8637.37.1.17
Bransen, D., Govaerts, M. J. B., Sluijsmans, D. M. A., & Driessen, E. W. (2020). Beyond the self: The role of co-regulation in medical students’ self-regulated learning. Medical Education, 54(3), 234–241. https://doi.org/10.1111/medu.14018
Bücker, S., Nuraydin, S., Simonsmeier, B. A., Schneider, M., & Luhmann, M. (2018). Subjective well-being and academic achievement: A meta-analysis. Journal of Research in Personality, 74, 83–94. https://doi.org/10.1016/j.jrp.2018.02.007
Burić, I., Sorić, I., & Penezić, Z. (2016). Emotion regulation in academic domain: Development and validation of the academic emotion regulation questionnaire (AERQ). Personality and Individual Differences, 96, 138–147. https://doi.org/10.1016/j.paid.2016.02.074
Ciere, Y., Jaarsma, D., Visser, A., Sanderman, R., Snippe, E., & Fleer, J. (2015). Studying learning in the healthcare setting: The potential of quantitative diary methods. Perspectives on Medical Education, 4(4), 203–207. https://doi.org/10.1007/s40037-015-0199-3
Cleary, T. J., Durning, S. J., Gruppen, L. D., Hemmer, P. A., & Artino, A. R. (2013). Self-regulated learning in medical education. In K. Walsh (Ed.), Oxford textbook of medical education (1st ed., pp. 465–477). Oxford Univ. Press. https://doi.org/10.1093/med/9780199652679.003.0040
Cruess, R. L., Cruess, S. R., & Steinert, Y. (2018). Medicine as a community of practice: Implications for medical education. Academic Medicine: Journal of the Association of American Medical Colleges, 93(2), 185–191. https://doi.org/10.1097/ACM.0000000000001826
Dai, D. Y., & Sternberg, R. J. (2004). Motivation, emotion, and cognition: Integrative perspectives on intellectual functioning and development. The educational psychology series. Lawrence Erlbaum Associates. http://lib.myilibrary.com/detail.asp?id=237498
de La Fuente, J., Sander, P., Martínez-Vicente, J. M., Vera, M., Garzón, A., & Fadda, S. (2017). Combined effect of levels in personal self-regulation and regulatory teaching on meta-cognitive, on meta-motivational, and on academic achievement variables in undergraduate students. Frontiers in Psychology, 8, 232. https://doi.org/10.3389/fpsyg.2017.00232
Deci, E. L., & Ryan, R. M. (2012). Self-determination theory. In P. van Lange, A. Kruglanski, & E. Higgins (Eds.), Handbook of theories of social psychology (Vol. 1, pp. 416–437). SAGE Publications Ltd. https://doi.org/10.4135/9781446249215.n21
Duffy, M. C., Lajoie, S. P., Pekrun, R., & Lachapelle, K. (2018). Emotions in medical education: Examining the validity of the Medical Emotion Scale (MES) across authentic medical learning environments. Learning and Instruction. https://doi.org/10.1016/j.learninstruc.2018.07.001
Dyrbye, L. N. M., Thomas, M. R. M., & Shanafelt, T. D. M. (2006). Systematic review of depression, anxiety, and other indicators of psychological distress among U.S. and Canadian Medical Students. Academic Medicine, 81(4), 354–373.
Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-value theory: A developmental, social cognitive, and sociocultural perspective on motivation. Contemporary Educational Psychology, 61, 101859. https://doi.org/10.1016/j.cedpsych.2020.101859
Edwards, A. J., Weinstein, C., Goetz, E. T., & Alexander, P. A. (2014). Learning and study strategies: Issues in assessment, instruction, and evaluation. Elsevier Science.
Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46(1), 6–25. https://doi.org/10.1080/00461520.2011.538645
Elliot, A. J., Murayama, K., & Pekrun, R. (2011). A 3 × 2 achievement goal model. Journal of Educational Psychology, 103(3):632–648. https://doi.org/10.1037/a0023952
Enders, C. K. (2022). Applied missing data analysis. Guilford Publications.
Fares, J., Al Tabosh, H., Saadeddin, Z., El Mouhayyar, C., & Aridi, H. (2016). Stress, burnout and coping strategies in preclinical medical students. North American Journal of Medical Sciences, 8(2), 75–81. https://doi.org/10.4103/1947-2714.177299
Finney, S. J., Pieper, S. L., & Barron, K. E. (2004). Examining the psychometric properties of the achievement goal questionnaire in a general academic context. Educational and Psychological Measurement, 64(2), 365–382. https://doi.org/10.1177/0013164403258465
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive—Developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906
Frajerman, A., Morvan, Y., Krebs, M.-O., Gorwood, P., & Chaumette, B. (2019). Burnout in medical students before residency: A systematic review and meta-analysis. European Psychiatry: THe Journal of the Association of European Psychiatrists, 55, 36–42. https://doi.org/10.1016/j.eurpsy.2018.08.006
Garnefski, N., Kraaij, V., & Spinhoven, P. (2001). Negative life events, cognitive emotion regulation and emotional problems. Personality and Individual Differences, 30(8), 1311–1327. https://doi.org/10.1016/s0191-8869(00)00113-6
Gehlbach, H., & Brinkworth, M. E. (2011). Measure twice, cut down error: A process for enhancing the validity of survey scales. Review of General Psychology, 15(4), 380–387. https://doi.org/10.1037/a0025704
Geiser, C., Götz, T., Preckel, F., & Freund, P. A. (2017). States and traits. European Journal of Psychological Assessment, 33(4), 219–223. https://doi.org/10.1027/1015-5759/a000413
Godefrooij, M. B., Diemers, A. D., & Scherpbier, A. J. J. A. (2010). Students’ perceptions about the transition to the clinical phase of a medical curriculum with preclinical patient contacts; a focus group study. BMC Medical Education, 10, 28. https://doi.org/10.1186/1472-6920-10-28
Hager, P. (2013). Theories of workplace learning. In M. Malloch, L. Cairns, & K. Evans (Eds.), The Sage handbook of workplace learning. This paperback edition first published (pp. 17–31). Sage.
Hayat, A. A., Shateri, K., Amini, M., & Shokrpour, N. (2020). Relationships between academic self-efficacy, learning-related emotions, and metacognitive learning strategies with academic performance in medical students: A structural equation model. BMC Medical Education, 20(1), 76. https://doi.org/10.1186/s12909-020-01995-9
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational Psychologist, 41(2), 111–127. https://doi.org/10.1207/s15326985ep4102_4
Hoerger, M. (2010). Participant dropout as a function of survey length in internet-mediated university studies: Implications for study design and voluntary participation in psychological research. Cyberpsychology, Behavior and Social Networking, 13(6), 697–700. https://doi.org/10.1089/cyber.2009.0445
Hope, V., & Henderson, M. (2014). Medical student depression, anxiety and distress outside North America: A systematic review. Medical Education, 48(10), 963–979. https://doi.org/10.1111/medu.12512
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., de Groot, E., Gilbert, M. C., Musu, L., Kempler, T. M., & Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139–151. https://doi.org/10.1080/00461520701416231
Kim, Y., Brady, A. C., & Wolters, C. A. (2020). College students’ regulation of cognition, motivation, behavior, and context: Distinct or overlapping processes? Learning and Individual Differences, 80, 101872. https://doi.org/10.1016/j.lindif.2020.101872
Kline, R. B. (Ed.). (2016). Methodology in the social sciences. Principles and practice of structural equation modeling (4th ed.). The Guilford Press.
Klingsieck, K. B. (2018). Kurz und knapp – die Kurzskala des Fragebogens „Lernstrategien im Studium“ (LIST) [A short version of the questionnaire on learning strategies in higher education (LIST)]. Zeitschrift Für Pädagogische Psychologie, 32(4), 249–259. https://doi.org/10.1024/1010-0652/a000230
Koenka, A.C. (2020). Academic motivation theories revisited: An interactive dialog between motivation scholars on recent contributions, underexplored issues, and future directions. Contemporary Educational Psychology,60, 101831. https://doi.org/10.1016/j.cedpsych.2019.101831
Kunter, M., Schümer, G., Artelt, C., Baumert, J., Klieme, E., Neubrand, M., Prenzel, M., Schiefele, U., Schneider, W., Stanat, P., Tillmann, K., & Weiß, M. (2002). PISA 2000: Dokumentation der Erhebungsinstrumente [PISA 2000: Documention of Assessment Instruments]. Materialien aus der Bildungsforschung: Nr. 72. Max-Planck-Institut für Bildungsforschung.
Loch, N., Hiller, W., & Witthöft, M. (2011). Der cognitive emotion regulation questionnaire (CERQ). Zeitschrift Für Klinische Psychologie Und Psychotherapie, 40(2), 94–106. https://doi.org/10.1026/1616-3443/a000079
Mayring, P. (2014). Qualitative content analysis: Theoretical foundation, basic procedures and software solution. https://www.ssoar.info/ssoar/handle/document/39517
McDonald, R. P. (1970). The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23(1), 1–21. https://doi.org/10.1111/j.2044-8317.1970.tb00432.x
Miele, D. B., & Scholer, A. A. (2018). The role of metamotivational monitoring in motivation regulation. Educational Psychologist, 53(1), 1–21. https://doi.org/10.1080/00461520.2017.1371601
Morris, C., & Behrens, M. (2013a). Work-based learning. In K. Walsh (Ed.), Oxford textbook of medical education (1st ed., pp. 209–220). Oxford Univ. Press. https://doi.org/10.1093/med/9780199652679.003.0018
Muthen, L. K., & Muthen, B. O. (1998–2017). MPlus (Version 8.6) [Computer software]. Muthen & Muthen. https://www.statmodel.com/index.shtml
Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and new findings. In Psychology of learning and motivation (Vol. 26, pp. 125–173). Elsevier. https://doi.org/10.1016/S0079-7421(08)60053-5
OECD. (2017). PISA 2015 results (volume III): Students’ well-being. OECD Publishing. https://doi.org/10.1787/9789264273856-en
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422
Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34(5 Pt 2), 1189–1208.
Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18(4), 315–341. https://doi.org/10.1007/s10648-006-9029-9
Pekrun, R., Muis, K. R., Frenzel, A. C., & Götz, T. (2018). Emotions at school. Ed psych insights. Routledge Taylor & Francis Group.
Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). Academic Press. https://doi.org/10.1016/B978-012109890-2/50043-3
Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385–407. https://doi.org/10.1007/s10648-004-0006-x
Pintrich, P. R., Smith, D. A. F., Garcia, T., & Mckeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (Mslq). Educational and Psychological Measurement, 53(3), 801–813. https://doi.org/10.1177/0013164493053003024
Poncelet, A., & O’Brien, B. (2008). Preparing medical students for clerkships: A descriptive analysis of transition courses. Academic Medicine: Journal of the Association of American Medical Colleges, 83(5), 444–451. https://doi.org/10.1097/ACM.0b013e31816be675
Richardson, M., Abraham, C., & Bond, R. (2012). Psychological correlates of university students’ academic performance: A systematic review and meta-analysis. Psychological Bulletin, 138(2), 353–387. https://doi.org/10.1037/a0026838
Roff, S. (2005). The Dundee ready educational environment measure (DREEM)—a generic instrument for measuring students’ perceptions of undergraduate health professions curricula. Medical Teacher, 27(4), 322–325. https://doi.org/10.1080/01421590500151054
Roff, S., & McAleer, S. (2017). Student learning environment. In J. L. Hodgson & J. M. Pelzer (Eds.), Veterinary medical education: A practical guide (pp. 511–524). Wiley Blackwell.
Roth, A., Ogrin, S., & Schmitz, B. (2016). Assessing self-regulated learning in higher education: A systematic literature review of self-report instruments. Educational Assessment, Evaluation and Accountability, 28(3), 225–250. https://doi.org/10.1007/s11092-015-9229-2
Rotthoff, T., Ostapczuk, M. S., de Bruin, J., Decking, U., Schneider, M., & Ritz-Timme, S. (2011). Assessing the learning environment of a faculty: Psychometric validation of the German version of the Dundee Ready Education Environment Measure with students and teachers. Medical Teacher, 33(11), e624–e636. https://doi.org/10.3109/0142159x.2011.610841
Rotthoff, T., Ostapczuk, M., Kröncke, K., Schneider, M., Decking, U., & Ritz-Timme, S. (Eds.). (2010). Erfassung des Lehr-/Lernklimas in der Medizinischen Ausbildung - Validierung des DREEM (Dundee Ready Education Environment Measure) in deutscher Sprache und ergänzender Messinstrumente. German Medical Science GMS Publishing House.
Sagasser, M. H., Kramer, A. W. M., Fluit, C. R. M. G., van Weel, C., & van der Vleuten, C. P. M. (2017). Self-entrustment: How trainees’ self-regulated learning supports participation in the workplace. Advances in Health Sciences Education: Theory and Practice, 22(4), 931–949. https://doi.org/10.1007/s10459-016-9723-4
Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729. https://doi.org/10.1177/0539018405058216
Schmitz, B., Klug, J., & Schmitz, M. (2011). Assessing self-regualted learning suint diary measures with university students. In B. J. Zimmerman & D. H. Schunk (Eds.), Educational psychology handbook series. Handbook of self-regulation of learning and performance (1st ed.). Routledge.
Schmitz, B., & Perels, F. (2011). Self-monitoring of self-regulation during math homework behaviour using standardized diaries. Metacognition and Learning, 6(3), 255–273. https://doi.org/10.1007/s11409-011-9076-6
Scholer, A. A., Miele, D. B., Murayama, K., & Fujita, K. (2018). New directions in self-regulation: The role of metamotivational beliefs. Current Directions in Psychological Science, 27(6), 437–442. https://doi.org/10.1177/0963721418790549
Schraw, G., & Lehman, S. (2001). Situational interest: A review of the literature and directions for future research. Educational Psychology Review, 13(1), 23–52. https://doi.org/10.1023/A:1009004801455
Schunk, D. H., & Greene, J. A. (Eds.). (2018). Educational psychology handbook series. Handbook of self-regulation of learning and performance (2nd ed.). Routledge/Taylor & Francis Group.
Schwinger, M., von der Laden, T., & Spinath, B. (2007). Strategien zur Motivationsregulation und ihre Erfassung [Strategies for the regulation of motivation and their assessment]. Zeitschrift Für Entwicklungspsychologie Und Pädagogische Psychologie, 39(2), 57–69. https://doi.org/10.1026/0049-8637.39.2.57
Schwarzer, R., & Jerusalem, M. (Eds.). (1999). Skalen zur Erfassungvon Lehrer- und Schülermerkmalen.: Dokumentation der psychometrischen Verfahren im Rahmen der Wissenschaftlichen Begleitung des Modellversuchs Selbstwirksame Schulen [Scales for assessing teacher- and student characteristics. Documentation of assessment instruments of the pilot project on self-effective schools]. Freie Universität Berlin.
Shuman, V., & Scherer, K. R. (2014). Concepts and structures of emotions. In L. Linnenbrink-Garcia & R. Pekrun (Eds.), International handbook of emotions in education (pp. 13–35). New York: Routledge Taylor & Francis Group.
Slavin, R. E. (2018). Educational psychology: Theory and practice (12th ed.). Pearson.
Strand, P., Sjöborg, K., Stalmeijer, R., Wichmann-Hansen, G., Jakobsson, U., & Edgren, G. (2013). Development and psychometric evaluation of the undergraduate clinical education environment measure (UCEEM). Medical Teacher, 35(12), 1014–1026. https://doi.org/10.3109/0142159X.2013.835389
EFS Survey. (2022). Unipark [Computer software]. Globalpark. Cologne, Germany. https://www.unipark.com/
Teo, A. R., Harleman, E., O’sullivan, P. S., & Maa, J. (2011). The key role of a transition course in preparing medical students for internship. Academic Medicine: Journal of the Association of American Medical Colleges, 86(7), 860–865. https://doi.org/10.1097/ACM.0b013e31821d6ae2
Teunissen, P. W., & Westerman, M. (2011). Opportunity or threat: The ambiguity of the consequences of transitions in medical education. Medical Education, 45(1), 51–59. https://doi.org/10.1111/j.1365-2923.2010.03755.x
Urdan, T., & Kaplan, A. (2020). The origins, evolution, and future directions of achievement goal theory. Contemporary Educational Psychology, 61, 101862. https://doi.org/10.1016/j.cedpsych.2020.101862
van Houten-Schat, M. A., Berkhout, J. J., van Dijk, N., Endedijk, M. D., Jaarsma, A. D. C., & Diemers, A. D. (2018). Self-regulated learning in the clinical context: A systematic review. Medical Education, 52(10), 1008–1015. https://doi.org/10.1111/medu.13615
Weinstein, C. E., Jung, J., & Acee, T. W. (2010). International encyclopedia of education. In International encyclopedia of education (pp. 323–329).
Weinstein, C. E., Acee, T. W., & Jung, J. (2011). Self-regulation and learning strategies. New Directions for Teaching and Learning, 2011(126), 45–53. https://doi.org/10.1002/tl.443
Westerman, M., & Teunissen, P. W. (2013). Transitions in medical education. In K. Walsh (Ed.), Oxford textbook of medical education (1st ed., pp. 372–382). Oxford Univ. Press. https://doi.org/10.1093/med/9780199652679.003.0032
White, C. B. (2007). Smoothing out transitions: How pedagogy influences medical students’ achievement of self-regulated learning goals. Advances in Health Sciences Education: Theory and Practice, 12(3), 279–297. https://doi.org/10.1007/s10459-006-9000-z
Wild, K. P., & Schiefele, U. (1994). Lernstrategien im Studium: Ergebnisse zur Faktorenstruktur und Reliabilität eines neuen Fragebogens [Learning strategies of university students: Factor structure and reliability of a new questionnaire]. Zeitschrift Für Differentielle Und Diagnostische Psychologie, 15(4), 185–200.
Willis, G. B. (2015). Analysis of the cognitive interview in questionnaire design. Oxford University Press.
Wirth, J., & Leutner, D. (2008). Self-regulated learning as a competence. Journal of Psychology, 216(2), 102–110. https://doi.org/10.1027/0044-3409.216.2.102
Wirth, J., Stebner, F., Trypke, M., Schuster, C., & Leutner, D. (2020). An interactive layers model of self-regulated learning and cognitive load. Educational Psychology Review, 32, 1127–1149. https://doi.org/10.1007/s10648-020-09568-4
Wolters, C. A. (1998). Self-regulated learning and college students’ regulation of motivation. Journal of Educational Psychology, 90(2), 224–235.
Wolters, C. A. (2003). Regulation of motivation: Evaluating an underemphasized aspect of self-regulated learning. Educational Psychologist, 38(4), 189–205. https://doi.org/10.1207/S15326985EP3804_1
Wolters, C. A., & Benzon, M. B. (2013). Assessing and predicting college students’ use of strategies for the self-regulation of motivation. The Journal of Experimental Education, 81(2), 199–221. https://doi.org/10.1080/00220973.2012.699901
Wolters, C. A., & Won, S. (2018). Validity and the use of self-report questionnaires to asess self-regulated learning. In D. H. Schunk & J. A. Greene (Eds.), Educational psychology handbook series. Handbook of self-regulation of learning and performance (2nd ed.). Routledge/Taylor & Francis Group.
Zimmerman, B. J. (2008). Attaining self-regulation. In M. Boekaerts (Ed.), Handbook of self-regulation [Nachdr.] (pp. 13–39). Academic Press. https://doi.org/10.1016/B978-012109890-2/50031-7
Acknowledgements
We thank the interview partners (Step 2), experts (Step 5) and students (Steps 6 and 7) for participating in our studies.
Funding
Open access funding provided by Austrian Science Fund (FWF). Austrian Science Fund, P 33913.
Author information
Authors and Affiliations
Contributions
ES: study design, data collection, data analysis, results interpretation, writing and review of the manuscript. SM: study design, data collection, data analysis, results interpretation, writing and review of the manuscript. TY: study design, data analysis, results interpretation, review of the manuscript. LDU, FP: study design, results interpretation, review of the manuscript. CP, PB, LS, UA: data collection, results interpretation. CK: data collection.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Ethical approval
The study was submitted to the Ethics Committee of the Medical University of Vienna. This committee makes decisions on studies involving human subjects at the University of Veterinary Medicine in Vienna. The decision was that no ethical approval was required according to the Declaration of Helsinki. The following ethical standards were met. Participation in the study was completely voluntary. Written consent was obtained for participation in the study and for the use of the data. Participants were assured that their responses would remain confidential and would only be used for scientific purposes. Complete anonymity was ensured by not publishing any data that would allow conclusions to be drawn about the identity of the respondents. Moreover, the study was approved and supported by the project steering committee, including the Vice-Rector for Study Affairs and Clinical Veterinary Medicine.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Disclaimer: Views are our own and not an official position of the institution.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Steinberg, E., Marsch, S., Yanagida, T. et al. Development and validation of the Workplace Learning Inventory in Health Sciences Education: a multimethod study. Adv in Health Sci Educ (2023). https://doi.org/10.1007/s10459-023-10295-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10459-023-10295-y