Introduction

There are a variety of training formats to develop research ethics and research integrity competencies (e.g. Steele et al. 2016; Katsarov et al. 2021). The issue lies in evaluating the effect of the training to learners – as programmes vary and the measurement instruments are different, it is challenging to conclude which format of training is most effective (Steele et al. 2016; Stoesz and Yudintseva 2018). The key might be in finding an aspect that many ethics trainings have in common, for instance reflection.

Reflection is a crucial part of ethics education as it supports the development of ethical sensitivity and ethical decision-making (Mustajoki and Mustajoki 2017; Löfström and Tammeleht 2023). For researchers to become reflective practitioners (Scön’s term since 1987), especially on topics pertaining research ethics and integrity (REI), various instructional methods can be used to support the development of reflection competence. For instance, talking about one’s work before, during, and after doing various activities (Schön 1987). Still, written reflection tasks may provide better results as writing offers a chance to pause and have an inner dialogue with oneself (Lázaro et al. 2022). This helps to become more aware of oneself as well as observe one’s learning process. While “reflection” is usually defined as “careful thinking” in dictionaries (e.g. Cambridge Dictionary), in adult learning context (as outlined by Knowles 1978) the entire adult education learning process is reflection.

Thus, it seems that written reflective learning diaries (also called logs or journals) might be useful in developing REI competencies as well as instruments for monitoring the learning process. A variety of terms are in use: learning diaries (Authors 2023), learning logs (Clarkeburn et al. 2002; Lázaro et al. 2022), reflective learning journals (Thorpe 2004), reflection journals (Mintz 2006), reflective diaries (Gibbs et al. 2007), also digital platforms provide means like blogs, apps or social media platforms (Voss 2012). In our study we use the term “learning diaries” to capture the idea of long-term commitment to reflection of own learning. Nevertheless, we will keep the original terms used by various authors we refer to.

Our choice of the term “research ethics and integrity” (REI) aims to combine various facets of ethics present in research institutions, combining both research ethics (as knowledge about guidelines and codes of conduct) and research integrity (common practices and behaviour in research community) as various training formats may pertain to one or the other or both (Tammeleht 2022; Tammeleht and Löfström 2023).

Prior research indicates (Watts et al. 2017; Stoesz and Yudintseva 2018; Katsarov et al. 2021) that using written reflective tasks is not very widespread in ethics courses, and their effectiveness is considered moderate. Nevertheless, there are some examples of reflective learning logs used in REI education (like Clarkeburn et al. 2002; Gibbs et al. 2007; Lázaro et al. 2022), but the specific analysis criteria were not outlined in those studies. It seems that there are limited criteria to evaluate the development of REI competencies based on reflective learning diaries. We also argue that monitoring the development of REI competencies in learning diaries over a longer period of time could provide insights to the effectiveness of REI courses.

The goal of our study was first, to increase understanding about how learning diaries capture development of REI competencies of participants of the micro-credential programme, and second, to explore their use as a measure of effectiveness of REI training in the long term (in our case, throughout one academic year). To do this, we present three different formats of learning diaries and provide criteria for analysing learning diaries, namely through (1) evaluating the levels of reflection, (2) evaluating the levels of understanding (based on the SOLO taxonomy), and (3) monitoring the content knowledge when discussing ethical issues.

Theoretical background

While reflection activities (e.g. in the form of learning diaries, logs or blogs) are considered important in developing ethical sensitivity (Mustajoki and Mustajoki 2017), they are not used very often in ethics education. For instance, Watts et al. (2017) outline in their meta-analytic review of ethics courses in sciences that out of the sample size of 150, 47 incorporated a (written) self-reflection activity and 18 used a web-based discussion (also written), both displaying a modest effect size (Cohen’s d of 0.4). (Explanation of Cohen’s d in Cohen 2013) Similarly, in their review of effective academic integrity instruction Stoesz and Yudintseva (2018) identified four out of 21 studies included in the review, which included a written assignment (discussion of ethical aspects) at the end of the course. In addition, Katsarov et al. (2021) pinpoints 10 studies with (written) reflection activities in the sample size of 75 (with a moderate effect size at Cohen’s d 0.6).

Nevertheless, learning diaries have been used in ethics education. Mintz (2006) used reflective journals effectively in ethics education as well as Thorpe (2004) who used Kember et al.’s (1999) reflection levels to evaluate the reflective learning journals. Clarkeburn et al. (2002) report using learning logs during a 4-week course (10 entries from participants), where the logs were used as individual assessment means. The participants of the course were asked to reflect on what they had done during their learning activities, what they had learned, reflect on their learning, and contemplate the impact on the next learning unit. While the logs were generally well-written, they displayed low engagement of ethical considerations and minimal reflective elements (Clarkeburn et al. 2002). The authors did not outline the criteria for evaluation for learning logs nor were the logs a focus of their study, thus, it is not possible to evaluate the effectiveness of the course based only on the logs.

Moreover, Gibbs et al. (2007) describe using reflective journals during a research ethics course, in which the learners were asked to reflect on their experiences and intuitions when faced with ethical dilemmas discussed during the course. The course facilitators provided cases and guiding questions for reflection. The diaries displayed the learning process and personal development but were not the basis for evaluation neither for personal development, nor for course effectiveness. Lázaro et al. (2022) describe using learning logs for reflective writing and metacognition during a bioethics course. The main objective for using logs in their study was to provide learners with a chance to pause for writing and thinking to encourage reflection. The authors also outline that in higher education context it is not enough to focus just on teaching content; to solve complex problems a more reflective and internal contemplation must be added to education. As in Gibbs et al. (2007), the purpose of the learning logs for Lázaro et al. (2022) was to monitor and evaluate the learning process by the facilitators as well as learners and took the form of dialogic reflection. Questions were provided to guide the writing process and feedback was provided later by the facilitators. The authors analysed the logs by evaluating the level of reflection, relevant content of the course as well as thoughts about different activities used during the course, no specific criteria were provided by the authors. The analysis of logs revealed an evolution of reflective thinking as well as improvement of ethical reasoning.

Researchers also point out some drawbacks of logs/diaries. Writing, especially reflective writing might be a challenge for many students and may even hinder learning/development (Lázaro et al. 2022). Still, gradual improvement might be possible and while students may start off as using merely “descriptive writing” (with no personal views in the logs), with practice and time the logs start to include “descriptive reflection” (mostly descriptive but with some reflective elements) and finally also “dialogical reflection” (asking questions and making connections). Admittedly, only about a quarter of the logs in that research reached the level of “critical reflection” where the students displayed personal development (Lázaro et al. 2022). Still, the authors conclude that learning logs should be used as they provide learners with a space to think, order thoughts and make sense of what has been learned. Gibbs et al. (2007) also emphasise that reflective journals may be more suitable for adult learners as they are more intrinsically motivated. Journals may also include very personal content which means sharing them should be voluntary. In addition, when course evaluation is based on learning logs, students may be under pressure and not be able to feel at ease with the task (Gibbs et al. 2007).

Even though reflective learning diaries are used to a certain extent in REI courses, there is limited information about how to monitor the development of REI competencies and the effectiveness of a REI course based on learning diaries. We posed the following research questions:

  1. (1)

    How can learning diaries be used to understand the evolution of reflection and content knowledge during a REI course?

  2. (2)

    How do learning diaries display long-term development of REI competences?

Method

In our evaluative case study, we used a micro-credential programme as the context of the research. We provide information about the three formats of learning diaries and evaluate the development of REI competencies of learners based on their learning diaries.

Case studies provide an opportunity to use a specific instance to illustrate a more general principle or concept (Cohen et al. 2007). A case provides information about a real-life situation at a specific timeframe and context of a group or individual by providing rich data for detailed analysis (Cohen et al. 2007). Case studies may not be generalisable, but there are measures to ensure reliability and validity of the method by providing detailed information about the case (Cohen et al. 2007). Our case study focused on REI competence development over a 1-year micro-credential programme. There were originally eight participants in the programme, out of whom five participants provided access to all their learning diaries and one case study document for this research. Three more participants provided access to some of their learning diaries or their case study document. We used authentic examples, that is, learning diary entries from the learning process to evaluate the development of reflection and REI competencies over a period of one semester of keeping learning diaries. We also evaluated a case study done by participants five months after the last learning diary entry to see the retention of competence levels.

Context and participants of the study

In 2022, a new micro-credential programme was opened to support development of REI competencies in researching organisations. The goal of the programme was to educate REI leaders who could solve various issues pertaining to research ethics and research integrity in their field and institution, and who could guide others. The aim of the programme was that after the successful completion, a participant recognises ethically sensitive issues of research and academic ethics; is able to improve REI practices in their chosen field; knows the main values and principles underlying academic and research ethics and integrity; and has acquired knowledge of the formation of a value-based organisational culture and the implementation of good research practices in the organisation.

The volume of the programme was 24 ECTS credits and it lasted for two sequential semesters. During both semesters, the learners were expected to complete 12 ECTS credits through participation in courses, during which they had to attend seminars, watch video lectures, read assigned materials, and complete written tasks. During the first semester, learners had to complete three courses (12 ECTS in total): “Critical thinking and argumentation”, “Basics of ethics” and “Research integrity: framework, requirements, values and principles of action”. During the second semester, learners had two courses to complete (12 ECTS in total): “Ethics in an organization” and “Workplace applications of practical ethics”. The courses included in the program helped to develop knowledge and competencies of various ethical issues related to research and organisational practices. The program concluded with a final summative task where each participant was expected to apply the acquired knowledge and competencies in solving practical ethical issues encountered in their daily work.

Most of the studies during the first semester were held online – either in study forums or online seminars. Since the courses in the first semester involved a substantial amount of independent work and face-to-face seminars were limited to just a few occurrences, we decided to support the learners with online learning diaries. For the second semester we developed tailored courses for this specific programme, keeping in mind the learners feedback related to the aspects of schedule. Therefore, all the seminars (held once a month) took place face-to-face and independent studies were supported via an online platform. (See more information about the development and organisation of the programme in Authors 2023).

Participants for our study were sought from the micro-credential programme described above. While most participants represented higher education institutions, they were from various backgrounds, including teaching, arts, medicine, and life sciences. Most of them had doctoral degrees and held a full-time position. The micro-credential programme started with eight learners in the autumn semester. One of the initially registered participants decided to withdraw before the courses started, and another dropped out four months later, right before the end of the semester. The remaining six learners successfully completed the entire programme after the spring semester. Five of them gave permission for the analysis of learning diaries and a case study. Additionally, three new learners joined during the second semester. One of them completed both of the spring semester courses through open academy and gave permission for the analysis of the case study. The other two were graduate students from the university and both finished one of the courses of the micro-credential programme. One of them gave permission for the analysis of the case study submitted in the spring semester.

Data collection

Data was collected in the form of learning diaries and one case study. There were three formats of written digital documents provided by participants:

  1. 1)

    Individual learning diaries – participants wrote their entries individually and could not see each other’s responses;

  2. 2)

    Learning diaries created in the forum platform – participants could read each other’s entries and provide their own response on the platform;

  3. 3)

    Individual task – a case study done by each participant individually.

Learning diary entries were added about once a week over a period of one semester (four months), there were four individual learning diary entries and six forum format entries. Facilitators provided questions and topics for reflection for each learning diary entry (e.g. Should all research and research data be available open access? or description of a short case and guiding questions) and the case study had a template (see the Appendix). Participants were not taught or directly guided to use ethical principles, analysis or approaches during diary keeping. In total 21 individual learning diary and 32 forum format entries were submitted by participants. At the start of the second semester ethical principles, analysis and approaches were discussed with the participants as a reflection of topics covered in the learning diaries. Case studies were submitted by seven participants five months after the end of diary keeping. Five participants submitted all three formats of documents. They were coded as P1-P5 to display individual development.

Research ethics

The current research followed the Estonian national code of conduct (Hea Teadustava 2017) as well as the all-European code of conduct for research integrity (ALLEA 2017). An ethics review is not required in Estonia for research such as this one, in which participants are adults who give their consent voluntarily, and which addresses neither sensitive topics nor interferes with the physical integrity of participants.

Consent for using the learning diaries for research purposes was sought retrospectively (an information letter and a digitally signed consent form). Participants were approached at the end of the programme with a request for consent for their learning diaries in order not to interfere with the learning during the course.

As pointed out by Thorpe (2004) and Gibbs et al. (2007), there are various ethical aspects to consider when collecting and analysing learning diaries. We took the following measures to uphold a high ethical standard:

  1. a)

    Learning diaries were initially implemented to support learners to make connections of various concepts they were learning in three different courses during the first semester. The diaries were not a basis for course evaluation, but they were summarised and discussed during the monthly meetings of the programme participants and served as a thread to tie the learned material into one.

  2. b)

    By the request of the participants the individual format of writing the learning diary entries was changed into a forum format – learners claimed that they would like to read each other’s entries, and this would help them understand better what they were learning.

  3. c)

    As adult learners, participants took responsibility for writing the entries on time. Nevertheless, some entries were superficial indicating the pressure of time or decreased motivation. As the problems did not continue, facilitators did not see the need to interfere.

  4. d)

    While the facilitators were ready to keep the personal content confidential in the group, but due to the participant-initiated forum format this became unnecessary. The group seemed to have reached a trusting work environment and felt at ease to express their views and reflections.

  5. e)

    In order to decrease the pressure and anxiety of evaluation, the diary entries were not used for assessment. In addition, permission to use the learning diaries for analysis and research purposes was sought retrospectively to gain more authentic content and relieve pressure. Participants were approached at the end of the programme with a request to provide informed consent for their learning diaries. Out of nine participants, eight provided their consent by digitally signing the informed consent document. Five people agreed to submit all their written work. There were three participants who granted permission to some of their written work but were present only during the first or the second semester.

Data analysis

Qualitative data analysis was implemented but mixed methods (including correlation) were used to display results and extract patterns. We utilised deductive content analysis (Cohen et al. 2007) for interpreting content of learning diary entries and MaxQDA 2022 programme (VERBI Software 2021) was used for coding and analysis. The unit of analysis was one entry by one person. This provided an opportunity to compare the development of the person as well as identify similarities and differences between learners.

Triangulation was implemented in the data analysis phase. Learning diaries and one case study were analysed from various perspectives: (1) the content knowledge (ethical principles, ethical analysis, ethical approaches); (2) development of reflective competence (based on Kember et al.’s (1999) reflection levels) and (3) level of understanding (based on the SOLO taxonomy by Biggs and Collis 1982). To enhance validity, we had multiple coders.

There were two coders who coded all entries and a third coder who analysed 83% of forum format entries (5 entries by 6 participants, N = 30). Inter-coder agreement was 96% for the two coders who coded all the data, and 78% with the third coder.

For the analysis of the learning diaries, we used the following criteria: content knowledge, levels of reflection and levels of understanding (example excerpts can be seen in Table 1).

Table 1 Explanation of the SOLO taxonomy (Tammeleht et al. 2019; 2022; Tammeleht and Löfström 2023) and examples from data
Full size table

Content criteria

One of the cognitive tools that researchers have at their disposal to analyse situations involving ethical dimensions is called ethical analysis (Mustajoki and Mustajoki 2017). The analysis starts with the recognition that the situation involves an ethical question or dilemma. The nature of the question or dilemma must be identified, as it is pertinent to the rest of the analysis process what kind of issue one is dealing with; is it for instance a situation that jeopardises the principle of non-maleficence, or puts people’s autonomy at stake? Ethical principles (Kitchener 1985) provide a solid framework for identifying and naming various ethical aspects, which may be at stake in a situation:

  • autonomy – respecting the right of individuals to make choices regarding their own lives and protecting their privacy;

  • doing no harm – non-maleficence and avoiding (mostly psychological or social) harm;

  • beneficence – contributing to the well-being of others;

  • justice – being fair and objective, respecting reciprocity and equality;

  • being faithful – fidelity, keeping promises, being truthful and respecting others.

Once the nature of the issue is clear, ethical analysis guides to the identification of the stakeholders affected by the situation, either directly or indirectly. Various stakeholders may have differing interest and needs and consequently the situation could mean a variety of different things for the stakeholders. In order to arrive at a sound conclusion, it is important to understand what is at stake and for whom.

There may be laws, guidelines and rules that set the parameters for the situation and its potential conclusions. Therefore, knowing which laws, guidelines, rules and moral rights and duties are in place and how they define what is legal, expected, and reasonable in a situation, is a great help and resource for the researcher solving an ethical question or dilemma. Only after these analytical steps is it meaningful to start considering options and possible solutions. Without carefully laying the groundwork any solutions proposed may fall short an optimal outcome. If the situation at hand poses a true ethical dilemma, one must bear in mind that no proposed solution will be fully optimal from all perspectives considered. Especially in these situations, a careful ethical analysis will provide sufficient tools for deciding about a situation that is inclined to leave an ethical residue no matter what the solution. All in all, ethical analysis contains the following steps:

  • identifying the ethical issues at hand (including ethical principles);

  • identifying various stakeholders – both primary and secondary;

  • recognising the rights and responsibilities, guidelines and rules;

  • providing possible courses of action and considering their implications;

  • finally, the decision-maker must reflect on their own positionality (Mustajoki and Mustajoki 2017).

We also identified ethical approaches (consequentialist, rule-based and virtue-based as outlined in Hursthouse and Pettigrove 2023) in case they were distinctly recognisable in the entries.

Levels of reflection

To understand qualitative variation in reflection, it is possible to distinguish between levels of reflection Kember and his colleagues (Kember 1999; Kember et al. 2000) draw primarily on the work of Mezirow (1991) and they propose the following levels:

  • non-reflective thinking – this means displaying habitual action and just repeating words from the prompt.

  • descriptive level – this included describing what happens and how.

  • analytical level – this includes reflection on experience, i.e., what is the meaning of something.

  • reflective/critical level – this means the learner displays change or redirection of their competence, for example taking a different view or realising a mistake and adjusting perspective.

Performing routine activities does not involve much reflection and is therefore referred to as habitual action. When routines are not sufficient for a task, or there are no suitable routines in place, thoughtful action aiming at understanding what is going on is needed. Reflection is yet not pertinent, and the activity is more descriptive than analytical. However, as the learner engages analytically, the level of reflection increases, and at the following, analytical level of reflection already moves beyond the merely descriptive and captures past experiences and meaning making. Finally, critical reflection involves elements of perspective transformation, that is, the analytical reflection and meaning-making processes lead to qualitatively fundamental change in thinking or in-depth insight. Critical reflection entails the recognition of own pre-defined beliefs and values, and understanding how those influence any perspectives taken. In the context of ethical analysis, merely routine habitual action could leave the decision makers own interest, beliefs, and values unattended. Analytical and critical reflection would in the context of ethical analysis involve attention to one’s own motives and how one’s beliefs and values influence the decision making, even critically scrutinising the validity of one’s supposed beliefs as a foundation for the decision making.

Levels of understanding

The Structure of Observed Learning Outcomes, that is, the SOLO taxonomy was used to evaluate the level of understanding of content knowledge. Biggs and Tang (2007) argue that the SOLO taxonomy is hierarchical and based on research of learning outcomes in higher education settings. The SOLO taxonomy levels are related to the learning outcomes the learners display (Biggs and Tang 2007) and it has been used to evaluate understanding in the REI context (Löfström 2012; Tammeleht et al. 2019, 2022). We used the following framework (see Table 1) for deductive analysis of understanding of content knowledge.

Results

We first scrutinised the coding results of learning diaries to understand evolution of reflection and content knowledge. Development of levels of reflection and understanding was analysed in individual learning diaries and forum format diaries separately to pinpoint differences.

Individual logs displayed reflection on the levels from non-reflective to analytical, critical level was not displayed (see Fig. 1 top). 38% of responses were on the analytical level, 33% on the descriptive level and 29% were non reflective entries. Forum format diary entries indicated a change in reflection levels (Fig. 1 bottom). Moving to this format from individual reflection, only 6% entries were on the non-reflective level. More than half of entries were on the analytical level, and about a third were on the descriptive level. Critical level of reflection was displayed by 9% of learners.

Fig. 1
figure 1

Screenshot of reflection levels in individual logs—top (N = 21 entries) and in forum format entries—bottom (N = 32 entries)

Full size image

Overall individual development can be seen in Fig. 2. P0 to P5 indicate codes for participants (N = 6). The first six columns (P0-P5) display code occurrences in individual diary entries and the last six columns indicate results in forum format entries. In forum format diary entries participants displayed not only higher levels of reflection but also more content knowledge topics (ethical principles and ethical analysis). Occasionally, it was also possible to deduct the ethical approach the participants utilised (virtue-based, rule-based or consequentialist) – the codes indicate that the most prevalent approach was rule-based. Also, individual differences can be seen between participants – while during the individual diary submission the differences were not so clear, they become more pronounced during forum format diary keeping. For instance, P4 reflects on analytical level and displays most often virtue ethics approach. In addition, P6 very clearly displays the rule-based approach, which seems to align with the principle of being just. More individual development patterns will be discussed below.

Fig. 2
figure 2

Screenshot of overall individual development of reflection levels and content knowledge (P0-P5 indicate participants, N = 6; numbers indicate frequency of codes, i.e. how many times learners indicated this aspect)

Full size image

Development of content knowledge understanding was identified based on the SOLO taxonomy. In individual diary entries participants mostly displayed understanding of the ethical principles on the relational level (meaning that they provided explanations pointing out interrelations and examples to illustrate their opinion). Occasionally, the multistructural level was displayed, but the unistructural level was not common (see Fig. 3 top).

Fig. 3
figure 3

Screenshot of levels of understanding (SOLO levels) in individual diary entries—top (N = 21) and in forum format diary entries—bottom (N = 32)

Full size image

In the forum format entries participants displayed more awareness of content knowledge on different levels of understanding. About half of participants discussed topics in the relational level, 15% also indicated an extended abstract level (relating the ethical issues to applications beyond the present case) (Fig. 3 bottom). Nevertheless, as many topics were new to participants, unistructural and even pre-structural levels were displayed in about 15% of entries, while about a quarter of responses were on the multistructural level.

We next analysed the display of long-term retention of REI competencies. This means that we looked at the overall results of all learning diaries and compared them to the results of case studies (submitted five months after finishing learning diaries). We also looked at individual development process of participants based on the levels of understanding (SOLO levels) and reflection levels.

Level of understanding and reflection of content knowledge in learning diaries was illustrated in Figs. 1 and 3. The results can also be summarised as overall results (N = 53 diary entries) of SOLO levels (Fig. 4 top) and reflection levels (Fig. 4 bottom). Figure 4 (top) indicates that 65% of participants display understanding on relational level or higher. About a quarter display multistructural level and 11% remain under the threshold level. Figure 4 (bottom) indicates that about a half of participants can reflect on the analytical level, 30% on the descriptive level. Critical level is displayed by 6% and non-reflective thinking by 15% of participants.

Fig. 4
figure 4

Screenshot of overall levels of understanding (SOLO levels) in learning diaries—top and overall levels of reflection in learning diaries—bottom (N = 53)

Full size image

Five months after finishing learning diaries the participants were given a case study with 3 specific tasks that were analysed separately. The overall results can be seen in Fig. 5 which indicates that the lowest levels of understanding and reflection are not present anymore. In addition, the rate of the highest levels of understanding and reflection have increased (19% and 14% respectively). About a half of participants display relational level of understanding and analytical level of reflection. About a third of participants can discuss on the multistructural and descriptive levels.

Fig. 5
figure 5

Screenshot of SOLO levels (top) and reflection levels (bottom) in case studies (overall, N = 21 tasks in the case studies)

Full size image

Individual development was monitored on participants that submitted all three formats of assignments. Figures 6, 7 and 8 display individual development: first of individual diary entries (I + number on the graphs), then of forum format entries (F + number on the graphs) and finally a case study (CS) with three tasks. Figures 6, 7 and 8 indicate that the SOLO and reflection levels fluctuate similarly. Occasional downturns and peaks seem to correlate between the SOLO and reflection levels. There are, of course, individual differences, for instance participant P5 (Fig. 8) seems to be the least inconsistent with the trends set by others, P2 (Fig. 6) and P4 (Fig. 7) indicate higher levels in SOLO levels than in reflection, while P5 (Fig. 8) occasionally displays very high levels of reflection. Other than participant P5, participants display retention of high levels of reflection and understanding.

Fig. 6
figure 6

Individual development of participant 2

Full size image
Fig. 7
figure 7

Individual development of participant 4

Full size image
Fig. 8
figure 8

Individual development of participant 5

Full size image

We also looked at correlation of average SOLO and reflection levels (of all submitted entries) achieved by participants. As indicated in Fig. 9, levels of understanding (SOLO) tended to be higher for most participants (participant 5 had almost the same average for both). The average SOLO level was relational for participants 1, 2 and 4; for participants 3 and 5 it was between multistructural and relational. The average reflection level is between descriptive and analytical, participants 3 and 5 are more descriptive; 1 and 2 are between the two levels and 4 is mostly analytical.

Fig. 9
figure 9

Scatter graph of participants’ average results of SOLO and reflection levels (numbers 1–5 at the top indicate participants 1–5 who submitted all formats of assignments; numbers 0–3 on the left indicate SOLO and reflection levels – see Table 1 for reference)

Full size image

Correlation of SOLO and reflection levels in individual and forum format entries indicated that the analytical level correlated mostly with the relational SOLO level. Still, relational level was also present with descriptive and non-reflective level. Multistructural level also occasionally correlated with analytical, descriptive and non-reflective level.

Discussion

We first scrutinised how learning diaries can be used to understand the evolution of reflection and content knowledge understanding during a REI micro-credential programme. Analysis of individual vs forum format diary entries indicated that when learners have a chance to work in a group, they tend to notice more relevant topics as well as display higher levels of understanding and reflection. This correlates strongly with previous studies (Smith and Leonard 2005; Cavanagh 2011; Larraz et al. 2017; Tammeleht 2022) that outline that working in a group and collaboration improves critical thinking and decision-making, especially in the context of ethics education. Pertaining to content knowledge and the levels of understanding we saw that in individual diary entries there were a limited number of ethical principles and no indications of ethical analysis topics. Forum format entries displayed all ethical principles and ethical analysis steps as well as indicated towards which ethical approach learners tended to incline.

Results also indicate that the reflection levels improve over time. While during individual diaries about a third of entries displayed non-reflective thinking, it decreased significantly during forum format and was not present in the case studies. By the same token, critical level was not present during individual entries, but was present in forum format entries and increased further in case studies. Lázaro et al. (2022) also indicate that levels of reflection improve over time but only about a quarter may reach the critical level of reflection, in our case 14% displayed the highest level of reflection.

Even though in some cases the relevant content was discussed in a non-reflective manner or on a low level of understanding (based on the SOLO taxonomy), this did not seem to be prevalent. The occasional low display of reflection and understanding was present only for individual entries and not consistent in one person’s submissions. This may indicate that the learners had occasionally not given the tasks full attention (due to e.g. time constraints, work overload or personal issues). Indeed, as Gibbs et al. (2007) also pointed out learning diaries may be more suitable for adult learners as they are more intrinsically motivated, and they take responsibility for their own learning. The results of individuals indicated that the average levels of both understanding and reflection were mostly relational and analytical, we can conclude that learners had obtained relevant knowledge, and analysis and reflection competencies needed for REI leaders (which was also the goal of the programme).

We also monitored how learning diaries display long-term development of REI competences and whether the obtained competencies were retained. For that we compared the average results of learning diaries and the results of case studies. In addition, we outlined the entire learning process based on the SOLO and reflection levels for individual learners (N = 5).

SOLO levels tend to indicate that the results are more consistent, and there is less extreme fluctuation in the individual learning process. Reflection levels tend to fluctuate more. This may indicate that the SOLO taxonomy as a tool can be considered a robust means for evaluation (Biggs and Tang 2007; Hattie and Brown 2004; Hook 2012). As also indicated by prior research, reflection may be also a more difficult competence to obtain (McAlpine and Weston 2000; Nevgi and Löfström 2015).

Individual learning process graphs also indicated that some learners were more consistent while others may develop at their own pace indicating rather modest levels initially but achieving good results by the end of the programme. Some fluctuations posed a question for both researchers and training facilitators – the occasional high levels alternated with relatively low levels of both understanding and reflection, which could not be explained without more detailed investigation (which we could not do due to programme having ended). We do not anticipate that participants’ level of understanding as a whole drops, but merely fluctuations are likely to display variation in how well different topics have been understood. Some themes may have been understood in less depth because of several reasons, such as the topic being more complex, or the learner having less prior knowledge on the topic. We may also speculate that due to time constraints or work overload learners had not given the learning diary task their full attention and just wrote something not to miss the submission altogether. Reasons for fluctuation could be explored in more detail in the future.

Case study conducted five months after finishing with learning diaries participants displayed very high retention of competencies obtained. All ethical principles and ethical analysis steps were present, learners also indicated ethical approaches they used in completing the tasks. Non-reflective thinking and unistructural/prestructural levels were not present. About two thirds of responses indicated analytical reflection level or higher, and relational SOLO level or higher. Indeed, this result could be explained by further development of REI competencies during the second semester with the help of independent reading, lectures and group discussions during seminar days. Overall, the high levels of reflection and understanding may indicate an effective learning experience. Prior studies (e.g. Watts et al. 2017) mostly conclude effectiveness of ethics training based on pre- and post-test where generally content knowledge was measured. Focusing on competence development and retention may provide a more accurate picture of the effectiveness of training.

Limitations

We are aware of limitations of our study. One of them is a small sample size, however, this was compensated by the multitude of data points from each participant. On the other hand, this limitation may also be an advantage – only with a small sample size is it possible to scrutinise the learning progress of individuals, especially over a long period of time. There may be subjectivity in interpretation, but the fact that three coders coded the data, and had relatively high inter-rater agreements suggested that the analytical framework was solid enough to withstand subjective interpretations. We are also aware that it may be challenging to provide a measure for evaluating course effectiveness across different trainings because trainings can have a variety of learning objectives. However, although this is a small-scale study with a limited number of participants, we think there is broader applicability of the levels of understanding and reflection which can be applied on any learning irrespective of training format and objectives.

Conclusion and recommendations

In this case study our goal was to find out how learning diaries could be used to understand the evolution of reflection and content knowledge during a REI micro-credential programme, and how learning diaries display long-term development of REI competences, which would hint at the effectiveness of the programme.

As a result of the research, we can outline criteria for analysis to be used in REI training programmes for learning diaries. The learning diaries and evaluation criteria could be used as one measure to monitor the evolution of learners’ progress as well as to evaluate the effectiveness of the training and/or course.

Results indicated that learning diaries submitted in the forum format provide learners an opportunity to asynchronously interact with their peers and this supports them in displaying more content knowledge on high levels of reflection and understanding. Submitting learning diaries during a longer period and making submissions repeatedly can improve writing and reflection competencies. The quality of learning diary entries may also depend on learner characteristics, such as commitment and motivation. Adult learners usually take responsibility for their learning, but time management and workload may affect the level of investment in the learning, that is, reflection and understanding. Still, keeping a learning diary can help retain obtained competencies over a longer period of time.

Specific evaluation or analysis criteria make it possible to see the learning process, identify problematic topics or misconceptions. Content criteria help identify relevant conceptions and aspects necessary for ethical decision-making. Monitoring them helps identify gaps in knowledge and needs for improvement. Content criteria can be added or deleted as necessary. Levels of reflection and understanding provide a common framework for evaluating content knowledge.

Based on the research results we make the following recommendations:

  • Learning diaries support making connections between various content criteria and enhance reflection competence – both are needed for ethical decision-making and development of ethical sensitivity.

  • Using learning diaries as one means of evaluation during REI training provides relevant information about the learning process and competence development.

  • In order to decrease the pressure and anxiety of evaluation, the diary entries should not be used for final assessment. They could be used to evaluate personal progress.

  • Learners should make submissions repeatedly over a period of several weeks or months – this way they get a chance to improve their reflection competence and deepen content understanding.

  • In addition to individual learning diaries, forum format diary entries could be considered – they may provide support to learners and enhance personal competence development. Facilitators should discuss the format with learners and not pressure them to choose this option if they oppose it.

  • Personal characteristics of learners may have an impact on the quality of reflection and understanding – being overwhelmed with work or poor time management skills may hinder submitting good quality entries.

We believe that using various measurement points and tools provide a more holistic picture of the learning process. Indeed, with large-scale trainings it may not be feasible to analyse various aspects in such great detail. Future studies could test out learning diaries in different REI training contexts as well as finding ways to automate analysis and evaluation.

On behalf of all authors, the corresponding author states that there is no conflict of interest (no third parties being influenced by the article content). Information disclosure, acknowledgements, information about funding and data availability are included on the cover page.