1 Introduction

An e-portfolio can be defined as “a digitized collection of artifacts including demonstrations, resources, and accomplishments that represent an individual, group or institution” (Lorenzo & Ittelson, 2005). Portfolios have been widely used in arts as a way to exhibit the skills and achievements of an artist, or in business to show the financial potential of an enterprise (Barrett, 2007). However, portfolios, mainly in their electronic format (e-portfolios), are gaining ground in educational settings, especially in the field of teacher education (Totter & Wyss, 2019). In this context, the portfolio takes a step further, and is more than a collection of artifacts because it encourages critical reflection, facilitates both formative and summative assessment, and reveals the acquisition of competences (Amaya et al., 2013; Mohammed et al., 2015; Ring et al., 2009). In addition, it is argued that the e-portfolio captures the learning process itself (Barrett, 2007) because the student’s progress can be monitored along the way (Gómez et al., 2013; Mohammed et al., 2015). It also allows the exchange of ideas and feedback (Lorenzo & Ittelson, 2005), key elements in formative assessment as well (Gedye, 2010), and one of the most important factors influencing learning (Hattie & Timperley, 2007; Wisniewski et al., 2020). Ultimately, some authors argue that it is most suitable for student-centred, rather than teacher-centred, learning environments (Clark & Eynon, 2009; Harun et al., 2021; Stefani et al., 2007) and allows students to be more active, self-regulated and autonomous in their learning process (Amaya et al., 2013).

Therefore, the rationale for using e-portfolio, as well as its potential benefits, are striking. Bearing in mind all these advantages, it is not surprising that the use of the e-portfolio is increasing in higher education institutions worldwide, especially in the United States (Clark & Eynon, 2009). In Spain, tentative steps forward include the creation of the “RED portfolio” (Barberà et al., 2009), although the use of the e-portfolio is not generalized across institutions (European Network of ePortfolio Experts & Practitioners, 2015).

Among the different reasons that may be hindering the dissemination of the e-portfolio in education, an important one is teachers’ resistance to use assessment methods other than the traditional exam, fearing the lack of objectivity (Litchfield & Dempsey, 2015). A way to overcome this resistance could be the demonstration of the extra benefits of the e-portfolio without the risk of losing the objectivity that academic achievement, or variables enhancing it, presents. Another hindrance, as Clark and Eynon (2009) note, is that the e-portfolio movement lacks a robust body of published practices. Although there is an important body of information about theoretical considerations and descriptions of practices, more empirical research about effective e-portfolio practices and their impact on student learning is needed (Kahn, 2014). In this sense, few published studies assess the impact of the e-portfolio on academic achievement and other related variables.

Academic achievement is currently a main concern in higher education, as it has important social, psychological, and economic repercussions (Mayhew et al., 2016). Two variables linked to academic achievement are academic engagement and academic self-efficacy. Academic engagement is defined as “a positive, fulfilling, and work-related state of mind that is characterised by vigour, dedication, and absorption” (Schaufeli et al., 2002). Academic self-efficacy is the belief in one’s capabilities to organize and implement actions leading to learning or performance at designated levels (Bandura, 1977). High levels of engagement and self-efficacy are related to an increased level of academic performance (Honicke & Broadbent, 2016; Lei et al., 2018).

According to Bandura (1986), self-efficacy is a cognitive variable that can influence and be influenced by behaviour and environment. And according to Schaufeli’s et al.’s (Schaufeli et al., 2002) definition of “state of mind”, engagement could also be considered a cognitive factor. Therefore, we could expect bidirectional influences between engagement and students’ behaviour and environment. That is, educational interventions, as the e-portfolio implementation, may lead to changes on students’ self-efficacy and engagement, and vice versa, students’ self-efficacy and engagement may predict the performance on the e-portfolio tasks.

It has been argued that the use of the e-portfolio promotes students’ active engagement in their learning (Barrett, 2007; Ring et al., 2009), although the empirical evidence is somewhat limited and based mainly on student’s opinion (see for example, Abuzaid et al., 2021; Felea & Stanca, 2019; Harun et al., 2012; Hegarty & Thompson, 2019), but not on validated measures. In addition, previous studies do not show whether students’ engagement predicts performance on the e-portfolio or vice versa.

There is more evidence, however, that the use of the e-portfolio promotes a higher self-efficacy on students. For instance, there is evidence that students who use portfolios present higher levels of self-effficacy (see for example, van Dinther et al., 2015; Ziegler, 2014; but see Franco et al., 2020 for contradictory evidence), and that there is an overlap between student’s self-reported competence in the portfolio and self-efficacy (Bergil & Sariçoban, 2017). These results appear to extend to the electronic portfolio as well. For example, Huang et al. (2011) found that internet self-efficacy significatively predicted student’s satisfaction with the e-portfolio. We note, however, that same as with engagement, previous studies have mainly employed qualitative methodologies and relied on the students’ surveys of opinion to reach to these conclusions (see for example, van Dinther et al., 2015) but not in quantitative studies using validated questionnaires (with the exception of the studies of Huang et al., 2011 and Ziegler, 2014). Furthermore, with the exception of Huang et al. (2011), who found that self-efficacy predicts satisfaction with the e-portfolio, no study to date explores the direction of the relationship between self-efficacy and the use of the e-portfolio, focusing especially in individual reflections as a key element of the e-portfolio. In words of Mackenna (McKenna et al., 2017) “More empirical research is required to establish whether […] e-portfolios have a positive effect on students’ perceived self-efficacy”.

All in all, it would be important to explore the bidirectional relationship between e-portfolio use and variables such as engagement and self-efficacy using validated questionnaires (in order to facilitate the comparison among studies) and determine if key elements of the e-portfolios, such as the individual reflections, are the elements contributing to these improvements.

1.1 Objectives

Considering the aforementioned antecedents, the main aim of this study was to provide initial evidence about the effectiveness of the e-portfolio as a suitable method in higher education. For this purpose, three specific objectives were set: 1) to explore whether achievement through individual reflections correlates with achievement through more traditional activities such as individual tests and group assignments; 2) to explore whether student’s personal variables predict performance in key elements of the e-portfolio such as the individual reflections, and whether its use has an effect on academic performance; and 3) to evaluate if the use of the e-portfolio during a semester changes students’ self-efficacy and engagement.

2 Methods

2.1 Participants

A total of 80 students of the Bachelor’s Degree in Psychology (University of Zaragoza), aged between 19 and 24 years, were enrolled on the Educational Psychology course in which the e-portfolio was implemented. Of these 80 students, six decided not to use the e-portfolio and one quit using it by the end of semester. Therefore, a final sample of 73 students (60 males and 13 females) participated in the study.

2.2 Instruments and measures

A repeated measures design was employed (see Fig. 1). Before and after the treatment phase, students were asked to fill out both engagement and self-efficacy questionnaires (see below). The treatment phase consisted of the e-portfolio implementation in the Educational Psychology course. The course syllabus included four main topics: History and Concept of Educational Psychology, Behavioural Models of Learning, Cognitive Models of Learning and Motivational Models in Education.

Fig. 1
figure 1

Schematic overview of the study design

Full size image

We employed the University of Zaragoza institutional tool for e-portfolios, Mahara (www.mahara.unizar.es). Mahara (www.mahara.org) is an open-source software created by the Government of New Zealand (Mohammed et al., 2015). It includes several features such as pages and collections, forums, and the typical features of a social network such as “walls” and the possibility of connecting with friends and joining groups. Importantly, it can be used both individually and in teams, allowing cooperative or collaborative learning.

Two instruments were selected to assess both pre- and post academic self-efficacy and engagement:

  • Self-efficacy. We employed the Perceived Self-Efficacy Specific for Academic Situations Scale (EAPESA; Palenzuela, 1983) to evaluate students perceived self-efficacy before and after the use of the e-portfolio. This scale consists of 10 items (e.g., ‘I feel confident approaching situations that test my academic ability’) assessed on a four-point scale (1 = never, 4 = always). Higher scores indicate higher perceived academic self-efficacy. The reliability of the scale in our sample, measured by Cronbach’s alpha, was .86 and .88 for pre and post, respectively.

  • Engagement. We employed an adaptation of the Utrecht Work Engagement Scale (UWES-S; Schaufeli et al., 2006) to the academic field. The original scale includes 17 items (e.g., ‘I can study for long periods of time’) which measure three factors, namely Vigour, Dedication, and Absorption, on a Likert-type scale ranging from 0 (never) to 7 (every day). We adapted the scale by replacing the work-specific terms with academic terms (i.e., in the item “I am enthusiastic about my job” the word “job” was replaced with “studies”). Preliminary analyses conducted in our laboratory with other samples revealed reliability problems with the factor Absorption (Cronbach alphas between .52 and .57; unpublished data), and with the factor loadings of some items of the other subscales. Therefore we eliminated the Absorption subscale and those problematic items, which lead to a scale composed of 13 items loading on one of two factors: Vigour and Dedication. The reliability of the scale in our sample, measured by Cronbach’s alpha, was .90 and .92 for pre and post, respectively.

Each one of the four course topics were assessed with the individual reflections’ activity of the e-portfolio, and with multiple choice tests.

  • Individual reflections. At the end of each topic, students wrote an individual reflection on their e-portfolio and share it with the teacher. This activity was intended to incorporate metacognitive reflection and reflective writing, key elements of the portfolio (Eynon et al., 2014) as they facilitate the integrative connection of learning in and out the classroom (Eynon et al., 2014; también nombráis a Dewey y Rogers). Individual reflections were expected to include 1) metacognitive analyses (e.g., statements about which concepts were harder to understand, understanding ratings, and reflections on how they could improve their understanding); 2) applications of the learned concepts and theories to real life through examples (e.g., the students could reflect on how several motivational theories influenced their own educational trajectory), and 3) critical visions of the topic (e.g., argued criticism of the learned concepts and theories or their applications to education). In total, the student had to write four individual reflections over the course, one per topic. The course instructor corrected the individual reflections with a rubric (see Table 1). Since the students are used to be graded on a 0 to 10 scale, the scores of the rubric were adjusted to 10. The scores in this activity were averaged and ranged from 0 to 10.

  • Multiple-choice tests. Students completed a multiple-choice test at the end of each topic (a total of 4). The tests, designed by the course instructor following the course’ syllabus, included about 10 questions with three response alternatives and were administered online (through the Questionnaire tool of the Learning Management System Moodle). Incorrect answers were penalized with 1/3 of the points of a correct answer. In total, the students had to complete four multiple-choice tests over the course. The scores of this activity were averaged and ranged from 0 to 10.

Table 1 Rubric employed to assess the students’ individual reflections
Full size table

In addition, students were required to conduct a group project, which consisted of an essay on a theoretical or practical resolution of an education-related problem. The teacher guided the students during the semester and used a rubric to assess the project. The rubric included key criteria such as the quality of the bibliography, APA normative adjustment, argumentation, grammar, and critical vision. The scores of this activity ranged from 0 to 10.

The final course grade could be obtained as it follows: up to 5 points through a series of activities programmed by the teacher (watching video lectures, answering questions, problem-solving, and class presentations); up to 2 points (adjusted from 10, see above) through the individual reflections collected in the e-portfolio; up to 2 points (adjusted from 10, see above) through the multiple-choice tests; and up to 1 point (adjusted from 10, see above) through the group project. In summary, we collected four measures of students performance: scores on individual reflections, scores on multiple choice-test, scores on the group project and lastly, the total score of the course.

2.3 Procedure

This study was designed following the ethical principles of the International Declaration of Helsinki. The participants were informed about the objectives of the study and data treatment. They had to give their consent for data analysis, following the Law 3/2018, of December 5, on the Protection of Personal Data and guarantee of digital rights.

The study included a pre-test and a post-test assessment, employing both the EAPESA and UWES-S (see Fig. 1). The pre-test (baseline) was conducted during the first class, when students did not even know the methodology of the course, whereas the post-test was conducted during the last class, after a whole semester working with the e-portfolio. The questionnaires were administered online using Google Forms. The order of administration was the same for all the participants: first, the EAPESA followed by the UWES-S.

In addition to pre-test administration, on the first day of the course, students were informed about the teaching method and the use of Mahara as a central tool in the course, course activities, and assessment criteria. Bearing this information in mind, the student could choose between continuous or final assessment (see Fig. 2). If they choose final assessment, they did not have to perform any activity related to the e-portfolio, performing a final exam at the end of the semester. If they choose continuous assessment, they had to carry out all the activities related to the implementation of the e-portfolio over the entire semester.

Fig. 2
figure 2

Procedure and teaching assessment options

Full size image

For students choosing continuous assessment, the course was structured in different group and individual activities. The group activities were carried out during the synchronous sessions, either online or in class. Students were required to carry out several activities, such as solving cases or making presentations. The individual activities were asynchronous and consisted of studying the material before the synchronous sessions (mainly watching videos and reading papers) and writing their individual reflections at the end of each topic (see the Instruments and measures section). Both the individual reflections and group activities were collected in the Mahara e-portfolio, which the teacher could access to provide feedback.

2.4 Data treatment and statistical analyses

Only 6 out of 80 students did not follow the course regularly and therefore, they were assigned to the final exam. Of the remaining 74, only one student left the continuous assessment and therefore, had to take the final exam. Thus, 92.5% of the students chose to work regularly with the e-portfolio, and 91.25% remained actively engaged with it by the end of the semester. We conducted Pearson correlations with the scores obtained by these 73 students in the individual reflections, final group project, and multiple-choice tests. As completing pre- and post questionnaires was voluntary, only 45 of the 73 students completed both pre- and post-tests for engagement and self-efficacy (some students completed the pre-test questionnaire but not the post-test and vice versa, and therefore their data were discarded). Once the data were collected, they were encoded for subsequent analysis with the statistical program SPSS-26 for Windows (George & Mallery, 2019).

The Shapiro-Wilk test was first performed to confirm the assumption of normality. In order to control type I errors, we employed the bootstrapping method, which also tends to yield the highest power (Hayes, 2009).

One of the main novelties in the employment of educational portfolios are critical reflections and metacognitive exercises. Due to the role that self-efficacy and engagement have on academic performance (Honicke & Broadbent, 2016; Lei et al., 2018), we aimed to explore if critical reflections could have a mediating role in this relationship. For this purpose, a mediational analysis was carried out through the macro PROCESS v3.5 (Hayes, 2017), putting initial engagement (pre) as the predictor or independent variable, the final grade as the outcome variable, and individual reflections as mediators. We first examined the correlations between assessment methods and self-efficacy and engagement. Mediation effects were calculated bootstrapped (k = 10,000), and bias-corrected 95% confidence intervals (CI) were estimated for the indirect effects (Preacher et al., 2007).

Lastly, in order to explore if the whole instruction changed students’ self-efficacy and engagement, we conducted a Student’s t test with pre and post scores on these variables.

3 Results

3.1 Relationship between achievement and assessment methods

To test whether a student’s academic achievement is similar regardless of the type of activity being assessed (that is, if a student with high performance in individual reflection also presents high performance in the other more traditional activities), Pearson correlations were calculated. There was a low-moderate positive correlation between the final group project with respect the individual reflections (r = 0.36; p < 0.05) and the final group project with the multiple-choice tests (r = 0.34; p < 0.05). The analysis revealed also a moderate positive correlation between multiple-choice tests and individual reflections (r = 0.49; p < 0.05). As Table 2 shows, the highest correlation occurred between individual reflections and multiple-choice tests.

Table 2 Pearson correlations between the three activities contributing to the continuous assessment
Full size table

3.2 Mediational role of individual reflections

Pearson’s correlations between the assessment methods and self-efficacy and engagement were first calculated. There was a low-moderate positive correlation between individual reflections and self-efficacy at the end of the course (post-test; r = 0.30; p < .05) and a moderate positive relationship between individual reflections and the two engagement subscales (pre-test r = 0.47; p < .01; post-test r = 0.49; p < .01). In the same line, the course grade showed a positive low-moderate association with self-efficacy (post-test, r = 0.31; p < .05) and engagement (pre-test r = 0.50; p < .01; post-test r = 0.48, p < .01). In addition, the multiple-choice tests presented positive but low correlations with engagement (pre-test r = 0.31; p < .05; post-test r = 0.36; p < .05). The final group project did not show any significant correlation with the tests. All correlations are shown in Table 3.

Table 3 Pearson correlations between assessment methods, course grade, self-efficacy and engagement (pre and post)
Full size table

Given these results, in order to explore if student’s personal variables (self-efficacy and engagement) predicted performance in the individual reflections and if this contributed to their final grade in the course, a mediational analysis was established. Our purpose was to explore if the individual reflections made in the e-portfolio had a mediating effect on the relationship between initial engagement (pre-test) and the final course grade.

The results of mediation analyses are illustrated in Fig. 3. Initial engagement exerted an indirect effect on the final grade through individual reflections (B = .469 SE = .1779; 95% CI [−.83, −.13]; p < 0.01). The same analysis was conducted with multiple-choice tests score as mediator, but the model was not statistically significant (B = −.186, SE = .1602; 95% CI [−.52, .11]).

Fig. 3
figure 3

Results of the mediational analysis. Indirect (a x b), total (c) and direct (c’) effects of initial engagement (predictor) on the final grade (outcome) via individual reflections made in the e-portfolio. ** p ≤ .01. *** p ≤ .001

Full size image

3.3 Changes in self-efficacy and academic engagement

To evaluate if there was a change in self-efficacy or engagement by the end of the course, Student’s t test were performed and the analysis revealed a significant increase in self-efficacy (Fig. 4a) [t (44) = −3.02, p < .005; d = .37] but not in engagement (Fig. 4b) [t(44) = −1.250, p = .218; d = 0.09].

Fig. 4
figure 4

Changes in a) the Self-Efficacy Specific for Academic Situations Scale (Self-efficacy) and b) the Utrecht Work Engagement Scale (Engagement). Pre- and post-test measures are presented with means ± st. dev. *p < .05 significant differences with respect the pre-test, n = 45

Full size image

4 Discussion

The use of the e-portfolio in education, especially in teacher education, has experimented a high increase in the last decades. However, there are still various unsolved questions about its implementation. This study was designed to answer three of them. The first one was whether individual reflection scores correlate with those obtained with more traditional assessment activities, such as multiple-choice tests or the final group project; the second one was whether student’s personal variables predicts performance in individual reflections and whether these contribute to improve the final grade in the course; the third one was whether the use of the e-portfolio during a semester changes the students’ self-efficacy and engagement.

Our results showed that individual reflection scores correlated positively with the performance on both the multiple-choice tests and the final group project. The highest correlations were found between the two individual activities, as expected, because the final project was a group activity carried out by students with likely different knowledge and skills. Therefore, at least in this experience, we can assert that individual reflections may work as good as other “objective” assessment methods, or that individual reflections yield the same individual variability as other objective assessments.

Both engagement and self-efficacy have been related to academic achievement (see, for example, the reviews of Honicke & Broadbent, 2016; Lei et al., 2018). Our study shows that both variables are relevant to understand performance on the e-portfolio but they operate in different ways. First, the implementation of the e-portfolio did not change the levels of engagement, as there was no change in engagement from pre- to post-test. However, engagement proved to be an important pre-requisite that had an indirect effect on the final course grade through the mediation of individual reflections, in line with the results of other studies (see Lei et al., 2018, for a meta-analysis of the relationship between student engagement and academic achievement). That is, it appears that a high initial engagement promoted a better performance in the individual reflection activity, which in turn contributed to a higher final course grade. Although there was no direct effect of initial engagement on the final course grade, the present results inform that individual reflections may play an important role in this relationship, as the total effect is the sum of many different paths of influence, direct and indirect. Future studies should address other paths that may contribute to the relationship, considering additional variables. One possible explanation to this indirect effect of individual reflections is that the e-portfolio is assumed to support self-regulated learning (Amaya et al., 2013). Critical reflections on the e-portfolio such as those encouraged in the individual reflections employed in this study have also been linked to self-regulated learning, as they improve as instructional strategies to develop self-regulated learning (Jenson, 2011). There is also a strong relationship between self-regulation and engagement (Sun & Rueda, 2012; Wolters & Taylor, 2012). Therefore, considering that self-regulated learning is a common link between engagement and reflective writing such as in the individual reflections, our results suggest that individual reflections denote a high level of self-regulation, one of the claimed but not yet experimentally demonstrated benefits of e-portfolio implementation. Further studies should confirm this idea as well as determine which specific element(s) of individual reflection are contributing the most to its mediational role in the relation between initial levels of engagement and the final grade of the course.

In contrast to engagement, our results revealed that pre-course levels of self-efficacy did not correlate with the final course grade or with the performance in activities such as individual reflections or multiple-choice tests. However, the implementation of the e-portfolio over a semester was associated with an increase in self-efficacy. In 1986, Bandura proposed the Triadic reciprocally model of causality, which explained human behaviour by the interaction of three factors: person, behaviour, and environment. Perceived self-efficacy is a personal factor, and is both a prerequisite to start and maintain an action—that is, to promote a change in the behaviour factor (e.g., someone with low perceived self-efficacy in a specific field is reluctant to change their behaviour)— and also a variable that is modified by the behaviour (e.g., if the person finally starts and completes the task successfully, perceived self-efficacy will increase). In this study, we found that the e-portfolio seems to operate in the second way, increasing students’ sense of self-efficacy. Although scarce, other studies have found an increase in self-efficacy after educational interventions. For example, Dunlap (2005) found an increase in self-efficacy after the implementation of a problem-based learning methodology in higher education students, and Banfield and Wilkerson (2014) reported a similar increase after introducing gamification as an experiential learning method. According to Bandura’s theory, changes in self-efficacy can also come from four sources: vicarious learning, verbal persuasion, physiological states, and performance accomplishments (Bandura, 1986). Whereas we cannot discard indirect changes in self-efficacy through the first three factors, the most probable factor in changing our students’ perceived self-efficacy was performance accomplishment. The positive correlation found between post-test self-efficacy and the final course grade supports this idea. Interestingly, in the final course grade, post-test self-efficacy correlated with the individual reflection scores, but not with other elements such as performance in individual multiple-choice tests or the group project. Therefore, a specific element of the e-portfolio, individual reflections, appears to be quite relevant to understand the changes in self-efficacy observed after the portfolio implementation. Interestingly, other studies testing for the effect of including self-reflection as part of the teaching methods found the same increases in self-efficacy (Bleicher & Lindgren, 2005; Dunlap, 2005). This agrees with the evidence showing that self-efficacy and performance can be increased, providing the students with opportunities to practice self-evaluation (Schunk & Pajares, 2009), and deserve more research in the future.

Although these are promising results, several limitations must be mentioned. First, as answering the questionnaires was voluntary, the sample size is limited (only 45 students answered both questionnaires). Moreover, only highly engaged and self-regulated students may have answered both questionnaires, and this a source of bias. Second, the study lacks a control group to compare the e-portfolio intervention with a more traditional teaching methodology. For this reason, we cannot conclude that there is a causal relationship between the use of the e-portfolio and the observed increase in self-efficacy. The third limitation refers to the mediational analyses conducted, as we collected measures at two temporal moments whereas the recommendations are to collect it at three moments: the independent variable at T1, the mediational variable at T2, and the dependent variable at T3. The last limitation refers to self-reported measures in general, as they are influenced by social desirability.

In spite of these limitations, our results are a first step in the empirical evidence for the use of the e-portfolio in higher education. Our results are promising because they show that the use of the e-portfolio, or more specifically a characteristic element of it, the individual reflections, are related to higher performance during the course as well as increased self-efficacy after implementation. The fact that students’ engagement predicts overall performance in the course through individual reflections implies that it would be important to know the starting level of engagement of students: it is possible that the e-portfolio is the most suitable method to improve performance only in the case of highly engaged students. On the other hand, our study suggests that the implementation of the portfolio could be a good method to increase self-efficacy. In short, it is necessary to deep into personal variables, such as self-efficacy and engagement, related to the use and benefits of the portfolio both before and after its implementation. Additional studies shall confirm and extend our findings.