Introduction

The process of making, which entails designing, creating, and sharing self-produced artefacts, has a long history in educational research (Halverson & Sheridan, 2014; Bieraugel & Neill 2017). Several educational theorists, such as Dewey, Piaget, and Papert, have previously advocated the significance of hands-on learning and experimentation with tangible artefacts in personally meaningful ways (Blikstein, 2013; Halverson & Sheridan, 2014; Bieraugel & Neill 2017; Schad & Jones, 2020). With the advances of new technologies, making has taken new forms and acquired a distinctive character manifested by the Maker Movement (Dougherty, 2012; Martin, 2015). According to Martin (2015), making activities refer to “designing, building, modifying, and/or repurposing material objects, for playful or useful ends, oriented toward making a “product” of some sort that can be used, interacted with, or demonstrated” (p. 2). Such activities have been practiced in places broadly known as Makerspaces. In general, makerspaces are seen as hands-on learning environments where learners explore project-based learning, participate in learning-by-doing activities, and develop innovations. Specifically, in makerspaces, participants can design and develop an idea and construct it into some physical or digital form (Halverson et al. 2014) with the use of traditional craft and hobby techniques (e.g., sewing, woodworking) and the manipulation of digital technologies, either for manufacture (e.g., laser cutters, 3D printers) or within the design (e.g., microcontrollers, LEDs) (Martin, 2015).

The philosophy of the Maker Movement has been widely adopted in K-12 education (Rouse & Rouse, 2022) and has been closely linked to STEM (Science, Technology, Engineering, and Mathematics) or STEAM (Science, Technology, Engineering, Arts and Mathematics) education (Martin, 2015; Clapp & Jimenez, 2016; Timotheou & Ioannou, 2019a,b). Beyond its broader link with STEM or STEAM, Maker Education is also claimed to provide learning opportunities which are not limited to subject domains (Papavlasopoulou et al. 2017; Iwata et al. 2020). For example, very recently, researchers have been exploring the relationship between making and 21st-century learning and how making projects could help students develop the necessary skills to achieve their career and life goals through the identification of their strengths and areas of interest (Freiman, 2020). Such skills are widely known as 21st -century skills and have become a necessity in today’s societies as they help learners navigate the challenges of the twenty-first century (Scott, 2015). Specifically, Ananiadou and Claro (2009) defined 21st century skills as the “skills young people will be required to have in order to be effective workers and citizens in the knowledge society of the 21st century.” Koul et al. (2021) argued that the multifaceted nature of Maker Education, which involves designing, innovating, collaborating and building, provides the “perfect context” for students to develop 21st-century skills (p. 76).

Assessing the 21st century skills, however, has proven challenging. Namely, the complex nature of 21st-century skills requires different types of assessments, not necessarily relying on paper-and-pencil or multiple-choice tests (Geisinger, 2016), but rather drawing on the learners’ perspective in the form of self-assessment and reflection (Care & Kim, 2018). In general, challenges in assessing the 21st century skills seem to be related to: (a) their complex nature, which creates difficulties in their conceptual understanding (Care, 2018; Nieveen & Plomp, 2018), (b) the learners’ lack of self-assessment and reflection skills (Bowler & Champagne, 2016; Siverno et al. 2021), and (c) the lack of documentation tools and processes for engaging learners in assessment and reflection about their development of these skills (Peppler et al. 2017). Furthermore, in the context of makerspaces, such issues have been largely under-explored (Rayna & Striukova, 2021; Freiman, 2020).

To address the above challenges, several researchers suggested the introduction of innovative forms of assessment situated in realistic settings that would integrate the existing know-how on the assessment of 21st-century skills with new technological tools (Care & Kim, 2018; Nieveen & Plomp, 2018). Extending on these ideas, the present study investigates the use of a novel digital self-assessment tool which was designed to capture the development of 21st-century skills, through self-assessment and reflection, during making. The work sought to illuminate the experiences of teachers and students who used the digital self-assessment tool in makerspace contexts. The overarching research question (RQ) of the study was:

RQ:

How do teachers and students experience the use of a novel digital self-assessment tool of 21st-century skills in makerspace contexts?

The RQ was approached with a particular focus on the abovementioned challenges i.e., complex nature of skills, lack of self-assessment and reflection skills, and lack of tools and processes. Findings from this work provide insights into the application of innovative forms of assessment of 21st -century skills that take advantage of technological means (Geisinger, 2016; Care & Kim, 2018). Additionally, findings promote a deeper understanding of how digital self-assessment tools can engage learners in self-assessment and reflection processes.

Background work

The potential of makerspaces to foster 21st century skills

During the past years, several educational frameworks for the development of 21st-century skills serve as points of reference to guide educational policy and practice into the successful preparation of learners in succeeding in work and life (e.g., P21  - Partnership for the 21st Century Learning, 2019). According to Larson and Miller (2011), the essence of 21st -century skills includes “strong communication and collaboration skills, expertise in technology, innovative and creative thinking skills, and an ability to solve problems” (p. 121). Additionally, Scott (2015) referred to three categories of 21st -century skills which are considered vital for the 21st -century workforce, namely “personal skills (e.g., initiative, creativity), social skills (e.g., teamwork, networking), and learning skills (e.g., managing, organizing)” (p. 2). The 21st -century skills “are transversal, have mobility, adaptability and accessibility across subject matter without being directly linked to a content base” (Kipp et al. 2018, p. 42). Rayna and Striukova (2021) argued that some 21st-century skills are hard to foster in a traditional classroom environment as they require learners’ activity in physical spaces, access to technologies, and multidisciplinary approaches in teaching and learning including hands-on learning.

An ever-increasing body of literature shows that makerspaces have the potential to foster the development of a range of 21st -century skills (Freiman, 2020; Iwata et al. 2020; Rayna & Striukova, 2021; Koul et al. 2021; Timotheou & Ioannou, 2021a, b). For example, Timotheou & Ioannou (2021a, b) provide evidence for the potential of makerspaces to support the enactment of learning and innovation skills. Also, Soomro (2022) presented evidence of how makerspaces foster creativity in a wide range of disciplines. Moreover, Rayna and Striukova (2021) argued that the half-digital-half-physical nature of makerspaces provides opportunities for the development of entrepreneurial and digital skills. Nevertheless, research is inconclusive as to how the 21st -century skills can be assessed in makerspace contexts.

Capturing 21st century skills in makerspaces through self-assessment and reflection

The nature of 21st-century skills is complex (Voogt et al. 2013) which makes their assessment challenging. According to Care and Kim (2018), 21st -century skills are demonstrated through actions; therefore, assessment needs to attend to actions and behaviors or enable inferences to be drawn from those. Additionally, these skills are often implicitly taught, and thus it can be difficult for teachers to measure or assess their development (Kipp et al. 2018). Indeed, Geisinger (2016) proposed that assessing 21st -century skills requires different types of assessment than paper-and-pencil or multiple-choice formats. Such types of assessment can be supported by practices which include the student perspective, such as self-assessment and reflection (Care & Kim, 2018). Peppler et al. (2017) found that self-assessment is one of the most common approaches used in makerspaces.

Hughes and Thompson (2022) view self-assessment and reflection as an essential part of making, arguing that students should be taught how to reflect on their own performance in terms of meeting the learning goals. Indeed, previous work has investigated how learners in makerspaces are encouraged to engage in reflective practices where they are called to assess their own learning (Oliver, 2016; Baykal et al. 2021; Hughes & Thompson, 2022). For instance, Bieraugel and Neill (2017) introduced a self-assessment tool (in both paper and electronic format) which captures students’ confidence in the development of exploration and fabrication technologies that take place in digital fabrication facilities. The authors used a combination of short-answer and binary-choice questions that called students to identify key component parts or give the details of a specified electrical appliance or electronic device. One of the main challenges they encountered was related to the representation of complex knowledge beyond factual information and the use of appropriate language that students would understand to consistently measure skills and knowledge on fabrication technologies. Additionally, Bowler and Champagne (2016) used question prompts to encourage young people’s reflection on the design, development, and use of technological artefacts in makerspaces to assess their ability to think deeply, critically, mindfully, and with a sense of responsibility about those artefacts. The authors noted that youth participants didn’t have prior experience in reflecting on the questions they asked themselves when making digital artifacts, so their responses were raw and intuitive. They proposed that preparation and time for reflection are needed if youth are to be engaged in self-prompting questions during their making activities.

In general, several issues pertaining to the application of self-assessment processes remain under-researched and are logical to arise in aiming to assess 21st -century skills in makerspaces. According to Greenstein (2012), self-assessment and reflection are important skills that can be used as assessment strategies to capture 21st-century learning. The essential elements of self-assessment include, among others, reviewing learning, providing evidence of learning and evaluating progress. However, younger learners may need more structured approaches and support to master the above (Greenstein, 2012). This view is consistent with previous studies showing that reflection is an obscure concept for young learners and that there is a “struggle in supporting or maintaining the reflection in activities” for school-aged children (Baykal et al. 2021, p. 1). Also, Siverno et al. (2021) studied elementary pupils’ reflections on written essays regarding collaborative design processes, team collaboration and their co-inventions during a 2-year project which involved digital and traditional fabrication technologies. The authors reported that the students remembered their major decisions during the project and reflected on their collaboration. However, there were cases where the students forgot to report or didn’t connect all the activities that contributed to their co-inventions (e.g., expert knowledge transfer during two visits to museums).

Other assessment issues may stem from the complex nature of the contexts and settings in which 21st -century skills are being developed. For instance, Bieraugel and Neill (2017) found that reflection is mainly supported in quiet places, such as libraries, in contrast to makerspaces which, per these authors’ findings, were rated low in reflecting as they are places with noise, movement distractions, and group work going on nearby. Also, a survey by Peppler et al. (2017) found that makerspace practitioners experience several barriers in their assessment practices which include but are not limited to the lack of access to dedicated technology for documentation, the lack of youth motivation to capture making, forgetting to capture work and lack of skills required for capturing, among others. What is more, making activities may be intentionally designed to be “play-like” and the metrics for success are mainly based on student interest, engagement, and excitement (Timotheou & Ioannou, 2019a; Weiner et al. 2018), rather than direct assessment of skills development. For more direct assessment of skills, teachers need different strategies and tools. Figueroa-Flores (2016), for example, proposed gamification as a promising approach to 21st century learning due to its potential to “reinforce not only knowledge but also important skills such as problem solving, collaboration, and communication” (p. 508).

Gamification as a concept in educational settings is mainly about adapting game design elements to be used for teaching and learning purposes in non-game contexts (Deterding et al. 2011). The most frequently adopted game elements include but are not limited to levels, achievements, badges, points, leaderboards, quests, and avatars (Buckley et al. 2018). In some cases, gamification is expanded to gameful design as a wholistic learning experience design in a classroom setting (Ioannou, 2018). Although not in the context of makerspaces, a previous study by Kipp et al. (2018) tested the use of a mobile application for the self-assessment of 21st century skills in K-12. The authors developed a gamification framework which uses pedagogical elements for the design of a digital self-assessment tool, known as SkillsTrack. The authors found that the tool successfully activated student literacy and created informal learning opportunities around the skills (Kipp et al. 2018). They also reported that the language used for 21st century skills for students of various age/grade/year levels was appropriate for some but too sophisticated for other students.

From the above, we can conclude that there is a need to examine how an assessment tool could support self-assessment and reflection in makerspace contexts, by addressing challenges related to the conceptual understanding of 21st-century skills as well as students’ engagement in the process of self-assessment and reflection. In line with Geisinger’s (2016) proposal that the exploration of technological means may address the challenge of assessing the complex constructs of 21st-century skills, in this study, we sought to investigate the design and enactment of a novel digital self-assessment tool for 21st-century skills. The digital tool was explicitly designed for use in makerspace contexts to address the challenges stemming from (a) the complex nature of 21st century skills which cannot be easily assessed, and (b) students’ lack of engagement in self-assessment and reflection in these contexts. The overarching research question of the study was “how do teachers and students experience the use of a novel digital self-assessment tool of 21st-century skills in makerspace contexts” and was approached with a particular focus on the abovementioned challenges.

Methodology

The authors employed qualitative research methodology. Specifically, an exploratory case study was conducted aimed at the in-depth investigation of the use of the digital self-assessment tool of 21st-century skills in different makerspace contexts. The study involved collecting data from multiple contexts to provide a comprehensive understanding of the phenomenon being studied e.g., in-school makerspaces vs. out-of-school makerspaces, making activities linked to school curricula vs. making activities as extracurricular work, technologies available for projects from 3D printers to microbits, very experienced vs. less-experienced teachers, duration of making activities and nature of the making projects. The study draws from the perspectives of all participants (teachers, students, researcher-observer) and focuses on how their different meanings illuminate their experiences with the tool (Yin, 2018).

Participants

Participants were six classes of students (n = 94) aged 12–18 years old, 47% boys and 53% girls, from three urban and two rural areas, who used the digital tool as part of their daily or weekly making activities. Participants were also three male and three female teachers (n = 6) and the researchers (authors of this work). Four of the teachers had previous experience in maker education and their students normally engaged in making activities on a weekly basis. The other two teachers had no directing experience implementing making projects with their students, although they personal experience (e.g., as makers) and sought to engage in maker education. Prior to the study, all teachers received training on the use of the digital tool, as well as specific guidelines on how to introduce the 21st-century skills and the tool to their students. The procedures of study implementation including data collection were known to the teachers before the pilots began. Ethical approval was granted by teachers, students and gradients.

Digital tool “Assessmake21”

Assessmake21 is a digital self-assessment tool which was introduced to support the self-assessment of 21st-century skills in makerspace contexts. The tool was developed as an expansion of a previous tool, known as SkillsTrack by Kipp et al. (2018), addressing K-12 students in general school contexts. The tool is web-based and runs on any browser. It includes a “Definitions” feature which provides information on each of the 21st century skills being assessed, namely a conceptual and operational definition of the skill as a construct, dimensions of the construct, and examples of possible evidence demonstrating the use of the skill. The conceptual and operational definitions and dimensions were drawn from widely known 21st-skills frameworks, such as the P21 (Partnership for the 21st Century Learning, 2019). The details of the tool design, including decisions on what skills to be assessed and what challenges/questions to be presented are discussed in parallel publications by the authors (to appear, Cyprus Interaction Lab).

To motivate students to participate in self-assessment and reflection, the tool incorporates gamification elements including challenges, progression in levels upon the completion of challenges, earning of batches, and visual feedback e.g., thumbs-up. The tool engages the students in the process of self-assessment and reflection on their 21st -century skills development, through the so called “tagging”. The “tagging” of skills involves the students self-selecting a skill they are using during a making activity. Tagging triggers, a series of reflection activities, so called “challenges”. For example, a challenge involves asking the student to create a “visual portfolio” by uploading photos of their work/artifacts and elaborating on how the use of the tagged skill allowed them to achieve the selected work/artifact. Figure 1 shows what the visual portfolio challenge looks like in the tool.

Fig. 1
figure 1

Asessmake21 - The visual portfolio challenge

Full size image

By completing challenges, students can progress through levels. Upon the completion of a level, students can request a badge, which is awarded by the educator. There is always a level for students to self-assess their skills (i.e., infinite process), giving them the possibility of continued use of the tool (see Fig. 2). The available options for tagging skills include five skills (Collaboration, Creativity, Problem-solving, Life/Social Skills, and Communication), the selection of which was a result of a design study presented in parallel work (to appear, Cyprus Interaction Lab).

Fig. 2
figure 2

Asessmake21 -The levels of skills’ progression in the tool

Full size image

The whole process is overseen by the teacher, who is responsible for setting up and managing a virtual classroom on the educator’s dashboard. Namely, the tool allows the teachers to register their students by generating a unique login code. They can also register their self-assessment approach, with three options: (i) when a parallel approach is selected, students are asked to engage with the tool during their making activities, by tagging skills and completing challenges throughout the class session, (ii) when a parallel tapping and challenges at the end approach is followed, students are called to tag the skills they perceive they are using during their making activity, but complete the challenges at the end of the making activity, and (iii) when an at-the-end approach is followed, students are called to rate their perceived skills usage and complete the challenges at the very end of a making activity. The results of the self-assessment and reflection are displayed on the educator’s dashboard, which offers a complete overview of the individual or classroom progress, including information and graphs.

Context and procedures

The study was conducted in six makerspaces located in three urban and two rural areas in [country name]. Four of these makerspaces (4 different teachers) were found in public schools i.e., in a formal educational setting. In this case, the making projects were held during the Design & Technology and Computer & Informatics classes and during free school hours. Each teacher followed a different activity plan, given the technologies available, the school curriculum, and their schedule (i.e., 1 to 2 h sessions per week). The other two makerspaces in the study operated as sites for extracurricular activities i.e., non-formal education. Activities were planned based on the technologies available and interest of the makerspace teachers. Across the six makerspaces, the activities or projects included programming, digital fabrication, robotics, circuitry, and arts and crafts. Figure 3 presents two examples of activities. Table 1 shows the activity plan followed in all six sites. The pilots and data collection lasted 6–7 months for all participating sides (early November 2021- late May 2022).

Fig. 3
figure 3

Examples of making projects: Students working on a task with LEGO EV3 robots (left), Students learning about geometry using 3D Pens (right)

Full size image
Table 1 Activity Plan for each Teacher/Group
Full size table

Data collection and analysis

The authors converged qualitative data mainly collected from semi-structured interviews with teachers and group interviews with students. The semi-structured interviews with teachers and students aimed to convey the overall essence of their experience with the use of the tool. The interview protocol of the semi-structured interviews is found in Table 2..

Specifically, individual interviews of approximately 45 min were conducted with all six teachers. Five group-interviews of approximately 40 min each were conducted with students representing five out of six sites participating in the pilot (the pandemic situation prevented our focus group with the sixth group). Moreover, researcher observations were used as a triangulation source, i.e., to confirm and enrich findings drawn from the interview data. Specifically, observations by two researchers (authors of the work) were held to gather rich contextual data about the students’ and teachers’ experiences with the tool. In total, three 60-to-80-minute observations were carried out by the researchers, in two in-school (formal education) makerspaces and one non-formal makerspace. Lastly, tool log files (e.g., student work in the digital tool) were used to draw data about students’ self-assessment responses and served as a second triangulation source.

Table 2. Semi-structured interview questions (teachers and students)
Full size table

Findings

The authors employed an inductive approach to qualitative data analysis aiming to discover themes and patterns related to the research questions (Patton, 2002). First, all textual data from the interviews and researcher-observations were transcribed verbatim and imported to NVivo software. Then, open coding was used to identify extracts and text segments of significance relevant to the research questions of the study i.e., tool supporting (or not) the assessment of the skills through self-refection, tool encouraging (or not) self-assessment and refection in the makerspace context, and tool presenting opportunities or challenges in this process. Open coding was an iterative process in which text segments were labeled with descriptive codes, which then formed categories, and were, lastly, organized into themes. The NVIVO software allowed researchers to handle the coding process of multiple data sets (textual data from interviews with teacher and students and researcher observations) aiming for triangulation evidence across datasets. Then findings from the textual data were further triangulated with evidence in tool log files, namely student uploaded pictures and artifacts along with textual responses to the challenges. To increase the credibility of the findings, three researchers (authors of the work) were involved in the analysis, cross-checking the consistency of the information extracted and triangulated. Four themes emerged from the analysis, driven by the research question of the study and the identified issues in the literature which motivated this work.

Theme 1: awareness of 21st -century skills development

Theme 1 concerned the perceived usefulness of the tool in advancing the students’ conceptual understanding and recognition of 21st -century skills. The reports under this theme were mostly positive although some concerns were raised too (see summary in Table 3.).

Both teachers and students explicitly referred to the conceptualisations of skills in the tool (e.g., conceptual and operational definitions) which helped the students learn what these skills are and be able to identify them during making.

“The fact that there are explanations, comments for each skill, and points and footnotes on what each skill includes is quite helpful … students learnt what the skills are, and how to detect them when they are occurring. Because the important thing is, first, to know each skill, then identify it and describe it in the way they answer and in the example they choose to upload” (Teacher, Group 6).

Teachers reported that the tool increased students’ awareness of 21st-century skills, helped them identify such skills, and provided evidence of their development during making. The students also thought that the tool helped them gain understanding of the skills they were developing and how they were progressing with these skills. For instance, one student explained:

“The tool helped us to understand the skills we needed to develop more. For example, in the beginning, we felt that our collaboration skills were not so good because we had not worked together as a team before. But as the days went by and we did this work, we found ourselves working together better and seeing our progress in the tool helped us know we were doing it right.” (Students, Group 1).

Also, the teachers positively perceived the challenges triggered by tagging skills, arguing that “the questions urged students to think about how a 21st -century skill helped them progress with the making activities.” (Teacher, Group 2). Overall, the teachers were optimist about the use of the tool, seeing it as a great starting point in understanding and raising awareness of 21st-century skills, e.g.,

“Along the way, it seems that they learnt to identify some skills; what they mean and how to recognize them. I can’t say that they learned to identify all of the skills, and all their characteristics, but I could see improvement, and I think that the more they use it [the tool], the more they will learn.” (Teacher, Group 6).

Per tool log-files, Fig. 4 illustrates an example of how a student used the tool to self-reflect on the use of collaboration. In this example, the student completed a challenge by uploading a picture of an example of collaboration, answering a set of questions about what they were doing and how their example demonstrated the skill, also rating how good the selected example was.

Fig. 4
figure 4

Example of a student’s self-reflection per tagging of collaboration

Full size image

Despite the positive feedback, raising awareness of using 21st-century skills was perceived as challenging at the same time. Both students and teachers reported difficulties in understanding the definitions of some skills. For example, it was perceived that the level of language used to explain the skills was higher than the students’ level of understanding. In fact, the teachers reported that students were often concerned about the meaning of the skills and were asking for explanations and help on how to respond to the self-assessment questions:

“The vocabulary was difficult; we were trying to simplify the terms so that they [students] could understand the skills…the language was above their level of language ability… Some terms were unclear to them; there were students who kept asking: What is communication? Is this collaboration or communication?“ (Teacher, Group 2).

It is worth mentioning that teachers referred to the limited options offered by the tool regarding the selection of 21st -century skills and recommended that the tool is expanded with more skills to allow the selection from a broader skills repertoire based on the goals and the activities of the making sessions.

Table 3. Theme 1 summary - Awareness of 21st-century skills development
Full size table

Theme 2: Engagement in self-assessment and reflection

Theme 2 concerned the perceived usefulness of the tool in engaging students in self-assessment and reflection. The reports under this theme were mostly positive although some concerns were raised too (see summary in Table 4.).

On the positive side of feedback, the self-assessment process via tagged skills followed by challenges was valued by both students and teachers who agreed that the process allowed students to think about what they were doing, what they wanted to learn, and what would be the final learning outcome. In the example below, the teacher explains that the tool helped students to engage in thinking processes and enabled them to realize what they were doing and to share their thoughts:

“…. Putting what they were thinking into writing was extra processing for the students… to realize what they did and why this was not 3 and neither 4, but 3.5.” (Teacher, Group 3).

Teachers also elaborated that the self-assessment process enabled the students to understand their skills progression and to approach the overall making session in terms of skills development. On a similar note, the students found it interesting that they could rate their level of skills development. For example, one student commented:

“We could rate to what extent we made use of certain skills, for example, in collaboration and creativity, and to see what level we made it to; it was really interesting seeing it.” (Student, Group 1).

Using the tool seemed to be an entirely new learning experience for the students. The self-assessment process, as opposed to a teacher-led assessment was positively perceived and was commented as an alternative way of expressing themselves through uploading of examples, answering questions, and rating how well their examples demonstrated the skills (see Fig. 5). Along these lines, the researcher observed that the use of the tool was engaging and that the students became better and better at using the tool as time passed by e.g.,

“Students’ wanted to engage. They were impressed with their own results, and this created a positive learning environment overall. The more they used it, the better they because at identifying the skills, tagging, and reflecting.” (Observation, Group 2).

On the negative side of feedback, teachers reported concerns about the developmental appropriateness of this kind of self-assessment approach, calling on the lack of students’ understanding of the process of self-reflection and students’ difficulties in self-expression. As explained, some students perceived the challenges as a questionnaire they had to complete, they didn’t know how to self-reflect, or found it hard to express themselves and put their thoughts in written form. That said, per teachers’ reports, some student reflections were superficial and without justifications for the skills development. Indeed, per researcher’s observations,

“Students couldn’t understand what they had to fill in, for example, how they solved a problem, to what extent they collaborated or how innovative their idea was … they rated [their skills] without really thinking about it; they gave random ratings in some cases.” (Observation, Group 5).

Furthermore, the teachers reported that the context of makerspaces and the nature of making activities didn’t favour the self-assessment process. They argued that the making activities drew the students’ attention, overshadowing self-assessment, which sometimes was perceived as an unnecessary interruption. The students’ reports confirmed that they often didn’t want to stop their making activities to self-reflect and they were so immersed in the experience that they forgot about self-assessment and the tool, e.g., “Sometimes I felt like I was getting distracted from my work” (Student, Group 3).

Table 4. Theme 2 summary - Engagement in self-assessment and reflection
Full size table

Theme 3: opportunities and challenges for teaching and learning

Theme 3 (see summary in Table 5.) concerned the perceived effects of the tool in the overall teaching and learning process. The reports under this theme were mostly on the positive direction, with some concerns being raised.

On the positive side, teachers reported that the tool placed students at the centre of the learning process and acknowledged that their role shifted from being the source of knowledge to becoming the moderator of knowledge. Moreover, the teachers argued that the tool was educational for themselves too, allowing them to better realize what 21st-century skills entail and how they could be developed in makerspaces. For example, a teacher commented that the use of the tool made her reflect on her teaching practice to deliberately facilitate the development of skills:

“I had to think about some issues regarding the skills and what activities or processes help students develop a certain skill. […] Certainly it [the tool] forces the teacher to think how they can help the students develop these skills. Therefore, it helps in changing or improving the lesson.” (Teacher, Group 4).

Τhe tool was further commented as beneficial in helping the teachers notice who is falling behind and adjust their practices. In teachers’ view, the tool was useful in generating self-assessment results for each student and for the whole class, making it easy and straightforward to see who and on what skills is progressing and who is falling behind, therefore adjust the teaching process accordingly. Indeed, the educator’s dashboard provided a comprehensive view of skills development e.g., “I was able to view the progress in skills development and examine the evaluation students were doing for themselves.” (Teacher, Group 2). Figure 5 illustrates the educators’ dashboard, showing for a selected group, which skills are mostly tagged across making sessions (top), and an overview of the skills tagged (bottom).

Fig. 5
figure 5

Educators’ dashboard – Summaries and reports on tagged skills

Full size image

The teachers further argued that tool forced them to follow a more goal-oriented teaching approach directly targeting the development of 21st-century skills during the making activities. Per teacher input, this practice was extended to student-oriented goals, as the students chose to peruse their development of skills and select examples to share in responding to challenges. The teachers noted students’ gains in confidence while doing so, e.g.,

“I saw that students became more confident with various initiatives and roles during making, which was also evident in how they answered to the questions and how they chose the picture they uploaded in the challenges” (Teacher, Group 1).

On the negative side, the tool was perceived to increase the workload of the teachers in course planning and to interrupt their overall activity, especially for those who used the parallel approach (i.e., reflecting while engaging in making) e.g.,

“We had time issues; it takes a while to use the tool. In an 80-minute lesson, we must leave some time for the challenges …students must read them carefully and not rush.” (Teacher, Group 6).

In general, teachers perceived the tool as a complete and usable and made explicit references to gamification. Yet, they were also interested in having a more customizable solution, e.g.,

“I think that it’s a complete and usable tool. It has a nice gamification approach for students to how they progress and earn badges, keeping the students active and engaged on task…And the connectivity between the educator’s dashboard and the student’s dashboard is direct and explanatory, so that the teachers remain engaged too, following their student’ progress.” (Teacher, Group 3).

“My first recommendation is to make the levels shorter, maybe to create my own levels as a teacher… fewer questions for each level, perhaps in different formats, so that it’s not draining for students (Teacher, Group 1).

Some challenges were reported about the use of the tool on a tablet (not reported in the case of using personal computer). Namely, a teacher reported that typing on a tablet was challenging for her young students (elementary school age). Another teacher referred to the difficulty of working simultaneously on two browser tabs on a tablet, namely in her case Assessmake21 was used together with Lego WeDo software needed for one of the activities, e.g.,

“Students were using the Lego WeDo software on the tablet and tended to forget about the self-assessment tool which was open in another tab on the same tablet.” (Observation, Group 2).

Table 5. Opportunities and challenges for teaching and learning
Full size table

Discussion

In this study, we sought to investigate the application of a novel digital self-assessment tool for 21st-century skills. The tool was designed for use in makerspace contexts to address the challenges stemming from (a) the complex nature of 21st century skills which cannot be easily assessed, and (b) students’ lack of engagement in self-assessment and reflection in these contexts. The overarching research question of the study was “how do teachers and students experience the use of a novel digital self-assessment tool of 21st-century skills in makerspace contexts,” and was approached with a particular focus on the abovementioned challenges.

The self-assessment process via tagging skills and taking challenges was positively perceived by the participants; it helped the students realize what they were doing, what they wanted to learn, what could be the outcomes and understand their skills progression. Also, the tool seemed to encourage students to take initiative, being responsible for selecting skills and completing challenges, and they gained confidence in doing this. Yet, situating self-assessment in makerspace contexts seemed to pose some challenges. It was reported that making activities overshadowed self-assessment and reflection. Specifically, working on making activities was far more interesting for the students than using a digital tool for reflection. This concern corroborates previous research findings demonstrating lack of motivation and interest in reflection as barriers to conducting self-assessment (Peppler et al. 2017; Baykal et al. 2021). As Bieraugel and Neill (2017) also found, makerspaces are rated low in reflecting as they are places with noise and movement distractions. On the other hand, the gamification elements incorporated in the tool seemed to be beneficial in this aspect, keeping students active and engaged, consistent will prior work reporting the benefits of gamification and gameful design (e.g., Ioannou, 2018). In conclusion, addressing students’ lack of engagement in self-assessment and reflection in makerspace contexts was one of the aims of this work. It was partially achieved by offering a positively endorsed self-assessment and reflection experience, despite the reported difficulties replicating previous findings. Future studies should continue to explore how to promote engagement in self-assessment and reflection in makerspace contexts.

Τhe use of the tool for self-assessment and reflection was valued by both teachers and students. The tool offered a new experience in assessing skills development (student-led instead of teacher-led) and an alternative way of students expressing themselves. However, in some cases the assessment was hindered by the lack of students’ knowledge and skills, namely lack of understanding of the process of self- reflection and difficulties in putting their thoughts into writing. These findings align with Peppler et al. (2017) who found that one of the barriers regarding portfolio assessment in makerspaces is having the requisite skills for capturing and self-reflection. The study suggests that the process of self-reflection and respective writing skill could be supported through more careful teacher-scaffolding as well as time on task i.e., allowing more time for the skills to grow. In essence, there was sufficient evidence in the data that the tool was a good starting point, and that the students became better and better in identifying the skills, tagging, and reflecting as they kept using it over time. Additionally, some step-by-step guidance and examples on how to approach the reflection process, could be taught explicitly as a process, perhaps before the introduction of the digital self-assessment tool. Similarly, Hughes and Thomson (2022) argued that students need to be taught how to reflect on their own performance in making contexts.

Moreover, the 21st -century skills conceptualisations incorporated in the tool allowed students to learn what these skills are and be able to identify them during their making practice. However, the vocabulary in the tool was difficult for some 12-year-old students. This finding is consistent with Bieraugel and Neill (2017) who reported challenges in the language used to present the complex knowledge that was being assessed with a self-assessment instrument. It also resonates with findings from an earlier study of SkillsTrack digital tool (Kipp et al. 2018) showing that the language used to describe the 21st century skills was too sophisticated for some students. In this sense, the study recognizes the need for introducing tools that are more developmentally appropriate for younger students, by simplifying the language used.

There was a general agreement that the use of the tool was beneficial not only to the students, but for the teachers too. The tool helped the teachers realize what 21st-century skills entail and how they could be developed in makerspaces. It also helped teachers to adjust their instruction by reflecting on their students’ progress, available through the teachers’ dashboard. Yet, the teachers’ workload in course planning was increased and the use of the tool required additional instructional time during making. Some difficulties with using tables were also reported. In line with direct suggestions presented by the teachers, the authors recognize the need for a more customizable solution where the teachers could reduce or modify questions and challenges considering their students’ progress and time availability. They should also be able to choose from a broader repertoire of skills. Limitations related to the use of some devices (in this case, tablets) should be considered when introducing an online tool in makerspace contexts.

Closing, in terms of study limitations, the participants of the study, namely teachers, were selected based on their interest in assessment in makerspaces. Thus, they may have had increased motivation to work on achieving positive results about the use of the digital self-assessment tool. The work should extend to include more teachers, perhaps less enthusiastic with assessment and makerspaces. Moreover, the representation of students from different age groups was unbalanced, with many more participants in primary school age than in secondary school age (see Table 1). We suggest that more studies are needed with older students to illuminate how this target population responds to the use of a digital self-assessment tool for 21st-century skills. It is possible that older students may not experience difficulties in grasping the meaning of 21st-century skills. We indeed traced the vocabulary issue only in the reports of 12-yrs old students and teachers, but we cannot justify the hypothesis of older students not experiencing difficulties, based on data from a single group of this age group. Concerns on subjectivity and reliability that often surround qualitative methods, apply in our work alike. These can be addressed with replication of work and evidence of transfer of the results in similar circumstances and context. The tool is available for use [blinded for review] and replication of findings is encouraged. On the positive side, however, this work took place in authentic educational settings, which brings high ecological validity to this study. Promoting student self-assessment and learning reflection for makerspace tasks is an important area for research and development with implications for future work in STEAM learning as well as future work on the mechanics and designs of digital self-assessment tools.

Conclusion

Through this exploratory case study of the use of a digital self-assessment tool of 21st-century skills in makerspaces, we can draw rich insights to guide future studies on the topic. Based on our findings, we encourage future studies to consider how a self-assessment tool for 21st-century skills could appeal to students and increase their motivation to participate in self-assessment and reflection. Research efforts should concentrate on making the conceptually difficult 21st-century skills concepts more understandable, especially to younger learners. Also, the process of self-reflection should be developed either via scaffolded and extended use of tools such as Assessmake21 or via direct instruction with examples together with or outside the use of a digital tool. Last, gamification of the tool and the experience as done in this work (e.g., progression in levels, badges) should be encouraged in future studies with similar goals. The results of the study provided rich insights to guide future research on the topic. These findings should be transferable to similar contexts and settings, while replication is encouraged.