Introduction

Makerspaces are mainly found in non-formal environments around the world (e.g. Halverson & Sheridan, 2014; Lindstrom et al., 2017). However, these have also started to be integrated in school settings (e.g. Vourikari et al., 2019). From a European perspective, it is argued that makerspaces are being utilised across Europe, and making activities are seen as conducive environments for promoting European Key Competences for Lifelong Learning, e.g. digital competence; competence in science, technology, engineering, and mathematics (STEM); entrepreneurship; analytical thinking; problem-solving; creativity, etcetera (Vourikari et al., 2019). It is argued that these kinds of skills (often referred to as twenty-first century skills) are developed during makerspace activities (e.g. Sheffield et al., 2017; Sheridan et al., 2014). How to assess the acquisition of twenty-first century skills by involvement in makerspace activities is a question being asked and emphasised by educators and policymakers (Vourikari et al., 2019). As the ability to assess is linked to evidence to inform policy making regarding integration of makerspaces and maker movement into formal education systems across Europe, scientific evidence is needed (ibid.).

However, it is challenging to assess development of skills such as these; there is, for instance, a lack of tools to assess development of creativity that is related to school contexts (e.g. Sawyer, 2003). In this study, we focus on how a recently developed digital self-assessment tool (DSAT) can be used to identify development of some twenty-first century skills when students have participated in makerspace activities. The tool has been developed within a European project [Anonymous during the review process]. In the study we are investigating a pilot test and implementation of DSAT. Since the paper is presenting the first implementation of DSAT we are interested in both how the tool is experienced by the users from a design perspective and also if use of the tool is supporting development of understanding some twenty-first century skills. A rich description of the development of the tool is presented by (REF Anonymous during the review process). Within the project, it was decided to focus on self-assessment of collaboration, creativity, problem-solving, life/social skills, and communication. The definition of these specific skills will be presented further on in this paper. Based on the above-mentioned reported interest in trying to assess outcomes of makerspace activities, in terms of development of twenty-first century skills, we pose the following research questions:

  1. 1.

    How do students and teachers respond to the design of the digital self-assessment tool?

  2. 2.

    How do students and teachers respond to use of a digital self-assessment tool in terms of understanding and identifying development of twenty-first century skills during makerspace activities in formal education settings?

Literature review

Integration of makerspace activities and maker culture in formal education

Makerspace activities were first presented as activities that take place in non-formal learning environments where the activities are voluntary and makers are free to make whatever they want (e.g. Sheridan et al., 2014). Typical features from makerspaces include activities being open-ended, collaborative, and experimental (Godhe et al., 2019; Nemorin & Selwyn, 2017; Sheridan et al., 2014). Maker activities are built on the learning theory of constructionism where learning is driven by learners’ own interest and open-ended discovery (Harel & Papert, 1991), in contrast to formal education, where there are intended learning objectives to be achieved. In addition, making is based on design-processes, where makers explore materials and processes, the doing being iterative. Hence, learning is considered as cyclic processes (Brahms & Crowley, 2016). Integrating makerspace activities into schools is not a matter of only copying and pasting what is going on in informal makerspaces into school settings. There are several challenges that need to be considered since school activities are directed by curricular goals, assessment, and organisational priorities (e.g. Godhe et al., 2019; Walan & Gericke, 2022). A key point in implementing makerspace activities in schools is that teachers relate the activities to learning outcomes and curricular goals. Consequently, being able to assess students’ performances becomes a challenge because of the absence of established measures to assess students’ learning during making processes (ibid.).

Still, there is a growing interest in integrating makerspaces into formal education (e.g. Oliver, 2016). One reason for this is that makerspace activities can fill gaps that exist in schools, to design and solve problems, as well as promote development of twenty-first century skills (Samuelsson Wardrip & Brahms, 2016). So far, there are few examples of actual integration of makerspace activities in schools (e.g. Godhe et al., 2019; Vongkulluksn et al., 2018). The ones that are reported often present positive outcomes (Vongkulluksn et al., 2018), but Godhe et al. (2019) argue that the reports present ideal conditions and not typical classrooms. Based on the fact that school activities connect to learning outcomes, a question is if one way of enhancing integration of makerspace activities into schools could be to make it possible to assess outcomes of makerspace activities?

Outcomes from makerspace activities—twenty-first century skills

In the European Framework of Key Competences for Lifelong Learning, skills that are deemed as important for education and future work include, for instance, creativity, problem-solving, collaboration, critical thinking and digital literacy (European Commission, 2019). These skills are often referred to as twenty-first century skills (e.g. Bell, 2010; Davies et al., 2011; Jang, 2016). As already mentioned, it has been argued that participation in makerspace activities can contribute to development of twenty-first century skills (e.g. Sheffield et al., 2017; Sheridan et al., 2014; Vuorikari et al., 2019). We are well aware that the term twenty-first century skills is somewhat anachronistic given as we are now in the third decade of this century. However, since this term is used in the literature, we will use the term throughout this paper.

There are examples of studies (e.g. Coşkun & Deniz, 2021; Novak & Wisdom, 2019) where they have studied outcomes of makerspace related activities, such as use of 3D printing. In these studies, it was reported that students developed twenty-first century skills (creativity, problem-solving, communication, and collaboration). In the study by Coşkun and Deniz (2021), students reported that working with 3D printing was also connected to learning technological information and design.

The focus of our study is the five skills that are included in the DSAT: collaboration, creativity, problem-solving, life/social skills, and communication. The tool also includes definitions of the skills (see "Appendix 1") since definitions can vary in different sources.

Here, we only give some examples of how the five skills that are used in the DSAT are described in literature in brief, since knowledge and research about the skills are fields in their own and can be explained and discussed to a much wider extent than in this study.

Collaboration, or collaborative learning, is explained as an active form of learning with two or more individuals working together in a physical or virtual environment (Kirschner et al., 2018). Marinez-Moyano (2006) defines collaboration as ‘the process of two or more people, entities or organisations working together to complete a task or achieve a goal’. Another example of the definition is that ‘Collaborative learning is an educational approach to teaching and learning that involves groups of learners working together to solve a problem, complete a task, or create a product’ (Laal & Laal, 2012). The definitions are quite similar, however, with some different wording.

Creativity is one of the skills often referred to when makerspace activities are discussed (Maslyk, 2016). Sawyer (2003) and Craft (2003) argue that the concept is difficult to define and assess. However, Sawyer (2003) summarises that creativity is a process over time, resulting in products that are novel and that these can emerge from elements that already exist, but are combined in new ways. Another example is Amabile (1996) who discusses creativity as a method of problem-solving, building on thoughts, or products. Moreover, Lucas et al. (2013) state that creativity is complex and multifaceted, learnable, possible to analyse at an individual level, and strongly influenced by context and social factors. Gauntlett et al. (2010) argue that in order to develop learners as creative thinkers, opportunities to create and demand for stimulating learning environments, engaging tools, and materials are needed.

Jonassen and Hung (2012) define problem-solving as the process of constructing and applying mental representations of problems to finding solutions to those problems that are encountered in nearly every context. Problems that are ill-defined and ill-structured are argued to develop a set of twenty-first century skills, including creativity and critical thinking (Lai & Viering, 2012). During the learning process of an ill-structured problem, students often get involved in challenges where the information that is required to solve the problem is not complete in the problem statement. Hence, they need to select from several solutions or pathways to follow. In this way, the students need to consider various ideas. Such learning situations recall dialectical aspects of both conventional and innovative analysing, where students engage in productive discourse, which ultimately deepens knowledge co-construction and creation of solutions (Tan et al., 2014).

Life/social skills are not that easy to define. In the Partnership for twenty-first century skills report (2009), it is argued that in today’s society, people need not only content and thinking skills. It is also a matter of knowing how to navigate through personal and working life. It is challenging to try and include all possible life/social skills that are in this category. For this reason, the skills that are mentioned in this sector are specifically the ones that can be developed in the maker learning contexts and are a combination of a number of frameworks (e.g. Partnership for 21st Century Skills, 2009; Kipp et al., 2018; Scoular et al., 2020). People who have developed these life/social skills share the following attributes: take an initiative and have self-direction, are flexible and adaptable, are productive, and have a sense of responsibility. Finally, they often take responsibility and inspire others to reach their best.

Communication skills are considered as important twenty-first century skills (e.g. European Commission, 2019; Lai & Viering, 2012; Partnership for 21st Century Skills, 2009; Vourikari et al., 2019). Communication includes several aspects, where one example of a definition, presented by Thompson (2020), is:

Operationalizing communication as a success skill which can be measured requires synthesising the interrelationships among reading, writing, speaking, and listening skills, the role of technology, and verbal and nonverbal proficiencies in various situations and cultures. (p. 11).

Lai and Vering (2012) argue that communication is included in interpersonal skills, along with social skills, teamwork, cultural sensitivity, etcetera. Hence, elements of communication may intertwine with elements of other skills. People who have good communication skills share the following attributes: they can express opinions, discuss, reason, speculate, argue and engage in debates. Moreover, they can listen attentively and communicate in diverse environments, and they are good at presentations using multiple media and technologies as well as inspiring and enthusiastic (ibid.).

Assessment of outcomes from makerspace activities—why and how?

Assessment of making activities in school situations has proven to be a challenge (Lin et al., 2020). Depending on which activities the children or students choose, outcomes differ with great variety. A literature review performed by Lin et al. (2020) shows that there are largely three areas that are assessed: understanding and construction of a content, feelings and attitudes that are nurtured in making activities, as well as commitment and collaboration. The content that is assessed is about different programming skills, with a focus on concepts and computer logical thinking, but also creativity and problem-solving abilities. In the development of computer logical thinking, formative assessment can guide students' understanding (Hadad et al., 2020). Maltese et al. (2018) also argued that formative assessment and feedback practices are important to increase students' commitment and learning in maker activities. In connection with feedback, the view of failure is also important. With a permissive environment where failures are taken advantage of in the makerspace activity, there is a potential to create knowledge and learning. Maker activities have this process built in and students are given the opportunity to train, among other things, their endurance (ibid.). Summative assessment, on the other hand, goes against the maker culture and the basic ideas of making (Hadad et al., 2020).

A precursor (SkillTrack) to the DSAT tested in our study has previously been used by students and teachers. In that tool, the idea was that it could be used in any activity, not only specifically in makerspace activities, but rather in different school subjects. The idea with that tool was to offer a formative assessment experience that would provide students with feedback, motivation, and a sense of progression in development and understanding of five skills: collaboration, communication, creativity, managing information, and managing themselves. It was found that the tool worked as an eye-opener to identify these twenty-first century skills, and that feedback provided as badges given to the students in the tool stimulated them. However, there were also some drawbacks with the tool, such as comments from both teachers and students that the language in the tool was too complicated and that students needed guidance from teachers when using it. One of the tasks in the tool was that students were supposed to upload pictures as exemplar evidence and self-assessment of a chosen skill. This task was appreciated by the students, who argued that this made them feel a sense of achievement (Kipp et al., 2018).

During the pandemic with COVID-19, new situations have arisen with teaching being moved from occurring in the same physical learning environments to being held online. As a consequence of students not having access to the same equipment in their activities and teachers not being able to actually see students during the activities, teachers needed to change the way they assessed students’ work (Samuelsson Wardrip et al., 2021). In their study, Samuelsson Wardrip and colleagues focused on maker-based learning with students making remotely. They investigated how teachers could monitor students’ progress, offer timely feedback, or infer what the students understood. It was shown that as a way to enhance assessment, students uploaded pictures and/or videos, showing how they progressed during the activities. The structural documentation and feedback gave each student a better opportunity to explain their designs and their development process. Teachers’ assessment became more oriented towards the students’ development of different skills, such as problem-solving and perseverance compared to earlier. This is interesting from several points of view; it is about assessing outcomes of activities typical in makerspace environments taking place with school students, and it is a new approach on how to assess. In addition, the results in the study showed that teachers changed what they assessed, focusing more on development of skills that are often referred to as twenty-first century skills.

The connection between makerspace activities and Swedish education

The curriculum for Swedish compulsory school (Swedish National Agency for Education, 2018) states that school must prepare students for active, creative, responsible participation in society. Education must also promote a lifelong desire to learn in all students. Learning to learn is a necessity when the development of society is constantly changing. In other words, students need to develop several types of skills, both digital and entrepreneurial, to be able to participate in this society. The school's responsibility is therefore to stimulate and promote interaction between students and create opportunities for communication. Furthermore, the school must also ensure that students can work with exploratory work that also stimulates creativity and curiosity. Students should be able to solve problems and turn ideas into action, both independently and together with others. Students should also have the opportunity to use digital tools for this creation and to communicate (Swedish National Agency for Education, 2018). All of these mentioned skills are, as previously described, considered as twenty-first century skills (e.g. Bell, 2010; Davies et al., 2011; Jang, 2016).

In Sweden, each subject has its own syllabus with different abilities to be developed, as well as different central content to be addressed. The central contents are divided into three different stages: grades 1–3, grades 4–6, and finally, for grades 7–9. This means that the teacher has the freedom to organise the teaching and decide when different central content is to be addressed within the three years in question. For this study, the central content for grades 7–9 (students at age 13–15) is in focus, and the classes that participate in the makerspace activities and try the DSAT are from these grades.

Makerspace activities are often related to science, technology, engineering, and mathematics (STEM) and the integration of these subjects (e.g. Kurti et al., 2014; Sheridan et al., 2014; Vossoughi et al., 2016; Vuorikari et al., 2019). However, from a Swedish perspective and connection to different school subjects, it is easy to connect makerspace activities with the subject technology.

If we take a closer look at the abilities and central content from the technology subject's syllabus, which are affected by the makerspace activities, we find that students should develop their technical knowledge and technical awareness, so that they can act in a technology-intensive world (Swedish National Agency for Education, 2018). Through teaching, students must solve problems and develop their own technical ideas and solutions that meet different needs (ibid.). Moreover, the central content includes:

  • How mechanical and digital technology interact.

  • Technical solutions that use electronics and how they can be programmed.

  • The different phases of technology development work: identification of needs, research, proposals for solutions, design, and testing. How the phases of the work process interact.

  • Own constructions where control and regulation are applied, among other things, with the help of programming.

  • How digital tools can be a support in technology development work, for example, to make drawings and simulations.

  • Documentation in the form of manual and digital sketches and drawings with explanatory words and concepts, symbols and dimensions, as well as documentation with physical and digital models.

We argue that this central content is typically also featured in makerspace activities. Swedish teachers teach programming, for instance, by using equipment, such as Micro:bits, Lego robotics in their teaching, equipment that often is found in makerspaces (e.g. Halverson & Sheridan, 2014; Oliver, 2016; Tan, 2019). Furthermore, we consider the central content referred to as different phases of technology development work as being similar to design processes that are typical in makerspace activities (e.g. Sheridan et al., 2014).

However, there are no learning outcomes that are explicit in terms of assessing twenty-first century skills, such as creativity, collaboration, etcetera, in the syllabus. Hence, teachers lack experience and motives from this perspective. Noteworthy, in makerspace activities in non-formal learning environments, there are no formal assessments of learning (Blikstein & Worsley, 2016). In this study, teachers test together with students how the use of a DSAT, used in connection with makerspace activities and teaching in technology, can be supportive in understanding and developing twenty-first century skills. Hence, the focus is shifted from assessment to self-assessment.

Self-assessment—why and how?

Andrade (2019) has made a systematic review on research articles about self-assessment, and she argues that self-assessment has been used in a variation of activities, such as showing a happy or sad face after a story has been told, estimating the number of correct answers on math tests, identifying strengths and weakness in a certain activity, etcetera. She also presents different researchers’ definitions of the concept, and these definitions have some things in common; it is all about self-assessment of one’s abilities, processes, and products. Hence, a broad conception. Still, according to Andrade (2019), this works because it is subject to influence of feedback from oneself. An important issue raised by Andrade is that there is one important thing missing in all the definitions, namely the purpose of the self-assessment. Andrade herself argues that self-assessment is feedback, stating:

… the purpose of feedback is to inform adjustments to processes and products that deepen learning and enhance performance; hence the purpose of self-assessment is to generate feedback that promotes learning and improvements in performance. This learning-oriented purpose of self-assessment implies that it should be formative: if there is no opportunity for adjustment and correction, self-assessment is almost pointless. (Andrade, 2019, p. 2)

Hattie and Timperly (2007) name and discuss different forms of feedback in a widespread article. Their description is related to learning objectives and feedback provided by others, not self-assessment. However, they also emphasise that feedback in the form of self-assessment is important because it develops and supports the student's self-confidence and insights into how to acquire one or more abilities.

Another example of the use of self-assessment, specifically connected to creativity, is found in the review by Bolden et al. (2020). They concluded that an emerging theme in the literature was the value of self-assessment in supporting student creativity. Furthermore, analysis of the studies reviewed suggested that technology can support student self-assessment, which, in turn, supports creativity. The examples they presented were about the use of 3D printing programmes and this kind of technological equipment. Bolden and colleagues (2020) also found that across the studies they reviewed, teachers were very much involved in supporting the students’ use of the various reflection and self-assessment techniques described and that teachers play an essential role in guiding and supporting students’ self-assessment.

Zimmerman (2002) has presented a model of how students can become self-regulated learners. In his model the term self-evaluation is used instead of self-assessment. Self-assessment and self-evaluation are two terms that seem to be used interchangeably. As already mentioned, self-assessment refers to the process of evaluating one’s own knowledge, skills and abilities in a particular area, with focus on a specific task or performance, and is often used to identify areas for improvement (formative assessment) (Andrade, 2019). Self-evaluation on the other hand, we believe is a broader term that refers to the process of reflecting on one’s overall performance, accomplishments, and goals. It takes into account not only specific tasks of skills, but also personal interest and values, feelings etcetera. (Zimmerman, 2002). Hence, we interpret self-assessment as more task-oriented, while self-evaluation is more holistic and reflective of one’s overall performance and impact.

In this study, we prefer to use the term self-assessment, since focus is on outcomes of specific tasks (makerspace activities) with students not having specific goals of learning objectives presented for them to achieve, but that development of twenty-first century skills could be expected outcomes. Hence, the reflections are task oriented and not as holistic as presented from a self-evaluation perspective according to Zimmerman (2002).

From a Swedish perspective, we find that in the National Curriculum for Compulsory School (Swedish National Agency for Education, 2018), it is written in the overall goals concerning assessment and grading that:

The goals of the school are that each pupil develops the ability to assess their own results and relates these and the assessments of others to their own achievements and circumstances. (p. 16).

Even though the word self-assess is not in the text explicitly, it could be assumed that the meaning of developing the ability to assess their own results is about self-assessment.

Moreover, the Swedish National Agency for Education (2020) makes the following recommendation to teachers:

Instead of getting students to compare themselves with other students, teachers should strive to get students to do their own self-assessments. Research shows that competition between students reduces the degree of motivation of most people, while self-assessment strengthens their responsibility for their own learning. [Translated to English].

Methodology

In this study, we used qualitative methods to collect and analyse data. Individual interviews with three participating teachers and 14 group interviews with in total 65 lower secondary school students. The reason for having many groups of interviews with students was because they worked with different kinds of makerspace activities and during different lengths of time as will be described later on.

Participants

Four teachers (all of them females in the ages of 40–45 years) from two different secondary schools volunteered to participate in this study. They had earlier been involved in a project about makerspaces, and they were especially interested in issues concerning assessment of makerspace activities and integration of these in their teaching. The teachers are teaching science, technology, and mathematics. They involved six classes in total in this study and the students from these classes that had accepted to participate and were allowed to do so by their parents. The classes ranged from grades seven to nine, which means that the students are 13–15 years old. From the six classes there were 14 groups of students participating in interviews, comprising 65 students in total. Three of the teachers participated in the individual interviews. The fourth teacher was not able to participate because she had to take a long time off from work. When referring to quotes from the teachers in the result section, we call them teacher 1–3 (T1–3, not including the fourth teacher who did not participate in the interview). When referring to student comments in the group interviews, we refer to the group, not individual students and quotations; instead, we present typical comments that arise from the groups. Hence, quotations will be referred to group G1–14.

Research context and process

First, meetings were held with the teachers to present the project, the DSAT, and discussions were held of what kind of makerspace activities they wanted to do with their classes. Plans were made of when and how different activities were going to be implemented. Thereafter, the teachers had training in how to use the DSAT. Information and consent letters were given to the students and their parents. All of the teachers wanted us researchers to come to their schools and inform the students about the project. The teachers themselves introduced their students to the DSAT, an application they could use in any digital device with internet connection (e.g. smart-phones, computers, Chrome books, or tablets). Each teacher used about 40 min in each class to present the DSAT for the students, providing them with login details and some guidance in the tool.

In the next step, the teachers chose different activities based on their students' pre-knowledge and interest in trying new things. Hence, two of the teachers first wanted to work with programming in Scratch and then continue with activities using Microbits. The third teacher wanted to focus on different activities with the design process in focus, having the students create different kinds of packages for a music festival. The fourth teacher was also interested in working with the design process and allowed her students to work with the design of an online application for smart-phones. After each session (a variation of number of sessions among the teachers), the students and the teachers used the DSAT with different challenges and tasks (found in the tool), aiming to assess what kind of skills they had been able to develop during the makerspace activities. All of the activities so far took place in the classrooms. The final activity was held together with the local makerspace in facilities at a university nearby. The reason for choosing to be at the university was simply because the makerspace localities were too small to gather a whole school class at the same time. For many years, there has also been collaboration between the university and the makerspace, with makerspace activities actually being held at the university.

The final activity was about programming, this time within Roblox. This session was held during a full day (five hours plus one hour of lunch break in the middle of the day) that ended with the students using the DSAT once again. After this final activity, data collections took place. These included group interviews with the students, and individual interviews with the teachers. Table 1 shows an overview of the whole process.

Table 1 Overview of activities in the project
Full size table

One teacher (T2) worked with the activities during a period of 7 weeks, while the other teachers only used about 2 weeks each for the project. In Table 2 an overview of what kind of makerspace activities, duration of working with the activities and the use of DSAT in the different schools is presented.

Table 2 Overview how teachers used DSAT and the student groups involved at each school
Full size table

The digital self-assessment tool

In the study conducted by Coskun and Deniz (2021), they used a self-assessment tool for students to report on their developments after working with 3D printing; their tool was in the form of a pre- and post-test questionnaire. In their study, the focus was on identifying outcomes of the specific use of 3D printers. In our study, we used a DSAT that was created by partners in the European project (Assessmake 21) through iterative design processes with representatives involved from four different countries. The idea with the tool is that it is supposed to be used by students (aged 12–18) to reflect on their development of twenty-first century skills in makerspace contexts, without specifying what kind of makerspace activity or project is taking place. There are five twenty-first century skills included in the tool: collaboration, creativity, problem solving, life/social skills, and communication. The reason for these skills being included in the tool was based on workshops and interviews with teachers, makerspace staff, and researchers during the European project Assessmake 21. These were skills that were argued to be most often identified in the context of makerspaces. The development and design of the DSAT used in this study are beyond the scope of this article, as they are fields of their own. However, a short description of the tool is necessary since it is a foundation of the study. The tool consists of one part for teachers and one for students. First, the teachers need to create classes in the tool with a list of their students. Then, the students get login credentials and get access to the student version.

The students respond to different kinds of challenges in the tool at different levels; after finalising a level for each skill, the students can ask the teacher to receive a digital badge, which confirms that a certain level has been reached by the student. The students respond to different questions that are supposed to reflect if they have understood the meaning of the skills and feel that they have developed these during the makerspace activity. Besides responding to questions, there are also tasks in the tool where the students are asked to take pictures and upload these to verify examples of when they feel they have been creative, collaborating, communicating, etcetera.

The tool has been designed with the purpose of being pleasing graphically, being easy to follow, and with a language easy for teenagers to understand. Furthermore, there are also definitions of each of the five twenty-first century skills that are included in the tool, so users can check their meaning in case they feel uncertain about the definitions. The definitions of the five twenty-first century skills included in the DSAT are found in "Appendix 1".

Finally, the tool is an online tool, making it possible to use with different kinds of devices, such as computers, Chrome books, smart-phones, etcetera. Figure 1 shows some examples from the tool and what it looks like.

Fig. 1
figure 1

Screen shots from the student version of the DSAT, with some examples from the content in the tool

Full size image

Data collection and analysis

Focus group interviews were held with 14 groups of students after the last session with makerspace activities as a means to capture deeper explanations of students' responses on the use of the DSAT. Individual interviews with three of the four teachers were also held after the last session of makerspace activities to capture their responses to the use of DSAT. Interview guides are found in "Appendix 2".

The interviews were analysed through thematic analysis, as described by Braun and Clarke (2006). Hence, the interviews were listened to several times, and the transcripts of the interviews were also repeatedly read, with the aim of making the researchers familiar with the material. During the readings, notes were taken of things that caught the researchers’ attention associated with the research question. The material was then tentatively coded, followed by contrasting, comparing and reviewing the codes against each other. Thereafter, the codes were continuously revised during the process to ensure that no notes were left without a code. The second author conducted the first analysis of the group interviews with the students, while the first author did the first analysis of the interviews with the teachers. The next step was to discuss and compare the results of the thematic codings until a consensus was reached. Since several parts of the coding from the interviews with the students were the same, or similar to the coding from the interviews with the teachers, we decided to merge the coding into common coding maps. The main themes were the same as well as part of some of the subthemes. The coding maps are presented in the result section as Table 3. Quotes from the interviews with students and teachers were originally in Swedish and have been translated into English.

Table 3 Coding map of student and teacher responses connected to the use of the DSAT
Full size table

Results

Coding of the interviews resulted in a focus on two main themes: The use of the DSAT and the twenty-first century skills [included in DSAT]. These themes were separated into subthemes and categories of responses.

The use of the DSAT

The first main theme, The use of the DSAT, was coded as shown in Table 3. We identified three subthemes in how students and teachers responded. Two of these subthemes were only found in the responses from the students, but not from the teachers, namely the purpose of using the tool and suggestions for improvements of the tool. Both students and teachers had many comments on functional issues with the tool.

First, we describe how the students understood the purpose of using the tool. In eleven of the groups we interviewed, the students described that the purpose of the tool was to evaluate and reflect upon their work during makerspace activities, or the results of this work. Here is an example of how students expressed this:

I think in retrospect, to be able to check later what we have done, to learn from it, maybe. (G3).

In another group, it was expressed as:

You assess yourself. Like, how good am I at being creative? It's good to see what stuff you can and what it is you cannot. (G14)

In one of the groups, the students argued that the purpose of the DSAT is that teachers can see how students develop:

The teachers find out how good they think we are and how well we work. (G5).

However, there were also students in nine of the groups who had not understood the purpose of the DSAT at all. Here is an example of how this was expressed:

I don’t understand why you should use it [the DSAT]. (G3)

However, we found that none of the teachers reported this aspect and reflected on the fact that there were students who had not grasped the purpose of using the DSAT.

Secondly, when the students reported more about practical aspects of using the DSAT, we found that there were different functional issues with the tool, most of them problematic. There were issues such as difficult language in the tool, loops of too many similar questions, and necessary to make too many clicks. Moreover, the students found that using the DSAT was time-consuming, and it was problematic to upload pictures. In addition, students reported that they sometimes were kicked out of the tool and no information was saved, and that the DSAT was not user friendly for smart-phones. Here is an example of how these problems were addressed:

It was complicated words…if you don’t read books 24/7 [all 24 h of the day and 7 days a week]…it was difficult to understand. (G12).

This experience was confirmed by all three teachers, with one of them stating:

There are too complicated explanations in the tool of the skills and too much information. It is not user friendly. (T2)

This teacher tried to support her students because they were struggling when they were using the DSAT and searching for the definitions of the skills, so she made posters with all the five skills included in the DSAT and put them on the walls in the classrooms.

In five of the groups, students declared that questions in the DSAT were repeated all over again, like a loop; the questions were too many and too similar, and this made them lose motivation in using the tool. One student commented:

Since it is the same questions that recur in the different skills, you answer almost all the same and you do not think before each new question. You simply repeat. (G8)

In addition, there are statements from five of the student groups that they had to click several times on the same symbol to advance in the programme. It was not perceived intuitively and not as something they were used to doing from other types of digital tools. With the combination of many repetitive questions and several necessary clicks, the DSAT was also perceived as time-consuming, which was also mentioned by another three groups of students. There were also arguments of how using the DSAT took time from working with the makerspace activities and that the students wanted to focus on these. Here are some examples of comments from student groups:

In many apps [digital applications], you can click once to move forward, but in this, you need to click twice or even more times to proceed. (G8). I think it's better to learn what to learn, instead of answering the questions [in the DSAT]. For this work [makerspace activity] requires quite a lot of time, and then you did not quite have time for everything. But imagine that you would have taken the lesson time you answered the questions and worked instead. (G12).

The teachers confirmed that they also experienced that it was annoying for their students that they needed to make so many clicks in the DSAT. On the other hand, the teachers did not comment on the time issue, but rather that students needed a lot of support in how to handle the tool. One teacher commented:

I found that I needed to support and guide the students a lot when they used the tool. Like I told them, you have to choose a skill and take pictures during the making when you believe that you are using this skill. It was not intuitive for them. (T1)

As noticed from the above quote and in the description of the DSAT, a feature of the DSAT is the ability to upload photos. Since the student groups worked with the tool at the end of the lesson, there were students from five of the interviewed groups who talked about how difficult it was to take pictures of something that had already happened and that it was difficult to know what to take pictures of. One of the groups expressed:

Yes, when we have done everything, you should take the pictures. The question is [in the tool], what do you do in the picture? I've already done what I did. That question is really only for while you work. You do what you do. (G13)

It should be noted that not all groups had come so far that they had the opportunity to upload photos. There were also reflections on handling of up-loading pictures and what kind of device to use for the DSAT. One of the teachers commented:

If the students did not upload a picture, they were stuck within an eternal loop of questions. Since they are asked in the tool to take pictures and upload, the best device to use is the smart-phone. The problem is that the tool is not user friendly with a phone. Text and pictures become weird. (T1).

In addition to the categories from the student responses and how their teachers confirmed these, the teachers had comments about the DSAT from their own perspective. The ones that we coded as having problematic functional issues were about not appreciating the feedback system in the DSAT, not finding the tool useful for assessment purposes, and the need to create new classrooms in the tool as soon as you started a new project. Here are some examples of comments:

I don’t like the feedback system with the badges; students are used to being assessed without badges, and I didn’t like that I couldn’t provide students with negative feedback. Sometimes when students requested a badge, I did not think that they deserved it, and there was no way of showing that in the tool or for them to move back. (T3).

I don’t want to use the tool in its current form. It needs to be simpler for the students, and we have so many other digital tools for assessment in school, we don’t need another one. (T1).

So far, the responses to the use of the DSAT have focused on problematic functional issues. However, there were positive comments made as well, albeit they were few. The teachers found it easy to create classrooms in the tool and add their students, and they also identified possibilities for students to create their own portfolio and how this could be useful for them in the future. One of the teachers expressed it like this:

Well, it was easy to create a class in the DSAT, and I can see that there are some possibilities in the tool for students to create their portfolios and this can be useful for them in the future. (T3).

Finally, we identified a third subtheme related to the use of the DSAT, suggestions for improvements in the tool. This was actually not a specific question that was posed to the students, but to the teachers. However, the only ones that did make any comments on this were students. They suggested that it would be helpful with some kind of instructional video, how to use the tool, or some written instructions. This suggestion was presented both in groups 2 and 11. Here is an example from one of the groups:

They could have made a video before, so it was possible to see how to use the tool. (G11).

Students in group 13 also suggested that the tool could include exercises to train the skills that they identified with the help of feedback from the tool as not so developed. In groups 8 and 9, students requested a result from the DSAT where they can read what they may need to think about to develop their skills. A student describes this through a parallel with sports training outside school:

I think with football… the coach says, what he thinks we are good at, what he thinks we are bad at, and then in training we try to do better. It can be the same. We write in the app [DSAT] what went well and stuff. (G8).

In addition, students in three of the groups suggested that the questions in the tool should be more open-ended to stimulate reflection. This is an example of how students commented on the suggestion:

Well, more difficult, no, not difficult, but more questions that make you reflect more deep, sort of…The questions were more of the nature, have you done this, not deep or any reflecting at all. (G9).

Now we have presented examples of how students and teachers have reflected on the design and the use of a DSAT. Students and teachers had similar experiences of the design and the use of the DSAT in many aspects. However, the students were the only ones that reported about what they thought about the purpose of using the tool as well as being the ones suggesting improvements. We did not identify any differences between the groups about their experiences of the design and the use of the tool, no matter how long they had tried the tool. The teachers and the students did not only respond to their experiences of using the tool, but also to what it was supposed to assess, namely, development of twenty-first century skills. To be more specific, five such skills. In the following section, we present results related to this perspective.

The twenty-first century skills

The second main theme, the twenty-first century skills, was coded as shown in Table 3 presented earlier. We identified two subthemes, understanding and development of twenty-first century skills. Most of the categories were covered by both students and teachers. One category was mentioned only by teachers.

First, students found the DSAT supportive to help them understand some of the twenty-first century skills. In two of the groups, students argued that the DSAT made them reflect on the skills:

You are supposed to think and reflect. Was there anything I developed during the lesson and that I felt I became better at doing? Maybe it can provide me with new ideas. (G8).

It [DSAT] makes you reflect. For instance, about how you collaborate with others…so you can improve…if you notice that you never collaborate, you can start doing that. (G14).

Even though students argued that the DSAT supported their understanding, in the sense of starting to reflect on the skills and their own need to develop these skills, other data did not support this. Some comments from the teachers:

Students can develop the skills without the tool. But if the tool is improved, it may help them to be aware of the skills. (T1).

I don’t think that the students really have reflected on the skills and their meaning; they were just doing what they were told and clicked around in the tool. (T3).

In addition, it was shown that students had difficulties understanding, especially the concept of life/social skills, despite the fact that explanations were available in the tool and that some groups had explanations printed on paper. In a direct question, whether the students thought they had been trained to develop planning, completing tasks, and setting goals for themselves, (included in the definition of the life/social skills), the answers varied and were vague, some students mentioned that they had some opportunities to train how to plan their project, while others said that they never had any time to plan. But most of all, the students just seemed confused and did not know what to say when this skill was discussed.

All three teachers meant that students’ understanding of the skills were not clear, even though they had used the DSAT, or the help of posters in the classroom, or as one of the teachers said:

We used one whole lesson only to talk about the skills. It’s important to make students aware of these, so we talked about the definitions. They know the words of the skills, like collaboration, but when you dig deeper, you realise that the students do not always really know. (T3).

The problem with overlap of the definitions and the understanding of some skills was exemplified between the skills collaboration and communication. As stated in one of the student groups:

You collaborate automatically and improve communication automatically when you collaborate. (G9).

In another group, they did not explicitly use the word collaborate when they discussed communication, but collaboration was implicit in their discussion, and they said:

It’s necessary to agree; all are different and want different things, so you need to communicate what you like most or if you put together different ideas, like choice of material, for instance. (G12).

There were also reflections showing that students were thinking about problem-solving in different ways, like from every-day perspectives, or not really understanding what it could be all about in a makerspace activity, or actually identifying problems during the activity, etcetera. In other words, a variation in students’ understanding of what problem-solving could be. Some examples:

Well, a problem can be that when you are a small kid, you cannot open doors, but when you become older, you solve the problem and understand how to do it. (G10).

In another group, they had identified a problem during the Roblox activity:

In Roblox, our things just fell all the time when we were testing the game. So, we needed to find a solution to that problem. (G1).

The teachers argued that it was problematic both for themselves and the students to separate the meanings of problem-solving and creativity.

It’s not easy for the students to separate the meaning of different skills; it’s difficult for me as well, and we spent one whole lesson only to discuss the definitions of the skills. (T2)

The definitions of problem-solving and creativity; they overlap in the tool, as well as in reality. (T3).

We also noticed that both students and teachers found it difficult to capture the skills in pictures. This was mentioned in five of the student groups and expressed, for instance, as:

It’s difficult to know what to upload. (G8).

This was confirmed by the teachers, and one of them simply said:

It is difficult to capture, for instance, collaboration through a picture. (T3).

In the second subtheme, development of twenty-first century skills, we found that both students and all three teachers believe that this is possible in makerspace activities.

From the students’ perspective, they argued that in the tasks that were carried out within the makerspace activities, there were great opportunities for creative solutions. A student in group 8 said that he usually did not perceive himself as creative, but in these activities, he first developed a small thing, which led to more and more things. Creativity is a skill that students link to how the task in the makerspace activity is formulated. In eleven of the interviewed groups, students discussed creativity. Students say that when the task is free and has no rules or boundaries, creativity can be developed. If there are no stated requirements, it is almost only a matter of creativity. One of the student groups described:

But you have to decide exactly what to do. If there are certain robots, you can choose completely, there is a requirement here and there, but otherwise, it's just a matter of saying what to do and doing it. And the same in Roblox, there were no strict demands, only to create a kind of track, as we did. No further requirements. Then it's just a matter of creativity. (G3).

Here is another example declared by one of the teachers:

The students were able to develop all the five 21st century skills in the makerspace activities. (T3).

Students and teachers were also united in the opinion that it is possible to develop twenty-first century skills in other activities, both in other school subjects and outside of school. From the student perspective, this was mentioned in all of the groups except one, and, for instance, expressed as:

You can be creative everywhere. (G4)

One teacher said:

There are several connections to different school subjects and the skills, like math, Spanish, etcetera. It’s possible to develop in many ways. (T2).

However, only the teachers were specific about development of twenty-first century skills, the makerspace activities and connection to learning objectives in the school subject technology:

There are connections between the activities and learning objectives in technology, it’s programming and design-processes; very clearly. (T2).

Finally, an opinion that students and teachers were united about was that these skills are important for the future. Here are some examples of how this was expressed among the students and teachers:

It’s a major part of the future. For instance, at work. There’s collaboration. You can learn to collaborate with anyone. (G14).

It has been an eye-opener for me, that it’s important for students to develop these skills, not only subject knowledge. (T2).

It’s great for the students to start to create a portfolio of their skills. Useful for them in the future. I would like to focus even more on students' development of life/social skills. This is something that they start to develop as teenagers. (T3).

Students and teachers responded similarly to how the makerspace activities and the use of DSAT could support development and understanding about twenty-first century skills. The only difference between teachers and students in this respect was that teachers also related to learning objectives in the curriculum, which the students did not reflect on.

Discussion

Based on the reported interest in finding ways to assess the development of twenty-first century skills (Vourikari et al., 2019), we investigated how a recently developed DSAT was experienced by students and teachers during pilot tests when they used the tool in connection with makerspace activities. However, since it was the first implementation of the DSAT, we also wanted to know how the tool itself was experienced by the users. We posed the following research questions:

  1. 1.

    How do students and teachers respond to the design of the digital self-assessment tool?

  2. 2.

    How do students and teachers respond to the use of a digital self-assessment tool in terms of understanding and identifying development of twenty-first century skills during makerspace activities in formal education settings?

Understanding the purpose of using a DSAT

First of all, we found that not all students had understood the purpose of using the DSAT. This was not even something that we had reflected on before the pilot tests as something that could be problematic. There can be several explanations for this. Perhaps the researchers and teachers did not explain this well enough during the introductions of the tool. On the other hand, the tool itself was probably not intuitive enough for students to understand. Another explanation can be that the students were not used to doing self-assessment activities. As we reported earlier, we do not know if and how much the students are used to self-assessments. In addition, many of the students only used the tool for a short period of 2 weeks. However, one of the classes used the tool for 7 weeks. In addition, the students worked with makerspace activities that were also new to them, so there was a lot of “news” for them at the same time. We cannot say for sure what actually made some students not understand the purpose of using the DSAT. Nonetheless, there were students who did understand that the purpose was for them to assess their own development of some twenty-first century skills. Some students also argued that the tool was supporting their teachers in their assessment of the students’ work; hence, a focus on assessment, not self-assessment.

Designing a DSAT is a complicated task

Second, it seems difficult to create a DSAT, as both the students and teachers in the study had several critical comments on functional issues related to how the tool was working, and not working. Comparing our results with the outcomes of using the precursor tool, SkillTrack, we notice some similarities and some differences. In their report, Kipp and colleagues (2018) presented that the tool had very complicated language, and teachers needed to support their students when using the tool. Hence, the same experiences as with the DSAT used in our study. On the other hand, students in their study seemed to appreciate both the system of rewards with badges and the uploading of pictures, but this was not as positive in our study. Not only was the aspect of uploading pictures complicated from a technical point of view, but the students in our study also found it difficult to capture development of twenty-first century skills through pictures. The idea of uploading pictures to capture development or processes was also used by Samuelsson Wardrip et al. (2021) with positive responses. Thus, the less positive experience of this in our study, can possibly be explained by the technical problems of doing this and making this more of a drawback. In addition, more technical problems are reported in our study, so developers of DSATs need to do a lot of work before tools like this can be used by students. When it comes to language issues, for instance, we interpret this as even though the translation from English to Swedish may not be perfect, sentences and explanations in the tool are too long and complicated. The students in our study also suggested that the number of questions in the tool could be fewer, but deeper, to increase opportunities for reflection. Furthermore, the students had suggestions for developers of DSATs to make videos showing how to use the tool. We also argue that if students need support and explanations to understand a skill, a teacher can adapt and individualise their explanations, which a tool cannot. Hence, no matter how great the DSAT is, there will probably always be a need for teachers to support at least some students.

Outcomes of using the DSAT and participating in the makerspace activities

Third, even though it seems there were many negative responses to the practical use of the DSAT, there were positive outcomes as well. The most important one being that both teachers and students became aware of twenty-first century skills and that the use of the DSAT supported this. This is in line with the outcomes of the use of Skilltrack (Kipp et al., 2018). In addition, participation in the makerspace activities in themselves was reported as supporting development of twenty-first century skills, as found in several studies (e.g. Sheffield et al., 2017; Sheridan et al., 2014; Vuorikari et al., 2019). The students and teachers in our study also argued that they believe these skills are important for the future. Being aware of twenty-first century skills and also believing that these are important and can be developed during makerspace activities is one thing, but really understanding the meaning of the skills is more complicated. Despite the detailed definitions of the skills in the DSAT, the students had some problems understanding the meaning of the skills. One reason for this being that the definitions overlap, and skills are sometimes deeply connected, for instance, when it comes to collaboration and communication. Other researchers have also found it problematic, for instance, to define creativity (e.g. Craft, 2003, 2008), but also that assessing outcomes of makerspace activities in school situations is challenging (e.g. Lin et al., 2020). In our study, the most complicated skill for the students to understand was life/social skills. A possible explanation, as we see it, may be that there is no clear Swedish word for the term. However, as one of the teachers reflected, this is something the students are just about to start to develop, and maybe they are not at the stage yet being able to self-assess or reflect on this skill. However, the teacher emphasised that supporting student development of this skill was something she wanted to work with more in the future. We also noticed that this skill is not mentioned in the literature as often as creativity, problem-solving, collaboration, and communication when outcomes of makerspace activities are discussed.

Self-assessment reflections

It has been argued that assessment of outcomes of makerspace activities is not suitable as summative, but preferable as formative (e.g. Maltese et al., 2018). Self-assessment seems to be a logical part, in this sense. Andrade (2019) argued that when working with self-assessment, it is important that there are opportunities for adjustment and correction. However, this was not possible in the DSAT used in our study, and something that both students and teachers commented upon. The students specifically asked for this function in the tool, namely that they wanted suggestions on how to improve the different skills. A teacher was also critical to the fact that it was not possible for teachers to give students negative feedback and not accept when students requested badges claiming that they had reached a certain level of development of a twenty-first century skill. This could, of course, be considered as some kind of conflict, not being a matter of self-assessment when a teacher, despite the idea of the DSAT to be a self-assessment tool, still is supposed to give feedback by providing students with rewards in the form of badges. Is that not some form of assessment, rather than self-assessment?

As we argued earlier, the teachers’ role is of importance even if students use a self-assessment tool, this is confirmed by Bolden et al. (2020) in their review and also by Hattie and Timperley (2007) who argued that feedback can be provided by peers, but the next step of feedback, which they refer to as feed forward, is the teacher's most important instrument for supporting learning. This form of feedback usually requires a knowledge of knowledge and the nature of the ability that the student, the participant, does not yet have.

Conclusions

This study was implemented in a Swedish context, investigating outcomes of the use of a DSAT, aiming to stimulate understanding and development of five twenty-first century skills when students were working with makerspace activities. Our results show that not all students understood the idea of using a DSAT, and that there were several problematic practical issues which need to be taken care of in future development of the tool. Students also presented suggestions on improvements that can be useful for designers of DSATs. We also conclude that it is not that easy to understand the meaning of the twenty-first century skills included in this study (creativity, problem-solving, collaboration, communication, and life/social skills) despite great efforts to explain their meaning with long explanations in the tool, extra lessons discussing the skills, etcetera. Skills overlap and connect, and it was especially difficult for students to relate to life/social skills. Still, both students and teachers believed that it is possible to develop the mentioned twenty-first century skills during makerspace activities. Teachers also found connections between the skills and learning objectives in the school subject they were teaching, technology, as well as in other subjects.

In addition, if students are not used to self-assessment, this is something they need to practise. The students need opportunities to practise and strengthen the ability to assess their efforts and where they can receive a response to the assessments they have made. Gradually, the student's ability to self-assess, to make realistic values and to assess their own achievements, knowledge and skills, things that they can and things that they do not yet master, develop. The ability to self-assess one's own work also increases the student's ability to see the importance of their own work efforts in achieving good results. We agree with other researchers (e.g. Bolden et al., 2020; Hatti & Timperley, 2007) that teachers’ role is essential, even when it comes to supporting students in self-assessment. A teacher can explain to individual students; however, a DSAT cannot, at least not this one, at this stage. Finally, our results confirmed that the self-assessment made students and teachers aware of the twenty-first century skills of focus in our study and that the makerspace activities supported this development and that both students and teachers argued that these skills indeed are important for the future.

Limitations and future research

It could be questioned whether the results of this study could have been different if the makerspace activities had taken place in a “real” makerspace and not, as in our case, in classrooms and at university localities. On the other hand, the idea with the DSAT was that it should work in any learning environment.

Another critical reflection is how long time is needed for testing a new tool before conclusions can be drawn on outcomes. This study was investigating pilot tests of DSAT and we noticed that the time using the DSAT varied, with only one teacher testing the tool with her students for a longer period of 7 weeks, while the others only worked 2 weeks with the project. We believe that future studies would benefit from being of a more longitudinal approach.

In addition, it could be of interest to use Zimmermans’ model of self-regulated learning and self-evaluation investigating more of students’ perspectives, such as goal settings, planning, self-efficacy, intrinsic interest and values, affective responses and so on. These aspects were not included in this study, but could provide more holistic perspectives on responses to the design and use of this DSAT or other tools developed with a similar purpose.