Introduction

Since the foundation of the first makerspace at the Massachusetts Institute of Technology (MIT) in 2012 (Dougherty, 2012), the number of workshops which offer access to digital fabrication technologies such as 3D printers and laser cutters have been steadily growing. The idea of designing and making “almost everything” with one’s own hands and prototyping innovations has gained increasing interest not only by hobbyists, but also by start-ups (Browder et al., 2017, 2019; Troxler & Wolf, 2017; Waller & Fawcett, 2014) and formal and informal educational institutions (Bullock & Sator, 2015; Craddock, 2015; Dougherty, 2016; Martin, 2015). In makerspaces, traditional crafting is combined with digital fabrication tools such as additive manufacturing (3d printing) and electronics (soldering stations). Makerspaces can not only be found as distinct workshops predominately in urban areas, but also in libraries, museums and increasingly in universities and schools.

The European Schoolnet’s Open Book of Educational Innovation (Licht et al., 2017) mapped innovative educational initiatives in Europe and highlighted several maker projects in schools as being highly innovative and promising for educational holistic changes. Similarly, the annual Horizon report (Becker et al., 2017), forecasting technological trends in education, listed 3D printing and making in general, already in 2017, as one of the future trends in (higher) education for next five years. Two years later, the annual report recognised that this had already been taken up in some distinct educational fields such as in public health courses (Alexander et al., 2019; Brown et al., 2020).

To date, a few schools are already equipped with their own maker tools and makerspaces, others cooperate with makerspaces in their regions either by organising visits to the makerspaces or installing pop-up makerspaces in schools (e.g. Daley & Child, 2015; Irie et al., 2019; Jaatinen & Lindfors, 2019).

While making at school is still in its infancy, making and the maker pedagogical approach have increasingly gained attention and interest by pedagogical scholars and educational institutions (Geser et al., 2019) and some have already jumped on this trend, especially in STEM (Science, Technology, Engineering, Mathematics) subjects but making has not been explored in entrepreneurship education.

At the same time entrepreneurship education is rarely offered at elementary schools but tends to be integrated in the curricula at a upper secondary school level starting at the age of 14 (OECD, 2008) and even at this educational level, entrepreneurship education is restricted mostly to business oriented schools. Thus, also impact studies on the effect of entrepreneurship education focus mostly on adults and university student. A few studies (e.g. García-Rodríguez et al., 2018) however show that young people would benefit from entrepreneurship education, especially in relation to the entrepreneurial attitude associated with entrepreneurship. Most entrepreneurship programmes for young people focus on learning about entrepreneurship and programmes exist with a gamification approach such as learning through tentative entrepreneurial activities such as ‘mini-companies’ (c.f. Oosterbeek et al., 2010). These programmes often lack the practical hands-on focus of developing a product from idea generation to prototyping and the final product.

In the framework of the European funded project DOIT, an educational programme for children between the ages of 6 and 16 that combines maker elements and entrepreneurial education was developed. While the DOIT programme was not meant to replace entrepreneurship education, by trialling the developed programme, it was the intention to investigate the opportunity and worthiness of this combined maker and entrepreneurship education approach. Thus, we argue that the combination of maker education with entrepreneurial education fosters skills and attitudes that are key for future entrepreneurs.

Correspondingly, this paper aims to evaluate the immediate outcomes of this programme on the development of entrepreneurial skills and attitudes among participating children. The specific research questions (c.f. study approach) refer to the creative development and potential changes in perceived self-efficacy of participants as well as the investigation of demographic patterns in the effect of the programme.

While the evaluation is specific to the programme, it may shed light on potential benefits for making as a concept in education in general and as an opportunity for entrepreneurial education specifically.

The paper is divided in five chapters. The introduction chapter includes a reference to the maker pedagogical approach and entrepreneurial education. The chapter ‘Programme and context’ describes the DOIT programme followed by the study approach and the empirical results. In the last chapter the results are discussed and summarised.

Maker pedagogy

Due to its rather short history, the maker approach as introduced in education is not well established yet, but rather, it builds upon other well-established pedagogical approaches. However, maker education is still under development gaining more and more shape with the development of ever more maker programmes for schools. What can be observed however, is that there are certain denominators in the pedagogical approach that educational maker projects typically share, and which best represent the “maker spirit”.

The “maker pedagogy” builds on constructive learning approaches: Learning by doing principles, social learning (Bandura, 1977; Rotter, 1982) engaging in collaborative and interdisciplinary teams, problem based learning (c.f. overview by Walker et al., 2015), and learning through trial and error, where mistakes are acknowledged as learning opportunities (Kaltman, 2010; Unterfrauner et al., 2018). Making is hands-on learning, where makers learn from others and from their own experiences (Bell, 2010; Bruffee, 1993; Kaltman, 2010). Thus, the pedagogy of making builds on several pedagogical pioneers, including reform and constructivist pedagogues, from Montessori (2013) to Piaget and Papert (Ackermann, 2001), which all support self-regulated learning (van Hout-Wolters et al., 2000), i.e. where learners decide on their learning goals as well as on time and the methods to achieve these goals. In such open learning settings, teachers acquire the role of tutors, assisting the learner in their learning paths.

In schools, making has been introduced in STEAM (Science, Technology, Engineering, Arts, Mathematics) subjects first, recognising the value of maker education by offering hands-on learning activities and learning with all senses (Dougherty, 2016; Hwang, 2017; Papavlasopoulou et al., 2016).

The integration of maker pedagogy in formal and informal education is often based on the goal to facilitate the development of twenty-first century skills required in rapidly changing, digital society (Ananiadou & Claro, 2009). However, making could also contribute to entrepreneurial thinking and developing core entrepreneurial skills and attitudes; an aspect which has been largely neglected in the “maker pedagogy” as well as entrepreneurship education. This component was firstly introduced in the DOIT project as an innovative model for entrepreneurial learning.

Entrepreneurial education and making

Entrepreneurship education is mostly directed towards high school and university students and rarely towards children in their primary school years. This is despite the fact that studies such as the study by Rosendahl and Huber et al. (2014) have shown that non-cognitive related skills especially, such as creativity, persistence and self-efficacy, integral for entrepreneurship are best developed already at an early age. Making supports many skills relevant to entrepreneurship: Setting goals and devising paths to achieve them, believing in one’s own capabilities, creativity, integrating skills of others (collaboration) and working interdisciplinary in iterative processes including resolving failures and problems through trial and errors. Nevertheless, entrepreneurship education especially with regard to related goals and methods is still an underdeveloped area in most European countries (Eurydice, 2016).

Acknowledging that maker principles perfectly match the conditions of entrepreneurial education, the DOIT project incorporated the maker pedagogical approach.

Programme and context

The European project DOIT (Entrepreneurial skills for young social innovators in an open digital world) was a 3-year project (from October 2017 to September 2020) with 13 partner organisations from 10 different European countries. It aimed to develop and test an early entrepreneurial education programme for children combining maker and entrepreneurial elements and social innovation (in the sense that innovations are being developed for the greater good addressing Sustainable Development Goals as defined by the UN (SDGs)) (Rosa, 2017). These broad goals were broken down to the children’s reality so that they could relate to them.

The DOIT programme was inspired by the definition provided by Eurydice on entrepreneurship education, which acknowledges “… learners developing the skills and mind-set to be able to turn creative ideas into entrepreneurial action. This is a key competence for all learners, supporting personal development, active citizenship, social inclusion and employability” (Eurydice, 2016).

The developed DOIT programme (c.f. Table 1) consisted of seven elements with a minimum course length of 15 h.

Table 1 DOIT programme elements (Hornung-Prähauser et al., 2018)
Full size table

The programme started with a sensitisation element where students could envision the scope of their possibilities for tackling challenges in their personal environments which relate to the SDGs. The second element was about exploring the challenge, followed by co-design: Students were asked to collect and select potential ideas for innovations. In the co-creation phase, students gathered in teams and developed first prototypes and iterated them to continuously improve them. In the next step, in the scaling up phase, concrete steps for envisaging the realisation of the final prototype were developed. In the last step, reaching out, the robustness of the prototype was tested with a bigger group of users and finally students shared their projects with the wider public.

To provide some examples of prototypes resulting from this programme, students, for instance, came up with the idea for a prototype that would send an alarm in case of risk of flooding of the little creek on the mountain pasture, or they developed a machine that transformed garbage in something useful. The focus did not lie on a fully functional final prototype, but the programme rather aimed at encouraging the development of own ideas, believing in them and at working together on a prototype that would illustrate the students’ ideas in a tangible way.

The DOIT programme was implemented in ten different European countries in both school and outside school settings, either as a pop-up makerspaces in schools or directly in a makerspaces.

Study approach

The DOIT programme was not oriented towards a commercial definition of entrepreneurship education, but rather, it worked on the basis of the more comprehensive definition provided by the EC thematic working group as cited above, which recognises entrepreneurship education as learners developing the skills and mind-set to be able to turn creative ideas into entrepreneurial action (Eurydice, 2016). Thus, we put skills and attitudes at the core of the definition in line with the definition of entrepreneurship education as provided by Lackéus (2015) who distinguishes between entrepreneurial attitudes (self-confidence, self-efficacy, sense of initiative, ambiguity tolerance, perseverance), entrepreneurial skills (creativity, planning, financial literacy, managing resources, managing uncertainty/risk, teamwork); entrepreneurial knowledge (assessment of opportunities, identifying with the role of an entrepreneur – self-reflection) and how-to knowledge (accounting, finance, marketing and communications). This is also very much in line with the Entrepreneurship Framework (Bacigalupo et al., 2016), which puts 15 competences such as self-awareness and self-efficacy, creativity, coping with ambiguity, uncertainty and risk at the core of their framework.

In line with these core principles, the evaluative dimensions for measuring the effect on a personal level were defined: Creativity, self-efficacy, teamwork and collaboration skills, dealing with uncertainty, perseverance, empathy and knowing others’ needs, motivation and sense of initiative, and planning and management skills. While the first two dimensions were assessed quantitatively and qualitatively, the other seven dimensions were assessed only qualitatively. In this paper we focus on the effect of the DOIT programme on the two former dimensions: Creativity and self-efficacy, which were to be found at the core of similar evaluation frameworks (c.f. Entrecomp framework—(Bacigalupo et al., 2016)).

The following definitions of the two dimensions covered in this paper guide the research design:

  • Creativity is perceived as the ability to find different solutions for the same problem in the sense of divergent thinking and to turn these possible solutions into new opportunities (Baer, 2014).

  • Self-efficacy is defined as the believe own’s abilities to solve problems and overcome obstacles. As Shane (2003) notes, the confidence in one’s own abilities increases the willingness to pursue entrepreneurial opportunities. Self-efficacy, pro-active behaviour and creativity are deemed essential not only for pursuing entrepreneurship, but also seem increasingly relevant for the labour market in general (Gensowski, 2018; Kautz et al., 2014). Self-confidence and self-efficacy are acknowledged as being key in pursuing own goals and ultimately support the development of entrepreneurial intentions and actions (Boyd & Vozikis, 1994; Izquierdo & Buelens, 2011; McLaughlin, 2019).

As this paper aims to understand and analyse the impact of the DOIT maker and entrepreneurial education programme on the participating students in developing non-cognitive skills, the following research questions guided the study:

  1. (1)

    Does the programme have an immediate impact on children’s creative development?

  2. (2)

    Does the programme have an immediate impact on children’s self-efficacy?

  3. (3)

    Do participants’ demographic data (gender and age) correlate with their skills development?

While the first two research questions are dedicated to detecting an overall change in non-cognitive skills, question 3 aims to analyse which specific groups eventually benefit more (or less) from the programme which might ultimately lead to recommendations on how to the set up such programmes to achieve maximum impact.

To measure the individual impact, a one-group pre/post design (Levine & Parkinson, 2014) was implemented. This design falls under quasi- experimental designs as the main premise of true experiments; namely the existence of a control or comparison group and the random selection and assignment of participants is missing. As a result, although one would be able to assume that the changes from the pre-test to the post-test are due to the participation of children in the DOIT programme, unlike in true experiments where such effects would be solely attributed to this participation; in this design outside factors cannot be controlled or ruled out. Nevertheless, this design is more reliable or provides more data than a one-group post-only design, which due to the lack of a pre-test cannot show changes in skills or attitudes. In this design, changes between measurements at the beginning of the programme (pre-test), i.e. the baseline, and those at the end of the programme (post-test), estimates the effectiveness of the DOIT programme. A true experimental design with a control group was not possible for different reasons, e.g. it was difficult to organise control groups outside the school settings since the control group participants would have no incentive to just complete the pre-test and post-test and not take part in the activities like the experimental group participants. As concerns the school setting, schools found the scientific argument not convincing to randomly assign children to the programme and the control group, having the latter one go through both instruments two times and not take part in any activities, others argued that the procedure would introduce a divide between those who can actually participate in the DOIT actions and others who cannot directly participate. The challenge of setting up control groups is quite common in quasi-experimental settings as the project was operating in (Bauman & Nutbeam, 2013). Nonetheless, an attempt to create a comparison group was made within the dataset including the participants who participated in up to 13 h of the programme’s activities and therefore did not fulfil the minimum requirement of 15 h. All participants that took part in the programme’s activities for at least 15 h formed the experimental group. To this end, the design is more reliable in as far as attributing the impact on the participants to the programme than a one-group post-only design.

Evaluative instruments–operationalisation of creativity and self-efficacy

To measure creativity, the Test for Creative Thinking-Drawing Production ((TCT-DP; in German: TSD-Z, (Urban & Jellen, 2010)), a standardised psychological test, was deemed as suitable as it is “language-free” and “culture-fair” (Weiner, 2003) and therefore can easily be enforced in different countries as well as on a broad age range. This test further provides for so-called parallel versions: A form A for the pre-test and a form B for the post-test. This test has been designed exactly for this kind of pre-post evaluation, where learning effects due to the simple repetition of tasks can be controlled. Furthermore, numerous studies have made use of this test in similar research designs (e.g. Greb et al., 2007; Karwowski & Soszynski, 2008; Maksić & Tenjović, 2008).

Furthermore, in addition to aspects of divergent thinking, the TSD-Z also considers qualitative, creativity-relevant components (such as composition and solution form, unconventionality, willingness to take risks). Drawing skills do not play a significant role in the processing of the TSD-Z. On the test sheet, some special figural fragments are provided, which are intended to encourage further drawing in a free, undefined manner. The final graphic product is scored with points on the basis of 14 evaluation criteria, which at the same time represent the test construct.

The test comes with very detailed instructions that have to be followed very closely by the facilitator in order to create a comparable setting. It foresees, for instance, that every participant is equipped with a pencil or black felt pen (no eraser, ruler or coloured pencils) and the facilitator keeps track of the time with a stopwatch since participants can win some extra points if they complete the drawing in less than 15 min (which is the maximum duration).

As for measuring self-efficacy, although there are several standardised published scales available that measure self-confidence, self-efficacy or related constructs, such as FSKN (Deusinger, 1986), FKK (Krampen, 1991), PALS (Midgley et al., 2000), and some sub-scales of PFK (Seitz & Rausche, 2004), none was deemed suitable for the purposes of the evaluation of the DOIT programme. Additionally, the Entrecomp framework (Bacigalupo et al., 2016), which has developed scales for entrepreneurial education, also covering self-efficacy, would have matched only the older age group. For the purpose of this study, the different available instruments either did not cover the age span of the participants or comprised incomprehensible items, especially for the younger children. For these reasons, we selected the most suitable items from different surveys and used them as basis for further development and as a source for inspiration to develop our own scale instead. The resulting 15 items were meant to measure three different aspects of self-efficacy (also in line with the general dimensions of the DOIT pilot actions): sub score 1, Self-efficacy in relation to peers (questions 2 + 6 + 7 + 10 + 14; subscore 2, Self-concept regarding own capabilities (i.e. questions 3 + 4 + 5 + 8 + 13; subscore 3, Self-concept regarding problem solving (i.e. questions 1 + 9 + 11 + 12 + 15). To these 15 questions we added one regarding the entrepreneurial intention.

After several iterations in strong collaboration with different practice partners, who were each in charge of implementing the DOIT programme in their respective countries, and pre-tests with children, we chose a 5-point Likert scale as most suitable answer format. Several children were involved to test the questions of the different scales that aimed to understand children’s preferences and needs, the attention span, and their preferred way of working. These pre-tests showed that visual scales (e.g. emoticons) and analogue scales did not work. With the visual scales, children often got confused whether they were being asked to rate whether they liked the question or whether they should rate the answer to the question that applied to them. With the analogue scale, especially younger children, marked mostly the extremes and did not understand that they could nuance their answers. This is very much in line with research by Mellor and Moore (2013) who experimented with different answer formats in questionnaires designed for children. Furthermore, in pre-tests with children it turned out that questions were easier to understand than statements and therefore the final version of the questionnaire comprised only of questions.

In the pre-tests, children were asked to rephrase how they understood the questions to check for the comprehensibility of the question. While completing the pre-tests, it was observed how the children answered the questions, their reaction to the questions, whether the answer format was intuitive enough and other occurring issues such as skipping a line. All the observations led to several reformulations as well as re-designs. The questionnaire had to be translated into the national languages as accurately and as closely as possible to the original wording. Accuracy checks of the translation were recommended to the practice partners by applying backwards translation: One member of the team translated the questionnaire from English into the national language and another one translated back from the national language into English. The two versions, the original version and the backwards translated version, were then compared against each other to detect inconsistencies.

Empirical results

Data sample

In total, between September 2018 and February 2020 1,002 children participated in the DOIT programme. For the analysis of the effects in relation to self-efficacy and creativity the sample had to be further cleaned taking into account only those participants that took part in at least 15 h of the DOIT programme’s activities. The data included the results of both the pre-test and the post-test as well as demographic data such as gender and age. This resulted in 759 complete datasets for the self-efficacy questionnaire and 618 complete sets for the creativity tests based on forms A and B described above. Incomplete creativity test and self-efficacy datasets were not taken into account (e.g. due to a missing answer in the questionnaire or either pre or post-test were not provided).

Overall, the gender distribution was fairly balanced with about 48% female participants in the self-efficacy dataset and about 46% in the creativity dataset (c.f. Fig. 1). The younger age group (6–10 years) represented 27% of the self-efficacy dataset and 28.8% in the creativity dataset. A more detailed breakdown of age and gender is shown in Table 2, providing an overview of the self-efficacy dataset after data cleaning (n = 759) and Table 3, 4, describing the participants in the creativity test.

Fig. 1
figure 1

Participants per age and gender (n = 774)

Full size image
Table 2 Overview of participants (self-efficacy dataset, n = 759)
Full size table
Table 3 Overview of participants (creativity dataset, n = 618)
Full size table

The DOIT programme was in most cases carried out in schools but slightly less than a quarter took place in other settings such as makerspaces, summer schools, youth centres etc. resulting in a sample of 73.6% of children coming from a school setting for the self-efficacy dataset and 82.5% for the creativity dataset.

In order to measure changes in the two evaluative dimensions, creativity and self-efficacy the post-test scores were compared against the pre-test scores using paired t-tests. The statistical analysis shows that both scores increased significantly with effect sizes of 0.21 for creativity and 0.08 for self-efficacy. According to Cohen (2013) an effect above 0.2 can be interpreted as moderate and thus as a change in a meaningful dimension.

The specific results in relation to the creativity and the self-efficacy score gains are described in the following sections.

Creativity

In average, the creativity score increased from the pre-test to the post-test by 1.59 points in the TCT-DP Test, which represents a significant increase (cf. Fig. 2). The distribution of pre-test values is indicated by the solid curve while the distribution of post-test values is indicated by the dotted curve in the figure below.

Fig. 2
figure 2

Pre-post comparison of creativity test

Full size image

The following table shows the results of the paired T-tests by demographic variables as well as by setting (Table 4).

Table 4 Creativity score key results
Full size table

Interestingly, in contrast with the assumption that creativity would correlate with age (Chan & Zhao, 2010; Claxton et al., 2005; Yeh & Li, 2008), the younger age group (6 to 10 years) did not show a significantly lower level of creativity in the pre-test than the older age group (11 to 16 years). A comparison of the pre-test and the post-test within the age groups however, shows that, the older age group gained significantly higher creativity scores than the younger participants.

In relation to gender differences, the creativity score does not differ between females and males in the pre- test. However, there is a significant increase in the scores of both genders from the pre-test to the post-test, with the scores of the female participants increasing to a higher extend than that of the male participants (c.f. Fig. 3).

Fig. 3
figure 3

Creativity scores in pre-test and post-test by demographic data

Full size image

The analysis also shows that the setting correlates with the creativity score gain. While the score increased significantly among the group of participants that took part in the DOIT programme in school, this was not the case with the group of participants that took part in the programme’s activities outside of school. However, this difference has to be interpreted with caution as it cannot be ruled out that the positive self-selection bias (Wainer, 2013) did not come into play. While the participation in the programme was compulsory for the participants in school, extracurricular activities usually attract children who are already interested in the subject and choose to join. This is reflected in the higher creativity score at the beginning of the programme, which however does not increase significantly in the post-test. One could argue that participants who showed a considerably high level of creativity at the beginning of the programme, do not have room to significantly increase their score due to a ceiling effect, which is quite a common phenomenon as other studies have shown (e.g. Harkins, 2001).

Thus, the results of the analysis show that specific groups benefitted more than others in terms of creativity from their participation in the programme: The older age group (11 to 16), females (more than males although both genders increased their scores) and pupils taking part in the programme at school.

Self-efficacy

In order to prove the reliability of the newly developed questionnaire, the Cronbach alpha was used as a measure for internal stability (Santos, 1999) which shows that the questionnaire is a sufficiently reliable instrument (pre: r = 0.728, post: r = 0.782).

The paired sample t-tests of the whole dataset (without breaking it into groups) shows a statistically significant increase on the scores from the pre-test to the post-test by 0.56 points (p = 0.000).

Table 5 shows the results of the paired sample t-tests by demography and setting for the self-efficacy dataset.

Table 5 Self-efficacy score key results
Full size table

The younger age group, i.e. children between 6 and 10 years, showed higher levels of self-efficacy already in the pre-test compared to the other groups. The older age group, i.e. participants from the age of 11 to the age of 16, increased their scores significantly from the pre-test (54.77) to the post-test (55.25), however, despite this increase, their score on self-efficacy at the end of the programme was still not as high as that of the younger age group at the beginning of the programme (55.48). Thus, in contrast to the creativity test where the older participants gained higher scores in the post-test, in the self-efficacy test, the younger participants outscore the older ones, although their self-efficacy score does not change dramatically from the pre-test to the post-test. This is line with studies on self-efficacy which shows that children tend to have higher self-efficacy scores until they reach around grade 7, where they are approximately 12 years old and the perception of their own capabilities in this area declines (c.f. Urdan & Pajares, 2006).

At the pre-test, females have an average self-efficacy score of 54.50 compared to 55.44 for males. This difference in the pre-test is however not statistically significant. Scores of both genders increase from the pre-test to the post-test, however, only the scores of the male participants increase significantly (the gain score for girls was 0.41 at a p-vale of 0.08 compared to 0.66 for boys at a p value of 0.01). As can be seen in Fig. 4, all four demographic groups improve their scores from the pre-test to the post-test, however, to different degrees.

Fig. 4
figure 4

Self-efficacy scores in pre-test and post-test by demographic data

Full size image

In terms of the setting, i.e. whether the participants took part in the DOIT programme within or outside school, it can be observed, like with the other groups, that the self-efficacy score increases from the pre-test to the post-test. Evidently, the baseline score for the group that participated in the DOIT programme as a voluntary extracurricular activity, i.e. out of school, was notably higher at the pre-test (58.24 compared to the average of 54.96 for all participants), further exhibiting the positive selection effect.

The composition of the self-efficacy questionnaire allowed for differential analysis of different facets of self-efficacy: Self-efficacy in relation to peers (sub-score 1), self-concept regarding own capabilities (sub-score 2) and self-concept regarding problem solving (sub-score 3). Five items each in the questionnaire were attributed to one of the sub-scores. In general, sub-score 1, self-efficacy in relation to peers, did not change significantly (p = 0.180, T = − 1.341, df = 775) from the pre-test to the post-test. However, the other two dimensions showed a significant increase from the pre-test to the post-test: Sub-score 2, self-concept regarding own capabilities (p = 0.019**, T = − 2.354, df = 775) and sub-score 3, self-concept regarding problem solving (p = 0.042, T = − 2.040, df = 775). The Fig. 5 below shows the pre-test and post-test values differentiated by these three dimensions.

Fig. 5
figure 5

Self-efficacy scores in pre-test and post-test by sub-score

Full size image

This result very much underlines the pedagogical approach that was followed in the DOIT programme: Expectation that participation would increase the belief in being able to do something with the own hands, from ideation to realisation and active problem solving. These are fundamental for developing an entrepreneurial spirit.

Limitation of findings

In order to attribute the changes in the participants scores from the pre-test to the post-test in both the creativity and self-efficacy test to the DOIT programme, one would have to control for confounding variables. As explained in earlier sections of this paper, this can best be done by a comparison of the scores to that of a control group that includes participants who were included in a control group through randomised assignment and did not participate in the programme. This would ensure that the participants in the control group would be as similar as possible to those in the experimental group (Bortz & Döring, 2013). As setting up a control group in this sense was not possible for reasons described above, the research team nevertheless attempted to create a comparison group within the dataset including participants that completed both measurements, but those who did not fulfil the minimum requirement of participating in the programme for at least 15 h. This comparison group comprised only 27 children who answered both the pre-test and the post-test but only attended between 2 and 14 h of the programme.

Despite the sample size of the comparison group being very small in comparison to the experimental group, comparisons between the groups were still attempted. The results show that when the scores of the creativity test were compared between these two groups, the experimental group increases their score significantly from the pre-test to the post-test, while the comparison group does not (p = 0.286, T = 1.094, df = 26). The same applies to the self-efficacy survey (p = 0.724, T = 0.333, df = 26). This suggests that the differences can be attributed to the DOIT programme. However, this assumption can only be made with a lot of caution as it can be argued that the sample size of the comparison group is too small to generate any statistically relevant results.

Besides comparing with a parallel group there are some more arguments that speak for a DOIT effect. The creativity test does have two different parallel forms, Form A for the pre-test and Form B for the post-test and thus score gains due to learning effects are avoided.

Furthermore, the creativity score and the self-efficacy score do not correlate (r = 0.031, p = 0.586) which further speaks against a simple learning effect due to the repetition of tasks.

Outlook: looking for optimal didactical settings

Although the didactical settings were organisationally and content-wise an open learning setting where the seven programme were all operationalised in different ways by the facilitators, each pilot site was assessed with the same measuring instrument and positive effects could be found in all countries. This open learning setting however makes comparisons between the different pilot sites impossible since the variety and diversity of variables cannot be controlled (like in a real experiment in a lab). Figure 6 shows all pilot sites ordered by their pre-post differences in the creativity tests. Although overall creativity increased, in Slovenia, for example, the median of the differences between pre- and post-test was -1, i.e., creativity decreased.

Fig. 6
figure 6

Means of pre-post differences within the creativity test

Full size image

The objective of this study was to evaluate the overall effect of the DOIT approach; hence the data of all participants irrespective of their location was merged and neither country-specific analyses nor country comparisons were made.

Since each pilot might have had its own specific pedagogical approach and maybe even a different set of machines and materials, that could determine the way the DOIT programme was implemented, it would be correct to assume that the DOIT programme worked better in some countries than others leading to different results in the participants’ creativity levels. However, a one-way ANOVA showed that ‘pilot site’ was not a variable that could establish a general pattern. Put differently, there were no statistically significant differences between group means [F(2, 469) = 1.47, p = 0.16]. As a result, this variable was not further investigated, and the final sample was compiled from all the pilot sites.

The possibility to provide recommendations on specific pedagogical designs, a more fine-grained data collection would be needed; including variables such as the duration of specific activities, technologies used, settings, topics, language and cultural factors. Also research on the local conditions and country specific factors would be worth to consider controlling for variables such as socio-economic factors and specific educational frameworks. This way, concrete recommended operationalisations of different elements within the DOIT approach could be provided to help increasing positive changes in creativity and self-efficacy.

Discussion and conclusion

Coming back to the research questions, we can affirmatively answer research question 1 and 2, i.e. whether the programme has an immediate impact on the creative development and self-efficacy respectively: The DOIT pilot did increase the creativity and self-efficacy of the participants at a statistically significant level. This effect can be attributed to their participation in DOIT as the comparison group does not show the same change. The participants of the quasi-control group did not gain any significant score increases in the two evaluative dimensions. Also learning effects due to the repetition of tests can be excluded, as parallel versions of the creativity test were applied in the pre-test and the post-test. Although this was not the case with the self-efficacy questionnaire, the lacking correlation of the self-efficacy and the creativity score gain speak against a learning effect.

Research question 3, i.e. whether the demographic data correlates with skills development, requires a more differentiated answer. The demographic data, gender and age of the participants, correlates with the skills development. The analysis shows higher effects on some participants and no significant effect on others. It is particularly the older age group (11 to 16) which seems to unfold their creativity potential in the framework of the DOIT programme, while the younger age group (6 to 10) seems to remain unaffected in terms of creativity. While both genders reach higher creativity scores at the end, it is particularly females who take up the creative spin of the programme. With self-efficacy, it is particularly the younger age group and males who seem to increase their scores to a higher degree.

The evaluation design follows a pre-post comparison of the scores, comparing the scores before and after taking part in the DOIT programme. Thus, the analysis cannot anticipate any long-term changes. However, we can conclude that “maker education” approaches with children can contribute, to some extent, to the development of creativity and self-efficacy as key for entrepreneurial skills and attitudes. Even if the programme was to be extended to more than 15 h, we can assume to have at least an effect of a similar size or more.

Although these open learning settings challenge students as well as facilitators, this type of learning settings adapts well to the next generations’ need for creativity and problem-solving skills in order to promote social innovations. We further conclude that maker elements in entrepreneurial education are worth to consider when designing new programmes as a way for hands-on learning complementing already existing entrepreneurship approaches. This is especially the case in entrepreneurship education for young people with a limited number of studies in relation to the effects on young people (e.g. Paço & Palinhas, 2011; Rosendahl Huber et al., 2014).

A better understanding of the relationship between age and the resulting benefits in creativity and self-efficacy would be a worthwhile follow up research. A more detailed collection of potentially influencing factors to control for would probably result in even more substantial effects and hint to local and country specific conditions to take into account such as socio-economic conditions as well as educational setting. Such insights could even better inform pedagogical strategies in open learning settings.

Conflict interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.