1 Introduction

Nowadays it is essential to be familiar with the competencies that are needed to function in a digital world (Iivari et al., 2020). These competencies can be acquired as early as primary school through learning programming (Stamati, 2020). Learning to program enables pupils to develop concepts from the world of computer science (Hogenboom et al., 2021). Primary education can provide a designated environment where programming can be taught, learned and practised (Sáez-López et al., 2019). Applications of programming can enhance pupils' problem-solving abilities (Sprankle & Hubbard, 2012). The ability to solve complex problems by making use of computer science concepts facilitates the development of computational thinking (CT) (Fanchamps, 2021; Nouri et al., 2020; Tedre & Denning, 2016).

Although developing computer-based programming skills is the most common approach to developing CT in schools, a systematic literature review of 125 studies focused on CT reveals that educators and teachers also deploy unplugged variants and alternatives in their prevailing teaching (Kalelioglu et al., 2016). However, while the effectiveness of computer programming for promoting CT skills is frequently studied (Lye & Koh, 2014), this is still only marginally applicable regarding the unplugged approach. Most studies with unplugged activities focused on promoting students' interest in computer science (Battal et al., 2021; Taub et al., 2012; Yildiz & Karal, 2021).

Contrary to common belief, it is not always necessary to apply digital tools and programming to develop skills associated with CT (Brackmann et al., 2017). Unplugged programming refers to learning computer science concepts without the use of digital technology (del Olmo-Muñoz et al., 2020). This is often called computer science unplugged (CSU) (Kalelioglu et al., 2016). A specific category of unplugged applications for learning computer science concepts are SmartGames that can train the memory, require and stimulate logical thinking, and can improve spatial awareness and problem-solving skills (Curzon & McOwan, 2017; Tsarava et al., 2018). By their nature, unplugged SmartGames contain characteristics that can be attributed to CT (Sharma et al., 2019; Sun et al., 2021). Therefore, the application of SmartGames to learn the principles of programming with a development on CT appears to be a promising concept for primary school practice. Furthermore, determining whether unplugged SmartGames can act as a trigger for learning appears to be essential (Prieta et al., 2013). In this regard, from previous research in CT learning, it is already known that in the application of plugged-in programming environments like on-screen simulations and robotics the perception, motivation and involvement of pupils is high (Slangen, 2016; Zapata-Cáceres & Fanchamps, 2021). Given the potentially expected results on CT, it is valuable to determine the effects on learning experiences when using unplugged SmartGames and whether a comparison with such plugged-in programming environments can be made.

Apart from unplugged activities by application of SmartGames, sense-reason-act (SRA) programming in visual environments with tangible or on-screen output facilitates the development of CT by means of complex programming concepts (Fanchamps et al., 2022a). Previous research has shown that the type of task design, the programming environment used and the type of output can be of characteristic importance (Fanchamps et al., 2022b). SRA-programming appears to be of distinctive importance regarding the acquisition of a deeper understanding of complex programming concepts, resulting in a higher level of CT (Fanchamps, 2021). SRA-programming can be seen as an anticipatory method of solving programming problems by applying complex programming concepts, using visual programming environments with tangible or on-screen output. Given previous research indicated that SRA-programming allows for a better understanding of complex programming concepts that subsequently leads to an increased development of computational thinking (Rodriguez et al., 2017), this research therefore investigates whether unplugged programming applied via SmartGames has the same impact on gaining an understanding of complex programming concepts, resulting in a higher level of computational thinking, compared to visual SRA-programming with either a tangible or visual, on-screen output. It also examines the characteristics of learning experiences created by applying different task environments (unplugged—plugged-in).

2 Theoretical framework

Computational thinking (CT) encompasses more than merely solving problems using concepts derived from the world of computer science. It refers to thinking processes in which problem-oriented issues and their solutions can be reformulated and re-arranged in a form that they can be effectively executed by an information-processing digital agent (Dummer, 2017). Computational processes are a way of understanding natural and social phenomena and for explaining and interpreting the world as a composition of information processes (Denning & Tedre, 2019). Computing, as an ancient human practice, required computational thinking to design procedures and artefacts to automate them. Many aspects of computational thinking existed before the electronic computer and have been refined over time (Tedre & Denning, 2016). As the use of computers became increasingly common in the middle of the last century, the use of CT became a prerequisite for proper handling them. This implied the necessity to find a manner to define and to teach CT. From this perspective Denning and Tedre (2019) have addressed six dimensions (methods, machines, computing education, software engineering, design, computational science) that describe the construct of CT.

From a philosophical perspective, CT seems elaborated to the essence of what we can achieve by interacting with computers, as extensions of our minds, to create and discover (Knuth, 1980). A similar conceptual approach was also taken by Papert (1980), who focused on two aspects of computation: 1) how to use computation to create new knowledge, and 2) how to use computers to improve thinking and patterns of access to knowledge. After computational thinking was relegated to the background, Wing (2006) generated new attention to computational thinking and formulated a modified approach, considering CT as the fundamental ability to solve challenging problems using skills abstracted from the world of computer science. From then on, CT has attracted renewed and prominent interest from scholars, and a multitude of definitions appear in the literature. In summary, these definitions for CT can be represented as “the mental skills and practices for: a) designing computations that let computers do tasks for us, and b) explaining and interpreting the world as a complex of information processes” (Denning & Tedre, 2019). In parallel with a further refinement of the definition of CT, the characteristics that define CT are also reducible from the literature, namely: problem decomposition, pattern recognition, data representation, generalisation, abstraction, and algorithms (Shute et al., 2017).

There are various ways to develop CT by means of educational activities (Hsu et al., 2018). On the one hand, this development can be achieved through the use of ICT-based activities. On the other hand, a development on CT can also be achieved without the application of digital technology using unplugged programming activities (Brackmann et al., 2017; Kuo & Hsu, 2020; Rodriguez et al., 2017).

Unplugged programming activities can be defined as learning to understand concepts derived from computer science without using digital technology (Bell et al., 1998). Unplugged programming has its origins in the movement of computer science unplugged (CSU) (Wahyudin et al., 2021). The rationale of CSUnplugged includes learning activities that teach computer science thinking without using digital technology. With this in mind, an approach emerged that introduced primary school pupils and teachers to off-line activities such as board games (Bell et al., 1998), and to help them place computer science in a broader perspective so that it encompasses more than just programming (Bell & Vahrenhold, 2018). In the studies undertaken by Lambert and Guiffre (2009), several positive results were obtained from CSUnplugged activities. Studies conducted by Rodriguez et al. (2017) and Thies and Vahrenhold (2013) with secondary school students showed that learning revenue on CT through unplugged activities indicated to be at least as effective as learning through the use of a digital approach. Furthermore, the study by Brackmann et al. (2017) found that primary school pupils who had participated in unplugged activities showed significant improvement in CT compared to pupils in the control group. Research conducted by del Olmo-Muñoz et al. (2020) shows that pupils can learn CT-skills by means of unplugged activities and that this form of unplugged learning is at least as effective as learning based on a computer aided approach. Moreover, studies conducted by Looi et al. (2018) and Relkin et al. (2020) show that primary school pupils, who had participated in unplugged activities, showed significant improvement on their level of CT compared to a group that used plugged programming activities.

In education, games can be used as a teaching aid in helping to explain or to reinforce the learning of computational concepts (Alvarez & Djaouti, 2011; Miller, 2008). The difference between 'normal games' and SmartGames is in the initial design process. Elements are added with the intention of evoking a variety of approaches and strategies to be employed or for acquiring specific content (De Freitas, 2018). SmartGames are logical thinking games which can stimulate the use of human memory, can improve spatial understanding and appeal to problem-solving thinking (Tsarava et al., 2018). SmartGames prompt thought processes among users for solving problems where the application goes beyond pure entertainment. SmartGames can be used in order to propose and understand computer concepts, such as algorithms, parallel processing or data transfer. According to (Bakan & Bakan, 2018), playing a SmartGame enhances the process of inquiry based learning because it can put users in the required learning situations. Moreover, it stimulates interaction among learners, and can facilitate mastering complex tasks (Hung et al., 2014). In education, the use of games can be implemented to teach particular aspects like cleverness, insightfulness, speed, automation, knowledge and for cognitive challenges (Alvarez & Djaouti, 2011). Most games contain a core of required dexterity, insight and skills and can therefore be used for this purpose. Such elements characterize the difference between "regular games" and SmartGames. In addition, elements can be added that can evoke the user's cognitive challenge.

Robotic and virtual programming environments are considered powerful tools for learning the basic programming concepts (Caci et al., 2013; López et al., 2021). When applying sense-reason-act (SRA) programming, the more complex programming concepts such as iterations, conditionals and functions with parameters are used (Basu et al., 2016; Werner et al., 2012). SRA-programming asks for a more thorough understanding of the underlying operating principles rather than restricted linear, sequential programming structures (Martinez et al., 2015). The application of SRA requires logical reasoning of the ''if … then …'', “if … then … else, ''wait … until …'', ''repeat … until …'', “while …” kind. The complexity is rooted in the ability to think in terms of scenarios, and to understand and apply the more complex concepts of programming (Popat & Starkey, 2019). Programming according to the SRA approach is characterised by a multi stage process in which information obtained from sensor readings (sense) is fed into a reasoning component which compares these external conditions with pre-set values (reason), and the subsequent process of interferences that lead to initiated actions (act) (Lith, 2006). From previous research (Fanchamps, 2021) we know that a variety of ICT-based programming environments can ensure a growth in computational thinking, and that primary school pupils achieve higher levels of CT when applying SRA-programming either using robots or using simulations of reality with visual, on-screen output. We also know that the application of SRA-programming with an impact on CT depends on the type of task design and the environmental conditions in which programming are to be performed (Slangen, 2016). In addition we know that the application of SRA-programming enables a thorough understanding of the more complex concepts of programming resulting in a significant impact on computational thinking (Fanchamps, 2021).

Building on the theoretical exploration above, This research aims to explore opportunities to establish educational improvement, and specifically focuses on comparing applications of plugged-in and unplugged learning materials in order to identify characteristics and features from a didactic perspective regarding the development on CT. Although thorough examination of the literature shows that unplugged programming in a comparison with ICT-based activities indicates to be at least as effective concerning a development on CT, it is questionable whether this is also demonstrable when SRA-programming is applied. Moreover, previous research has shown that SRA-programming can achieve enhanced development on CT through an increased understanding of complex concepts of programming such as the application of iterations, conditionals and functions. Therefore, we want to know whether the application of SmartGames can achieve similar effects. We hypothesise that the application of unplugged programming activities offered through SmartGames can have an equal impact on gaining insight into underlying complex programming concepts, and generate identifiable development in CT compared to visual plugged-in SRA-programming with tangible or on-screen output. In addition, we conjecture that through the application of unplugged SmartGames there is an influence on acquired learning experiences similar to the application of plugged-in SRA-programming. Our conceptual model, as displayed in Fig. 1, summarizes the relationships and interconnections between the independent and dependent variables, with some relationships being reciprocal.

Fig. 1
figure 1

Schematic representation of the conceptual model

Full size image

2.1 Research questions

Grounded on the previous exploration, our main research question is: Whether and to what extent can unplugged programming activities applied via SmartGames, in comparison to SRA-programming tasks with tangible or visual output, show a similar development on complex programming concepts resulting in a development on computational thinking, and what aspects of learning emerge by means of such different task environments?

2.2 Sub-questions:

  • Can unplugged SmartGames provide a full-fledged substitute for developing computational thinking?

  • Does unplugged programming via SmartGames promote identifiable development on CT compared to SRA-programming with tangible or on-screen output?

  • Do unplugged activities contain more aspects of CT development than just programming components?

  • What indications regarding different aspects of learning (knowledge, experiences, added value, motivation, collaboration) can applications of various CT programming environments provide?

2.3 Hypothesis

  • Unplugged programming applied by SmartGames shows equivalent development on computational thinking in comparison with plugged-in SRA-programming.

  • Programming via unplugged SmartGames activities provide a better understanding of complex programming concepts compared to visual SRA-programming with tangible or on-screen output.

  • The application of unplugged SmartGames activities produces more positive identified learning effects comparing SRA-programming with tangible or on-screen output.

3 Method

This comparative study should be seen as an exploratory approach to gain a better understanding of the effects of unplugged and plugged-in learning environments on CT development. As illustrated in Fig. 2, an exploratory mixed-methods design was conducted in which, by application of a pre-/post-test questionnaire survey and semi-structured interviews, both quantitative and qualitative data were obtained to (a) determine the effect of the interventions and comparing the impact; (b) to assess the associated hypotheses; and (c) to investigate the research questions. For the dependent variables, this includes among primary school pupils 1) pre-assessment of computational thinking skills (CTt), 2) for the intervention either the application of unplugged SmartGames activities, or two different visual programming environments with either tangible output (Lego EV-3) or on-screen output (Bomberbot), and 3) post-assessment of computational thinking skills (CTt) combined with semi-structured interviews.

Fig. 2
figure 2

Research design

Full size image

In addition to the data obtained by applying the CTt, we also collected qualitative data by means of semi-structured interviews (open ended interview / one-to-one) (Creswell, 2008) with 4 standard questions (‘What would you like to share about working with SmartGames / Lego / Bomberbot?’, ‘How would you feel about working with SmartGames / Lego / Bomberbot at school?’, ‘What have you gained from working with SmartGames / Lego / Bomberbot?’, ‘How did you experience working together on the tasks with your buddy?’), and subsequent follow-up questions (e.g. ‘Can you explain your answers a little more precisely?’, ‘Can you give an example?’, ‘Why do you think so?’, ‘What would you like to do differently yourself now that you have performed the tasks?’, ‘And why?’, ‘Would you like to do the tasks more often?’, ‘What did you gain the most and/or the least from?’, ‘What do you know now?’, ‘What can you do better now?’, ‘How do you feel about doing the tasks in dyads?’, ‘Could you have done this task alone and by yourself?’, ‘How would you prefer to do it?’, ‘Is there anything you would have liked to change?’), asking pupils about their experiences at the end of the programming sessions. These questions where upfront benchmarked with a focus group of educational research experts. The aim of these interviews is to find out a) whether the application of unplugged SmartGame activities and/or plugged-in SRA-programming can facilitate pupils in their opinion about content development and knowledge gained, b) to find out how pupils stand in relation to the task and working with either SmartGames, Lego EV-3 and Bomberbot, and c) to determine whether pupils have different interesting learning experiences in different settings, and what these experiences contribute to supporting the hypotheses posed in this study.

3.1 Participants

This study was conducted with primary school pupils ranging in age from 8 to 13 years, from grade 5 and 6Footnote 1 (N = 35), from a primary school in the Netherlands, and of which three experimental groups (SmartGames n = 12 / Lego EV-3 n = 10 / Bomberbot n = 13) were randomly composed. None of the participating pupils were familiar with programming beyond the use of basic computer programs such as Word, PowerPoint and the Internet, and none of them had previously taken a CT test. All pupils from the primary school involved were arbitrarily assigned to one of three experimental conditions (SmartGames, Lego Ev-3, Bomberbot). This ensured that each experimental group was a realistic reflection of the population concerned.

3.2 Materials

To answer our research questions, we used three different programming intervention setups which vary in characteristics. These consisted of the application of 1) unplugged SmartGames, 2) Lego EV-3 Mindstorms© robots with tactile output, and 3) Bomberbot© with visual on-screen output.

3.2.1 Unplugged smartGames

As an application for unplugged programming, several different SmartGames are used. These games can be played without the use of a computer and are characterised by required logical thinking and strategic action taking. As a prerequisite, the selected SmartGames should have an increasing level of difficulty and ascending level of abstraction, and should be playable with two players against each other. First of all, we selected a number of educational SmartGames varying in structure and complexity, which are appropriate for pupils in the age group of 9–12 years. Subsequently, in order to determine relevant characteristics with respect to logic, critical and creative thinking, as addressed by Jonassen (2000) (evaluating, imagining, elaborating, evaluating, analysing, connecting), of the pre-selected SmartGames, an educational games expert was consulted who, specifically with the population concerned, determined what the build-up in difficulty and level of abstraction of each game should be, and what transfer could be made from each game to which content-related link to other school subjects. In addition, the findings and recommendations of the game expert were benchmarked with a focus group of experts who helped determine what the structure in application should be. Figure 3 shows the selected unplugged SmartGames.

Fig. 3
figure 3

Selected unplugged SmartGames

Full size image

Based on these three starting points, we determined the selection and sequence of the SmartGames applied. These are games that support learning regarding 1) how to recognise a problem, 2) how to identify the most important details of a problem, 3) how to divide a problem into small, logical steps in order to create a problem-solving process, and 4) to evaluate the learning process afterwards. To promote the development of CT skills, a logical increase of difficulty is a prerequisite and the 'luck element', as often used in games, should be avoided. Also, these games should enable collaborative learning processes. Since pupils play two games in one hour, an additional criterion was that the explanation of a game should not exceed three minutes. Based on these criteria, a game specialist was subsequently consulted to arrive at a final selection of eight games. Subsequently, the finally selected games were played beforehand by an expert-focus team in order to determine which characteristics of computational thinking skills can be promoted by playing each game. Table 1 shows the selected games and renders the allocated computational thinking characteristics for each game.

Table 1 Allocated Computational Thinking characteristics per SmartGame
Full size table

3.2.2 Plugged-in lego mindstorms EV-3

Lego EV-3 Mindstorms robots are used as a tangible programming environment. Characteristic for this visual oriented programming environment is that a program to be executed by a robot is constructed by using the "drag and drop method" by which programming commands must be arranged onto a worksheet in the correct sequence. Each robot is equipped with various sensors (tactile sensor, ultrasonic sensor, colour sensor) that allows the robot to react to changing conditions in the task environment by means of its program to be constructed based on sensor input (e.g. manoeuvring through various labyrinth setups, avoid obstacles, follow a line, etc.). Applied in this way, reactive SRA-programming enables the application of thinking in parallel routines. Figure 4 shows a SRA-programming solution devised using Lego Mindstorms software, and Fig. 5 shows the Lego EV-3 robots with sensors mounted used.

Fig. 4
figure 4

SRA-programming Lego EV-3 Mindstorms©

Full size image
Fig. 5
figure 5

Used robots Lego EV-3 Mindstorms©

Full size image

3.2.3 Plugged-in bomberbot

Bomberbot is a purely visually oriented programming environment that is a simulation of reality. This environment also uses the “drag and drop” programming method to construct a program to be executed by a simulated robot mounted with virtual sensors. This robot must manoeuvre through a labyrinth, solve tasks, smash rubies and open treasure chests. This programming environment also enables the application of SRA-programming by means of thinking in parallel programming routines. Figure 6 shows an example of SRA-programming in Bomberbot.

Fig. 6
figure 6

SRA-programming in Bomberbot©

Full size image

3.2.4 Computational thinking assessment

To determine the level of CT between the pre- and post-assessments, we used the validated Computational Thinking test (CTt) (Román-González et al., 2017). This test makes it possible to generate information on the level of solving CT tasks, the understanding of the computational concepts involved, and the understanding of complex programming concepts (hypotheses 1 and hypotheses 2). All pupils involved in this research completed this questionnaire individually. The questionnaire contained a total 28 items that relate to the various computational concepts involved, i.e. basic directions (28 items), loops (repeat times: 13 items, repeat until: 12 items), conditionals (if-simple: six items, if/else-complex: four items, while: four items) and functions (simple functions: four items, functions with parameters: zero items). The existence or nonexistence of nesting can be derived (19 items). The computational task required (completion: nine items, debugging: five items, sequencing: 14 items) to provide the right solution for each of the 28 questionnaire items can be inferred. To determine the reliability of the scale, we calculated Cronbach's alpha. It should be noted that a value for Cronbach's alpha of 0.70 is considered an acceptable reliability factor (Santos, 1999). The developers of the CTt indicate that for fifth and sixth graders (N = 176), Cronbach's alpha should be α = 0.721 (Román-González et al., 2017). We identified a value of Cronbach's alpha of α = 0.686, which almost complies with the required level of internal consistency for our scale in this particular sample. An explanation for this slightly lower value for Cronbach's alpha can be found in the fact that the research sample size was smaller than that for which the designers of the CTt validated the test. In addition, the age category of the research population included in this study was towards the lower limit of the test, which may be the reason for the lower reliability. Taking this into account, the CTt performed almost as reported by the original authors, and the measurement results obtained were therefore used as such.

3.3 Procedure

As a pre-test assessment (Fig. 2), all three groups completed the CTt. Subsequently, the group that was to use unplugged SmartGames first received a basic explanation regarding the game approach. Furthermore, the application of SmartGames consisted of five one-hour sessions. In each hour, pupils played two games against each other in dyads. After half an hour, pupils switched to a new, more difficult game. A high form of motivation is a prerequisite for playing the games. To stimulate this motivation, an additional competition element was built in. Each pupil noted their own score for each game and gave an indication of how they experienced each game (“a lot of fun”, “boring”, “difficult”). At the beginning of each session, pupils were shown the score everyone had achieved. The fifth session was a final challenge in which pupils, according to the achieved score-ranking, played against each other with the game that had received the highest score. The group that were to program with Lego EV-3 Mindstorms robots received basic instruction and then completed 15 programming tasks, including five final challenge assignments in five one-hour sessions, by applying SRA-programming. The group that was to use Bomberbot completed, after a short introduction, 10 programming missions in five one-hour sessions using SRA-programming, each consisting of 15 programming tasks including challenge assignments.

To analyse the quantitative data collected from the questionnaire survey, a variance analysis (Anova) and Levene's test were initially conducted for all variables to distil preliminary indications and to assess whether equal variances should be assumed. Since specific hypotheses were formulated in this study, a subsequent contrast analysis was carried out to demonstrate possible significant effects and to confirm or reject these hypotheses. To make the magnitude of the effects visible, Hedges g was calculated.

The pre- and post-measurement results from the CTt were entered into SPSS for quantitative data analysis, and the effects of the independent variables on the dependent variables were assessed. The differences in values were determined by comparing the means. In all our statistical analyses, a significance level of 5% (p =  ≤ 0.05) was assumed. The nature of the data met the conditions for the assumption of normality, indicating that the distribution of sample means (across independent samples) was normal. We tested whether assumptions of the homogeneity of variances were violated (p ≤ 0.05). Degrees of freedom were calculated, and a bootstrapping procedure was applied to re-estimate the standard error of the mean difference. Based on statistical analysis, differences between the averages (means) were studied. Subsequently, it was determined whether the 0-value was within the confidence interval. To assess the normality of the variables, the Shapiro–Wilk test was applied (p = 0.766), which was above the chosen alpha level of α = 0.05. Furthermore, the null hypothesis could not be rejected, as there was evidence that the tested data were normally distributed. The histograms also indicated a normal distribution to which could be assumed that all variables were normally distributed. The value for the extent of the effect size (Hedges g) for different sample sizes was calculated (it should be noted that g = 0.2 can be considered a small effect size, g = 0.5 represents a medium effect size, and g = 0.8 indicates a large effect size) (Field, 2013).

4 Results and data analysis

Our main research objective, to determine whether and to what extent unplugged programming activities applied via SmartGames, in comparison to SRA-programming tasks with different types of output (using Lego EV-3 with tangible output and Bomberbot with on-screen output), show a similar development on complex programming concepts resulting in a development on computational thinking, and what aspects of learning emerge by means of such different task environments, is answered by analysing the means for all the variables measured in this research. We aimed to explore whether; (i) unplugged programming applied by SmartGames shows equivalent development on computational thinking in comparison with plugged-in SRA-programming; (ii) programming via unplugged SmartGames activities provide a better understanding of complex programming concepts compared to visual SRA-programming with tangible or on-screen output; and (iii) the application of unplugged SmartGame activities produces more positive identified learning effects comparing SRA-programming with tangible or on-screen output.

4.1 Differences computational thinking development

To provide a structured overview regarding differences in CT development pertaining to the three experimental groups, our study uses a subdivision into CT subcategories. These are derived from the same subdivision established by (Román-González et al., 2017) in the validated CTt used. Table 2 displays the data for each of the CT subcategories regarding the three intervention groups.

Table 2 Differences computational thinking development
Full size table

The analysis of the averages obtained from pretest–posttest assessment (Table 2) shows that applying unplugged SmartGames, in comparison to applying plugged in SRA-programming using Bomberbot or Lego EV-3 showed (a) increased values for solving CT tasks successfully, (b) increased values regarding control over loops, conditionals and functions, (c) increased values for the use of nesting, and (d) increased values regarding sequencing, completion and debugging for the required task application. Both the group that applied unplugged SmartGames as well as the groups that applied SRA-programming with Bomberbot or Lego EV-3 showed higher average scores (M) for all identified variables in the post-assessment. The mean values should be interpreted as being the average number of correctly answered questions per respondent, standardised to a value of zero to one. This progression can also be derived from the calculated percentage values for each separately measured variable, calculated for the three different intervention groups. This percentage calculation was included as such because the three different intervention groups differed in respondent size (SmartGames: n = 12; Lego EV-3: n = 10; Bomberbot: n = 13). The percentage values per sub-category were calculated for the number of items per sub-category divided to the total number of items in the questionnaire, multiplied by the calculated mean (M) (for example calculation pre-test loops “repeat times”: 13/28 × 0.47 = 21.82%). By illuminating characteristic differences between intervention groups by calculating percentages, an objective comparison could be made regarding the effect and impact on CT by applying either unplugged SmartGames or both plugged-in SRA programming environments. For all combined sub-categories, percentages were calculated based on the weighted averages. The category "loops: combined" is an amalgamation of the subcategories "loops: repeat times" (13 items) and "loops: repeat until" (12 items). Since both sub-categories have an overlap of three items, the percentage calculation for the category "loops: combined" is based on a total of 22 items. The category "conditionals: combined" is an amalgamation of the sub-categories "conditionals: if-simple" (6 items), "conditionals: if/else" (4 items) and "conditionals: while" (4 items). Since there are two items that overlap between these three sub-categories, the percentage calculation for the category "loops: combined" is based on a total of 12 items. Grounded on the data obtained, it can be claimed that the application of unplugged SmartGames and plugged-in SRA-programming with Bomberbot or Lego EV-3, equally cause this identified increase. This is despite the fact that for some CT sub-characteristics in the post-assessment only a slight increase in measured values could be observed. Proportionally, all three intervention groups performed better on all measured CT sub-characteristics comparing the pre- and post-assessment of computational thinking.

4.2 Understanding of complex programming concepts

To better understand the impact of the different interventions applied, concerning unplugged SmartGames or plugged-in SRA programming, and whether this led to an increased understanding of complex programming concepts, a contrast analysis on all variables was performed (Table 3).

Table 3 Contrast analysis with a comparison of unplugged SmartGames and SRA-programming for all groups
Full size table

The contrast analysis of the total number of correctly solved CT tasks (28) shows that there was a significant difference, with a large measurable effect, between the groups that applied SRA-programming using Lego EV-3 compared to Bomberbot t (33) = 2.499, p = 0.018, g = 0.936. No significant difference could be measured between the groups that used unplugged SmartGames and applied SRA-programming using either Lego EV-3 (p = 0.233) or Bomberbot (p = 0.200).

A contrast analysis of the differences in the application of “loops: repeat times” shows that there was significant difference with a large measurable effect, between the groups that applied SRA-programming using Lego EV-3 compared to Bomberbot t (33) = 2.122, p = 0.041, g = 0.815. No significant difference could be measured between the groups that used unplugged SmartGames and applied SRA-programming using either Lego EV-3 (p = 0.420) or Bomberbot (p = 0.188).

A contrast analysis of the differences in the application of “functions” shows that there was a significant difference with a large effect between the group that used unplugged SmartGames and SRA-programming using Bomberbot t (33) = –2.300, p = 0.028, g = 0.920, No significant difference could be measured between the group that used unplugged SmartGames and SRA-programming using Lego EV-3 (p = 0.589), and no significant difference could be measured between the group that applied SRA-programming using Lego EV-3 or Bomberbot (p = 0.114).

A contrast analysis of the differences in the required task “debugging” shows that there was a significant difference, with a large measurable effect, between the groups that applied SRA-programming using Lego EV-3 compared to Bomberbot t (33) = 2.430, p = 0.021, g = 0.976. No significant difference could be measured between the groups that used unplugged SmartGames and applied SRA-programming using either Lego EV-3 (p = 0.308) or Bomberbot (p = 0.162).

4.3 Identified learning effects attributed

A qualitative analysis by means of the semi-structured interviews conducted provided additions and further explanations to what our indications from the quantitative data showed. These interviews were conducted after the final work session in which pupils applied either unplugged SmartGames or SRA programming by usage of Lego EV-3 or Bomberbot. Pupils were asked about a) what they would like to share about working with SmartGames / Lego / Bomberbot, b, how they feld about working with SmartGames / Lego / Bomberbot at school, c) what their gain was from working with SmartGames / Lego / Bomberbot, and d) how they experienced working together on the tasks in dyads. Regardless the programming environment used, pupils showed a very high motivation to participate with each of the programming environments offered. The most common feelings pupils expressed when talking about their experiences were those of happiness and joy, and that solving problems made them feel smart. All pupils who applied SRA-programming by using Lego EV-3 robotics or by application of unplugged SmartGames described these as enjoyable and fun. Of the pupils who applied SRA-programming by using Bomberbot, 50% agreed with this opinion, while the rest of this group found it sometimes difficult or boring. Pupils who applied unplugged SmartGames specifically expressed their growth in knowledge and skills, namely: "being able to think about strategies better” (25%), "targeted application of skills in the classroom” (25%), and “being better able to predict what the other person is doing” (25%). The answers of the pupils who applied Lego EV-3 robotics were more general in nature: 70% indicated that they could handle the robot controller better, 20% felt that they could handle programming better. Pupils who applied Bomberbot had the most difficulty putting into words what they had learned. Although 55% said they had learned more about programming in general, however, 29% said they had no idea where or how they had developed. Regarding the question posed to pupils about their experience of working together in dyads, all pupils who applied unplugged SmartGames (100%) perceived it as pleasant and enjoyable, followed by the group that applied Bomberbot (86%). Remarkably, only 50% of the pupils who applied Lego EV-3 robotics expressed the same experience.

5 Discussion

The fundamental premise of our research was to investigate whether and to what extent unplugged programming activities applied via SmartGames, in comparison to SRA-programming tasks with tangible or visual output, show a similar development on complex programming concepts resulting in a development on computational thinking, and what aspects of learning emerge by means of such different task environments.

An examination of the contrast analysis performed shows that the influence of different types of programming environments applied, either unplugged SmartGames or plugged-in SRA programming, can have significant effects on the development of multiple sub-characteristics of CT. Noteworthy are the significant values we have indicated with an * in Table 3.

Further interpretation of the reported results reveals that, depending on the characteristics of the different programming environment applied, a significant development in sub-characteristics of CT can be observed. A plausible explanation can be found in the fact that Bomberbot, characterised by its visual, on-screen output, promotes the use of iterative, cause-and-effect reasoning more effectively by means of its applied task design, and that unplugged SmartGames, through the strategic and logical thinking required as game elements to ultimately be smarter than an opponent in order to win, facilitates the understanding and application of functions. Also, when playing a game, forward-thinking is always required to anticipate what the opponent might do and what steps in the thinking scenario the opponent might have included. This strengthens our conviction that the application of unplugged SmartGames develops more aspects of CT than just programming components. The difference in focus to more directly perceive the impact of the programming intervention in Bomberbot can also explain the significant results found. Bomberbot appears to be more targeted towards handling parallel programming interventions, whereas Lego EV-3 still leaves much room for finding programming solutions by means of deterministic programming. It is also noticeable that Bomberbot incorporates incentives that encourage pupils to keep searching for the most optimal and efficient programming solution, which seems to have a determining effect in the development of sub-characteristics of CT.

From our qualitative findings obtained distilled from the semi-structured interviews, it can be inferred that participants who applied unplugged SmartGames or Lego EV-3 were highly motivated and enthusiastic during all the programming sessions. The level of motivation among participants who applied Bomberbot was lower for half of the participants. From the participants who applied unplugged SmartGames could 75% indicate which (CT) skills they had developed. From the participants that applied Bomberbot could 55% indicate a development on CT, and from the participants that applied Lego EV-3 could 50% indicate this development. In this regard, it is valuable to mention that the formulation of what was learned by the groups that either applied Bomberbot or Lego EV-3 was more general in nature (‘learnt more about programming’), than the group that applied unplugged SmartGames who were able to articulate more specifically their development gone through. Furthermore, participants were mostly very positive about working in pairs. Of the group that applied Bomberbot, 86% said they enjoyed working together, while of the group that applied Lego EV-3, 50% said they positively valued working together. Working together in pairs is a prerequisite for playing the SmartGames. All participants in this group (100%) reported enjoying this collaboration. As a valuable side effect, respondents in this group also noted that they got to know their classmate better.

Our research includes explanations for the observed results on CT that could be established by applying unplugged SmartGames and SRA-programming. Examples include SmartGames to solve complex problems more strategically, tangible robotics to understand iterations better and for debugging CT tasks, and on-screen simulations to generically learn more about programming. Pupils working with these kind of interventions develop demonstrable cognitive skills/tools that enable them to tackle issues transversely.

Only a limited number of studies have been conducted on the effects of SRA-programming and the impact on CT through the application of different types of interventions (Fanchamps, 2021; Slangen, 2016). Consequently, little is still known about the added value of SRA-programming regarding development on CT. Our study seeks to shed more light on this. In part, hypotheses and questions are results of theoretical exercises.

Our findings indicate that the power and possibilities regarding the application of unplugged SmartGames in education should not be underestimated in terms of their contribution to a development focused on computational thinking as claimed by Rodriguez et al. (2017) and Thies and Vahrenhold (2013). The exploratory research conducted shows that growth in a multitude of characteristics related to CT is perfectly possible without the application of digital computer technology as also stated by Brackmann et al. (2017) and del Olmo-Muñoz et al. (2020). Significant results were found on specific sub-characteristics of CT even by applying unplugged SmartGames in a direct comparison with SRA-programming by applying programmable robots or on-screen simulations of reality. This paves the way for learning environments where no digital technology or computer equipment are available, or where it is required to adapt to learners' needs.

Similar to SRA-programming, the application of unplugged SmartGames requires anticipating unforeseen situations which trigger up following actions, and the characteristics of specific tasks in which circumstances can change/vary which asks for more strategically handling. This involves an appeal to imagining what could possibly happen in a SmartGame, how to react to such changing circumstances and to be able to devise generic and specific approaches and solutions in advance. This manifests itself in strategies called for while playing a SmartGame. Herein, the explanation can be found regarding a demonstrable development on all CT characteristics that is closely aligned with the SRA approach as included in this study. In fact, as also stated by Brennan and Resnick (2012) and Denning and Tedre (2019), this goes in parts even beyond developing aspects of CT that relate only to programming and computing.

In addition, we noticed and obtained indications that discussing the qualitative questions asked in the semi-structured interviews in this study may have led to an increased and progressive awareness of the pupils. In our view, besides retrieving research data, this also harbours a pedagogical function of encouraging pupils to articulate from awareness the progression of their own development with increase in skill. Something we successively register as an increase in CT. It would be valuable in follow-up research to further investigate this assumed awareness effect for CT learning.

6 Conclusions

From an examination of the quantitative data obtained, it can be inferred that the group that applied unplugged SmartGames (based on the differences identified between pre- and post-assessment) showed equivalent growth for all variables compared to the application of the two other experimental SRA-programming conditions (Bomberbot or Lego EV-3). This supports our assumption that the application of unplugged SmartGames can also lead to an over-all increase in CT. A further interpretation of the available data shows that there was a notable increase on specific sub-characteristics of CT by application of either unplugged SmartGames, or by application of SRA-programming using the two different visual programming environments.

Comparing the application of SRA-programming with Bomberbot or Lego EV-3, it can be inferred that the application of Lego EV-3 causes a significant and large effect on the development on CT in general. Furthermore, the comparison between the application of SRA-programming with Bomberbot or Lego EV-3 shows that the application of Lego EV-3 is the main culprit in terms of the significant effect found on the application of “loops: repeat times” and for the CT task “debugging”. The application of unplugged SmartGames is found to have a significant effect on the application of "functions" in a comparison with the application of SRA-programming with Bomberbot. It can therefore be assumed that by applying unplugged SmartGames and both visual SRA-programming environments with different types of output (tangible – on-screen), pupils (1) solved more CT tasks correctly; (2) solved more questions correctly which required the application of loops, conditionals, functions and nesting; and (3) showed an increase in the application of required CT tasks (e.g., completion, debugging, and sequencing). This suggests that, like the application of SRA-programming, the application of unplugged SmartGames also contributes to a better understanding of complex programming concepts.

The hypothesis that unplugged programming applied by SmartGames shows equivalent development on computational thinking in comparison with plugged-in SRA-programming can be confirmed. This can be inferred from the increase in the total number of correctly solved CT-tasks and from the substantial progression visible on all measured CT sub-characteristics. It should be noted that, despite an increase on all measured values, a significant development could only be identified for the application of "functions" by application of unplugged SmartGames in a direct comparison to the application of SRA-programming with either Bomberbot or Lego EV-3.

The hypothesis that programming via unplugged SmartGame activities provide a better understanding of complex programming concepts compared to visual SRA-programming with tangible or on-screen output can be confirmed for the value "functions" compared to Bomberbot. The values for "loops: repeat times" and the CT-task "debugging" did indicate increased values for the application of unplugged SmartGames, but proved significant only in the comparison between SRA programming with Bomberbot and Lego EV-3.

The hypothesis that the application of unplugged SmartGame activities produces more positive identified learning effects comparing SRA-programming with tangible or on-screen output can only partly be proven. Our analysis shows that the application of unplugged SmartGames contributes to the (further) development of CT on the basis of their characteristics (strategic thinking, logical reasoning) and ensures a high level of engagement, and working in dyads is also highly valued. Our qualitative analysis also confirms that the pupils' perceptions as distilled from semi-structured interviews corresponded most closely to the development of their CT skills and the opportunities embedded in them as established in the quantitative assessment. It can be inferred that pupils show evidence of having become more aware of the skills we classify as CT skills and which are the meaningful steps for pupils to develop themselves further. Noting that an explanation for this can also be found in the language pupils use when applying SmartGames because it fits games, while the language used when applying Lego-EV3 and Bomberbot differs from everyday language in terms of CT.

7 Limitations and follow-up research

Several limitations and further considerations can be identified regarding the results of this research. Due to these, a certain lack of generalisability of the results obtained should be taken into account.

This study involved a relatively small target group population. Despite being aware of this limited research sample, this study has produced interesting findings that can provide valuable leads for further research. In order to increase the reliability and probative value of the qualitative data, it is recommended to repeat this study with larger groups of respondents.

Due to the age range of the respondents included in this study and the use of the validated computational thinking test (CTt), it is valuable to repeat the study with pupils older than the population included in this study who are towards the lower limit of the test. This is to further increase the validity and reliability of the results obtained.

It may be the case that pupils were already familiar with some of the selected and applied unplugged SmartGames, or may have worked with Lego EV-3 robotics or Bomberbot before. This may have undesirably affected the results identified.

This study used the application of visual programming environments that differ in type of output. It is recommended to investigate whether other types of programming environments generate similar outputs to those currently observed.

The in this research applied quantitative assessment instrument was more attuned to the use of digital programming environments. It would be worthwhile to replicate this research with other CT assessment tools which also adapt more to unplugged programming environments.

It would also be interesting in subsequent research to compare a development on CT by applying unplugged SmartGames with non-SRA-based programming environments.