1 Introduction

Korea’s informatics education started with the proliferation of ICT (Information Communication Technology). As educational informatization develops, the presence of informatics education was used as an index for measuring the informatization of schools [1]. Using ICT in education contributed to enhancing the effects of education itself [2, 3]. In particular, ICT changed the paradigm of education and had a positive infuence, such as promoting collaboration, sharing, participation and self-directed learning [4–6]. Emphasis of ICT is for more efficient teaching and learning, and recently produced the key phrase of ‘Smart Education’ [7].

In this way, the development of IT affected education, and now that wireless personal computing is widespread after continuous evolutions, it is leading changes in education. In particular, informatics has gone through five iterations since the first educational guideline was produced, and informatics education is now preparing for a new takeoff. The first ICT education operating guideline was announed in 2000. It was revised in 2005 with an emphasis placed on the concept and principles of computer science rather than ICT [8]. For the revised curriculum of 2007 the subject name was changed to ‘Informatics’, and the focus of the curriculum was to be on problem solving abilities [9]. The fourth iteration of 2009 revised the curriculum, but ‘Informatics’ lost ground for education in elementary schools, became an elective in middle schools, and became a subject difficult to acces in high schools as it was changed from a general elective to intensive elective. In the fifth revised curriculum of 2015, the focus began to be to foster “creative manpower of the future” with an emphasis on problem solving abilities. In other words, SW education was emphasized, and in elementary schools, ICT, which has been a 12-h unit in practical arts since 2019, was changed to a 17-h basic SW education program. In middle schools, it will be changed from an elective to a required subject longer than 34 h in 2018, and in high schools it will be changed from an intensive elective to a general elective in 2018 [10].

SW education aims to be maker education that will enable students to make something on their own and express themselves. That is, the revised curriculum of 2015 aims to enhance computational [11]. Computational thinking, emphasized for SW education, is presented in different forms in different countries as they define it. For example, the CSTA (Computer Science Teacher’s Association) & ISTE (Information Society for Technology in Education) proposed 10 elements for problem-solving in computational thinking [12]. CAS (Computing at School), in charge of informatics education in the UK, proposed 5 elements: problem decomposition, pattern recognition, abstraction, pattern generalization and algorithm design [13]. As an element of computational thinking, Code.org classified 4 problem-solving elements: abstraction, algorithm, problem decomposition and pattern recognition [14]. Even if different countries define the elements of computational thinking (CT) somewhat differently, it is clear that the purpose of SW education is to develop CT.

Korea is emphasizing SW education in informatics, and in the long run it is operating research schools and advanced schools for SW education in preparation for operation of a regular curriculum in elementary, middle and high schools in 2018 and 2019. Accordingly, this study provided SW education in elementary schools in Korea, and measured the achievement level of SW education. It also aims to extract the factors affecting the achievement level of SW education. The purpose is to show which areas focus must be emphasized to help SW education take root in public education starting in 2018. To achieve the objectives of this study, Chapter 2 analyzed previous CT-related studies, Chapter 3 proposes research methods for conducting this study. Chapter 4 derived research results based on the research methods, and Chapter 5 described the suggestions and conclusion of this study.

2 Background

2.1 Concept and Element of Computational Thinking

CT, emphasized in informatics education, was first mentioned by Seymour Papert [15, 16]. Computational thinking can solve complex large-scale problems with algorithms. It is also used to improve efficiency [17]. The characteristics defined in computational thinking are problem decomposition, pattern recognition/data representation, generalization/abstraction and algorithms. That is, it can produce results by decomposing problems, identifying variables related to data representation, and generating algorithms. If only these characteristics are taken into consideration, it seems that it can be accomplished through programming courses. CT does not simply represent programming, but is related to the entire process of starting with a given problem and solving it through programming.

Google for Education classified CT into four different types at the Computational Thinking Workshop at Nebraska University. In other words, abstract thinking is creating and using different levels of abstraction to understand problems by creating relevant models of the real world. Algorithmic thinking is for seeking and finding the most efficient and effective method for solving a problem. The other two types are logical thinking and scalable thinking. Scalable thinking is said to decompose a large problem into smaller problems that are easier to model and compose complex solutions from simpler algorithms and components [5].

CSTA & ISTE [12] divided the core elements into data collection, data analysis, data representation, problem analysis, abstraction, algorithm formation, procedures, automation, simulation and parallelization based on the result of a study by Barr [6] [12].

It is argued that the basic concept of CT is “to develop the models and simulations for the problems to be solved” [18]. That is, to solve a problem, it is necessary to consider the interactions between several elements existing in the problematic situation and develop a model or simulation with a system.

It is part of the effort to disseminate computational thinking among K-12 students. The ISTE (International Society for Technology in Education) working group selected three domains and developed examples related to computational thinking. These three domains are game design, simulations, and models and robotics. This group proposes a frame for developing students’ CT, which consists of three stages: the use stage, the modify stage and the create stage [19].

In a bid to create the framework of the curriculum for computer science education at the university level, DePaul University’s Perkovic and three others defined the detailed concepts through modification and supplementation based on the 7 principles of computing as defined by Denning which are: computation, communication, coordination, recollection, automation, evaluation and design [20] (see Table 2).

Both Tables 1 and 2 present concepts concerning computational thinking and keywords. CSTA & ISTE approached the concept of CT from the viewpoint of data processing. They have abstractions and algorithms that are necessary for data processing or automation. On the other hand, Table 2 focuses on content that helps lend coherent meaning to the keywords. For example, a typical dictionary definition of the word “communication” means having people think the same or understand each other through verbal or written interaction, but from the viewpoint of computer science, network functions like instant messages, senders/receivers and protocols must be considered. Likewise, the concept of computational thinking is presented but approaches are different. In consideration of the above, for the sake of consistency when CT-related abilities are measured, evaluation questions must be made from the viewpoint of either CSTA & ISTE or Perkovic et al.

Table 1 Elements of computational thinking
Full size table
Table 2 Computer science education for University
Full size table

In a study using Scratch, Brennan and Resnick [21] created three dimensions to evaluate computational thinking (see Table 3) [21, 22].

Table 3 CT dimensions
Full size table

Brennan and Resnick [21] chose Scratch as the target language, and used the CT evaluation method to propose a scenario for project analysis (code analysis), interviewing about works, and design. In particular, they said that the project analysis method can be used to evaluate the concept of thinking.

Discussion about the elements of CT, which can be the objective of SW education and the evaluation method after education, is ongoing. CT cannot be measured as one of the thinking skills, but it is necessary to clarify its components. Accordingly, this study regarded components of CT as evaluation of achievement, rather than measuring of CT.

2.2 Related Work

It can be said that discussion of CT in informatics education started when algorithmic thinking was first discussed. Then the concept was further defined by many scholars. As mentioned above, research on ICT and informatics education began to be conducted in 2007 [8, 23, 24]. This study will focus on CT and present Brennan and Resnick’s research on CT evaluation and SRI PACT.

Brennan and Resnick [21] first developed a tool for measuring computational thinking that can be used as part of the process of improving computational thinking through programming. The tool interacts with the media being used on the premise that design-based learning activities support development of computational thinking [21]. CT evaluation elements can be largely divided into the following three.

The computing concepts include the basic concepts of programming that learners must understand for automation. The practice of computing is a supplemental measure taken because the computing concept cannot explain computational thinking sufficiently. The practice of computing makes it possible to check learners’ learning process, i.e. what they learned and how while they write code (script) which is a product. Learners gradually develop what they understood in the learning process, their relationship with other factors, and various techniques, which is difficult to evaluate using the computing concept.

To evaluate the development of computational thinking by students conducting design activities using Scratch, there are three approaches: portfolio analysis, product-based interviews, and design scenarios. First, if Scratch is used when a learners’ portfolio is analyzed, it is possible to analyze which computing concepts learners used often and which they did not use often. This method has the advantage of enabling a quick evaluation of computational thinking, whereas it is difficult to know the practical aspects regarding how to apply the concepts because of the heavy dependence on learners’ products. Second, learners are interviewed about products. In the interview process, questions are asked to examine how learners’ computational thinking develops. Questions about the background of activities, such as their thoughts about Scratch and what they do currently with Scratch and whether they give help to and receive help from others, and questions about the creative project activities, such as how they got their ideas about their projects and how they started them, are included. The interview method can improve the weakness of using blocks to evaluate computational thinking. The weaknesses of the interview method are that there must be enough time for interviews, and that only what students remembered can be evaluated as the interviews are dependent on students’ memories. Third, the development of computational thinking is evaluated based on the designed scenario. One of the designed projects is selected to be analyzed and how it can be expanded is then explained. Any functional errors found will be modified. This method makes it possible to systematically understand learners’ analysis, expansion, debugging and remixing competency, checks whether they know various computing concepts, and whether they can use the concepts. As we have to wait until learners’ debugging or expansion of activities are completed, however, it will take a long time, which is a disadvantage, and if the subject of a given project is a subject that learners are not interested in, there may be deviations.

As the study conducted by Brennan and Resnick [21] proposed a method for evaluating the development of learners’ computational thinking in the Scratch programming environment, there must be different standards depending on the programming environment.

Second, the Principled Assessment of Computational Thinking (PACT) is a study on computational thinking evaluation conducted as part of the computer science program supported by the National Science Foundation of the U.S. [25, 26]. This study developed the framework for evaluating the learning of computational thinking based on Exploring Computer Science (ECS), a model curriculum for learning computer science [27].

PACT tried to find out what knowledge and competencies learners must learn according to the evidence-based evaluation design, determine which class activities they must conduct, and then make evaluation indexes based on the evidence supporting learning that appeared in the class activities. Computational thinking is not individual thinking that occurs inside the individual, but aims at the characteristics that appear while learners learn knowledge of computer science, competency and attitude, and practice. The practice of computational thinking, proposed in PACT, is largely divided into the concepts of computer science, research competency and noncognitive areas. The concepts of computer science include algorithms, programming, abstraction, debugging, testing, variables and repetition. Research competencies include evaluation, exploration, analysis, explanation, elaboration and modeling. Also, noncognitive competencies include communication, competency, cooperation, leadership, self-efficacy, concept of self and tenacity. Table 4 shows the components of computational thinking proposed in PACT.

Table 4 Components of PACT’s CT
Full size table

Computational thinking can be promoted when the elements of each area are in harmony. For example, let’s say you are analyzing computing work with several people. The concepts of computer science, which are required at this time, are algorithms and programming, and research competencies, i.e. analysis and explanation competency, and noncognitive competencies such as communication competency and cooperation [27].

According to PACT, for learners’ evaluation, the concepts of computer science, research competencies and noncognitive competencies related to what must be learned must all be extracted, the characteristics of what learners say, do and produce must be found, and it must be possible to predict what learners can produce. Also, it must be possible to provide the evaluation situation for accomplishing the goals that learners must reach. It is also necessary to create an evaluation environment consisting of diversified evaluation levels so that difficulty may be adjusted. SRI’s PACT proposed a framework for evaluating computational thinking based on learners’ responses that may appear in the classroom environment.

3 Experimental Method

In the experimental method, how the achievement test tool was developed, how the students’ perception survey was developed, the subjects, and how the test and class were conducted were all described.

3.1 Instrument

3.1.1 Achievement Test

The achievement test measurement tool was composed of four SW competencies: analysis competency, design competency, implementation competency, and inferential competency. Each SW competency consisted of subordinate elements as follows:

Analysis competency was composed of data collection, data analysis, data representation and problem decomposition.

  • Design competency was composed of abstraction and algorithm.

  • Implementation competency was composed of automation.

  • Inferential competency was composed of simulation and parallelization.

The achievement test for the elementary school students, who were the subjects of this study, were presented with 16 questions as shown in Table 5. The category of Analysis competency had four questions (25 %), Design competency 7 questions (43.7 %), Implementation competency one question (6.3 %), and Inferential competency four questions (25 %).

Table 5 Test questions
Full size table

3.1.2 Investigation of Students’ Perceptions

The questions for investigating students’ perceptions of SW education were as follows:

First, attitude toward SW education, second, satisfaction with SW education, and third, the effect of SW education. The reliability of the questions was estimated with Cronbach α. As Cronbach α provides the most conservative estimate, this method is suitable for determining reliability.

As a result of estimating the reliability, the reliability coefficient for each element exceeded .70. So it can be said that the reliability coefficient used in this study is reliable (see Table 6).

Table 6 The questionnaire for students’ perceptions of SW education
Full size table

3.2 Research Process

Here, development of the achievement test tool and survey questions are described.

3.2.1 Procedure for Developing Achievement Test Questions

This study was conducted according to the following procedure (for starters, the procedure for developing the questions for the achievement test is described):

In step 1, for the SW education achievement test, literature review was conducted for CT-related concepts. Based on the concepts derived through the literature review, the 1st expert meeting was held. Participants in the meeting were 15 experts related to contents and evaluation, i.e. five informatics teachers who were teaching SW-related contents, five college professors and five evaluation-related experts.

In step 2, based on what was discussed at the 1st meeting, the framework for evaluating the concepts was established. The developed framework was reviewed by the experts, and used as the framework for developing questions.

In step 3, questions were developed according to competencies and contents. Competencies were: analysis, design, implementation and inferential competency. 16 elements were selected as contents. According to the selected elements, one or two questions were developed. In total, 25 questions were developed.

In step 4, the content validity of the developed questions was tested, and a preliminary test conducted. In the preliminary test, as the subjects were elementary school students, we made sure that the contents were not too difficult to answer. Accordingly, the preliminary test was conducted for 20 elementary school students.

In step 5, what was discovered in the preliminary test was modified, and the reliability of each question was calculated. 16 questions were finally selected.

3.2.2 Survey Development Procedure

The survey questions used in this study were developed as follows:

The elements of the survey were extracted from related studies. The extracted elements were reviewed by the 15 experts who verified the evaluation questions. The content validity was reviewed, and like the achievement test, a preliminary test was administered to 20 elementary school students. As a result of the preliminary test, questions that students did not understand or were likely to be distorted were excluded.

The reliability of the preliminary test was verified, and 20 questions were selected for this test.

3.3 Research Subjects and Process

The subjects of this study was students taking classes in SW research schools, which are funded by the Ministry of Education of Korea. A total of 68 schools were selected as SW research schools from among schools that submitted proposals. Among the selected schools, only elementary schools were chosen for this study.

In other words, of the 68 schools, 45 are elementary schools and 23 are middle schools. This study conducted the research only in the 45 elementary schools. All the students in classes participating in the research school project in the 45 elementary schools are the subjects in this study. Accordingly, a total of 4573 students participated, and among them 4397 students participated in both the achievement test and the survey. Among the participants, those whose responses were inconsistent or provided confusing responses were excluded, so in the end data from 4221 students were analyzed (see Table 7).

Table 7 Subjects
Full size table

The achievement test and survey regarding SW education were conducted on the web from October 26, 2015 to November 3, 2015.

3.4 Designing and Conducting the Class

To conduct this study, research schools were operated, and achievement about SW education was examined.

As research schools conduct the project over a 2-year period, education is provided on an annual basis. Table 8 shows the annual plan for 2015.

Table 8 Annual plans of research schools
Full size table

According to the annual plan shown in the Table, lectures were first given on contents that would help students understand software, then, second, students learned the contents of code.org that helped them experience programming.

Third, students experienced the education programming language (EPL) in earnest. For the EPL, I had students use Scratch for games and storytelling.

Fourth, in the last 3 weeks of the Scratch activities, Python was used for programming.

Fifth, from October to December, Aduino was utilized as well as Scratch to let students experience physical computing. The focus of the physical computing was placed on understanding input and output.

The topics and contents of the Scratch activities conducted in June and September based on the annual plan are as shown in the Table 9.

Table 9 Experience of programming (Scratch_game and storytelling)
Full size table

As the students were new to programming, I gave concrete forms step-by-step from using blocks to storytelling.

I set up the schedule for each stage as described above, and taught classes according to the specific teaching–learning plan.

The course was taught for 1 or 2 h a week for a total of 34 h. The sample teaching and learning course plan for the classes is shown in Table 10.

Table 10 S/W curriculum teaching and learning course plan
Full size table

As shown in Table, the teaching and learning course plan includes evaluation of the unit.

As described above, students directly participated in the class, and various unplugged activities were conducted in addition to robot or Makey Makey activities (Fig. 1).

Fig. 1
figure 1

Robot and Makey Makey activities

Full size image

Also, the students will be able to make samples of the algorithm, and they will be able to draw the flow chart at the beginning of the class. The flow chart is shown in Table 11.

Table 11 Algorithm and coding sample
Full size table

4 Results

4.1 SW Education Achievement Test

The result of the SW education achievement test conducted at the elementary schools was as follows: The average of the SW achievement test was 54.238 points (out of a possible 100 points), the average of analysis competency, a subordinate element, was 13.895 points (out of a possible 23 points), the average of design competency was 27.546 points (out of a possible 47 points), the average of implementation competency was 3.204 points (out of a possible 6 points), and the average of inferential competency was 9.629 points (out of a possible 24 points). Meanwhile, the average of attitude was 72.853 points (out of a possible 90 points) (Table 12).

Table 12 Result of the SW education achievement test
Full size table

As a result of using t-tests to compare the differences in the average scores of SW achievement tests between male and female students in elementary school, female students had higher scores than male students. However, there was no statistically significant difference (see Table 13).

Table 13 Difference in SW achievement by gender
Full size table

As a result of analyzing the SW achievement test scores of elementary school students by grade, 6th graders had higher scores than 5th graders, and the difference was statistically significant at a significance level of .001. That is, as school ages increased, the achievement level also increased (see Table 14).

Table 14 Difference in SW achievement by grade
Full size table

4.2 Languages Used in SW Education

We also analyzed what languages were used during software education in elementary schools, see Table 15.

Table 15 Languages used (multiple responses’)
Full size table

The analysis result shows that 59.3 % of all students responded ‘Scratch,’ and 66.2 % responded ‘Entry.’ 2.3 % responded ‘Java’, and 1.7 % responded they learned ‘C’.

4.3 Correlation Coefficient

As for students’ perceptions of SW education, the correlation among their attitude toward SW, satisfaction with SW education, the effect of SW education, the SW achievement test, and the subordinate elements of the SW achievement test (analysis competency, design competency, implementation competency, inferential competency) were all analyzed, the results are shown in Table 16.

Table 16 The correlation among the attitude toward SW, satisfaction variables and the achievement test
Full size table

As a result of the correlation analysis, satisfaction with SW education and attitude showed the highest level of correlation, i.e. .735, and it was statistically significant. Next, the effect of SW education and attitude had a correlation coefficient of .730. That is, learners who thought SW education was effective can be said to show a high score in attitude as well.

The correlation between the SW achievement test result and the effect of SW education was the highest, .215. That is, it can be concluded that the achievement test score of learners who thought SW education was effective was high.

4.4 The Influence of Students’ Attitude Toward SW on Academic Achievement

Among students’ perceptions of SW education, how much influence such variables as the attitude toward SW, satisfaction with SW education and the effect of SW education had on the SW achievement test was analyzed, the results are shown in Table 17.

Table 17 The influence of perceptions of SW on attitude
Full size table

As a result of analyzing the influence of students’ variables on their attitude toward SW, the perception that SW education was effective had the most influence (β .205) followed by grade (β .134). The influence was statistically significant. That is, the higher the grade and the more positive their perception of the effect of SW education, the higher their achievement level.

What follows is the result of analyzing the influence of students’ perceptions of SW education and attitude on academic achievement (see Table 18).

Table 18 Influence of students’ perceptions of and attitude toward SW education on academic achievement
Full size table

The analysis result shows that satisfaction with SW had the most influence on academic achievement (β .377). That is, it can be concluded that as satisfaction with SW education increases, academic achievement also increases. Next, the effect of SW education had a statistically significant influence on academic achievement (β .302). Grade, gender and attitude also had a positive influence on academic achievement.

5 Conclusions

The purpose of this study is to make students ready for informatics education that will start in middle schools in 2018 and in elementary schools in 2019. In other words, as informatics has become a mandatory subject that must be taught for more than 34 h in middle schools, this study intended to provide SW education and conduct achievement tests to find the variables affecting academic achievement in preparation for when the subject is offered as a part of the regular curriculum. The class was offered to 4221 elementary school students, and their attitude toward SW education, satisfaction with SW education and its effect were analyzed. The results were as follows:

First, there was no statistically significant difference in achievement by gender, but the achievement level of female students was higher.

Second, the higher the grade, the higher the achievement level.

Third, the result of the correlation analysis showed that learners’ attitude was highly correlated with achievement and satisfaction. In particular, satisfaction also had a high level of influence on academic achievement.

Countries around the world have tried to reinforce informatics education since 2010, and many countries like Japan, the U.S., the U.K. and India revised their curriculums to reinforce SW education. Korea also began to make efforts to reinforce SW education through its revised curriculum of 2015. However, what is important in education is to make people perceive the necessity of education. That is, it is necessary to help students perceive the necessity for themselves, and display their competency as makers. In particular, we must not forget that fostering makers is not fostering technicians, but that it is education for improving self-expression.

The difficulties experienced during this study can be summarized as follows:

First, teaching and learning methods need to be changed. A great difficulty that was encountered during the class was that though students have a high interest in SW education there was a difference in level among the students. That is, if there are 25 or more students in a class, teachers cannot properly conduct the class well if there are too many coding mistakes. In programming classes, teaching assistants are required, but schools cannot provide them. If teachers fail to take the level of individual students into consideration when they teach classes, it will be necessary to find a solution for students who fall behind. For example, teaching and learning methods need to be changed, which might lead to, for example, developing specialized learning modules for different levels.

Second, if the level of satisfaction with SW education was high, the achievement level was also high. Students’ satisfaction encompasses various aspects, including the infrastructure, teachers’ expertise and cooperative learning. This means that if any one element is lacking, the satisfaction level may be lowered. Accordingly, it is necessary in the future to further clarify what kind of support teachers must provide, and what kind of efforts the office of education is required to make.

Third, programs need to be diversified. The languages that were used most frequently in elementary schools during this study were Scratch® and Entry®. All the other languages were used by less than 10 % of the students. To ensure that SW education will be more effective and students able to display their competencies as makers, not only block-structured languages, but also textual languages must be accessible at the same time. This study targeted elementary school students, but according to the revised curriculum of 2015, middle schools are also using block-structured languages. That is, we must take into consideration the fact that accessibility to textual languages may be decreased. Accordingly, if SW education is provided, it is necessary to clarify what the purpose of it truly is. For example, we must make efforts to select programming languages in consideration of whether SW education is simply to arouse the interest of students or to reinforce their competency as makers.

This study analyzed the influence of attitude, including satisfaction with SW education, on academic achievement in pilot schools with public SW education in the offing. As many countries have become aware of the necessity of SW education, if there are countries providing education similar to that of Korea, they should learn from this study.