Improving Grade 7 Students’ Achievement in Initial Algebra Through a Technology-Based Intervention

Abstract

Digital technology plays an increasingly important role in daily life, mathematics education and algebra education in particular. To investigate the effect of a technology-rich intervention related to initial algebra on the achievement of 12–13 year old Indonesian students, we set up an experiment. The experimental group’s intervention focused on equations in one variable and is characterized by an alternated use of paper-and-pencil and digital work, and by the intertwining of word problems and bare algebra problems. The control group was taught in a regular way without digital tools. Students from eight classes in four schools took part in a pre-test and a post-test. The results showed that the experimental group’s (n = 131) gain score was significantly higher than the control group’s (n = 119) score, with a medium effect size. Also, a school factor was found to affect student achievement. The qualitative analysis of student written and digital work during the teaching experiment corroborated the quantitative results. Both results confirm the effectiveness of this type of technology-rich intervention for enhancing student achievement in algebra.

Introduction

Algebra is a core topic in secondary school mathematics curricula and is recognized as a gateway either to advanced study or to professional work in today’s society (Katz 2007; Kendal and Stacey 2004). Mastering algebra is crucial for students’ futures all over the world. Indonesia is no exception to this. The 2007 Trends in International Mathematics and Science Study (TIMSS), however, showed that Indonesian students scored low on algebra and were in 36th position out of 48 participating countries (Mullis et al. 2008). Similarly, in the 2011 TIMSS, Indonesian students were ranked 38th out of 42 countries in the domain of algebra (Mullis et al. 2012). This raises the question of how to improve Indonesian student performance in the algebra domain. What are possible approaches to enhancing students’ conceptual understanding and skills in algebra, and how effective would such an approach be?

Information and Communication Technology (ICT) plays an increasingly important role in daily life, in education, and mathematics education in particular. The US National Council of Teachers of Mathematics (NCTM), for instance, claimed that, “technology is an essential tool for learning mathematics in the 21st century, and all schools must ensure that all their students have access to technology” (2008, p.1). Also, there is research evidence that the use of ICT can have positive effects both on students’ mathematics achievement (Li and Ma 2010) as well as on their perception of mathematics (Bakker et al. 2015; Barkatsas et al. 2009) and can support students in both exploring and expressing mathematical ideas (Ghosh 2012). In algebra education, ICT use contributes significantly to its learning and teaching (Rakes et al. 2010). For example, use of digital tools in algebra education can promote students’ development both of symbol sense and of procedural skills (Bokhove and Drijvers 2010b), can be effective for improving conceptual understanding and procedural skills of secondary school students (Bokhove and Drijvers 2012) and may foster the development of the notion of function (Doorman et al. 2012). Furthermore, use of a digital environment can support students’ mathematical problem-solving skills and can contribute to their ability to solve informal algebra problems (Kolovou et al. 2013; Van den Heuvel-Panhuizen et al. 2013).

In response to the worldwide use of technology in education, the Indonesian Ministry of National Education set up a policy that introduced ICT as a new subject for secondary schools (Depdiknas 2007). Furthermore, the accompanying curriculum documentation proposed integrating the use of ICT in all school subjects, including in mathematics. Even if ICT is not a panacea for all problems, and its impact is subtle and dependent on the learning environment (such as technological infrastructure, task design and the didactical approach), this integration is expected to enhance the quality of the learning and teaching of mathematics, and to improve student achievement in particular. However, the integration of digital tools in mathematics teaching is relatively new in Indonesia (PPPPTK Matematika 2013). As a consequence, the potential of ICT for enhancing the quality of mathematics education in Indonesia is still unexploited.

The Context of the Study

To understand the context of the study, we first provide some background information about mathematics education, teacher education, teaching practices and the use of technology in Indonesia. As a developing country, Indonesia is struggling in enhancing the quality of its education, as is reflected in the low TIMSS scores mentioned above. In international comparative studies, Indonesian student performance was in the lower positions.

To explain the low Indonesian results for mathematics and algebra on international comparative tests, we have proposed two hypotheses (Jupri et al. 2014a). First, in spite of mathematics curriculum revisions over the last decades, most mathematics teaching seems to be still traditional (e.g., Sembiring et al. 2008), in the sense that ‘drill-and-practice’ approaches are prevalent, including memorization of algebraic formulas. As such, there is a discrepancy between the intended and the implemented curriculum. Second, Indonesian students immediately start to learn algebra in a formal way in the first semester of grade VII (Depdiknas 2006), while the students did not learn any algebraic topic at the primary school level. As a consequence, they are not prepared to study algebra. According to Mohandes (2000), low Indonesian performance in the TIMSS studies in both science and mathematics are caused by student absenteeism and the large number of subjects taught in school.

Teacher education in Indonesia consists of a 4-year bachelor program. For mathematics teacher education, the content of this program includes mathematics courses (70 %), didactics courses (20 %) and general courses, including a 6-month internship in a school and a bachelor thesis. Prospective secondary school teachers in the mathematics education program obtain limited training for the use of digital technology in courses with titles like Multimedia in Mathematics Teaching. Practicing mathematics teachers may receive such training from the center of in-service teacher training (PPPPTK¹). With a bachelor degree in mathematics, one can also become a licensed mathematics teacher after a 1-year teacher training program.

Concerning Indonesian teaching practice, there are 35–45 students in an average class. Due to this large class size, class management is an important issue for teachers. Teaching approaches which are prevalent in Indonesian classrooms are teacher-centered (Sembiring et al. 2008). In mathematics lessons, for instance, the common lesson structure is as follows: the teacher explains the mathematical concept under consideration, next s/he gives some worked examples relating to the concept, then provides an exercise, and finally closes the lesson and provides homework; students listen, take notes and do exercises.

As a consequence of ICT being introduced as a new compulsory subject for secondary schools in 2007, computer laboratories are now available in most secondary schools. However, the integration of ICT into other school subjects, such as mathematics, is not yet mandatory (Depdiknas 2007). This means that computers are mainly used for the ICT subject itself. Other subjects can only use the computer laboratory to a limited extent. In mathematics education, calculators are not permitted in courses and in formative and summative tests; nevertheless, digital tools are occasionally used in the lessons. This limited integration of ICT in education is in contrast with what is happening in Indonesian society. As the fourth most populated country in the world, Indonesia is a big market for various technological products, for instance, from Japan, South Korea, China, European countries and the United States. Furthermore, more than 80 million Indonesians are now accessing the internet (Kemkominfo 2014) for various purposes, such as for mobile communication with messenger services and for social medias.

Theoretical Framework and Research Question

The theoretical framework of this study concerns both difficulties in initial algebra and the role of ICT in algebra education.

Difficulties in Initial Algebra

As a first step before being able to improve algebra education, we should have a clear view on what is hard in initial algebra. Therefore, based on an earlier literature review study (Jupri et al. 2014a), we identified five categories of difficulties in initial algebra. First, difficulties in applying arithmetical operations in numerical and algebraic expressions (abbreviated as ARITH) concern difficulties in adding or subtracting similar algebraic terms (e.g., Herscovics and Linchevski 1994; Linchevski 1995), difficulties in applying the associative, commutative, distributive, and inverses properties, and difficulties in applying priority rules for arithmetical operations (e.g., Booth 1988; Bush and Karp 2013; Warren 2003).

Second, difficulties in understanding the notion of variable (which we term VAR) include different views on the different roles it can play: the role of a placeholder, a generalized number, an unknown, or a varying quantity (Booth 1988; Bush and Karp 2013; Herscovics and Linchevski 1994).

Third, the difficulties in understanding algebraic expressions (we term this AE) include parsing, the ‘expected answer’, the lack of closure and the gestalt view of algebraic expressions (Arcavi 1994; Thomas and Tall 1991).

Fourth, the difficulties in understanding the different meanings of the equal sign (we term this EQS) concern difficulties in dealing with the equal sign, which usually invites a calculation in arithmetic, while it is a sign of equivalence in algebra (Bush and Karp 2013; Herscovics and Linchevski 1994; Kieran 1981).

The fifth and final category is that of mathematization (termed MATH), which concerns the difficulty of transforming any kind of reality in the world of the problem situation to the world of mathematics, and vice versa, and to reorganize and to (re)construct the symbolic world of mathematics (Treffers 1987; Van den Heuvel-Panhuizen 2003). By ‘reality’, we mean real life, but also mathematical situations that are meaningful and imaginable in mind (Freudenthal 1991; Gravemeijer 1994; Van den Heuvel-Panhuizen 2000, 2005; Van den Heuvel-Panhuizen and Drijvers 2014). The activity of transforming a realistic problem into a symbolic mathematical problem through observation, experimentation and inductive reasoning is called horizontal mathematization, while the activity of reorganizing and (re)constructing within the world of symbols which includes solving the problem, generalization of the solution and further formalization is called vertical mathematization (Treffers 1987; Van den Heuvel-Panhuizen 2003). These two activities are complementary during the process of mathematical activity (De Lange 1987).

The aforementioned five categories serve as a point of departure for analyzing observable student difficulties in learning of algebra. To understand and explain the background of these difficulties better, in a further study we used the lens of operational and structural views on algebraic activity (Jupri et al. 2014b). This lens originates from Sfard’s theory of reification, that is, a transformation of a process performed on an accepted mathematical object to become a new object. According to Sfard (1991), an abstract notion, such as an algebraic expression, can be perceived in two different complementary ways: operationally as a process and structurally as an object. For example, the equation $\frac{x-3}{5}+4=11$ can be conceived operationally as comprising a sequence of processes of arithmetical calculation, i.e., subtract 3 from a certain given number (x), next divide by 5, and finally add 4 to get 11, and it can be seen structurally as an equivalence between two objects, namely the algebraic expressions $\frac{x-3}{5}+4$ and 11. According to Drijvers (2003), flexibility in switching this process and object view of algebraic expressions signifies a mature understanding of algebraic thinking. In this study, this structural–operational duality is used to understand student activity while solving equations, as well as the difficulties encountered while doing so.

The Role of ICT in the Learning and Teaching of Algebra

In which way might the use of ICT contribute to the learning and teaching of algebra? In responding to this question, Drijvers et al. (2010) distinguish three didactic functions of technology in algebra education: as providing a tool for doing mathematics, as offering an environment for practicing skills, and as presenting an environment for developing algebraic concepts. With the first function, technology acts as an assistant to carry out algebraic routine procedures, such as expanding algebraic expressions and drawing graphs, and the user does not necessarily know and understand how the technology produces the outcomes, but may be prompted to find this out. The reconciliation of ICT-tool techniques and paper-and-pencil methods may be a particular educational challenge (e.g., Kieran and Drijvers 2006). In practice, while doing algebra, a student can take the initiative as to whether or not to use the technology to carry out the routine procedural work: s/he is probably able to carry out the routine procedures by hand, but may opt not to spend energy on that and to outsource the work to the technology (Drijvers et al. 2010).

As an environment for practicing algebraic skills (the second function of technology in algebra education), technology may offer immediate feedback to students’ responses, solutions and strategies (e.g., Bokhove 2010). With this function of technology, randomization of tasks may allow for variation and to avoid repetition (e.g., Bokhove and Drijvers 2010b). Moreover, the compatibility of problem-solving strategies in the technological environment and in the paper-and-pencil environment is crucial for transfer of notation and skill to the latter environment (e.g., Bokhove and Drijvers 2010a). In practice, a student can determine by her-/himself, depending on her/his mastery, when to use technology or not for practicing (Drijvers et al. 2010).

With the third function (as an environment for developing concepts and mental models), technology aims to evoke a specific thinking process and to guide the development of algebraic thinking. In doing so, the technology, for instance, helps to visualize a concept of an equation and to present it in a dynamic way (e.g., Drijvers and Barzel 2012) or to generate various examples for provoking exploration and generalization (e.g., Kieran and Drijvers 2006). According to Beeson’s glass box principle (1998), the transparency of the representations and techniques of the ICT environment is crucial for fostering conceptual understanding, because it provides an opportunity for students to perceive how the technology produces mathematical outcomes. In practice, this didactical function is guided by the teacher, as this function of technology requires a careful didactical analysis of the relationship between the use of the tool with its representations and techniques on the one hand, and the mathematical thinking and skill that the students are supposed to acquire on the other (Drijvers et al. 2010).

For the purpose of this study (i.e., improving student achievement in initial algebra), the use of technology is devoted to the second and the third function: practicing algebraic skills and developing algebraic concepts. Ideally speaking, these two functions go hand in hand and are supported by ICT in an integrated way: conceptual understanding underpins the acquisition of skill, and the mastery of procedural skills, in turn, may strengthen conceptual understanding. Therefore, we look for tasks involving ICT tools in which the representations and actions are closely related to the targeted conceptual development. Next, problem-solving activity should be routinized so as to foster procedural skills. As we describe in “Applets” sub-section these considerations guided the choice for specific applets and for specific tasks.

Research Question

This study addressed the following research question:

• Does an intervention with digital technology enhance students’ performance in initial algebra?

This general research question was further specified as follows:

• The intervention in this study concerns a teaching arrangement in which digital work and paper-and-pencil work are used alternately (a blended approach). Also, bare algebra tasks and word problems are used in an intertwined way, rather than addressing applications at the end of the learning process.

• The digital technology in this study is the Digital Mathematics Environment (DME) which was developed at the Freudenthal Institute, Utrecht University, the Netherlands. Our work involves four applets in particular, called Algebra Arrows, Cover-up Strategy, Balance Model and Balance Strategy. These four applets provide opportunities for both concept development and procedural work, and are described in more detail in “Applets” sub-section.

• The domain of initial algebra addressed in this study includes a central topic in this domain, i.e., equations in one variable and related word problems, which is in line with the content of the Indonesian mathematics curriculum. In this curriculum, algebra is introduced to grade VII students (12–13-year-olds). The algebra topics in this grade include linear equations and inequalities in one variable, simplifying algebraic expressions, proportions, and sets (Depdiknas 2006). At the primary school level, students were taught arithmetic and geometry, but no algebra. Therefore, the topic of linear equations in one variable is new for the participants in this study.

As a conjecture, we hypothesize that students who experience an intervention with the digital technology will outperform their peers who are engaged in regular learning settings. In addition to this, we wonder whether a school factor may play a role. Even if the role of the teachers is important in the intervention, the focus of this study concerns both the alternate and integrated use of digital technology and paper-and-pencil, and the task design, i.e., the balanced use between bare problems and word problems.

Methods

To investigate the influence of the ICT-rich intervention on students’ algebra performance in the topic of equations in one variable, a pre-test–post-test–control group experiment was set up. In this section, we address the sample, the instruments and the learning environment, the intervention procedure, and the data collection and analysis.

Sample

In total, 266 grade seven students (12–13-year-olds) were involved in this experiment: 139 students were in the experimental group and 127 students in the control group. The experimental group included four classes from two schools (two classes from each school), and the control group also included four classes, from two other schools (also two classes from each school). The schools were all located in one of the regencies – including suburban and rural areas in a balanced manner – in the Central Java province, in Indonesia. All schools came from similar socioeconomic backgrounds, and followed the same Indonesian mathematics curriculum.

The Indonesian education authorities have a school accreditation system, according to which schools are graded. The highest grade is A (excellent). In the case of the religious schools in this study, schools with grade A are usually have a good reputation because they have various forms of academic and non-academic achievement and are chaired by respected clerics. As a consequence, parents tend to send their children to A-certified schools. From the schools in this study, one school in the experimental group and one school in the control group were A-certified schools and the other two schools were not. We assume that the A-certified schools and the non-A-certified schools are representative of other schools, as the certification is officially conducted using the same criteria. So the classes at A-certified schools are expected to be representative for other A-certified schools, not for all Indonesian classes. Taking this school categorization into consideration, which from here onward is called the school type, we wondered whether it too might influence student achievement.

The background of classifying the school type into A-certified and non-A-certified schools is as follows. Initially, we classified the schools as respectively favorite and non-favorite schools. A favorite school, according to people’s perception, is a school that has a good reputation because, inter alia, it is chaired by a respected cleric (an influential figure in the community), has various academic and non-academic forms of achievement, and has good infrastructure and facilities. However, this categorization is difficult to measure. Therefore, we used the accreditation grade which is officially used as a measure of school quality in Indonesia.

Instruments and Learning Environment

The learning arrangement consisted of four applets in the DME that come with digital tasks embedded for the learning of equations in one variable, the daily intermediate paper-and-pencil assessment tasks, and the paper-and-pencil tasks for the pre-test and post-test. A teacher guide informed the teacher about this learning environment and the tasks to be undertaken.

Applets

The designed learning sequence included work with four applets: Algebra Arrows, Cover-up Strategy, Balance Model and Balance Strategy. These four applets are embedded in the DME, which is a web-based electronic learning environment. The DME, and the four applets in particular, have been developed by Peter Boon at the Freudenthal Institute (Boon 2006). The DME provides: (i) interactive digital tools for algebra, geometry and other domains; (ii) a design of open on-line tasks and appropriate immediate feedback; (iii) access to the environment at any time and place, as long as technological infrastructure and conditions are met; (iv) storage for student work (Drijvers et al. 2013). According to four groups of criteria (algebra didactics, theories on tool use, assessments, and the general characteristics of digital tools), the DME is considered suitable for research in algebra education addressing the co-emergence of procedural skills and conceptual understanding (Bokhove and Drijvers 2010a). The DME is also found to be suitable for algebra education because of the mathematical soundness, i.e., the correct display of algebraic notations, the ease of use and the storage of student work.

The Algebra Arrows Applet

The Algebra Arrows applet is designed to offer students the possibility of constructing and using chains of operations on numbers and algebraic formulas, and as such to foster students’ view of function as an input–output chain of operations representing a dependency relationship (Doorman et al. 2012). If f is a given function, f(x) = c represents an equation in one variable and it can be interpreted as: which input value in the chain of operations defined by f provides $c$ as an output value? Figure 1 shows how the equation $\frac{a-9}{7}=10$ is solved using this interpretation and the Algebra Arrows applet. Solving an equation through this applet, then, comes down to applying a reverse strategy, that is, a process of undoing a series of operations. As such, in line with the operational view of algebraic expressions, this applet invites students to see an equation as a series of operational processes (Jupri et al. 2014b).

Fig. 1

Equation solving with the Algebra Arrows applet applying the reverse strategy

The Cover-up Strategy Applet

The Cover-up Strategy applet was designed to solve equations in one variable of the form f(x) = c through global substitution (Wenger 1987), by subsequently selecting a part of the expression in an equation with the mouse and finding its value. Figure 2 shows a scenario of using the Cover-up applet to solve the equation $\frac{18}{5a-2}=6$. In step 1, a student follows the first hint provided, namely choosing a part of the equation that should be highlighted first. In step 2, the student highlights the expression 5a − 2 and the expression 5a − 2 = ⋯ appears automatically in the next line. In step 3, the student fills in 3 and the applet gives a yellow tick mark signifying a correct action (otherwise a cross mark emerges). This scenario continues until step 6 and ends with a = 1 as the solution of the equation (signified by a green tick mark and the final feedback: “The equation is solved correctly!”). In practice, a student does not necessarily follow all these six steps, but may also take shortcuts, such as going directly to step 5 after step 3.

Fig. 2

An equation-solving scenario using the Cover-up Strategy applet

In applying the cover-up strategy, a student should first see the equation as an equivalence of two objects (algebraic expressions). Next, s/he should be able to identify a sub-expression within the equation that is to be covered and will be assigned a numerical value. In this way, the strategy seems to invite students to develop a structural view of equations and expressions (Jupri et al. 2014b). This is the main reason for using the Cover-up Strategy applet after the Algebra Arrows applet. Another reason is that the Cover-up applet can be used to solve a wider repertoire of equations than just linear ones, which is not the case for Algebra Arrows applet.

The Balance Model Applet

The Balance Model is an applet that provides equations and the corresponding virtual dynamic models, which can be used for solving the equations. In this case, it works with linear equations in one variable of the form f(x) = g(x), i.e., equations with the variable appearing in both sides of the equations. Mathematically, this type of equation is an extension of linear equations of the form f(x) = c addressed in the tasks drawing on the Algebra Arrows applet and the Cover-up Strategy applet. Therefore, work with Balance Model applet was placed after that using the previous two applets. Figure 3 shows an equation-solving scenario with the Balance Model applet. In step 1, given a balance model, a student is required to write an equation that represents the model. In step 2, the student moves one block (representing the value 1) as an initial step to solving the equation 4x + 1 = 2x + 23. Step 3 represents a condition in which the student has moved a block of value 1 from each side of the equation. This scenario proceeds until step 5 which ends with x = 11 (signified by the emergence of feedback: “The equation is solved correctly!”). Again, a student does not necessarily follow all these steps consecutively and s/he may find shortcuts. The students are expected to perceive solving the equation both as a process of applying the same operation to both sides and as an equivalence of two algebraic equations; operational and structural perspectives are thus intertwined.

Fig. 3

An equation-solving scenario with the Balance Model applet

The Balance Strategy Applet

The Balance Strategy applet can be used to solve linear equations in one variable of the form f(x) = g(x) using strategies that have been developed with the Balance Model applet – namely, do the same operations to both sides of the equations – without providing models in the solution processes. As a design decision, this applet provides an abstraction of the Balance Model applet, and is therefore to be used after it. Figure 4 shows how the equation 5x − 11 = 2x + 13 is solved by means of the Balance Strategy applet. Equations may be provided in the task or can be set up by the student from word problems. Similar to the case of the Balance Model applet, both operational and structural views of equations play an important role in this applet.

Fig. 4

Equation solving with the Balance Strategy applet

Types of Tasks

The tasks addressed in the intervention consisted of two types: bare problems and word problems. Bare problems are tasks that are not related to contexts either within mathematics or other subjects, such as 5x − 11 = 2x + 13, whereas word problems are. To develop an integrated and balanced view of the topic, and not to consider word problems at the end of the teaching sequence only, students work alternately on these two types of problems. Of course, throughout the intervention, these problems are ordered from relatively easy to more difficult within each type.

Daily Intermediate Assessment, pre-Test and Post-Test

The daily intermediate assessment was carried out for 15–20 min in each of four lessons, in which each student was required individually to write down and show his or her solution on paper. The reason for using paper-and-pencil work was that students in Indonesia are not allowed to use technology in formative and summative tests. Table 7 (Appendix 1) presents the tasks used in the daily intermediate assessments at the end of each of the applet tasks.

The tasks used for the pre-test and the post-test were the same. However, we used some different words or phrases in the questions, such as: rather than using the word “equation”, we used the word “an expression” in the pre-test, and rather than using the phrase “find a value” we used the phrase “find a solution of …” in the post-test. The pre-test and post-test were each administered in 60 min, during which time each student was required to write down and show his or her solution on paper individually. Table 8 (Appendix 2) presents the tasks used in the pre-test and post-test. Cronbach’s alpha for the pre-test was α = 0.65 (acceptable); for the post-test it was $\alpha =0.81$ (good).

Intervention Procedure

The intervention for the experimental group included: (i) an individual 60-min paper-and-pencil pretest; (ii) four 80-min lessons on equations in one variable, partly via whole-class teaching and partly in groups (of 3–5 students of mixed ability), in which the students work on the designed tasks making use of the four applets described above; (iii) an individual 60-min paper-and-pencil post-test, similar to the pre-test. Each of the four lessons ended with a daily intermediate paper-and-pencil assessment after each use of the applet. Even if the results of these daily intermediate assessments in this study did not contribute to students’ final grade, as they were not reported to the teachers, students worked seriously on these tasks. Probably, they expected the results would contribute to their grade, as is usually the case in similar situations.

The control group worked on a 60-min paper-and-pencil pre-test, attended the regular teaching on equations in one variable without digital technology but including exercises and daily tests, and a 60-min paper-and-pencil post-test, similar to the pre-test. In the control condition lessons, the teacher explained the concept of equations in one variable with the corresponding examples and provided exercises, while the students took notes and did the exercises. The control condition and the experimental condition shared the topic, whole-class instruction by the teacher, and the daily written assessment; the differences concerned the intertwining of bare algebra tasks and word problems – in the control condition word problems were used as applications of the concept at the end of the learning process – and the alternation of working with the applets and working with paper-and-pencil.

In total, the extent of the intervention included six meetings: two in which the students were tested and four lessons (see Table 1). The work in each of the four lessons consisted of three parts: a paper-and-pencil task, a digital task, and a paper-and-pencil daily intermediate assessment as well as reflection. The teacher organized these three parts according to the teacher guide. In the paper-and-pencil work, the teacher posed problems and guided whole-class discussion. In the digital work, the teacher demonstrated how to use the applets, guided students into the group digital tasks, and led discussion. While the teacher demonstrated techniques in the use of an applet, students (in group) followed the demonstration. For accessing the DME during group work, each group was given a unique account so that the digital work was stored and could be retrieved either by the teacher or by the researcher for the analysis. Next, students were requested to individually do the paper-and-pencil daily intermediate assessment tasks. Finally, the teacher guided students to reflect upon the lesson.

Table 1 Intervention set-up

In carrying out the teaching intervention, the teachers were supported by a teacher guide provided by the researcher. The teacher guide contained teaching strategy steps (introduction, demonstration, digital activity and daily intermediate assessment), problems to pose in the whole-class discussion with corresponding solutions and predictions of student responses, problems using the applets with their corresponding answers, problems for daily intermediate assessments and their corresponding solutions, and a guide for accessing the DME and the applets. This teacher guide was explained to the teachers by the researcher prior to intervention. The teachers in the intervention classes had already been involved in the small-scale pilot experiment with the Algebra Arrows and the Cover-up Strategy applets in the previous year with other students (Jupri et al. 2014b; Jupri and Drijvers, accepted). In the event that the teachers encountered technical obstacles during the demonstration of the applets, or during the digital group activity, an external research assistant was available to provide help.

During the intervention lessons, the researcher, while videotaping one group of students in each class for the purpose of data collection, helped these students by acting as a substitute teacher, while the teacher took care of the other groups of students. In this way, each group received appropriate guidance during the learning process. The group that was videotaped in each class was based on the teacher’s recommendation, i.e., consisted of a mixed-ability group students who were communicative in front of a camera during the intervention. One observed group consisted of female students only, while the other three observed groups were all male. Even if female and male students were mixed in experimental classes, there were no mixed-gender groups – this is common in religious schools in which the teachers decide on group composition.

Data collection

The data that were collected from each class in the control group consisted of individual written student work from the pre-test and the post-test. Of the 139 students in the experimental group 131 students did both pretest and posttest; and of the 127 students in the control group, 119 did both pretest and posttest. In addition, the data that were collected from each experimental group consisted of video recordings of one group, student digital work, student written work from four daily intermediate assessments and field notes. Table 2 provides an overview of collected data.

Table 2 Overview of data

Data Analysis

To analyze the quantitative data, we used statistical methods with the help of SPSS software. First, as the tasks in the pre-test and post-test were the same, we used gain scores (the difference between post-test and pre-test scores) as the dependent variable. The Kolmogorov–Smirnov and Shapiro–Wilk tests showed that the gain scores were not normally distributed (p <.01). However, as the sample size was large and the P–P and Q–Q plots did not show crucial variations, we nevertheless applied parametric techniques to analyze the data, including $t$ -tests and ANCOVA (Field 2009). A further exploration of the collected data showed that there was a significant difference between the pre-test scores of the experimental and the control group (t(199) = 5.93, p <.001). Also, there was a significance difference in the pre-test scores for A-certified and non-A-certified groups (t(248) = 2.26, p <.05).

Second, using $t$ -tests for two independent samples, we compared the gain scores for the experimental and control groups. Third, an ANCOVA was carried out with school type (i.e., A-certified and non-A-certified schools) as a covariate. Fourth, to investigate relationships between achievement in the daily intermediate assessments, the pre-test and the post-test, a correlation analysis was carried out. Finally, a descriptive qualitative analysis on written and digital student work, as well as observations during the lessons in the experimental condition, was carried out with the help of Atlas.ti software so as to complement the aforementioned quantitative analysis.

Results

This section addresses the results of both quantitative and qualitative analyses. The quantitative results, provided in “Overall comparison between experimental and control group”–“Correlations between daily intermediate assessment, pre-test and post-test” sections, include a comparative analysis of gain scores between the control and experimental groups; the effect of school type on student achievement; the correlation between the results of daily intermediate assessment tests, pre-test and post-test scores. The descriptive qualitative analysis, presented in “Illustrative student work during the intervention” section, includes an analysis of student written work, which is backed up by the observation data collected during the intervention.

Overall Comparison Between Experimental and Control Group

Figure 5 shows a bar graph of the mean gain score for each class of the experimental and control groups. The graph shows that the classes in the A-certified schools had better gain scores than those in non-A-certified schools, and that within the A-certified schools and the non-A-certified schools the classes in the experimental group benefited more from the intervention than the classes in the control group.

Fig. 5

Mean gain score for the experimental and control classes

Table 3 provides the descriptive statistics of the mean gain scores and the result of the t-test for independent samples grouped by condition. The results show that the mean gain score of the experimental group was significantly higher than that the control group (Cohen’s d = 0.32). As such, the hypothesis that the experimental group would outperform the control group was confirmed by the data.

Table 3 Mean gain score in experimental and control groups, t-test results and effect size

The Effect of School Type

A-certified schools have a selective admission procedure and, as a consequence, have better-qualified students than non-A-certified schools. Even if better-qualified students are not necessarily good in mathematics, they are often highly motivated in the learning process. As such, one might conjecture that students from A-certified schools will benefit more from the intervention than students from non-A-certified schools. To investigate this, the data were split up with respect to school type: two schools were A-certified schools and the other two were not.

Table 4 shows the mean gain score in the experimental and control groups for each school type. In the experimental group, the students from the A-certified schools benefited more from the intervention than the students from the non-A-certified schools. However, students from the non-A-certified schools also gained from the intervention. Within these schools, the students in the experimental group did better than the students in the control group. Nevertheless, the experimental students from the non-A-certified schools had a lower mean gain score than students from the A-certified schools in the control group. These results suggest that even if the intervention affects student achievement, the school type is also an important factor influencing the results.

Table 4 Mean gain score in experimental and control group per school type

Table 5 shows the results from the ANCOVA test on the dependent variable Gain (mean gain score), with Condition (experimental or control group) as the independent variable and School type (A-certified or non-A-certified) as covariate. The results show that both Condition and School type had a significant effect on the improvement of the students’ achievement with small to medium effect sizes. However, there was no significant interaction between Condition and School type. In other words, the hypothesis that students in A-certified schools would benefit more from the intervention than students in non-A-certified schools was not confirmed.

Table 5 Results from the ANCOVA test on the dependent variable Gain, with Condition as the independent variable and School type as covariate

Correlations Between Daily Intermediate Assessment, pre-Test and Post-Test

Table 6 presents correlations between the daily intermediate assessment scores (ScoreL1, ScoreL2, ScoreL3, and ScoreL4), pre-test scores and post-test scores. All correlations are significant and positive.

Table 6 Pearson correlations between intermediate assessment scores (ScoreL1–ScoreL4), pre-test and post-test scores

Illustrative Student Work During the Intervention

To illustrate the improvement of student performance, as shown in the quantitative results, we describe the work of one group of students during the four lessons of the experimental intervention. This group consisted of five 12–13 year old male students of mixed ability from an A-certified school and is considered to be representative of other groups that were video-recorded during the intervention. These five students are named in this article Saiful, Danang, Rafi, Syafii and Taufiq. For each lesson, we start the description with a typical task from the lesson, taken from the daily intermediate assessment, and the student results. We interpret these results in terms of the theoretical framework described in “Theoretical framework and research question” section and, if necessary, back this up with appropriate evidence from the lesson observation. Hereafter, we look at the results of these students’ written work on a task from the pre-test and post-test that is quite similar to the one from the daily intermediate assessment.

Lesson 1: Algebra Arrows

Figure 6 shows two examples of written student work on Task 3 of the daily intermediate assessment after the Algebra Arrows work. All five students in the observed group solved this task correctly, all of them by using the reverse strategy.

Fig. 6

Taufiq’s (left) and Saiful’s (right) written work on Task 3 of Lesson 1

In Fig. 6 (left part), Taufiq first transformed the word problem into an informal equation: … − 3 : 5 + 4 = 11. Next, he solved the equation using the reverse strategy as shown in the line below. Though the final answer is correct, Taufiq made a notational mistake in the use of the equal sign, i.e., rather than writing 11 − 4 = 7; 7 × 5 = 35; and 35 + 3 = 38, the student wrote 11 − 4 = 7 × 5 = 35 + 3 = 38.

In Fig. 6 (right part), rather than first transforming the word problem into an equation, Saiful directly used the reverse strategy to solve it, i.e., 11 − 4 × 5 + 3 = 38. Next, he checked the answer by substituting it into the equation: … − 3 : 5 + 4 = 11, that is, by replacing the dots with the answer. Even if the answer is correct, the written notation of the reverse strategy is not appropriate as this violates the priority rules of arithmetical operations. The proper notation for the solution should be (11 − 4) × 5 + 3 = 38. The immediate use of the reverse strategy for solving the word problem was probably a direct consequence of the learning process in the digital work in which this group used the same strategy directly, as for instance shown in Fig. 7 and described in the corresponding excerpt below.

Fig. 7

An example of student digital work in the Lesson 1

Observer: Here, the unknown number to find is not given yet. So, you should determine for yourself, with, for instance, a, b, c or n. This task is similar to Task 1, is not it?

[The students choose a as the variable. It is interesting to note here that rather than creating an equation to represent the word problem, students directly apply the reverse strategy to solve the problem.]

Danang: [Puts a into the input box, clicks 3 from the table.] This [3] should be multiplied by 6, next subtracted by 3.

Saiful: [Then] divide by 5.

Danang: [He enters the solution process into the computer to find the unknown number, that is, 3. He puts this into the answer box and presses enter. The answer is correct as shown in Fig. 7.]

A pre-test and post-test task similar to this daily intermediate task is Task 4 (see Appendix 2). In the post-test, two students from this group used the reverse strategy directly, while three used the reverse strategy after formulating equations: four of these students solved it correctly. In the pre-test, there was no correct answer for Task 4.

These results reveal two points. First, even if the notation included in the Algebra Arrows applet did not emerge in written student work while solving problems, the two types of reverse strategies used by students in the daily intermediate assessment and in the posttest seem to follow from the use of the applet. Second, the pretest and posttest results of the students in this group show positive gain scores and as such illustrate the improvement of student achievement.

Lesson 2: Cover-up Strategy

We consider Task 3 of the daily intermediate assessment Lesson 2 – i.e., solve for positive $a:\frac{24}{{\left(a+2\right)}^2-1}=3-$ as a typical task for recognizing student understanding of the cover-up strategy. Out of all five students who used the cover-up strategy, three solved Task 3 correctly. Figure 8 presents two examples of written student work on this task. The left part shows a correct solution and the right part shows an incorrect one.

Fig. 8

Saiful’s (left) and Rafi’s (right) written work on Task 3 of Lesson 2

In Fig. 8 (right part), Rafi was successful in applying the Cover-up strategy for the first step, i.e., determining the part of the equation to cover and filling in a numerical value for it, namely (a + 2)² − 1 = 8. However, in the next two steps, Rafi made mistakes. In step 2, rather than fill in 9 for the value of (a + 2)², Rafi assigned 7, which is an additive inverse mistake. This suggests that rather than using the Cover-up strategy, Rafi used the reverse strategy in an incorrect way for this step. In step 3, Rafi seems not to understand how to find the inverse of a square: he subtracted 2 from 7 to get 5 rather than find a square root.

A similar difficulty in applying the Cover-up strategy was observed during the digital group work, i.e., students used an improper reverse strategy to solve an equation that can be solved more easily with the Cover-up strategy. This was probably the origin of student difficulties that were observed in the daily intermediate assessment, as described in the following excerpt.

The group is working on Task 5a in the digital activity, i.e., solve for $a:\frac{18}{5a-2}=6$. The observer (i.e., the researcher who acts as a substitute teacher) finds that students have difficulties in identifying the expression to cover in the first step.

Observer: Okay, what is the first step you should do?

[The students follow step 1 as given in the task, but they are not sure which part of the equation should be covered first.]

Observer: Which part of the equation should you cover first? [The students are still hesitating.]

Saiful: 6 × 18 = 108, and 108 + 2 = 110, next $\frac{110}{5}$.

[He used an incorrect reverse strategy rather than the cover-up strategy to deal with the equation. So, the observer suggests students follow the step 1 properly.]

Observer: Just choose and follow step 1. [After some guidance, the students are finally able to apply the cover-up strategy to solve the equation and their solution is shown in Fig. 9.]

Fig. 9

An example of digital student work in Lesson 2

Task 6 of the pre-test and post-test, i.e., solve for $m:\frac{64}{3\left(m+1\right)-1}=8$ is similar to the daily intermediate Task 3 of this lesson. Four students solved this post-test task correctly using the cover-up strategy. In the pre-test, two students ended up with correct answers and seemed to use an informal guess-and-check strategy.

The increase in the number of students who got correct answers in the post-test as compared to the pre-test is in line with the improved achievement reported earlier. Two additional remarks are noteworthy. First, the written work in both the intermediate assessment and the post-test reveals that students have consistently used the cover-up strategy in ways that are quite similar to the cover-up strategy in the applet environment. This suggests a transfer of the applet strategy to the paper-and-pencil environment. The transparent and visual character of the Cover-up Strategy applet may explain this. Second, mistakes in written student work concern the arithmetical category of difficulties, including calculation errors and errors in applying inverse properties (see “Difficulties in initial algebra” subsection), but they had nothing to do with difficulties in the algebraic expressions or the variable category. This suggests that the applet invites students to develop a structural view of algebraic expressions rather than an operational one.

Lesson 3: Balance Model

Task 3 of the daily intermediate assessment Lesson 3, i.e., solve for x : 3x + 22 = 6x + 1, is a typical task for recognizing student understanding in Balance Model work. The results show that all five students solved this task correctly. Four students presented a solution process similar to the one they had learned– indicating that it had influenced their thinking and action– and one student provided the final answer only. While solving this task, students apparently had a visual image of an equation as a balance in mind. So, solving an equation comes down to maintaining the equilibrium of the balance and to finding an answer as the weight of an object.

A digital task similar to this daily intermediate task is Task 7, in which students were required to write an equation from the given model and then to solve it: the equation to solve was 4x + 1 = 2x + 23. While solving this equation, the students performed an action on the model in each step (by moving a bag representing x or a block representing a weight) and represented the action in the form of an equivalent equation. After this group arrived at the equation 3x = 2x + 11 and one of the students moved a bag (representing x), one of other students concluded that x = 11. Their solution to this equation is shown in Fig. 10.

Fig. 10

An example of digital student work in Lesson 3

Task 8 in the pre-test and post-test, i.e., solve for x : 12x − 11 = 4x + 13, is quite similar to Task 3 of the daily intermediate test of this lesson. The result showed that all five students in this group solved this post-test task correctly using the balance model strategy, whereas only two students solved it correctly in the pre-test, using an informal guess-and-check strategy.

In addition to the improvement of student achievement, a point to note from these results concerns the balance strategy that students used in their written work. Even if the students did not have balance models at hand during their paper-and-pencil work, their solution strategies seem to follow the Balance Model applet approach directly. This is in line with the study by Vlassis (2002) on the balance model for solving linear equations in one variable; solving an equation boils down to maintaining an equilibrium of the left and right sides of the equation; finding a solution comes down to finding a numerical value of the variable, representing the weight of an object in a balance.

Lesson 4: Balance Strategy

A typical task to determine student understanding in the Balance Strategy work is Task 3 of the daily intermediate assessment, i.e., solve for x : 9(x − 1) = 2(x − 1) + 21. There are at least two different methods to implement the balance strategy for solving this equation. First, students can subtract 2(x − 1) from both sides of the equation to obtain 7(x − 1 ) = 21 in the first step, next divide both sides by 7 and finally add 1 to find x = 4 as the solution. This first method is actually a combination of the balance and the cover-up strategies. To do this, a structural view of the algebraic expressions in the equation plays an important role. Second, students can initially apply the distributive property to remove the brackets in the equation to get 9x − 9 = 2x − 2 + 21 and next carry out the balance strategy (i.e., for instance, add 9, subtract 2x, and divide by 7 to both sides, respectively) to get the solution x = 4. The results show that all five students solved this task correctly using the latter method and that no student used the former one. This suggests that the integration of the balance strategy and other equation-solving strategies in this task is subtle; it was not observed in these students’ written work.

Another way to see student understanding in this Balance Strategy work is by analyzing student work in solving word problems in algebra. Our observation showed that it was often difficult for students to transform word problems into appropriate equations. This difficulty is partly caused by, for instance, an inability to translate phrases into correct algebraic expressions. From the perspective of mathematization, such difficulty concerns understanding problems and formulating corresponding mathematical models. In the excerpt below, we provide an example of observation for a word problem task showing student difficulty in formulating a mathematical model.

Students are working on the following task:

• Father is 39 years-old now. If two times Tom’s age is added to his father's age, the result is equal to 5 times Tom’s age 3 years later. How old is Tom now?

After reading the task, students tried to represent the word problem in an equation. The observer read the task phrase-by-phrase to guide students in representing the problem in an equation.

Observer: Two times Tom’s age…

Saiful: 2t

Observer: Okay, good! Now, it is added to the father’s age.

Saiful: 2t + 39 = ⋯

Observer: Good! Now, it is equal to five times Tom’s age 3 years later.

Danang & Rafi: [So, it is 2t + 39 =] 5t + 3

Observer: Which one should be multiplied by 5?

Danang: t.

Observer: Is it only $t$ or ($t+3$)? Please enter what you wrote.

Saiful: [He types $2t+39=5t+3$, and presses enter.] Incorrect!

Observer: It says ‘five times Tom’s age 3 years later’. So, what should be multiplied by 5?

Saiful: t + 3

Observer: Okay, so it means 5 times (t + 3). [The students represent it correctly as: 2t + 39 = 5(t + 3)]

Observer: Good! [Next the students remove the bracket in the equation.]

Saiful: So, now it is $3t+39=5t+15$

Observer: Good! Now, what is next?

[Next students solve the equation as shown in Fig. 11.]

Fig. 11

An example of digital student work in Lesson 4

Task 7 of the pre-test and post-test concerns a word problem that requires students to formulate an appropriate equation and then solve it (see Appendix 2). The post-test result shows that four students solved this task correctly using the balance model and in combination with the balance strategy, while only two students solved it correctly in the pre-test using an informal guess-and-check strategy.

From these observational findings, we retain two points. First, the uncommon use of a combination of equation-solving strategies in student work seems to be a consequence of the absence of tasks that require students to do so. Inserting tasks that can be solved with more than one strategy in digital work is apparently not enough to influence student thinking and strategies. Apparently, the integration of different equation-solving strategies requires specific attention. Second, with regard to student difficulties in transforming word problems into appropriate equations, intensive attention from the teacher during the learning process of transforming word problems into equations seems necessary.

Conclusion and Discussion

In this article, we set out to answer the following research question for the case of equations in one variable and the related word problems by means of using four applets embedded in the Digital Mathematics Environment:

• Does an intervention with digital technology enhance students’ performance in initial algebra?

The first hypothesis was that students who were engaged in the intervention with digital technology would outperform their peers in the regular learning setting in solving equations in one variable and related word problems. The results of this study confirm this hypothesis: that is, students in the experimental group had a significantly higher mean gain scores than the students in the control group, with a small to medium effect size.

The second hypothesis concerned the impact of school factor on student achievement: that is, students from A-certified schools were expected to benefit more from the intervention than students from non-A-certified schools. The results show that both the condition and the school type had a significant main effect on the students’ gain score with small to medium effect sizes. However, there was no interaction effect between the condition and the school type. Thus, the second hypothesis was not confirmed by the data.

Concerning the results of this study, we note six important points. First, the medium effect size (Cohen’s d = 0.32) found in this study – which aims to improve student achievement– is in line with the results of a recent review study on the methods of instructional improvement in algebra education (Rakes et al. 2010). In that review study, studies that focus on conceptual understanding show an observed weighted effect size of more than twice the magnitude of the effects of studies focusing on procedural work (Cohen’s d = 0.47 and Cohen’s d = 0.21, respectively). In sum, the effect size of the present study, which lies in between the two effect sizes of the review study, is in agreement with the results of other studies into improvement of algebra education and the use of technology in algebra education in particular.

Second, concerning the school type as an important component that affects student achievement, the result of the present study is in line with the study on the effect of on-line tasks for algebra in the domain of linear and quadratic equations (Drijvers et al. 2014). In that study, students from schools with good ICT facilities – which also applies to the A-certified schools in the present study (see Fig. 5) – performed better than students from schools with less-prepared ICT facilities. However, a point that should be taken into account is that categorizing school type by A-certified and non-A-certified reflects the perceived quality only to a limited extent. Therefore, the influence of school type on student achievement and the effect of a particular intervention need further investigation using measurable characteristics that include both perceived qualities and objective evaluation data.

Third, in spite of the success in improving student achievement in this experiment, the duration of the intervention was not long in terms of the use of the applets. Each of the four applets was only used in one 80-min lesson. Actually, we think this is quite short, even if Rakes et al. (2010) found that the duration of an intervention does not account for differences in effectiveness on student achievement. Furthermore, because of the limited duration of the intervention, it was quite difficult to determine a specific applet’s role in the development of student conceptual understanding and procedural skills. Therefore, we are left with a number of questions. Does each of the four applets used in this study influence student development in an equal manner? Does the combination of the four applets for developing student conceptual understanding in the topic of equations in one variable produce an optimum effect? What is the effect of the use of an applet on student algebraic skills when used over a longer period?

In line with Artigue (2002), who claims that instrumental genesis – i.e., a process of a tool becoming an instrument – is a time-consuming process, using an applet more extensively might result in even more solid conceptual understanding and procedural skill. This might be detected, for instance, through the transfer of applet’s notations or visualizations to paper-and-pencil work (Bokhove and Drijvers 2010a; Kieran and Drijvers 2006). Fig. 12 shows two examples of such transfer, which were not that frequently observed in our study: the left part shows the transfer of visualization of the Balance Model applet, while the right depicts the transfer of notation from the Balance Strategy applet.

Fig. 12

Transfer from applet visualization to paper-and-pencil notation

Fourth, we would like to stress that the use of technology in the experimental classes was only one aspect of the teaching intervention as a whole. Each lesson included an introduction of the topic delivered interactively by the teacher, i.e., the teacher interacted with students in a whole-class discussion, a demonstration of the use of the applet, a digital group work session and a daily intermediate assessment. Thus, the results of the present study cannot be attributed to the digital technology only, but should take into account the effect of the intervention as a whole. We have, for example, the impression that group work efficiency and motivation during the lessons in class Experimental 2 (Exp2) contributed to the high gain in this class (see Fig. 5).

Fifth, as the integration of ICT in mathematics teaching is a complex enterprise (Lagrange et al. 2003), we expect the teacher to be an important factor in the success of the intervention. In our experiment, this role included the ability to demonstrate how to use the applets to the whole class as effectively as possible; to help students when encountering technical obstacles; in particular, to guide students acquisition of conceptual understanding and procedural skill in algebra through working with specific digital tools. As this teacher factor was not systematically investigated in this study, we acknowledge that we here only have a limited view of the factors that explain the intervention’s success.

Sixth, and finally, the qualitative findings described in “Illustrative student work during the intervention” section to illustrate and corroborate the quantitative results are still limited and focus only on the students’ written and digital work, which we analyzed with respect to difficulties in algebra and solution strategies. We did not yet, however, consider to a larger extent (and in a more concrete manner) the impact of the technology-based intervention on students’ conceptual understanding and procedural skill from an instrumental approach perspective (e.g., Artigue 2002; Trouche 2004; Trouche and Drijvers 2010). This type of impact of technology needs to be further elaborated to conceive more clearly the interrelationship between student thinking and the use of the digital tools.

Appendices

Appendix 1

Table 7 Tasks used in daily intermediate paper-and-pencil assessments

Appendix 2

Table 8 Tasks used in pre-test and post-test