The second section introduces two research projects that draw equally from qualitative and quantitative methods in a mixed-method design. As an introduction, a condensed theoretical framework is presented that we refer to again in the discussion of the two research projects.
In recent decades, researchers in mathematics education have expanded their repertoire with research designs that suffice for and can tackle especially multifaceted research problems by integrating different methodological practices, eventually with a mixture of qualitative and quantitative research questions suitable for a complex empirical and theoretical research topic (Johnson and Onwuegbuzie 2004; Hart et al. 2009). However, a review of the latest literature reveals many different theoretical justifications for and definitions of mixed-methods research (MMR), some of which are contradictory (cf. Kelle 2008; Teddlie and Tashakkori 2009).
Although Denzin (1970) originally referred to a combination of multiple forms of qualitative research methods only (Denzin 2012), especially his concept of triangulation has often been referred to in German-speaking countries as a theoretical research background for integrating and employing several (i) data sources, (ii) investigators, (iii) theories, and (iv) methodological practices—both qualitative and quantitative—within a single research design. Remote from its original understanding, the term triangulation is frequently referred to with two different meanings (Schulz 2010; Kelle and Buchholtz 2015, p. 331): (a) “triangulation as a mutual validation of research results” and (b) “triangulation as an integration of complementary perspectives on the subject being investigated to create a more complete image of the research domain.”
Being suitable for both of these meanings, the importance of MMR can be justified by taking a closer look at the specific strengths and weaknesses of qualitative and quantitative methods, and how a balance can be struck by different types of MMR designs (Kelle 2008; cf. Schulz 2011): Qualitative research is restricted to the careful choice of a few cases. Moreover, the analysis of unstandardized data is highly dependent on the individual researcher. This “immediately raises questions about the generalizability of findings and about the intersubjectivity of interpretations” (Kelle and Buchholtz 2015, p. 336). On the other hand, qualitative methods are often useful for exploration, detection, and discovery. They help us “construct new theoretical concepts, categories, and sometimes even whole theories about the domain under study” (ibid.). Quantitative research relies on standardization and requires reliable and objective (observer-neutral) data. This enables the investigation of many cases, which contributes to a theoretically grounded basis for generalizations. Quantitative methods are ideally suited for a theory-driven approach, “whereby precise hypotheses are formulated at the onset, then operational definitions are formulated for the concepts these hypotheses comprise, measurement instruments are constructed, and data are collected and analyzed subsequently” (ibid., p. 334). They are therefore appropriate tools to examine clear-cut causal statements. However, the researchers’ theoretical knowledge or that of the corresponding discipline may be limited. Furthermore, context-bound patterns, structures and rules may form an integral part of particular life worlds under investigation that may be unknown to the researchers, especially if the socio-cultural background of researchers and respondents differs. Such a lack of knowledge may cause severe problems when formulating hypotheses, defining variables, and creating research instruments. Specifically, when examining causal statements, important variables may be “omitted with low levels of explained variance as a consequence”, intervening variables may be overlooked, or functional relations between certain variables may be incorrectly specified “so that causal processes underlying the investigated phenomena are not adequately understood” (ibid., p. 335; cf. Kelle 2008; Schulz and Wirtz 2012).
Being driven by awareness of the specific strengths and shortcomings of mono-methodological research designs, MMR designs may be used as strategies “for overcoming each method’s weaknesses and limitations by deliberately combining different types of methods” (Brewer and Hunter 1989, p. 11). Meanwhile, various designs and manifold categorizations for MMR designs exist and are discussed in German-speaking countries (e.g., Kelle 2008; Kuckartz 2014; Kelle and Buchholtz 2015; Burzan 2016). To discuss many of them would clearly go beyond the scope of this chapter. Instead, we present a concise view into the world of MMR designs that suffices as a framework for discussing the MMR design examples discussed later in this chapter. We concentrate on four prototypical MMR designs:
In a sequential qualitative-quantitative design, a qualitative study is first used to develop theoretical concepts and hypotheses. Their generalizability is then examined in a quantitative study. A widely used example for this kind of MMR design is when measurement instruments or interventions are developed on the foundation of existing qualitative findings, which enable their operationalization in a quantitative study.
In a sequential quantitative-qualitative design, for example, incomprehensible statistics may lead to their further investigation with the help of qualitative data and methods. Another example is that a quantitative pre-study may support the selection of cases for a qualitative study.
In a parallel qualitative-quantitative design, for example, researchers may interview or observe the same individuals simultaneously while applying different methods. In an experimental setting, a learning progress might be examined in a summative assessment via quantitative methods while the underlying learning processes could be validated or identified via qualitative methods.
A complex qualitative-quantitative design integrates the partial qualitative and quantitative studies systematically, eventually by performing several research cycles altogether. This allows researchers to benefit from the many advantages of both sequential and parallel designs.
Complex designs require extraordinary effort that often can be coped with by well-coordinated teams only. If individual researchers hope to create a MMR design, they will require expertise in both qualitative and quantitative research methods. Furthermore, they should be aware of the specific strengths and weaknesses of qualitative and quantitative research methods so as to combine and integrate them productively and well targeted.
In the following, we present and discuss two examples of MMR designs from Germany: Kathleen Philipp made use of a sequential mixed-methods design (Philipp 2012; Philipp and Leuders 2012). She analyzed students’ strategies while solving several mathematical problems and developed a competence model about experimental thinking in mathematics. This laid the groundwork for an intervention study that confirmed that experimental skills in mathematics can be fostered effectively.
Prediger and Wessel (2013) and Wessel (2014) implemented an integrated, parallel mixed-methods design. They fostered students’ understanding of fractions, and scaffolded the learning processes by facilitating students’ abilities to talk about fractions and their meaning. Their intervention’s efficacy was investigated by a randomized controlled trial and qualitative analyses of the videotaped teaching-learning processes.
9.3.1 Example 3: Understanding and Teaching Experimental Thinking in Mathematics
Knowledge acquisition in the natural sciences is based on experiments. Experimental thinking includes the generation and confirmation of hypotheses. There are similar processes of knowledge generation in mathematics: relationships between different kinds of objects are explored, hypotheses are generated and examined by producing and analyzing examples—as described by e.g. Euler or Pólya. This is the starting point of the project by Philipp et al. (2012) described below.
Despite the analogy, knowledge acquisition in mathematics is quite seldom associated with experimentation (e.g., Bartolini Bussi 2009). Proof and the axiomatic method dominate when establishing mathematical knowledge is discussed. However, that neglects the essential role that experimental thinking in mathematics plays in (i) generating and confirming hypotheses by examining examples before coming to any final proposition, and (ii) in arriving at a mathematical proof.
Concerning the natural sciences, scientific discovery can be described as a process of Dual Search (SDDS, Klahr and Dunbar 1988). The search in two problem spaces (one hypothetical, the other experimental) shapes hypothesis generation, experimental design, and the evaluation of hypotheses.
In their first study, Philipp et al. analyzed the experimental activities of pupils in mathematical problem situations (Leuders et al. 2011). A theory about experimental thinking in mathematics was developed relying on those findings while keeping the theoretical background of experimental thinking in mind. In the following study, experimental thinking skills in mathematics were identified and operationalized. This measurement was used in a pre-post test design to assess the learning effect of an intervention that fostered students’ skills in experimental thinking.
The first study’s cohort consisted of nine students (prospective teachers) and twelve pupils (grades 3 and 4). They were videotaped while solving mathematical problems. The problems had been chosen to be open-ended and to trigger many hypotheses and examples to solve the problem. One of those problems was the “sums of consecutive numbers” (see Fig. 9.4: “stair numbers”).
The analysis made use of Grounded Theory. Procedures were identified and coded. The system below of four core procedural categories became apparent via axial coding: (1) generating examples, (2) structuring examples, (3) generating hypotheses, and (4) testing hypotheses. These four core categories are interrelated in the experimental-thinking process while trying to solve a problem (Fig. 9.5).
In the next quantitative study, the intervention was developed according to approved design principles for fostering problem-solving strategies (Bruder 2003) and included four phases: the introduction of problems, explaining problem-solving procedures, reflecting on procedures and applying them to other situations. Participants were sixth graders (middle school): 126 students were assigned to the intervention group, and 101 students to the control group. The intervention lasted about three weeks. A pre- and post-test with a control group were administered immediately before and after the intervention, a follow-up test about six weeks later. The pre-, post- and follow-up-tests included items for measuring “structuring examples” and “testing hypotheses”. The analysis revealed a significant increase in large effects in both areas, which remained stable over 6 weeks. The large effect sizes support the authors’ claim that such procedures can be fostered effectively in regular mathematics classrooms.
Interview with Kathleen Philipp (KP), about her experiences within the MMR design described above:
How did you come up with your research question?
KP: “When I was a teacher instructing my pupils in class, I often observed curiosity in children’s mathematical thinking. I was given the chance to take part in a large research project as a PhD student on the topic of “experimentation” in mathematics or natural sciences. Being embedded in that research context, our research goal was to analyze the significance of experimentation in mathematics, and I was pleased to be able to take a closer look at the experimental thinking of students in the mathematics classroom.”
How did your theoretical background influence the research design?
KP: “The theoretical background concerning processes that are characterizable as experimental approaches was very helpful for me to understand the importance of such processes. When Euler, and later Polya, described experimental processes, I realized it was worth trying to see whether children can do mathematics in a similar manner. One of the challenges in the beginning was to find tasks for students that allowed them to find their own ways to solve a problem, even when they were unfamiliar with such tasks. It was also clear that the interesting processes are internal ones and that it would be hard to observe and analyze them, in particular when young children are involved. To overcome these challenges, I used different methods to collect data.
The theoretical background of the two problem spaces from Klahr and Dunbar helped me understand the complex processes many students demonstrated in these settings. It was very obvious that examples and hypotheses play an important role in mathematical experimentation. But there was something else we needed to understand. I refined the existing theory about two problem spaces and identified a third space that characterizes the kind of strategies used to switch between the two spaces, i.e., between examples and hypotheses.”
How did you choose and develop your research strategies?
KP: “First I identified the experimental strategies students use when exploring mathematical problems. When I noticed that students do mathematics in a manner resembling that of mathematicians, I wondered if it would be possible to teach such experimental strategies in class. To investigate this, I had to think about a means of measuring such strategies. Then I developed tasks that would reveal whether experimental strategies were being applied to find a solution. The third step was to develop a teaching unit in the standard curriculum in which I could integrate the training of experimental strategies. Finally, I had the teaching unit taught in several classes. All classes were taught by their regular mathematics teachers. At the end, I compared how frequently the respective students used experimental strategies successfully with classes that had not participated in the training session but who had been taught the same contents by relying on the school textbooks.”
Which kinds of triangulation(s) did you implement in your mixed-method design?
KP: “I triangulated several theories and data sources. To generate and analyze my data, I applied both qualitative and quantitative methods.”
Why did you implement those kinds of triangulation? Can you describe the interplay of or roles of ‘theory development’ and ‘theory confirmation’ in your study?
KP: “The development of my research project required different methods. To identify experimental strategies, I had to take a qualitative approach, because the aim was to investigate and understand students’ approaches in depth, and to create a foundation for developing a theory of experimentation while learning mathematics. I therefore applied the Grounded Theory to analyze the students’ approaches, which has the advantage of enabling you to develop a theory from your data. Thus, as is typical when using Grounded Theory, it was feasible to identify core strategies that students apply in experimentation.
Quantitative methods were necessary to measure the use of such experimental strategies, which made it possible to investigate a larger group of students (about 230) and make comparisons with a control group.
On the one hand, different methods were necessary to find answers to the specific research questions. On the other hand, the measuring and fostering of experimental strategies validated the theory. I relied on different data sources to observe and assess experimental strategies.”
Was implementation in the classroom important from the start?
KP: “Yes, implementation in the classroom was important for me personally. Working as a teacher for many years, it was my research concern to link my research findings to actual practice. In my opinion, it is crucial that research results in our field be relevant in mathematics education.”
How did the project influence your later research projects personally?
KP: “For me, this research project was fundamental to the evolution of my research interests. It is still important to me that research results be closely associated with practice. Another aspect is that I have a deep interest in understanding people’s thinking and their thought processes.”
9.3.2 Understanding and Teaching Fractions
Our description of the fourth project example follows Prediger and Wessel (2013) and Wessel (2014) who work on language challenges and conceptual mathematical challenges for students with low language proficiency. Language proficiency is the key determinant of mathematics achievement—not immigrant status or being multilingual (Heinze et al. 2009; Prediger et al. 2018). Students with low language proficiency encounter reading obstacles as well as difficulty in developing conceptual understanding (Prediger et al. 2018).
To support students’ constructions of meanings for new mental objects and relationships (Steinbring 2005), the authors draw upon the principle of relating verbal, symbolic, graphic, and concrete representations (Duval 2006; Lesh 1979; Bruner 1967). Conceptual learning processes involve the acquisition of new linguistic, graphic, and symbolic means for expressing mathematical concepts and their meanings (Schleppegrell 2010). This illustrates how deeply interwoven mathematics learning and language learning are (Pimm 1987).
The principles of relating representations has been extended to relating different linguistic registers also, such as establishing first- and second-language registers, mathematical technical registers, a school language register, and the students’ everyday registers (see Fig. 9.6: Prediger and Wessel 2013). “A register can be defined as the configuration of semantic resources that a member of a culture typically associates with the situation type… in a given social context” (Halliday 1978, p. 111).
The school register is characterized by context-reduced and complex linguistic means. It differs from the everyday register by appearing conceptually written even if medially oral. Unlike the technical register, the school register is seldom taught explicitly in school. In particular, this is problematic for underprivileged language learners who acquire only everyday language and vocabulary at home but nevertheless require the school register for higher thinking skills (Clarkson 2009), while privileged (first- or second-) language learners are exposed to the school register already at home (Schleppegrell 2004; Cummins 2000).
An intervention was designed that included translations between all registers and related them to each other. The choice of suitable activities for each moment in the learning process was guided by the design strategies of “pushed output” (Swain 1985) and scaffolding (Hammond and Gibbons 2005). The language- and mathematics-integrated intervention aimed to initiate the construction of mathematical meaning in combination with opportunities to establish the required language means (Prediger and Wessel 2013: Fig. 9.7):
The intervention study took the mixed-methods approach to investigate effects from two research questions in a triangulating manner (Prediger and Wessel 2013, p. 443): “(1) To what extent do students who participate in the language- and mathematics-integrated intervention improve their achievement in the fraction test? (2) What is the situational potential of the intervention activities to initiate students’ constructing of meanings and activating of linguistic means in the school and technical register?”
The study sample consisted of 72 seventh-grade second-language learners with below average math performance and limited German language proficiency. The intervention consisted of six lessons lasting 90 min in 2-to-1 sessions (two students with one teacher). A pre- and post-test randomized controlled trial design with a control group was realized (Fig. 9.8).
The control group was taught by their regular teacher with the usual fraction textbook repetition program. The 18 × 540 min were video-taped, and relevant episodes were selected according to research question 2. The data corpus for the qualitative analysis comprised selected videos, all written materials from the interventions, and the teachers’ lesson plans.
The quantitative analysis of the learning gains revealed a low-to-medium effect (intra-group effect size measured by d = 0.42) in the control group, but a very strong effect (d = 1.22) in the intervention group. Those differences became significant in a later study with a larger cohort (Prediger and Wessel 2018).
Quantitative analysis of learning effects was complemented by a qualitative analysis to exploratively reconstruct the situational potential of the design strategies and instructional activities in the learning processes (Prediger and Wessel 2013: Fig. 9.8). The qualitative analysis included in-depth analysis of the videotaped teaching–learning processes: (1) the aim of sequential analysis was to reconstruct individual and interactional processes of meaning construction for fractions, (2) several qualitative coding procedures for students’ utterances were conducted to capture different linguistic aspects, and (3) students’ development of mathematical meaning and linguistic means were reconstructed in their interaction with concrete elements of the instructional design.
This qualitative analysis identified the specific potential of macro- and micro-scaffolding in combination with the design strategy of pushed output.
The MMR design was reflected in the study findings (Prediger and Wessel 2013, p. 453): “Whereas these quantitative results alone might have been explained on the basis of the intensity of individual teaching, the case studies gave some first insights into how the design elements offered fruitful learning opportunities, since they showed the deep interconnection of linguistic and mathematical learning processes: …With the help of macro and micro scaffolding, low-achieving students successfully proceeded from their everyday language to a language for thinking and talking about structural relationships.”
Interview with Susanne Prediger (SP), about her experiences within the above
described MMR design:
How did the research question emerge? Was it a rational decision?
SP: “A research question is also a matter of personal choices, but it is of course based on a rational decision. Regarding the challenges of language learners in mathematics: by 2011, we had a situation where we had a lot of statistical evidence of connections between language proficiency and mathematics learning. But we did not know much about how to foster language learning. We thus chose to continue to dig deeper and beyond the current descriptive knowledge, to find answers about how to foster language learners and what could work for them.”
How did the theoretical background influence the research design?
SP: “That it definitely does. In our case it’s how you conceive of the role of language in the learning process. When we consider language only as relevant in its communicative function, it might suffice to gather knowledge about how to approach word problems and alleviate language difficulties in word problems. But we start from the assumption that language has a deep cognitive function within mathematical learning processes. If we want to delve deeper into that, we need to study the learning processes, and see how the interplay between language and mathematics really works. That is why we had to initiate mathematics- and language-integrated processes. In everyday classrooms, we observed hardly any conceptual learning opportunities for students with low language proficiency, which created the need to establish an instructional design specifically for those students as a research context for our research questions.”
What kinds of triangulations did you implement in your mixed-methods design?
SP: “First of all it is a triangulation of data, because we wanted on the one hand to gather data about the intervention’s function to see if it works. To test what works, you need a pre- and a post-test, you need to conduct a randomized controlled trial. But if you want to learn how it works, you really need data on the actual processes, meaning video data and qualitative analysis.
On the theoretical level, we preferred working with networking theories. It is a similar idea, but it is not just about putting the two next to another. You really have to integrate the different theories. Regarding this topic of language and mathematics education, of course you need theories about mathematics learning and you need theories about language and its role, and they have to be combined.”
Can you describe the interplay between ‘theory development’ and ‘theory confirmation’ in your study?
SP: “In the classical sense we only discuss confirming theories, which is all about hypotheses and validating them. The randomized controlled trial is the research approach of choice for validating hypotheses. In our case, we could validate the hypothesis that students with low language proficiency learn better when they are given language- and mathematics-integrated learning opportunities.
However, before you can validate a hypothesis, you must develop categories to characterize phenomena and to generate hypotheses to be validated. This is where qualitative, explorative research approaches are most suitable.
In our research, we really combined these two approaches: we validated a hypothesis that had been quite general and invested a lot of time and energy to generate a theory on a more fine-grained level. For this, we had to develop the categories in which we could formulate phenomena and refined hypotheses. The importance of generating categories is often underrated.
In the present project the validation of the hypothesis that a language- and content-integrated intervention might be more effective was only a starting point. To elaborate upon our theory, we had to investigate the processes in depth to understand what exactly happened in the learning processes. And this qualitative research enabled us to generate categories that we could use as a starting point and for initial hypotheses for the next project. Within the next project, which is now being funded by the German Research Foundation, we again have some hypotheses that we can validate or reject, and we will generate new questions and new aspects to be studied in greater depth. (Prediger and Wessel 2018)
Thus, there’s always interplay between hypothesis generation and its validation. Indeed, the categories are really the most important element in terms of theory generation. There is a worthwhile article by Cobb and diSessa, who talk about ontological innovations. Their article puts emphasis on the generation of categories as the major output of research.”
What do you personally prefer: to start with a qualitative study, or take a quantitative approach?
SP: “(Laughs) Neither nor - it’s more like going backward and forward. Sometimes, like in this project, there was quantitative evidence of a problem with language proficiency and mathematics achievement. Thus, we started to understand what the problem was qualitatively, and conducted design research cycles to devise an intervention that we could put to the test in a randomized controlled trial. Yet I would never want to conclude with a randomized controlled trial, because we need to understand how things work and what really happens in these processes. So, it really is a backward-forward movement. I’m really pleased that we have funding for two successive projects, including the one described here. In both we dug deeper and deeper into the details about what students really need, into what kinds of fostering in language terms they require, and into the type of fostering that is genuinely effective.
It’s like the question about the hen and the egg - you can’t tell where to start, you have to take a combined, cyclical approach.”
Many researchers in mathematics education consider themselves to be engaged in an applied discipline, where research should contribute to improving education. Learning mathematics with understanding is a complex process, and (Hart et al. 2009, p. 39) “mixed methods research may be an appropriate response to calls for greater generalizability of results while maintaining enough detail about the processes of teaching and learning to be valid and useful.” This idea on the purpose and use of MMR in mathematics education was expressed clearly by the two researchers we interviewed, Kathleen Philipp and Susanne Prediger. It goes together with the concern expressed by Hart et al. (ibid.), that research in mathematics education does not only need to know “if particular educational experiments improve learning with understanding but also how those results are achieved and why we can expect them to be replicated elsewhere.”
The two examples above and the interviews with the researchers illustrate how MMR in particular can contribute a great deal of both scientific and practical knowledge to the evidence on “what works” (cf. Chatterji 2005): in particular, the MMR project discussed by Kathleen Philipp (cf. Philipp 2012; Philipp and Leuders 2012) exemplified how the use of systemic, contextually-grounded studies in early phases may be followed by more sharpened, analytic experimental studies in later research phases. The MMR project discussed by Prediger and Wessel (2013) and Wessel (2014) underscored that the combined use of more than one research method may help us reveal patterns, develop new constructs and deepen the understanding of relationships between constructs, especially causal relationships. However, as Prediger expressed it in her interview, it is not necessary “to end with a randomized controlled trial, because we want to understand how the things work and what really happens in the processes.”