Keywords

This chapter’s simulation at a glance

Domain

Teacher education

Topic

Diagnosing student understanding and misconceptions of decimal fractions

Learner’s task

Taking on the role of a teacher, student, or observer during a (simulated) diagnostic task-based interview on decimal fractions

Target group

Mathematics teacher education students with some prior pedagogical content knowledge of decimal fractions

Diagnostic mode

Individual diagnosis by teachers

Sources of information

Teacher–student interaction based on diagnostic tasks

Special features

Based on current research on students’ understanding of decimal fractions; trained actors or peer students as standardized students in a role-play simulation

5.1 Introduction

5.1.1 Diagnosing Mathematical Understanding in Direct Teacher–Student Interaction

Results from educational research emphasize the importance of teachers’ diagnostic competences for adaptive teaching and thus for improved student learning (e.g., Behrmann & Souvignier, 2013). Consequently, teacher training standards (e.g., KMK, 2004) highlight diagnostic competences as a central goal of teacher education at university. This, in turn, results in a need for evidence-based training methods.

Diagnosing is understood here as the goal-directed accumulation and integration of information to reduce uncertainty when making educational decisions (cf. Heitzmann et al., 2019) such as adaptive teaching, lesson planning, or student assessment (Schrader, 2013). While previous research has focused on judgment accuracy (i.e., the match between teachers’ expectation concerning a student’s performance on a test and that student’s actual performance on that test; Spinath, 2005), recent work suggests the need to include more qualitative evaluations of the learner’s understanding, misconceptions, and strategies (e.g., Herppich et al., 2017) and to also study the diagnostic process that leads to teachers’ judgments.

We conceptualize diagnostic competences as the collection of teachers’ individual resources that enable them to attend to and interpret students’ mathematical thinking in a variety of situations (Jacobs et al., 2010; Nickerson et al., 2017; Weinert, 2001). While diagnostic situations in teachers’ practice may vary substantially (Karst et al., 2017), most arise within student-teacher interactions in the classroom (Klug et al., 2013), have the formative assessment of student learning as their goal, and are closely intertwined with the teacher’s pedagogical actions (Kaiser et al., 2017).

Based on the above definition, the accuracy and effectiveness of teachers’ diagnoses in a range of situations—in terms of reducing the uncertainty of the pedagogical decision at hand—serves as the primary indicator for observing diagnostic competences. However, indicators from the diagnostic process itself might also provide insights into a teacher’s diagnostic competences. Firstly, it might be considered to what extent a teacher’s interaction with a student is indeed suited to generate interpretable evidence about students’ mathematical thinking, for example, by posing diagnostically rich (“probing”) questions (van den Kieboom et al., 2014). On the other hand, research on teachers’ noticing suggests that it would be worthwhile to examine the depth with which teachers process their observations during diagnosing. Seidel and Stürmer (2014), for example, propose to differentiate between the mere description of relevant aspects of a situation “without making any further judgements” (p. 745); explanations , in which teachers link their observations to concepts and theories from their professional knowledge; and predictions , in which teachers draw conclusions about the consequences of the “observed events in terms of student learning” (p. 746).

Despite some first results from interventions fostering in-service teachers’ judgment accuracy (e.g., Thiede et al., 2018), research that focuses on pre-service teachers, takes a broader view of diagnostic competences, and examines the role of characteristics of the diagnostic process for the final diagnosis as well as for development of diagnostic competences is still sparse.

5.1.2 Role-Play-Based Simulations to Foster Diagnostic Competences

University-based teacher education has traditionally put an emphasis on conveying professional knowledge, which is assumed to underlie competences such as diagnostic competences. Professional knowledge is often differentiated into content knowledge, pedagogical content knowledge, and pedagogical knowledge (e.g., Kleickmann et al., 2012). Based on the assumption that “integration is simple and builds up automatically” (Harr et al., 2014, p. 1), these knowledge domains are often taught separately, thus leaving “the challenge of integration to the individual teacher” (Harr et al., 2014, p. 1). Additionally, content from these knowledge domains is first encoded as declarative knowledge, but practice is an essential prerequisite for the transformation of this declarative content into procedural knowledge and skills (Anderson, 1982, 1987). However, many teacher education programs seem to lack this linkage between knowledge acquisition and practice (Beck & Kosnik, 2002; Fraser, 2007). As a result, weak connections between the knowledge domains are frequent, and pre-service teachers are likely to struggle to use this knowledge in practice (Alles et al., 2018).

In the same vein, Shavelson (2012) proposes using “holistic, real-world problems” (p. 58) to assess competences such as diagnosing. He mentions both task authenticity (e.g., Seidel et al., 2010) and the feeling of immersion into the task situation (“presence,” cf. Schubert et al., 2001, pp. 266ff.; Frank, 2015) as characteristics of valid learning and assessment tasks. However, situations involving diagnosis in everyday teacher practice are often characterized by a complex interaction between managing student-teacher interactions, making diagnoses, making pedagogical decisions, and enacting pedagogical interventions. This complexity may overstrain unexperienced learners and impede their learning. If such situations are included in early phases of teacher education, a central problem is to find a balance between tasks’ authenticity and their complexity (Seidel et al., 2015).

Therefore, Grossman et al. (2009b) propose a “decomposition of practice ” into its basic components in order to reduce complexity and so-called approximations of practice , which enable pre-service teachers “to engage in practices that are more or less proximal to the practices” (p. 2056) of their future profession. Based on the above, Seidel et al. (2015) provide a differential clarification of the relation between these two goals:

“Novices are faced with a myriad factors to be taken into account in the initial experiences of teaching. Thus, the acquisition of professional practice is not characterized by simply increasing the quantity of classroom teaching practice – the most complex form of teaching practice – but by building up a series of approximations to a practice that increases in complexity and that allows for systematically linking elements of professional knowledge to corresponding elements in professional practice.” (p. 86)

Similar to this idea, using diagnostic interviews to separate diagnostic demands from pedagogical decision-making and practice in initial teacher education has been proposed repeatedly in the past (Grossman et al., 2009a; Schack et al., 2013) and may be one form of decomposition . Furthermore, simulations of such interviews, e.g., in the form of role-plays, may provide an effective way to control complexity in this kind of learning situation. However, knowledge of learning processes in such simulations and factors influencing their effectiveness in initial teacher education is still scarce, and numerous research gaps exist regarding how these competences can be supported through instruction. Consequently, research on feasible and efficient learning environments to help mathematics pre-service teachers acquire those competences seems to be justified.

Medical education has recently studied role-play-based simulations as learning environments (e.g., Lane et al., 2008; Stegmann et al., 2012), and has also addressed their potential feasibility and benefits for teacher education (Gartmeier et al., 2015). Within the context of those studies, role-play-based simulations have proven an effective means of fostering communicative competences, especially during early phases of their acquisition (Berkhof et al., 2011; Lane & Rollnick, 2007). Further studies indicate that learning by observation and active role-taking (e.g., Stegmann et al., 2012) may foster the acquisition of competences within such simulations.

5.2 A Role-Play-Based Simulation of Diagnostic One-on-One Interviews

This chapter presents the conceptualization and development of a role-play-based simulation of one-on-one diagnostic interviews in DiMaL, a project that aims to study pre-service teachers’ learning processes in such simulations and their effects in university-based pre-service teacher education.

5.2.1 Selection of the Diagnostic Situation

In line with prior approaches to fostering pre-service teachers’ diagnostic competences (McDonough et al., 2002), and to ensure that the simulated interviews will successfully target central aspects of diagnostic competence as part of pre-service teachers’ training, it was necessary for the simulations to represent real-life job demands (Shavelson, 2012). Therefore, we focused on decimal fractions as the interview content. Research on students’ errors and misconceptions regarding decimals has a long tradition in mathematics education (Brueckner, 1928; Heckmann, 2006; Steinle, 2004) and is addressed in university-based teacher education in many countries (e.g., Lortie-Forgues et al., 2015; Ministry of Education, 2008; Padberg & Wartha, 2017). We placed focus on three areas of knowledge of decimal fractions that have been reported as particularly difficult in the past:

  1. 1.

    Principles of number representation in the decimal place value system, including the application to comparing decimals.

  2. 2.

    Flexible and adaptive use of calculation strategies for all four basic arithmetic operations.

  3. 3.

    The meaning of basic arithmetic operations One-on-one diagnostic interviews diagnostic situation in real-world situations (e.g., partitive and quotative situations for division by rational numbers).

5.2.2 Use Scenarios for the Simulation

Simulations , as approximations of practice, may serve as learning opportunities within teacher education (Grossman et al., 2009a), but can also deliver formative and summative information about pre-service teachers’ diagnostic competences (Shavelson, 2012). When developing the simulation, we anticipated two different use scenarios:

In a learning scenario , the simulation serves as an approximation of practice to support meaningful learning in university-based teacher education. In this scenario, participants engaging in the simulation take on one of three roles: One participant acts in the teacher role, while a second participant takes on the role of the simulated student (grade 6). A third participant may take on the role of an observer (cf. Stegmann et al., 2012) who watches and reflects on the diagnostic interview enacted by the participants in the other two roles.

In an assessment scenario , the goal of the simulation is to derive information about pre-service teachers’ diagnostic competences from the diagnostic process as well as the final diagnosis proposed. In this scenario, all participants take on the teacher role. For standardization purposes, the student role can be played by teacher education students who are specially trained to act as standardized sixth graders during the simulation. Apart from the standardization of the student role, the simulation follows the same procedure in both use scenarios.

5.2.3 Overview of the Simulation

We decided to construct an interactive role-play-based live simulation to approximate the diagnostic situation without too many restrictions on teachers’ questions and simulated sixth graders’ responses. Based on experiences with live simulations in medical education and teacher education (e.g., Gartmeier et al., 2015; Stegmann et al., 2012), the simulation was developed in close collaboration with a partner project from medical education (cf. Chap. 9).

A preparation phase for the simulation (15 min., cf. Fig. 5.1) acquaints participants with the technical aspects of the simulation and with the relevant content for their role. Participants in the teacher role study a set of diagnostic tasks that they can use during the interview, those in the student role study a description of the case profile they will enact later, and those in the observer role study an observation script. All participants can make notes for each task. A “fiction contract” informs participants about the natural restrictions of a simulation setting and asks them to engage in the simulation as they would in a comparable real interview as much as possible.

Fig. 5.1
figure 1

Overview of content and tasks for each role in the simulation phase

Full size image

In the subsequent interview phase (30 min., cf. Fig. 5.1), participants in the teacher and student roles engage in the role-play-based simulation of the diagnostic interview, starting with a short introductory dialogue. The participant in the teacher role selects tasks, presents them to the simulated student, observes the answer, and has the opportunity to ask further probing questions. The participant in the teacher role is instructed to start the interview by selecting at least one sub-task from each of three initial screening tasks . Before they proceed to subsequent tasks, they are asked to provide an intermediate diagnosis. The participant in the student role works on the tasks as described in their case profile, while the participant in the observer role watches and analyzes the simulated interview using the observation script. Participants in the teacher and observer roles can take notes.

During the final report phase (15 min., cf. Fig. 5.1), participants in all three roles are asked to individually compose a structured report diagnosing the simulated student’s knowledge and misconceptions based on the preceding interview. Participants in the student and observer roles are asked to take the perspective of the interviewer here, and to interpret the interview from this perspective.

Participants are guided through the simulation using a web-based simulation environment. During the interview phase, the participant in the interviewer role selects tasks from his or her computer screen (on the right-hand side of Fig. 5.2), while the participant in the student role can see the case profile on his or her own computer screen (on the left side of Fig. 5.2). The participant in the observer role can see the observation script. Moreover, there are text fields for the interviewer and the observer to take notes during the interview. Whenever the interviewer selects a task, the student’s solution based on his or her case profile is displayed on the student’s screen. The tasks are also presented on a tablet PC in the center of the setup, and the student solves the tasks by writing on the tablet PC with a digital pen (see the middle of Fig. 5.2). The tablet PC also records what is on its own screen and the verbal discourse during the simulation.

Fig. 5.2
figure 2

Setup of the simulation environment from the observer’s perspective, showing the student (left) and the teacher (right)

Full size image

5.2.4 Development of Simulation Materials

Diagnostic Tasks

Based on prior research on students’ understanding of decimal fractions (e.g., Steinle, 2004), we designed a set of 16 diagnostic tasks for the simulation. Some tasks are diagnostically sensitive to typical errors and misconceptions, while other tasks can be solved without deeper understanding of decimals. The first three tasks are screening tasks that address one of the three main areas of knowledge regarding decimals. Each of these screening tasks contains some sub-tasks that are diagnostically sensitive and some sub-tasks that are not. The subsequent tasks may provide additional information for each of the main areas of decimal knowledge and can be selected freely by the interviewer after completion of the screening tasks.

Case Profiles

In order to address knowledge and misconceptions in each of the three areas of decimal knowledge, we defined four student case profiles based on prior research on students’ misconceptions regarding decimal fractions. Each case profile includes strong misconceptions about decimal fractions in one area of knowledge, partial misconceptions in a second area, and quite robust knowledge in the remaining area. The profiles contain detailed descriptions of the students’ knowledge and misconceptions regarding decimals. A student solution and student justification for this solution for each of the 16 diagnostic tasks and each of the four case profiles were constructed that reflected each student’s specific case profile. These solutions and justifications are displayed to the participant in the student role during the interview whenever a task is selected by the interviewer.

Structured Report Format

Three different prompts were designed to obtain participants’ final diagnoses. As a first rough diagnosis, participants are asked to enter a main and two alternative short descriptions of the student’s main problems and provide information about their certainty in their diagnosis. The second prompt asks for a more extensive report to be given to the “teacher” of the simulated student, highlighting the students’ understanding, misconceptions, and first ideas for specific instructional support for this student. Based on the concept of professional vision (e.g., Seidel & Stürmer, 2014; van Es & Sherin, 2002), participants are instructed to describe what they have observed during the interview, to explain their observations using knowledge from their university courses, and to predict possible consequences. The third prompt requires participants to judge to what extent the simulated student has mastered each of the three areas of knowledge on decimals in a closed answer format.

5.2.5 Training Actors to Play Standardized Students

When using the simulation to assess participants’ diagnostic competences, having the simulated students act in a standardized way based on the case profiles becomes crucial. To achieve this, the pre-service teachers working as standardized students received a 10-hour acting training in three sessions. The training concept was based on prior research on micro-teaching events (Seidel et al., 2015) and role-play simulations in medicine (Stegmann et al., 2012). After a brief introduction to misconceptions on decimals and diagnosing mathematics skills, the actors were introduced to the interview setting, the technical environment and the diagnostic tasks, and the case profiles. Before the second session, they studied the case profiles in detail. The second session contained an active training with three diagnostic interviews, feedback by the project staff, and discussion of challenges and uncertainties in playing the simulated students. The third session comprised two test simulations in which a member of the project staff played the interviewer and provided feedback to the actors based on video recordings of the simulation. The training sought to maximize the simulation’s authenticity and ensure an accurate depiction of the case profiles.

5.2.6 Measures Derived from the Simulation

Diagnostic Process

The simulation offers three ways to gather data on the interviewer’s diagnostic activities (Fischer et al., 2014) within the simulation. (1) The diagnostic sensitivity of the sub-tasks selected by the interviewer from the screening tasks serves as a measure of how well participants in the teacher role generate diagnostic evidence . (2) Analyzing to what extent participants in the teacher and observer roles describe, explain or predict based on their observations relates to the diagnostic activities of evaluating evidence and drawing conclusions . It also enables an analysis of how deeply participants process the diagnostic evidence (Seidel & Stürmer, 2014; van Es & Sherin, 2002). (3) Participants in the teacher role may pose verbal questions to their simulated students to elicit information about their mathematical thinking. The extent to which these questions allow participants to infer students’ knowledge of decimals relates to the interviewers’ evidence generation, but also involves the creation of artifacts (tasks and questions).

Diagnostic Product

The diagnoses included in the final report provide information about the accuracy of the diagnosis, i.e., its match to the actual student profile. Moreover, the certainty ratings after the screening tasks and during the report phase allow this accuracy is accuracy to be related to the reduction of uncertainty during the diagnostic process. Finally, the descriptions, explanations, and predictions in the participants’ written reports show how deeply the participants process diagnostic information when communicating and scrutinizing their conclusions.

5.3 Ensuring a Suitable Approximation of Practice

In order to ensure that the simulation can serve as an appropriate approximation of practice for learning and assessment at university, three aspects are currently under study. The associated studies focus on the assessment use case for the simulation.

Usability of the Simulation

A pilot study with N = 6 mathematics pre-service teachers as participants was conducted. In the experimental session, the participants completed the whole simulation as described above twice; the case profiles were randomly assigned. Based on this pilot, several technical changes were made to the simulation platform to increase the usability of the system. For example, it turned out that some interviewers attempted to inform the simulated students about the correct solutions to the tasks (cf. van den Kieboom et al., 2014 for similar results). On the one hand, this may reflect high immersion into the simulation’s diagnostic task and an experience of authenticity, which have been put forward as important characteristics of valid learning and assessment tasks (cf. Shavelson, 2012). On the other hand, focusing on instruction instead of diagnosis might decrease the learning opportunities for diagnostic competencies. For this reason, we included an additional button titled “explanation alert” on the student’s computer screen in the interview system. If the interviewer starts to provide an explanation, the participant in the student role can select this button; after a certain amount of time, a warning message appears on the interviewer’s screen instructing them to remember that their goal is to diagnose students’ understanding.

Content of the Simulation Materials

We are conducting a video-based survey among experts in mathematics education and educational sciences to obtain evaluations of the diagnostic tasks and case profiles included in the simulation. This includes the comprehensibility of the instructions given to the interviewer role, the authenticity of the case profiles, and the standardized implementation of these case profiles by the trained pre-service teachers.

Participants’ Perception of the Simulated Situation

As Shavelson (2012) highlights the importance of task authenticity and the experience of immersion into the task situation resulting from the simulation environment as characteristics of valid assessment and learning tasks, we will investigate these factors in a first validation study with mathematics pre-service teachers and practicing mathematics teachers. Using adaptations of established scales (Frank, 2015; Seidel et al., 2010), we will study participants’ perception of authenticity and presence during the interview.

5.4 Conclusions and Outlook

In this chapter, we presented the conceptualization and development of a simulation that combines and extends three lines of research: Firstly, the simulation goes beyond the traditional conceptualization of teachers’ diagnostic competences as judgment accuracy (Spinath, 2005) by including process features (Herppich et al., 2017). Based on data from the simulated diagnostic interviews, it is possible to describe both diagnostic processes during the interview and products of the diagnostic process. Secondly, the interviews are implemented as role-play-based simulations and represent approximations of practice focusing on a separable facet of teachers’ professional work. Such a decomposition has been proposed for early phases of teacher education in particular (Grossman et al., 2009a). Thirdly, and in order to address the first two points, we make use of results from a long tradition of research on students’ misconceptions of decimal fractions (e.g., Brueckner, 1928; Steinle, 2004). Prior research has also shown that mastering professional demands in one-on-one interviews such as those in our simulation poses a challenge to beginning mathematics teachers. Such demands include, for example, selecting diagnostically sensitive tasks or asking rich “probing” questions (van den Kieboom et al., 2014).

One main feature of the simulation is that it is developed for two different use scenarios: assessment and learning. The use of authentic and realistic tasks has been put forward in the literature as a criterion for valid assessment in higher education (Shavelson, 2012), making it desirable to study their added value beyond traditional paper-and-pencil-based assessments in teacher education. Moreover, such simulations have been proposed as a means to transform teacher education to focus more on teachers’ professional tasks as a means of learning and applying professional knowledge (Grossman et al., 2009a).

Studying the suitability and effects of simulations for these two scenarios is the main goal of the DiMaL project. This will be addressed in further studies, including an expert survey to evaluate the diagnostic tasks, instructions, case profiles, and their implementation in terms of comprehensibility, authenticity, and standardization. Further studies will also focus on participants’ perception of the simulated situation in terms of authenticity and immersion into the diagnostic task (Shavelson, 2012). Moreover, relations between pre-service teachers’ professional knowledge, characteristics of their diagnostic process, and the accuracy and effectiveness of their diagnoses will be studied.

Nevertheless, the implementation of such simulations in pre-service teacher education still leaves a set of open questions. Despite the added value of such simulations over more traditional forms of instruction, open questions remain regarding the effects of the specific role taken within the simulation (teacher, student, or observer) on students’ learning (cf. Stegmann et al., 2012 for medical diagnosis). Moreover, research from other fields has highlighted the need for reflection in order to learn from such simulations (e.g., Mamede et al., 2012, 2014). Future studies will address the four questions introduced by Fischer, Chernikova & Opitz (2022) and Opitz et al. (2022). To optimize learning (overarching research question 2), we will investigate a video-based implementation of the simulation, which will provide more guidance and structure than the role-play simulation, as well as additional support in the form of reflection prompts and knowledge activation prompts during the simulation. Moderation effects by prior knowledge on the effects of these scaffolds will be investigated (overarching research question 3). Together with an in-depth investigation of learning process measures (overarching research question 1), the results will be used to implement adaptive support for the two scaffolds (overarching research question 4).