The participants of this study were 304 first-time LAs who were enrolled in the LA pedagogy course in one of five semesters, Spring 2014 to Spring 2016 from 13 departments, primarily working in science, mathematics, engineering, and psychology courses. The distribution of LAs by department and semester is shown in Table 1. Note that departments represent the courses that LAs worked in, not necessarily their majors.
All first-time LAs who completed both the first and last teaching reflection of the semester (described below) were selected to be part of our analysis. Table 1 includes only LAs who completed both first and last reflection. Due to our focus on LA growth, it was important for us to have pre/post-responses from each LA in the study. A total of 304 LAs out of 455 participated in the study.
Teaching reflections have been assigned in the LA pedagogy course consistently since 2002 and provide a way for LAs to reflect on their experience and thoughts about the pedagogical topics discussed. LAs are asked to write weekly responses that reflect on their opportunities to implement the pedagogical strategies in their LA sessions, any concerns, questions, or triumphs they might be experiencing, and their ideas about what teaching, learning, and being an LA means to them.
In all semesters in this study, LAs have been asked identical questions in the second and 15th week of the semester, which serve as pre/post-views. Spring 2014 is the only exception, with the “post” questions being asked in week 13 instead. We found that the Spring 2014 data was similar in nature to that of the other semesters analyzed. We concluded that the two-week disparity did not influence students’ responses. The two reflection questions that we were interested in were “What does it mean to teach?” and “What does it mean to learn?,” hereafter referred to as the teach prompt and the learn prompt. We chose these questions because they had the potential to provide insight into how LAs viewed students with respect to the teaching/learning process and whether they would be able to articulate the essential pedagogical principles without being asked directly. We investigated the research question: To what extent did LAs refer to essential pedagogical principles in written responses to questions about teaching and learning? We also investigated the sub-question of how deeply the language of the pedagogical principle of students’ ideas was taken up by LAs. All LAs that participated in the study responded to both teach prompts and both learn prompts for the semester. The only exception to this was in Spring 2016 when five LAs completed the pre/post-teaching questions, but not the learning questions because they were permitted to choose two of four questions. Overall, we analyzed 1206 teaching reflections across the five semesters.
Coding and analysis
Three essential pedagogical principles that were learning goals for the pedagogy course were student ideas, constructing knowledge, and formative assessment; hence, these are the focus of our methodological approach to measuring LA growth in their first semester in their LA role. Our analytical approach involved descriptive coding using a priori codes representing student ideas, constructing knowledge, and formative assessment. Although we took a deductive approach to coding, we did not initially compose formal definitions for each of our a priori codes. Rather, the researchers coded the responses to the teach and learn prompts separately, first relying on their own conceptions of what these codes meant. The responses to each of the prompts were coded holistically, that is, each response could be coded at most once for each of the three principles. Then, the researchers came together to compare codes and developed formal and concrete definitions (Table 2) before re-analyzing all responses using the analytical method of simultaneous coding (Miles, Huberman, Saldaña 2014). We felt that this was appropriate because while the essential pedagogical principles are distinguishable, they are not mutually exclusive, and we found that LA responses could include some aspects of one or all of the principles. Furthermore, since we were studying LA growth in learning, it made sense that their responses might not be a clear expression of just one thought, but that they might be an amalgamation of different notions.
Finally, we developed subcodes for our student ideas code, which we called resource, obstacle, and vocabulary. These codes largely emerged inductively from our data, as we tried to refine our understanding of the student ideas language present in LAs’ responses. However, we found that the subcodes we generated aligned with prior literature (e.g., Hammer 1996), and thus we clarified the definitions of these codes by building on prior studies.
We must acknowledge that the pedagogical principles were challenging to objectively code, and many iterations of coding were required to achieve definitional guidelines and clarity and reliability. We discuss some of our difficulties with particular codes below.
When coding for student ideas, we were looking for acknowledgement of at least one of the following statements: (a) students come in with ideas, (b) these ideas can impact the way the new information is taken up by students, (c) these ideas can be built upon or related to when teaching new information, or (d) the new information may cause some kind of rearrangement of a pre-existing framework. These four statements become progressively more complicated, for example, student idea code (a) simply requires some vocabulary use around student ideas, while (b), (c), and (d) require that LA acknowledged the use of student ideas in some way.
Certain words presented a clear picture (to us) of how the LA was thinking about student ideas. Such words or phrases included “prior, already know, mental model, pre-conceived notions, misconceptions, belief, old understanding, existing knowledge, previous knowledge, and student ideas.” We found that these words could be easily aligned with our code definitions and were thus ideal for standardizing our coding process. Examples of responses coded as student ideas can be seen in Table 2.
When coding for student ideas, it was often unclear whether LAs thought that students came into the classroom with ideas of their own or if the LAs thought that students’ ideas were simply regurgitations of ideas they were presented with in class. Also, if an LA assumed that whatever was taught in a previous or current class was taken up exactly as the instructor intended, statements about “prior knowledge” present a gray area. We resolved this issue by developing a coding rule that there must be a clear acknowledgement of ownership by the student of the ideas or there must be a clear acknowledgement that the ideas pre-dated the class. For example, the following responses were coded as student ideas:
“To help students understand the knowledge that they already have…” (LA reflection, Spring 2016)
“Learn what the student thinks and how they formulate ideas.” (LA reflection, Spring 2016)
In both of these responses, we see ownership and/or an indication that the student ideas were held prior to their interaction with the LA. By using the phrase “they already have,” the first LA indicates possession of the knowledge by the student and also that the knowledge was held by the student before the class. In the second example, although there is no obvious timeline, by saying “they formulate ideas,” the LA seems to be attributing the ideas to the student and not indicating that the student is just repeating the ideas they were given by someone else.
There were also some words used by LAs when referring to student ideas or their use that made coding difficult, such as “expanding, experience, grow(th), develop, opinions, information, thought process, real lives, level, shaping, or frame of mind.” These words were particularly difficult to code because it was not always clear what was meant by them, for example, whether the LA was crediting the student with authorship and control over their ideas or whether the LA was speaking in a more “acquisition-like” framework (Sfard 1998). For difficult-to-code responses such as these, we relied heavily on the context of the statements, making judgments that were later checked between the two coders.
Student ideas subcodes
We further divided the student ideas code (which was coded most often) into three subcodes: resource, obstacle, and vocabulary based on our own observations during the coding process. The purpose of these subcodes was to see how LAs were taking up the notion of students’ ideas with respect to their use in teaching and learning. We were interested in not only their recognition of student ideas but also in their ability to see them as a resource for learning.
We coded as resource when we saw evidence of LAs’ recognition of the benefit and usefulness of student ideas in LAs responses to the teach prompt or the learn prompt. An example of a resource-framework can be seen in the following definition of teaching:
“To help students understand the knowledge that they already have and how to use that information to understand new topics and concepts.” (LA reflection, Spring 2016, emphasis added)
Here, we noted that the LA was implying a clear use for a student’s prior knowledge in helping the student comprehend new ideas, and thus this response was coded student ideas as resources.
We coded obstacle as a recognition of student ideas that were viewed as something that needed to be overcome or changed to accomplish teaching and/or learning. For example, the following response demonstrates LAs’ views of student ideas as obstacles:
“I think that learning is the integration and application of knowledge within a pupil. The integration aspect depends on the teacher recognizing the learners current misconceptions and dispelling them with the new accurate knowledge.” (LA reflection, Spring 2014)
In the statement above, while the LA clearly acknowledges that a student has ideas when learning, the LA also implies that those ideas need to be moved out of the way to allow for new ideas to take their place. Thus, this response was coded student ideas as obstacles.
Finally, we coded vocabulary where we saw a recognition of student ideas, but there was no apparent value assigned to these ideas, and no explicit use of the ideas was recognized or mentioned. Most often, this coincided with the use of the term “mental models,” hence our choice of the “vocabulary” code descriptor. However, there were a few other instances where we found LAs acknowledging that students had ideas without explicitly identifying their functionality or usefulness and without using the mental models descriptor. We felt that such answers contained vocabulary consistent with language used in the course to discuss student ideas. Thus, we justified including these types of responses in the vocabulary category if they did not reflect a resources or obstacle perspective on student ideas. An example of this can be seen in the following excerpt:
“To teach is to bring students to a further point of knowledge than they began with. If a teacher can instill a sense of learning and interest in a subject, they know how to teach.” (LA reflection, Fall 2015)
While this response acknowledges that students begin with some knowledge, it assigns no particular value to this knowledge relative to its use in teaching or learning, and thus we conservatively coded this response student ideas as vocabulary.
We note that LA responses could occasionally be double-coded with these subcodes if one part of their response indicated that they were thinking about student ideas as obstacles, while in another section of their response, they demonstrated thinking about student ideas as resources. We argue that this is acceptable because any ideas that LAs have throughout the course are likely to be fusions of different concepts and language since they are still learning. An example of a double-coded response follows:
“Learning is listening to, actively-engaging with, and gaining new or different knowledge, integrating that knowledge into your current mental model, addressing any conflicts or misconceptions that arise and understanding the gained information.” (LA reflection, Spring 2014)
Here, we observe the LA viewing the integration of new knowledge with current knowledge as a critical resource for learning; however, they also use “misconceptions” terminology which implies more of an obstacle framing of student ideas. Thus, the above response was coded as student ideas as resources and student ideas as obstacles.
Our second code of interest for this study was constructing knowledge. When coding for constructing knowledge, we were looking for acknowledgement of at least one of the following ideas: (a) learning consists of the integration of new ideas into a framework, (b) knowledge can be built by the student, (c) students use prior knowledge to understand new ideas, or (d) use of the word constructivism. We note that responses of type (c) were always double-coded with student ideas.
For category (a), we struggled with whether or not the “framework” that new ideas could be integrated into necessitated double-coding with student ideas. For the most part, we found that the student ideas code was implied by these responses, but occasionally the framework mentioned by LAs was too vague to definitively code student ideas as well. For example,
“To learn is to construct a world view in which one intakes and accept knowledge that ‘works’ and rejects knowledge that doesn’t work.” (LA Reflection, Spring 2016)
In this response, a framework was implied as a way to accept or reject knowledge, but we did not see explicit mention of student ideas. Thus, this response was solely coded as constructing knowledge.
Similar to our coding of student ideas, we uncovered some problem words when coding for constructing knowledge, namely, “expanding, experience, grow(th), develop, exploration, thought process, shaping, learning through discovery, applying, connecting, and information.” We found that these words could be interpreted in two different ways, with one empowering the student to build their own knowledge (as per constructivism) or with a student being led to information (more of an acquisition lens). Our coding of these terms became very context dependent. We did find that there were some consistently clear words that indicated constructing knowledge such as “building, integration, creating meaning, and construct.” Whenever these terms appeared in responses, we found clear evidence of a constructivist framework.
When coding for formative assessment, we were looking for responses that included at least one of the following: (a) use of the phrase “formative assessment”; (b) demonstration of building the bridge (as described in the pedagogy class); (c) acknowledgement of where the student starts, where they need to go, and how to help them get there with respect to content; or (d) mention of finding out why a particular student does not understand a concept. For the building of the bridge in part (b), we noticed that several students specifically referred to the bridge picture that we used to introduce formative assessment in the pedagogy class. For responses including the bridge to be coded, we required that the language used be student-focused (i.e., not about the teacher improving their practice) and that LAs included at least two components of the bridge in their response (e.g., where the student starts relative to where they need to be or where the student starts and how they could be guided down a path in the right direction).
We found formative assessment particularly challenging to code given the number of components necessary to create the full formative assessment picture. It was difficult to decide at what point an LA was beginning to use formative assessment language even if they had not developed the full concept yet. In particular, we found that students began using some goal-setting language, which mimicked the language in the formative assessment article we distributed (Moss and Brookhart 2010), but there was no acknowledgment of student ideas. For example,
“You set a goal of where you want your students to be by the end of the course, and the way in which you teach them will help them get to the goal or not.” (LA Reflection, Fall 2014)
To resolve this, we decided that any response that incorporated two components of the bridge, even if it did not acknowledge where students were starting from, was indicative of early stages of an LA expressing the concept of formative assessment and thus goal setting language was coded as a formative assessment if the response indicated a goal and a path to achieve the said goal. Additionally, some LAs did not describe a complete bridge, even though they acknowledged the student’s starting place and the desired ending place.
“As a teacher it is important to understand where your students are coming from and where they need to go in terms of their understanding.” (LA Reflection, Spring 2015)
Again, we saw two components of the bridge here and thus decided that this response and similar responses were indicative of the early stages of expressing formative assessment.
We also discovered that finding formative assessment in responses to the “What does it mean to learn?” question was rare. Instead, LAs seemed to rely more heavily on metacognition as a way to check in on learning progress with respect to this question. For example,
“This [learning] includes: having an open mindset, assessing your current knowledge and knowledge gained periodically through the learning process, and reflecting on your own studying and learning techniques to determine what strategies are the most effective for you.” (LA Reflection, Spring 2016)
Though we were not specifically coding for metacognition in the study reported here, based on the patterns seen in LA reflections as well as our informal observations of LAs during class discussions, it seemed that the language associated with this concept (which we also discuss in the pedagogy course) was being taken up as the learning analog of formative assessment in teaching. However, metacognition was not one of the principles we focused on in this study, and thus we excluded such responses from our formative assessment coding.
To ensure the reliability of this study, all responses were coded independently by two different researchers, one semester at a time. After analyzing a given semester, the researchers came together to resolve any conflicting codes. We found that the resulting number of code changes was minimal, only 1.5% for student ideas, 1.2% for constructing knowledge, and 0.4% for formative assessment. After resolving any conflicts, we developed formal definitions for each of the codes and then analyzed all responses again. We found that this also resulted in a small number of changes to the code counts. Although the coders were instructors for sections of the pedagogy course for two of the semesters included in this study, section numbers and names were blinded during coding to minimize the potential for researcher bias.
We acknowledge that this study is restricted in scope and thus has several limitations. First, this research is meant to serve as a beginning assessment of what pedagogical content students (LAs) learned while enrolled in a course about pedagogy. It is intended to mirror studies that seek to evaluate learning of any content (e.g., physics) by comparing pre- and post-measures of students’ understanding. This study is not intended to determine the extent to which LAs used these ideas in practice, although such a study could be warranted, and in fact, is taking place.
We also acknowledge that a validity argument could have included member checking with previous LAs; however, this would not have been practical given the span of the data and the numbers of subjects. We took great care to operationalize our codes and coding process to make conservative estimates of the meaning being made by participants in the study.
Additionally, our initial review of the data revealed that LAs’ reflections included large numbers of references to student learning as acquisition (Sfard 1998) rather than construction. An inductive coding scheme could have provided more nuanced insights into what LAs take away from the course and the LA experience. However, the purpose of this study is to investigate the extent to which LAs took up the pedagogical concepts addressed in the pedagogy course as discussed previously.
Finally, in attempts to find an efficient measure for assessing LAs’ understanding of the pedagogical principles that were taught in class, we asked LAs general questions, “what is teaching and what is learning.” Our inductive coding allowed us to establish coding rules that accurately reflected LAs’ ways of applying each pedagogical principle to these general questions. The coding rules provide insight into the ways LAs were thinking as well as the extent to which their thinking was different pre- and post-instruction. The resulting graphs shown in the following section provide visualizations of the distributions of these responses, allowing for inferences about LAs understanding of pedagogical principles. While our ultimate goal is to understand how much LAs learn throughout their LA experience, this work is purely descriptive and we are not trying to make causal claims about the growth of LAs as a result of the pedagogy course. The coding percentages presented in the figures and findings are intended to lend insight into trends in LAs’ responses, which are exemplified in the methodology section. Our analysis is largely qualitative, illustrating trends in LAs’ responses to a very difficult, abstract question, intended to assess deep understanding of a concept.
To address our primary research question, (to what extent did LAs refer to essential pedagogical principles in written responses to questions about teaching and learning?), we calculated the percentage of responses to both the teach prompt and the learn prompt that were coded student ideas, constructing knowledge, or formative assessment each semester and then averaged them across the five semesters in our study (Fig. 3). In almost all cases, LAs answered both the teaching and the learning questions, so there were approximately two responses per LA for week 2 and two responses per LA for week 15. Since there were five LAs that discussed their views of teaching but did not respond with their definition of learning in Spring 2016, the results are reported in terms of percentage of responses instead of a percentage of students.
In week 2, a few LAs came into the pedagogy class with some constructivist or student ideas language already; about 4% of LA responses demonstrated some ideas about constructing knowledge, while 6% of LA responses indicated that students have pre-existing ideas. We did not see any indications of LAs using or understanding formative assessment in week 2 responses. For all three concepts, we saw positive growth on average throughout the semester. Our data indicates that the LAs showed the most growth around student ideas, with an 11% increase in the number of responses coded as student ideas between weeks 2 and 15. Growth was seen in constructing knowledge and formative assessment as well, with 3 and 4% increases in responses coded as these concepts, respectively.
How the language associated with the concept of student ideas is taken up by LAs
Since LAs demonstrated the most growth with respect to the concept of student ideas, we analyzed the responses coded as student ideas in more detail. In many responses, we noticed that LAs were specifically using the term “mental models” which was a key word in the Redish (1994) reading handed out in four of the five semesters (the article was not assigned in Spring 2016). In fact, in the four semesters that the Redish article was distributed, between 30 and 60% of the responses that were coded as student ideas in week 15 explicitly contained the term “mental models” (Fig. 4).
While these numbers vary among the four semesters, it is clear that at least 30% of LAs who indicated the importance of student ideas in their responses also took up the term mental models in their thinking about the concept each semester. Note that there are no mentions of mental models in Spring 2016, the semester that we did not assign the Redish article. This indicates that the mental models language was likely coming from the article distributed in the pedagogy course and not from some external source. Figure 4 also shows that the mental models language seems to be greater in the Fall than in the Spring semesters.
By investigating the breakdown of student ideas codes by semester, we determined how the growth in student ideas aligned with LA use of the mental models language. For each of the five semesters, we looked at the percentage of teaching and learning responses coded as student ideas in week 2 relative to the percentage of responses coded as the concept in week 15 (Fig. 5).
Though the number of LAs coming in with student ideas language varies quite significantly across the semesters, the growth from week 2 to 15 is around 12–13% of responses each semester except for Spring 2016 where we only see a 5% growth. This seems to be correlated with the absence of a mental models article in that semester. Note that the growth in student ideas codes does not necessarily follow the same pattern as mental model usage across Fall versus Spring semesters. While we saw that mental model language was consistently used in responses coded as student ideas from Spring 2014 through Fall 2015, the take-up of the term was particularly high in Fall 2014 and Fall 2015 (58 and 50%, respectively). However, the growth in the number of responses coded as student ideas is nearly invariant (around 12 or 13%) for all four of the semesters in which the Redish article was used, that is, we do not see substantially higher growth during the Fall semesters. Thus, it seems that an increased use of mental models terminology in a given semester does not correlate directly to more growth in student ideas codes during that semester. Only when there is a complete absence of mental models language do we see a change in student ideas growth; the number of student ideas codes decreases dramatically in Spring 2016 when the term mental models were not introduced in the pedagogy course. However, even though LAs were not using the term mental models explicitly as frequently in Spring 2014 and Spring 2015 as in the Fall semesters, the ideas and language in the article may have still facilitated discussion around student ideas, leading to the same amount of growth as in other semesters.
How deeply the language associated with the concept of student ideas is taken up
Our data indicated that LAs were beginning to use discourse around student ideas, but our initial coding did not indicate how deeply that learning went. We wanted to see if LAs actually found student ideas useful in teaching and learning. In order to investigate this, we took two analytical approaches to measuring the sophistication of LAs’ responses: first using the resource framework described by Hammer (1996) and then analyzing in terms of the formative assessment scale introduced by Gray (2013).
Is the language related to student ideas taken up in a resource framework?
We looked at the distribution of responses coded as student ideas that used resource or obstacle language surrounding student ideas as well as responses that were simply using the vocabulary without putting any value on the student ideas (as described previously in our definition of the student ideas as vocabulary subcode). This distribution is shown for each semester from Spring 2014 to Spring 2016 (Fig. 6).
Note that the columns do not sum to 100% for every semester because some LA responses were double-coded as resource and obstacle as explained in the “Methodology” section. The data indicates great variation in the ways that LAs talked about student ideas across the semesters. For Spring 2014, Fall 2015, and Spring 2016, over half of the student ideas were talked about using resource language. In fact, on average across all semesters, 55% of student ideas responses used resource language. Note that a 55% average of resources coded within responses already coded as student ideas actually amounts to only 9% of total LA responses. In Fall 2014, however, obstacle language was more dominant, and in Spring 2015, there was a fairly even distribution between all three subcodes. If we define “deep” learning to mean obstacle or resource language (i.e., referring to student ideas in a more than superficial way), we see that over 60% of LA responses that discussed student ideas demonstrated deep learning.
Comparing LAs’ learning of student ideas
Another way that we assessed the depth of LAs’ learning regarding student ideas was by looking at student ideas codes in conjunction with formative assessment and constructing knowledge. We argue that if the value of student ideas is truly learned, this will lead LAs to a better understanding of the latter concepts because they represent the positive utilization of these student ideas in the teaching and learning processes. To that end, we looked at the percent of week 15 responses that were coded with only student ideas, double-coded with only student ideas and constructing knowledge, double-coded with only student ideas and formative assessment, or triple-coded with all three. These percentages were then averaged across all semesters (Fig. 7).
From this graph, we see that approximately 10% of responses were only coded as student ideas, and the percent of responses that were coded as student ideas in conjunction with either of the other concepts was 5% or less. This seems to indicate that even though LAs begin to take up the language of student ideas throughout a semester in the pedagogy course, they do not necessarily extend that concept to constructing knowledge or formative assessment.