Investigating teachers’ understanding through topic modeling: a promising approach to studying teachers’ knowledge

Abstract

Examining teachers’ knowledge on a large scale involves addressing substantial measurement and logistical issues; thus, existing teacher knowledge assessments have mainly consisted of selected-response items because of their ease of scoring. Although open-ended responses could capture a more complex understanding of and provide further insights into teachers’ thinking, scoring these responses is expensive and time consuming, which limits their use in large-scale studies. In this study, we investigated whether a novel statistical approach, topic modeling, could be used to score teachers’ open-ended responses and if so, whether these scores would capture nuances of teachers’ understanding. To test this hypothesis, we used topic modeling to analyze teachers’ responses to a proportional reasoning task and examined the associations of the topics identified through this method with categories identified by a separate qualitative analysis of the same data as well as teachers’ performance on a measure of ratios and proportional relationships. Our findings suggest that topic modeling seemed to capture nuances of teachers’ responses and that such nuances differentiated teachers’ performance on the same concept. We discuss the implications of this study for education research.

Appendix

Ratios and proportional relationships assessment

Tasks
A photograph is enlarged to make a poster. The photograph is 10 cm wide and 16 cm high The poster is 25 cm wide, how high is the poster? Explain your answer If 6 ml of paint was needed for the original photograph, how much paint will be needed for the enlarged photo? Explain your answer
Raymond wanted to know the cost of buying different numbers of songs for his MP3 player. The cost of each song is the same. Let s represent the possible number of songs Raymond could buy Let d represent the amount of money, in dollars, Raymond would need to buy the songs. Fill in the table for all missing values of s and d
Number of songs s	Amount of money ($) d
2	2.50
	5.00
7
	22.50
Yasmin went to the store to buy a new purse. The purse she wanted was on sale for 40% off the original price, and the salesperson offered her an additional discount of 15% off the sale price. The salesperson told Yasmin that this was a great deal and that she was getting a discount of 55% off the original price. Do you agree with the salesperson? Please explain.
A recipe requires $\frac{1}{4}$ cups of flour for every $\frac{1}{3}{ }$ batch of cookies. How many batches of cookies can be made with ${5}\frac{1}{2}$ cups of flour? Explain how you solved the problem.
This graph shows the relationship between the number of gallons of gasoline used (g) and the total cost of gasoline (c). 1. How much money will be paid if 12 gallons of gasoline are used? 2. Write an equation to find the cost for any given amount of gasoline used. Explain how you found the equation.
Of following word problems, which represent equivalent ratios? Select all that represent equivalent ratios. If a men paint the outside of a house in b minutes, then how many minutes d would it take c men to paint the same house, if all the men work at the same rate? A leaky faucet was dripping water into a bucket. There was already some water in the bucket before Latika started collecting data. She found that there were a ounces of water in the bucket after b minutes. How many ounces of water c will be in the bucket after d minutes? Bob and Marty run laps together because they run at the same pace. Today, Marty started running before Bob came out of the locker room. Marty had run a laps by the time Bob had run b laps. How many laps c had Marty run by the time that Bob had run d laps? Explain your reasoning.