Skip to main content
SpringerLink
Log in

Automated Guidance for Thermodynamics Essays: Critiquing Versus Revisiting

  • Published:
Journal of Science Education and Technology Aims and scope Submit manuscript

Abstract

Middle school students struggle to explain thermodynamics concepts. In this study, to help students succeed, we use a natural language processing program to analyze their essays explaining the aspects of thermodynamics and provide guidance based on the automated score. The 346 sixth-grade students were assigned to either the critique condition where they criticized an explanation or the revisit condition where they reviewed visualizations. Within each condition, the student was assigned one of two types of tailored guidance based on the sophistication of their original essay. Both forms of guidance led to significant improvement in student understanding on the posttest. Guidance was more effective for students with low prior knowledge than for those with high prior knowledge (consistent with regression toward the mean). However, analysis of student responses to the guidance illustrates the value of aligning guidance with prior knowledge. All students were required to revise their essay as an embedded assessment. While effective, teachers involved in this study reported that revising is resisted by students and does not align with typical, vocabulary-focused classroom writing activities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Azevedo R, Bernard RM (1995) A meta-analysis of the effects of feedback in computer-based instruction. J Educ Comput Res 13(2):111–127

    Article  Google Scholar 

  • Blanchard M, Southerland S, Osborne J, Sampson V, Annetta L, Granger E (2010) Is inquiry possible in light of accountability? A quantitative comparison of the relative effectiveness of guided inquiry and verification laboratory instruction. Sci Educ 94(4):577–616

    Article  Google Scholar 

  • Butcher PG, Jordan SE (2010) A comparison of human and computer marking of short free-text student responses. Comput Educ 55(2):489–499. doi:10.1016/j.compedu.2010.02.012

    Article  Google Scholar 

  • Chang H-Y, Chang H-C (2013) Scaffolding students’ online critiquing of expert- and peer-generated molecular models of chemical reactions. Int J Sci Educ 35(12):2028–2056

    Article  Google Scholar 

  • Chang H-Y, Linn MC (2013) Scaffolding learning from molecular visualizations. J Res Sci Teach 50(7):858–886. doi:10.1002/tea.21089

    Article  Google Scholar 

  • Chang H-Y, Quintana C, Krajcik JS (2010) The impact of designing and evaluating molecular animations on how well middle school students understand the particulate nature of matter. Sci Educ 94(1):73–94

    Google Scholar 

  • Chase C, Chin D, Oppezzo M, Schwartz D (2009) Teachable agents and the protégé effect: increasing the effort towards learning. J Sci Educ Technol 18(4):334–352

    Article  Google Scholar 

  • Chin C (2006) Classroom interaction in science: teacher questioning and feedback to students’ responses. Int J Sci Educ 28(11):1315–1346

    Article  Google Scholar 

  • Chinn CA, Brewer WF (2001) Models of data: a theory of how people evaluate data. Cognit Instr 19(3):323–393

    Article  Google Scholar 

  • Clark D, Jorde D (2004) Helping students revise disruptive experientially supported ideas about thermodynamics: computer visualizations and tactile models. J Res Sci Teach 41(1):1–23

    Article  Google Scholar 

  • Clauser BE, Kane MT, Swanson DB (2002) Validity issues for performance-based tests scored with computer-automated scoring systems. Appl Meas Educ 15(4):413–432

    Article  Google Scholar 

  • Cuthbert AF, Slotta JD (2004) Designing a web-based design curriculum for middle school science: the WISE ‘houses in the desert’ project. Int J Sci Educ 26(7):821–844

    Article  Google Scholar 

  • Davis EA (2003) Prompting middle school science students for productive reflection: generic and directed prompts. J Learn Sci 12(1):91–142

    Article  Google Scholar 

  • de Jong T, Linn MC, Zacharia ZC (2013) Physical and virtual laboratories in science and engineering education. Science 340(6130):305–308. doi:10.1126/science.1230579

    Article  Google Scholar 

  • diSessa AA (1988) Knowledge in pieces. In: Forman G, Pufall P (eds) Constructivism in the computer age. Hillsdale, Lawrence Erlbaum Associates, New Jersey

    Google Scholar 

  • diSessa AA (1996) What do ‘‘just plain folk’’ know about physics? In: Olson DR, Torrance N (eds) The handbook of education and human development: new models of learning, teaching, and schooling. Blackwell, Cambridge, pp 709–730

    Google Scholar 

  • diSessa AA (2006) A history of conceptual change research: threads and fault lines. In: Sawyer K (ed) Cambridge handbook of the learning sciences. Cambridge University Press, Cambridge, pp 265–281

    Google Scholar 

  • Donnelly DF, Linn MC, Ludvigsen S (2014) Impacts and characteristics of computer-based science inquiry learning environments for precollege students. Rev Educ Res 84(4):572–608. doi:10.3102/0034654314546954

    Article  Google Scholar 

  • Erickson GL (1985) Heat and temperature: part A. An overview of pupils’ ideas. In: Driver R, Guesne E, Tiberghien A (eds) Children’s ideas in science. Open University Press, Milton Keynes, pp 55–66

    Google Scholar 

  • Gerard LF, Linn MC (2013) What kind of feedback improves understanding? A comparison of knowledge integration and generic feedback on students’ scientific explanations. Paper presented at the annual meeting of the American Education Research Association. San Francisco, CA

  • Gerard L, Matuk C, McElhaney K, Linn MC (in press) Automated, adaptive guidance for K-12 education. Educ Res Rev. doi:10.1016/j.edurev.2015.04.001

  • Gerard L, Ryoo K, McElhaney K, Liu L, Raferty A, Linn MC (in press) Automated guidance for student inquiry. J Educ Psychol

  • Haertel GD, Means B, Penuel W (2008) Technology tools for collecting, managing, and using assessment data to inform instruction and improve achievement. In: Smolin L, Lawless K, Burbules N (Eds), NSSE yearbook 2007, Information and communication technologies: considerations for teachers and teacher education. Chicago, National Society for the Study of Education

  • Hamp-Lyons L (2001) Fourth generation writing assessment. In: Silva T, Matsuda PK (eds) On second language writing. Lawrence Erlbaum Associates Inc, Mahwah, NJ, pp 117–125

    Google Scholar 

  • Havnes A, Smith K, Dysthe O, Ludvigsen K (2012) Formative assessment and feedback: making learning visible. Stud Educ Eval 38(1):21–27

    Article  Google Scholar 

  • Hmelo-Silver C, Duncan R, Chinn C (2007) Scaffolding and achievement in problem-based and inquiry learning: a response to Kirschner, Sweller, and Clark (2006). Educ Psychol 42(2):99–107

    Article  Google Scholar 

  • Höffler TN, Leutner D (2007) Instructional animation versus static pictures: a meta-analysis. Learn Instr 17(6):722–738

    Article  Google Scholar 

  • Hovardas T, Tsivitanidou OE, Zacharia ZC (2014) Peer versus expert feedback: an investigation of the quality of peer feedback among secondary school students. Comput Educ 71(1):133–152

    Article  Google Scholar 

  • Hutchison P, Hammer D (2010) Attending to student epistemological framing in a science classroom. Sci Educ 94(3):506–524

    Google Scholar 

  • Hyland F (1998) The impact of teacher written feedback on individual writers. J Second Lang Writ 7(3):255–286

    Article  Google Scholar 

  • Jordan S (2012) Student engagement with assessment and feedback: some lessons from short-answer free-text e-assessment questions. Comput Educ 58(2):818–834

    Article  Google Scholar 

  • Kali Y (2006) Collaborative knowledge building using the Design Principles Database. Int J Comput-Support Collab Learn 1(2):187–201

    Article  Google Scholar 

  • Kegel CAT, Bus AG (2013) Links between DRD4, executive attention, and alphabetic skills in a nonclinical sample. J Child Psychol Psychiatry 54(3):305–312

    Article  Google Scholar 

  • Kluger AN, DeNisi A (1996) The effects of feedback interventions on performance: a historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychol Bull 119(2):254–284. doi:10.1037/0033-2909.119.2.254

    Article  Google Scholar 

  • Landauer TK, Laham D, Foltz P (2003) Automated scoring and annotation of essays with intelligent essay assessor™. In: Shermis MD, Burstein J (eds) Automated essay scoring: a cross-disciplinary perspective. Lawrence Erlbaum, Mahwah, NJ

    Google Scholar 

  • Leacock C, Chodorow M (2003) c-rater: automated scoring of short-answer questions. Comput Humanit 37(4):389–405

    Article  Google Scholar 

  • Lee H-S, Liu OL (2010) Assessing learning progression of energy concepts across middle school grades: the knowledge integration perspective. Sci Educ 94(4):665–688

    Article  Google Scholar 

  • Lewis E (1996) Conceptual change among middle school students studying elementary thermodynamics. J Sci Educ Technol 5(1):3–31

    Article  Google Scholar 

  • Lewis EL, Linn MC (1994) Heat energy and temperature concepts of adolescents, adults, and experts: implications for curricular improvements. J Res Sci Teach 31(6):657–677. doi:10.1002/tea.3660310607

    Article  Google Scholar 

  • Linn M, Eylon B-S (2011) Science learning and instruction: taking advantage of technology to promote knowledge integration. Routledge, New York

    Google Scholar 

  • Linn MC, Songer NB (1991) Teaching thermodynamics to middle school students: what are appropriate cognitive demands? J Res Sci Teach 28(10):885–918. doi:10.1002/tea.3660281003

    Google Scholar 

  • Linn M, Songer N, Lewis E, Stern J (1993) Using technology to teach thermodynamics: achieving integrated understanding. In: Ferguson D (Ed), Advanced educational technologies for mathematics and science SE—1 (Vol 107, pp 5–60), Springer, Berlin. doi:10.1007/978-3-662-02938-1_1

  • Lipnevich A, Smith J (2009) Effects of differential feedback on students’ examination performance. J Exp Psychol Appl 15(4):319–333

    Article  Google Scholar 

  • McNeill KL, Krajcik J (2012) Supporting grade 5–8 students in constructing explanations in science: the claim, evidence and reasoning framework for talk and writing. Pearson Allyn & Bacon, New York, NY

    Google Scholar 

  • Mortimer E, Scott P (2000) Analysing discourse in the science classroom. In: Millar R, Leach J, Osborne J (eds) Improving science education: the contribution of research. Open University Press, Buckingham

    Google Scholar 

  • National Research Council (2012) A framework for K-12 science education: practices, crosscutting concepts, and core ideas. National Academy Press, Washington, DC

    Google Scholar 

  • Nuthall G (2005) The cultural myths and realities of classroom teaching and learning: a personal journey. Teach Coll Rec 107(5):895–934

    Article  Google Scholar 

  • Osborne J (2010) Arguing to learn in science. The role of collaborative, critical discourse. Science 328:463

    Article  Google Scholar 

  • Page EB (2003) Project essay grade: PEG. In: Shermis MD, Burstein J (eds) Automated essay scoring: a cross-disciplinary perspective. Lawrence Erlbaum Associates, Mahwah, NJ, pp 43–54

    Google Scholar 

  • Ruiz-Primo MA (2011) Informal formative assessment: the role of instructional dialogues in assessing students’ learning. Stud Educ Eval 37(1):15–24

    Article  Google Scholar 

  • Ryoo K, Linn MC (2012) Can dynamic visualizations improve middle school students’ understanding of energy in photosynthesis? J Res Sci Teach 49(2):218–243. doi:10.1002/tea.21003

    Article  Google Scholar 

  • Ryoo K, Linn MC (2014) Comparison of specific and knowledge integration automated guidance for concept diagrams in inquiry instruction. In: Paper presented at the International Conference of the Learning Sciences (ICLS) meeting, Boulder, CO

  • Sandoval WA, Millwood KA (2005) The quality of students’ use of evidence in written scientific explanations. Cognit Instr 23(1):23–55

    Article  Google Scholar 

  • Sato E, Linn MC (2014) Designing critique to improve conceptual understanding. In: Penuel B, Jurow S, O’Connor K (eds), Proceedings of learning and becoming in practice, Vol 1, p 471–478 International Society of the Learning Sciences

  • Scalise K, Timms M, Moorjani A, Clark L, Holtermann K, Irvin PS (2011) Student learning in science simulations: design features that promote learning gains. J Res Sci Teach 48(9):1050–1078

    Article  Google Scholar 

  • Segedy JR, Kinnebrew JS, Biswas G (2013) The effect of contextualized conversational feedback in a complex open-ended learning environment. Educ Tech Res Dev 61(1):71–89

    Article  Google Scholar 

  • Shen J (2010) Nurturing students’ critical knowledge using technology-enhanced scaffolding strategies in science education. J Sci Educ Technol 19(1):1–12

    Article  Google Scholar 

  • Sukkarieh JZ, Blackmore J (2009). C-rater: automatic content scoring for short constructed responses. In: Proceedings of the 22nd-International FLAIRS Conference. Sanibel Island, FL: Association for the Advancement of Artificial Intelligence

  • Swift JN, Gooding CT (1983) Interaction of wait time feedback and questioning instruction on middle school science teaching. J Res Sci Teach 20(8):721–730

    Article  Google Scholar 

  • Tiberghien A (1980) Models and conditions of learning. An example: the learning of some aspects of the concepts of heat. University of Leeds Printing Services, Leeds

    Google Scholar 

  • Tsai C-C, Lin SSJ, Yuan S-M (2002) Developing science activities through a networked peer assessment system. Comput Educ 38(1–3):241–252

    Article  Google Scholar 

  • Van der Kleij FM, Eggen TJHM, Timmers CF, Veldkamp BP (2012) Effects of feedback in a computer-based assessment for learning. Comput Educ 58(1):263–272

    Article  Google Scholar 

  • Vygotsky L (1978) Mind in society: the development of higher psychological processes. Harvard University Press, Cambridge, MA

    Google Scholar 

  • Wang H-C, Chang C-Y, Li T-Y (2008) Assessing creative problem-solving with automated text grading. Comput Educ 51(4):1450–1466

    Article  Google Scholar 

  • White BY, Frederiksen JR (2000) Technological tools and instructional approaches for making scientific inquiry accessible to all. In: Jacobson MJ, Kozma RB (eds) Innovations in science and mathematics education: advanced designs for technologies of learning. Lawrence Erlbaum Associates, Mahwah, NJ, pp 321–359

    Google Scholar 

  • Williams M, DeBarger AH, Montgomery BL, Zhou X, Tate E (2012) Exploring middle school students’ conceptions of the relationship between genetic inheritance and cell division. Sci Educ 96(1):78–103

    Article  Google Scholar 

  • Zhang ZH, Linn MC (2013) Learning from chemical visualizations: comparing generation and selection. Int J Sci Educ 35(13):2174–2197. doi:10.1080/09500693.2013.792971

    Article  Google Scholar 

  • Zwiep S, Straits W (2013) Inquiry science: the gateway to English language proficiency. J Sci Teach Educ 24(8):1315–1331

    Article  Google Scholar 

Download references

Acknowledgments

This material is based upon work supported by National Science Foundation (NSF) Grant Number 1119670 (CLASS: Continuous Learning and Automated Scoring in Science). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. The authors gratefully acknowledge the teachers and students who participated in this study, and the reviewers who gave feedback on this paper.

Author information

Authors and Affiliations

  1. Graduate School of Education, University of California, Berkeley, CA, USA

    Dermot F. Donnelly, Jonathan M. Vitale & Marcia C. Linn

Authors
  1. Dermot F. Donnelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan M. Vitale
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcia C. Linn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dermot F. Donnelly.

Appendices

Appendix A: Student Guidance

REVISIT Condition

  1. 1.

    c-rater (students scoring a 3 or 4)

Background: Students in this condition have noted that metal is the best conductor, but have most likely not noted anything about the temperature of the spoons being the same, but feeling different.

Revisit Guidance

You are on the right track. Now, revise your explanation using evidence from the finger/bowl activity. Explain the relationship between how the spoons feel and their temperature. [Hyperlink to visualization—currently Step 3.7—Bowl/Finger simulation]

  1. 2.

    c-rater (students scoring a 0, 1, or 2)

Background: Students in this condition have non-normative ideas or need to extend their explanation further by including a comparison of the spoons.

Revisit Guidance

Good start. Now, improve your answer using evidence from the finger/bowl activity. Explain which spoon will feel the hottest and why. [Hyperlink to visualization—currently Step 3.7—Bowl/Finger simulation]

CRITIQUE Condition

  1. 1.

    c-rater (students scoring a 3 or 4):

Background: Students in this condition have noted that metal is the best conductor, but have most likely not noted anything about the temperature of the spoons being the same, but feeling different.

Critique Guidance

“The metal spoon conducts heat the fastest and the three spoons feel different.”

[Multiple Choice Selection + Open Response] The scientific evidence in the response can be improved by…

  1. (a)

    Explaining that the spoons feel different because they are different temperatures.

  2. (b)

    Explaining that the spoons feel different, because they have different conductivity and NOT because they are different temperatures.

  3. (c)

    Explaining that the metal spoon conducts heat the fastest so it feels the hottest.

Improve your own explanation based on your choice.

  1. 2.

    c-rater (students scoring a 0, 1, or 2)

Background: Students in this condition have non-normative ideas or need to extend their explanation further by including a comparison of the spoons.

Critique Guidance

“Some spoons heat up faster than others.”

[Multiple Choice Selection + Open Response] The scientific evidence in the response can be improved by…

  1. (a)

    Explaining that the plastic spoon conducts heat the fastest.

  2. (b)

    Explaining that the metal spoon conducts heat fast.

  3. (c)

    Explaining that the metal spoon is a better conductor.

Improve your own explanation based on your choice.

Appendix B: Student and Teacher Interview Topic Guides

Student Topic Guide

  • How did you find this question [Step 4.7]: easy, okay, or difficult?

  • Can you explain your answer?

Guidance Step [Step 4.8]

  • How did you find this question: easy, okay, or difficult?

  • Revisit condition: Did you revisit the step it suggests?

  • Critique condition: Can you explain your choice?

  • If you put three spoons made of metal, plastic, and wood in hot water for 30 min, then take the temperature of them, would they be all the same or different temperatures? Would they feel the same?

Teacher Topic Guide: Subset of Questions Asked

  • How was your experience using WISE?

  • What are your classroom norms around writing in science?

  • What characteristics/features did you focus on when examining the students’ written explanations?

  • Were there ideas in the unit your students struggled with? What?

  • Examine critique/revisit guidance. What did you like about the automated guidance? What did you think needed improvement?

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Donnelly, D.F., Vitale, J.M. & Linn, M.C. Automated Guidance for Thermodynamics Essays: Critiquing Versus Revisiting. J Sci Educ Technol 24, 861–874 (2015). https://doi.org/10.1007/s10956-015-9569-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10956-015-9569-1

Keywords

Search

Navigation