Eliciting Informative Feedback in Peer Review: Importance of Problem-Specific Scaffolding

  • Ilya M. Goldin
  • Kevin D. Ashley
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6094)


In a controlled experiment using Comrade, a computer-supported peer review system, student reviewers offered feedback to student authors on their written analyses of a problem scenario. In each condition, reviewers received a different type of rating prompt: domain-related writing composition prompts or problem/issue specific prompts. We found that the reviewers were sensitive to the type of rating prompts they saw and that their ratings of authors’ work were less discriminating with respect to writing composition than to problem-specific issues. In other words, when students gave each other feedback regarding domain-relevant writing criteria, their ratings correlated to a much greater extent, suggesting that such ratings are redundant.


computer-supported peer review ill-defined problem-solving 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Topping, K.: Peer assessment between students in colleges and universities. Review of Educational Research 68, 249–276 (1998)Google Scholar
  2. 2.
    Cho, K., Schunn, C.D.: Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers and Education 48 (2007)Google Scholar
  3. 3.
    Sluijsmans, D.: The use of self-, peer- and co-assessment in higher education: a review of literature. Educational Technology Expertise Centre Open University of the Netherlands, Heerlen (1998)Google Scholar
  4. 4.
    Goldin, I.M., Ashley, K.D., Pinkus, R.L.: Teaching Case Analysis through Framing: Prospects for an ITS in an Ill-defined Domain. In: Workshop on Intelligent Tutoring Systems for Ill-Defined Domains, 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan (2006)Google Scholar
  5. 5.
    VanLehn, K., Lynch, C., Schulze, K., Shapiro, J., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., Wintersgill, M.: The Andes Physics Tutoring System: Lessons Learned. International Journal of Artificial Intelligence and Education 15 (2005)Google Scholar
  6. 6.
    Koedinger, K., Anderson, J., Hadley, W., Mark, M.: Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education 8, 30–43 (1997)Google Scholar
  7. 7.
    Voss, J.: Toulmin’s Model and the Solving of Ill-Structured Problems. In: Arguing on the Toulmin Model: New Essays in Argument Analysis and Evaluation. Springer, Heidelberg (2006)Google Scholar
  8. 8.
    Russell, A.: Calibrated Peer Review: A writing and critical thinking instructional tool. In: Invention and Impact: Building Excellence in Undergraduate Science, Technology, Engineering and Mathematics (STEM) Education. American Association for the Advancement of Science (2004)Google Scholar
  9. 9.
    Gehringer, E.: Strategies and mechanisms for electronic peer review. In: 30th Annual Frontiers in Education Conference, vol. 1, pp. F1B/2-F1B/7 (2000)Google Scholar
  10. 10.
    Zhi-Feng Liu, E., Lin, S., Chiu, C., Yuan, S.: Web-based peer review: the learner as both adapter and reviewer. IEEE Transactions on Education 44, 246–251 (2001)CrossRefGoogle Scholar
  11. 11.
    Masters, J., Madhyastha, T., Shakouri, A.: ExplaNet: A collaborative learning tool and hybrid recommender system for student-authored explanations. Journal of Interactive Learning Research 19, 51–74 (2008)Google Scholar
  12. 12.
    Hsiao, I., Brusilovsky, P.: Modeling peer review in example annotation. In: 16th International Conference on Computers in Education, Taipei, Taiwan, pp. 357–362 (2008)Google Scholar
  13. 13.
    Gouli, E., Gogoulou, A., Grigoriadou, M.: Supporting self-, peer-, and collaborative- assessment in e-learning: the case of the PEer and Collaborative ASSessment Environment (PECASSE). Journal of Interactive Learning Research 19, 615 (2008)Google Scholar
  14. 14.
    Nelson, M., Schunn, C.D.: The nature of feedback: how different types of peer feedback affect writing performance (2008)Google Scholar
  15. 15.
    Walvoord, M.E., Hoefnagels, M.H., Gaffin, D.D., Chumchal, M.M., Long, D.A.: An analysis of Calibrated Peer Review (CPR) in a science lecture classroom. Journal of College Science Teaching 37, 66–73 (2008)Google Scholar
  16. 16.
    Pinkus, R., Gloeckner, C., Fortunato, A.: Professional knowledge and applied ethics: a cognitive science approach (under review)Google Scholar
  17. 17.
    McNamara, D., Kintsch, E., Songer, N., Kintsch, W.: Are good texts always better? interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction 14, 1–43 (1996)CrossRefGoogle Scholar
  18. 18.
    Wooley, R., Was, C.A., Schunn, C.D., Dalton, D.W.: The effects of feedback elaboration on the giver of feedback. In: Love, B.C., McRae, K., Sloutsky, V.M. (eds.) Proceedings of the 30th Annual Conference of the Cognitive Science Society, pp. 2375–2380. Cognitive Science Society, Washington (2008)Google Scholar
  19. 19.
    Hübner, S., Nückles, M., Renkl, A.: Prompting cognitive and metacognitive processing in writing-to-learn enhances learning outcomes. In: 28th Annual Conference of the Cognitive Science Society (2006)Google Scholar
  20. 20.
    Nückles, M., Hübner, S., Renkl, A.: Enhancing self-regulated learning by writing learning protocols. Learning and Instruction 19, 259–271 (2009)CrossRefGoogle Scholar
  21. 21.
    King, A.: ASK to THINK-TEL WHY: A model of transactive peer tutoring for scaffolding higher level complex learning. Educational Psychologist. 32, 221–235 (1997)CrossRefGoogle Scholar
  22. 22.
    Cho, K., Cho, Y.H.: Learning from ill-structured cases. In: 29th Annual Cognitive Science Society Conference, p. 1722. Cognitive Science Society, Austin (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ilya M. Goldin
    • 1
    • 2
  • Kevin D. Ashley
    • 1
    • 2
    • 3
  1. 1.Intelligent Systems ProgramUniversity of PittsburghPittsburgh
  2. 2.Learning Research & Development CenterUniversity of PittsburghPittsburgh
  3. 3.School of LawUniversity of PittsburghPittsburgh

Personalised recommendations