Integrating Real-Time Drawing and Writing Diagnostic Models: An Evidence-Centered Design Framework for Multimodal Science Assessment

  • Andy Smith
  • Osman Aksit
  • Wookhee Min
  • Eric Wiebe
  • Bradford W. Mott
  • James C. Lester
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9684)


Interactively modeling science phenomena enables students to develop rich conceptual understanding of science. While this understanding is often assessed through summative, multiple-choice instruments, science notebooks have been used extensively in elementary and secondary grades as a mechanism to promote and reveal reflection through both drawing and writing. Although each modality has been studied individually, obtaining a comprehensive view of a student’s conceptual understanding requires analyses of knowledge represented across both modalities. Evidence-centered design (ECD) provides a framework for diagnostic measurement of data collected from student interactions with complex learning environments. This work utilizes ECD to analyze a corpus of elementary student writings and drawings collected with a digital science notebook. First, a competency model representing the core concepts of each exercise, as well as the curricular unit as a whole, was constructed. Then, evidence models were created to map between student written and drawn artifacts and the shared competency model. Finally, the scores obtained using the evidence models were used to train a deep-learning based model for automated writing assessment, as well as to develop an automated drawing assessment model using topological abstraction. The findings reveal that ECD provides an expressive unified framework for multimodal assessment of science learning with accurate predictions of student learning.


Assessment Multimodalilty Evidence-centered design 



This work is supported in part by the National Science Foundation through Grant No. DRL-1020229 and the Social Sciences and Humanities Research Council of Canada. Any opinions, findings, conclusions, or recommendations expressed in this report are those of the authors, and do not necessarily represent the official views, opinions, or policy of the National Science Foundation or the Social Sciences and Humanities Research Council of Canada.


  1. 1.
    VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46, 197–221 (2011)CrossRefGoogle Scholar
  2. 2.
    Shute, V.J., Kim, Y.J.: Formative and stealth assessment. In: Spector, J.M., Merrill, M.D., Elen, J., Bishop, M.J. (eds.) Handbook of Research on Educational Communications and Technology, pp. 311–321. Springer, New York (2014)CrossRefGoogle Scholar
  3. 3.
    Bennett, R.E.: Formative assessment: a critical review. Assess. Educ. Principles, Policy Pract. 18, 5–25 (2011)CrossRefGoogle Scholar
  4. 4.
    Desmarais, M.C., Baker, R.S.J.D.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User Adap. Interact. 22, 9–38 (2011)CrossRefGoogle Scholar
  5. 5.
    Minogue, J., Wiebe, E., Bedward, J., Carter, M.: The intersection of science notebooks, graphics, and inquiry. Sci. Child. 48, 52–55 (2010)Google Scholar
  6. 6.
    Mislevy, R.J., Haertel, G.D.: Implications of evidence-centered design for educational testing. Educ. Measur. Issues Pract. 25, 6–20 (2006)CrossRefGoogle Scholar
  7. 7.
    Nelligan, T., Helms, M., Polsley, S., Linsey, J., Ray, J., Hammond, T.: Mechanix : a sketch-based educational interface. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, pp. 53–56. ACM (2015)Google Scholar
  8. 8.
    Bollen, L., van Joolingen, W.: SimSketch: multi-agent simulations based on learner-created sketches for early science education. IEEE Trans. Learn. Technol. 6, 208–216 (2013)CrossRefGoogle Scholar
  9. 9.
    Jee, B.D., Gentner, D., Uttal, D.H., Sageman, B., Forbus, K., Manduca, C.A., Ormand, C.J., Shipley, T.F., Tikoff, B.: Drawing on experience: how domain knowledge is reflected in sketches of scientific structures and processes. Res. Sci. Educ. 44, 859–883 (2014)CrossRefGoogle Scholar
  10. 10.
    Burrows, S., Gurevych, I., Stein, B.: The eras and trends of automatic short answer grading. Int. J. Artif. Intell. Educ. 25, 60–117 (2014)CrossRefGoogle Scholar
  11. 11.
    Graesser, A.: Using latent semantic analysis to evaluate the contributions of students in autotutor. Interact. Learn. Environ. 8, 1–33 (2000)CrossRefGoogle Scholar
  12. 12.
    Sukkarieh, J., Blackmore, J.: C-rater: automatic content scoring for short constructed responses. In: Proceedings of the 22nd International FLAIRS Conference, pp. 290–295 (2009)Google Scholar
  13. 13.
    Dzikovska, M., Nielsen, R., Brew, C.: Towards effective tutorial feedback for explanation questions: a dataset and baselines. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Montreal, Canada, pp. 200–210 (2012)Google Scholar
  14. 14.
    Leeman-Munk, S., Smith, A., Mott, B., Wiebe, E., Lester, J.: Two modes are better than one: a multimodal assessment framework integrating student writing and drawing. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M. (eds.) AIED 2015. LNCS, vol. 9112, pp. 205–215. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  15. 15.
    Gobert, J.D., Pedro, M.A.S., Baker, R.S.J.D., Toto, E., Montalvo, O.: Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds. J. Educ. Data Min. 4, 111–143 (2012)Google Scholar
  16. 16.
    Rupp, A., Levy, R., Dicerbo, K.E., Sweet, S.J., Crawford, A.V., Calico, T., Benson, M., Fay, D., Kunze, K.L., Mislevy, R.J., Behrens, J.: Putting ECD into practice: the interplay of theory and data in evidence models within a digital learning environment. J. Educ. Data Min. 4, 49–110 (2012)Google Scholar
  17. 17.
    Min, W., Frankosky, M.H., Mott, B.W., Rowe, J.P., Wiebe, E., Boyer, K.E., Lester, J.C.: DeepStealth: leveraging deep learning models for stealth assessment in game-based learning environments. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M. (eds.) AIED 2015. LNCS, vol. 9112, pp. 277–286. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  18. 18.
    Mislevy, R.J., Almond, R.G., Lukas, J.F.: A brief introduction to evidence-centered design. ETS Research Report Series, 16 (2003)Google Scholar
  19. 19.
    Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Andy Smith
    • 1
  • Osman Aksit
    • 2
  • Wookhee Min
    • 1
  • Eric Wiebe
    • 2
  • Bradford W. Mott
    • 1
  • James C. Lester
    • 1
  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA
  2. 2.Department of STEM EducationNorth Carolina State UniversityRaleighUSA

Personalised recommendations