Learning Analytics to Support Teachers’ Assessment of Problem Solving: A Novel Application for Machine Learning and Graph Algorithms

  • Philippe J. GiabbanelliEmail author
  • Andrew A. Tawfik
  • Vishrant K. Gupta


In contrast to well-structured problems which have pre-defined, correct answers, complex real-world problems are often ill-structured problems (ISPs). The open-ended nature of ISPs creates considerable barriers to assess and guide students in forming better solutions, which results in low adoption levels of inquiry-based learning. Students can structure and represent their knowledge for an ISP in the form of knowledge maps or causal maps, which articulate relevant concepts and their causal relations (i.e., antecedents and consequents). Assessing such maps can involve a referent-free evaluation (e.g., to encourage the creation of maps with high density of concepts) or a comparison to an expert map used as reference. This chapter starts with a review of theories and tools to compare a student’s map to the expert map. Previous approaches often compared individual connections (e.g., scoring the number of connections that a student has/misses in contrast with the expert) or general map metrics (e.g., one map is denser than the other). In contrast, the problem of comparing two maps has been studied in network theory and graph theory for several decades, yielding categories of algorithms that are currently underutilized in educational research. This chapter reviews three categories of algorithms (i.e., graph kernel, graph editing distance, graph embedding) in light of their application to assessment and student success. We discuss an implementation of these algorithms through a new set of digital tools, designed to support a community of practice in problem-based instruction.


Causal maps Conceptual models Ill-structured problems Graph comparison Mental representation Systems thinking 



Vishrant K. Gupta acknowledges funding support from the College of Liberal Arts and Sciences at Northern Illinois University. The authors thank Kaspar Riesen for sharing his implementation of beam search.

Author Contributions

PJG designed the project and supervised VKG. PJG and AAT wrote and revised the manuscript. VKG wrote the software.


  1. Arvind, V., & Jacobo, T. (2005). Isomorphism testing: Perspectives and open problems. Bulletin of the European Association for Theoretical Computer Science, 86, 66–84.Google Scholar
  2. Ausubel, D. G. (1963). Cognitive structure and the facilitation of meaningful verbal learning. Journal of Teacher Education, 14(2), 217–222.Google Scholar
  3. Axelrod, R. (1974). Structure of decision: The cognitive maps of political elites. Princeton, NJ: Princeton University Press.Google Scholar
  4. Barrows, H. (1996). Problem-based learning in medicine and beyond: A brief overview. New Directions for Teaching and Learning, 1996(68), 3–12.Google Scholar
  5. Bax, E. T. (1994). Algorithms to count paths and cycles. Information Processing Letters, 52(5), 249–252.Google Scholar
  6. Bex, F., & Bench-Capon, T. J. (2014). Understanding narratives with argumentation. In COMMA (pp. 11–18).Google Scholar
  7. Budzynska, K., Janier, M., Kang, J., Reed, C., Saint-Dizier, P., Stede, M., & Yaskorska, O. (2014). Towards argument mining from dialogue. In Computational Models of Argument: Proceedings of COMMA 2014 (pp. 185–196).Google Scholar
  8. Carletti, V., Foggia, P., Saggese, A., & Vento, M. (2018). Challenging the time complexity of exact subgraph isomorphism for huge and dense graphs with VF3. IEEE Transactions on Pattern Analysis and Machine Learning, 40(4), 804–818.Google Scholar
  9. Chang, K.-E., Sung, Y.-T., Chang, R.-B., & Lin, S.-C. (2005). A new assessment for computer-based concept mapping. Educational Technology & Society, 8(3), 138–148.Google Scholar
  10. Clariana, R. B. (2010). Deriving individual and group knowledge structure from network diagrams and from essays. In Computer-based diagnostics and systematic analysis of knowledge (pp. 117–130). Boston, MA: Springer.Google Scholar
  11. Clariana, R. B., Engelmann, T., & Yu, W. (2013). Using centrality of concept maps as a measure of problem space states in computer-supported collaborative problem solving. Educational Technology Research & Development, 61(3), 423–442.Google Scholar
  12. Dufresne, R. J., Gerace, W. J., Hardiman, P. T., & Mestre, J. P. (1992). Constraining novices to perform expert like problem analyses: Effects on schema acquisition. Journal of the Learning Sciences, 2(3), 307–331.Google Scholar
  13. Ericsson, K. A. (2005). Recent advances in expertise research: A commentary on the contributions to the special issue. Applied Cognitive Psychology, 19(2), 233–241.Google Scholar
  14. Ertmer, P. A., Stepich, D. A., York, C. S., Stickman, A., Wu, X. L., Zurek, S., & Goktas, Y. (2008). How instructional design experts use knowledge and experience to solve ill-structured problems. Performance Improvement Quarterly, 21(1), 17–42.Google Scholar
  15. Eseryel, D., Ifenthaler, D., & Ge, X. (2013). Validation study of a method for assessing complex ill-structured problem solving by using causal representations. Educational Technology Research & Development, 61(3), 443–463.Google Scholar
  16. Foggia, P., Percannella, G., & Vento, M. (2014). Graph matching and learning in pattern recognition in the last 10 years. International Journal of Pattern Recognition and Artificial Intelligence, 28(1), 1450001.Google Scholar
  17. Ge, X., & Land, S. (2003). Scaffolding students’ problem-solving processes in an ill-structured task using question prompts and peer interactions. Educational Technology Research & Development, 51(1), 21–38.Google Scholar
  18. Giabbanelli, P. J., & Baniukiewicz, M. (2018). Navigating complex systems for policymaking using simple software tools. In P. J. Giabbanelli, V. K. Mago, & E. I. Papageorgiou (Eds.), Advanced data analytics in health (pp. 21–40). Berlin, Germany: Springer.Google Scholar
  19. Giabbanelli, P. J., & Crutzen, R. (2014). Creating groups with similar expected behavioural response in randomized controlled trials: A fuzzy cognitive map approach. BMC Medical Research Methodology, 14(1), 130.Google Scholar
  20. Giabbanelli, P. J., Flarsheim, R. A., Vesuvala, C. X., & Drasic, L. (2016). Developing technology to support policymakers in taking a systems science approach to obesity and well-being. Obesity Reviews, 17, 194–195.Google Scholar
  21. Giabbanelli, P. J., & Tawfik, A. A. (2018). Overcoming the PBL assessment challenge: Design and development of the incremental thesaurus for assessing causal maps (ITACM). Technology, Knowledge and Learning.
  22. Gray, S. A., Hilsberg, J., McFall, A., & Arlinghaus, R. (2015). The structure and function of angler mental models about fish population ecology. Journal of Outdoor Recreation and Tourism, 12, 1–13.Google Scholar
  23. Grotzer, T. A., Kamarainen, A. M., Tutwiler, M. S., Metcalf, S., & Dede, C. (2013). Learning to reason about ecosystems dynamics over time: The challenges of an event-based causal focus. Bioscience, 63(4), 288–296.Google Scholar
  24. Gupta, V. K., Giabbanelli, P. J., & Tawfik, A. A. (2018). An online environment to compare students’ and expert solutions to ill-structured problems. In Proceedings of the 2018 Human Computer Interactions (HCI) conference. To appear.Google Scholar
  25. Hays, J. R., & Simon, H. A. (1974). Understanding written problem instructions. In L. W. Gregg (Ed.), Knowledge and cognition (pp. 167–200). Hillsdale, NJ: Erlbaum.Google Scholar
  26. Herrington, J., Reeves, T. C., & Oliver, R. (2014). Authentic learning environments. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 453–464). New York, NY: Springer.Google Scholar
  27. Hjaltason, G. R., & Samet, H. (2003). Properties of embedding methods for similarity searching in metric spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(5), 530–549.Google Scholar
  28. Hmelo-Silver, C. (2013). Creating a learning space in problem-based learning. Interdisciplinary Journal of Problem-Based Learning, 7(1), 5. Google Scholar
  29. Hmelo-Silver, C., & Barrows, H. (2006). Goals and strategies of a problem-based learning facilitator. Interdisciplinary Journal of Problem-Based Learning, 1(1), 4. Google Scholar
  30. Hmelo-Silver, C., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-novice understanding of complex systems. Journal of the Learning Sciences, 16(3), 307–331.Google Scholar
  31. Hmelo-Silver, C., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28(1), 127–138.Google Scholar
  32. Hong, Y.-C., & Choi, I. (2011). Three dimensions of reflective thinking in solving design problems: A conceptual model. Educational Technology Research & Development, 59(5), 687–710.Google Scholar
  33. Horváth, T. (2005). Cyclic pattern kernels revisited. Lecture Notes in Computer Science, 3518, 791–801.Google Scholar
  34. Hung, W. (2011). Theory to reality: A few issues in implementing problem-based learning. Educational Technology Research & Development, 59(4), 529–552.Google Scholar
  35. Hung, W. (2015). Problem-based learning: Conception, practice, and future. In Y. Cho, I. S. Caleon, & M. Kapur (Eds.), Authentic problem solving and learning in the 21st century (pp. 75–92). Singapore: Springer.Google Scholar
  36. Ifenthaler, D. (2010). Relational, structural, and semantic analysis of graphical representations and concept maps. Educational Technology Research & Development, 58(1), 81–97.Google Scholar
  37. Ifenthaler, D. (2011). Bridging the gap between expert-novice differences: The model-based feedback approach. Journal of Research on Technology in Education, 43(2), 103–117.Google Scholar
  38. Ifenthaler, D. (2012). Determining the effectiveness of prompts for self-regulated learning in problem-solving scenarios. Educational Technology & Society, 15(1), 38–52.Google Scholar
  39. Ifenthaler, D. (2014). AKOVIA: Automated knowledge visualization and assessment. Technology, Knowledge and Learning, 19(1–2), 241–248.Google Scholar
  40. Ifenthaler, D., Masduki, I., & Seel, N. M. (2011). The mystery of cognitive structure and how we can detect it: Tracking the development of cognitive structures over time. Instructional Science, 39(1), 41–61.Google Scholar
  41. Inselberg, A., & Dimsdale, B. (1990). Parallel coordinates: A tool for visualizing multi-dimensional geometry. In Proceedings of the First IEEE Conference on Visualization (pp. 361–378).Google Scholar
  42. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41–49.Google Scholar
  43. Jeong, A. (2014). Sequentially analyzing and modeling causal mapping processes that support causal understanding and systems thinking. In D. Ifenthaler & R. Hanewald (Eds.), Digital knowledge maps in education: Technology-enhanced support for teachers and learners. Chapter 13 (pp. 239–251). New York, NY: Springer.Google Scholar
  44. Jonassen, D. H. (1991). Objectivism versus constructivism: Do we need a new philosophical paradigm? Educational Technology Research & Development, 39(3), 5–14.Google Scholar
  45. Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–94.Google Scholar
  46. Jonassen, D. H. (2011). Learning to solve problems: A handbook for designing problem-solving learning environments (1st ed.). London, England: Routledge.Google Scholar
  47. Ju, H., & Choi, I. (2017). The role of argumentation in hypothetico-deductive reasoning during problem-based learning in medical education: A conceptual framework. Interdisciplinary Journal of Problem-Based Learning, 12(1), 4.Google Scholar
  48. Juszczyszyn, K., Kazienko, P., & Musiał, K. (2008). Local topology of social network based on motif analysis. Lecture Notes in Computer Science, 5178, 97–105.Google Scholar
  49. Kim, K., & Clariana, R. B. (2015). Knowledge structure measures of reader’s situation models across languages: Translation engenders richer structure. Technology, Knowledge and Learning, 20(2), 249–268.Google Scholar
  50. Kim, N. J., Belland, B. R., & Walker, A. E. (2017). Effectiveness of computer-based scaffolding in the context of problem-based learning for STEM education: Bayesian meta-analysis. Educational Psychology Review, 30(2), 397–429.Google Scholar
  51. Kotovsky, K., Hayes, J. R., & Simon, H. A. (1985). Why are some problems hard? Evidence from Tower of Hanoi. Cognitive Psychology, 17(2), 248–294.Google Scholar
  52. Krabbe, H. (2014). Digital concept mapping for formative assessment. In D. Ifenthaler & R. Hanewald (Eds.), Digital knowledge maps in education: Technology-enhanced support for teachers and learners. Chapter 15 (pp. 275–297). New York, NY: Springer.Google Scholar
  53. Lavin, E. A., Giabbanelli, P. J., Stefanik, A. T., Gray, S. A., & Arlinghaus, R. (2018). Should we simulate mental models to assess whether they agree? In Proceedings of the 2018 Spring Simulation Multi-Conference, Annual Simulation Symposium (SpringSim-ANSS).Google Scholar
  54. Lazonder, A., & Harmsen, R. (2016). Meta-analysis of inquiry-based learning: Effects of guidance. Review of Educational Research, 87(4), 1–38.Google Scholar
  55. Loyens, S., & Rikers, R. (2011). Instruction based on inquiry. In Handbook of research on learning and instruction (pp. 361–381). New York, NY: Routledge.Google Scholar
  56. Mahe, P., & Vert, J.-P. (2009). Graph kernels based on tree patterns for molecules. Machine Learning, 75(1), 3–35.Google Scholar
  57. Malhi, L., Karanfil, O., Merth, T., Acheson, M., Palmer, A., & Finegood, D. T. (2009). Places to intervene to make complex food systems more healthy, green, fair, and affordable. Journal of Hunger & Environmental Nutrition, 4(3–4), 466–476.Google Scholar
  58. Meadows, D. H. (2008). Thinking in systems: A primer. Hartford, VT: Chelsea Green Publishing.Google Scholar
  59. Mueller, L. A. J., Dehmer, M., & Emmert-Streib, F. (2013). Comparing biological networks: A survey on graph classifying techniques. In A. Prokop & B. Csukás (Eds.), Systems biology. Dordrecht, The Netherlands: Springer.Google Scholar
  60. Nariman, N., & Chrispeels, J. (2015). PBL in the era of reform standards: Challenges and benefits perceived by teachers in one elementary school. Interdisciplinary Journal of Problem-Based Learning, 10(1), 5.Google Scholar
  61. Olney, A. M., Graesser, A. C., & Person, N. K. (2012). Question generation from concept maps. Dialogue & Discourse, 3(2), 75–99.Google Scholar
  62. Passmore, G. J. (2004). Extending the power of the concept map. Alberta Journal of Educational Research, 50(4), 370–390.Google Scholar
  63. Perera, D., Kay, J., Koprinska, I., Yacef, K., & Zaiane, O. R. (2008). Clustering and sequential pattern mining of online collaborative learning data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759–772.Google Scholar
  64. Riesen, K. (2015). Structural pattern recognition with graph edit distance. In Advances in computer vision and pattern recognition. Berlin, Germany: Springer.Google Scholar
  65. Riesen, K., Emmenegger, S., & Bunke, H. (2013). A novel software toolkit for graph edit distance computation. Lecture Notes in Computer Science, 7877, 142–151.Google Scholar
  66. Riesen, K., Fischer, A., & Bunke, F. (2014). Combining bipartite graph matching and beam search for graph edit distance approximation. Lecture Notes in Computer Science, 8774, 117–128.Google Scholar
  67. Ruiz-Primo, M. (2000). On the use of concept maps as an assessment tool in science: What we have learned so far. Revista Electronica de Investigacion Educativa, 2(1), 29–52.Google Scholar
  68. Sanz, J., Navarro, J., Arbues, A., Martin, C., Marijuan, P. C., & Moreno, Y. (2011). The transcriptional regulatory network of Mycobacterium tuberculosis. PLoS One, 6(7), e22178.Google Scholar
  69. Savery, J. (2006). Overview of problem-based learning: Definitions and distinctions. Interdisciplinary Journal of Problem-Based Learning, 1(1), 3.Google Scholar
  70. Schmidt, H. G., Rotgans, J. I., & Yew, E. (2011). The process of problem-based learning: What works and why. Medical Education, 45(8), 792–806.Google Scholar
  71. Sedki, K. (2018). Formalizing arguments from cause-effect rules. In International conference on industrial, engineering and other applications of applied intelligent systems (pp. 279–285). Basel, Switzerland: Springer.Google Scholar
  72. Sen, S., Li, T. J.-J., Lesicko, M., Weiland, A., Gold, R., Li, Y., … Hecht, B. (2014). WikiBrain: Democratizing computation on Wikipedia. In Proceedings of the International Symposium on Open Collaboration (OpenSym).Google Scholar
  73. Shervashidze, A., Vishwanathan, S. V. N., Petri, T., Melhorn, K., Borgwardt, K. (2009). Efficient graphlet kernels for large graph comparison. In Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, PMLR 5 (pp. 488–495).Google Scholar
  74. Simon, H. A., & Newell, A. (1971). Human problem solving: The state of the theory in 1970. The American Psychologist, 26(2), 145–159.Google Scholar
  75. Skyttner, L. (2006). General systems theory. River Edge, NJ: World Scientific.Google Scholar
  76. Sole-Ribalta, A., Serratosa, F., & Sanfeliu, A. (2012). On the graph edit distance cost: Properties and applications. International Journal of Pattern Recognition and Artificial Intelligence, 26(5), 1260004.Google Scholar
  77. Tamim, S., & Grant, M. (2013). Definitions and uses: Case study of teachers implementing project-based learning. Interdisciplinary Journal of Problem-Based Learning, 7(2), 3. Google Scholar
  78. Tarjan, R. (1973). Enumeration of the elementary circuits of a directed graph. SIAM Journal on Computing, 2(3), 211–216.Google Scholar
  79. Tawfik, A. A., Gill, A., Hogan, M., York, C. S., & Keene, C. W. (2018a). How novices use expert case libraries for problem solving. Technology, Knowledge and Learning, 1–18.
  80. Tawfik, A. A., Law, V., Ge, X., Xing, W., & Kim, K. (2018b). The effect of sustained vs. faded scaffolding on students’ argumentation in ill-structured problem solving. Computers in Human Behavior, 87, 436–449.Google Scholar
  81. Tawfik, A. A., Rong, H., & Choi, I. (2015). Failing to learn: Towards a unified design approach for failure-based learning. Educational Technology Research and Development, 63(6), 975–994.Google Scholar
  82. Trumpower, D. L., Filiz, M., & Sarwar, G. S. (2014). Assessment for learning using digital knowledge maps. In D. Ifenthaler & R. Hanewald (Eds.), Digital knowledge maps in education: Technology-enhanced support for teachers and learners. Chapter 12 (pp. 221–237). New York, NY: Springer.Google Scholar
  83. Trumpower, D. L., & Goldsmith, T. E. (2004). Structural enhancement of learning. Contemporary Educational Psychology, 29, 426–446.Google Scholar
  84. Vento, M. (2015). A long trip in the charming world of graphs for pattern recognition. Pattern Recognition, 48, 291–301.Google Scholar
  85. Walker, A., & Leary, H. (2009). A problem based learning meta analysis: Differences across problem types, implementation types, disciplines, and assessment levels. Interdisciplinary Journal of Problem-Based Learning, 3(1), 6.Google Scholar
  86. Weinerth, K., Koenig, V., Brunner, M., & Martin, R. (2014). Concept maps: A useful and usable tool for computer-based knowledge assessment? A literature review with a focus on usability. Computers & Education, 78, 201–209.Google Scholar
  87. Wijnen, M., Loyens, S. M. M., Smeets, G., Kroeze, M. J., & Van der Molen, H. T. (2017). Students’ and teachers’ experiences with the implementation of problem-based learning at a university law school. Interdisciplinary Journal of Problem-Based Learning, 11(2), 5.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Philippe J. Giabbanelli
    • 1
    Email author
  • Andrew A. Tawfik
    • 2
  • Vishrant K. Gupta
    • 3
  1. 1.Computer Science DepartmentFurman UniversityGreenvilleUSA
  2. 2.Department of Instruction and Curriculum LeadershipUniversity of MemphisMemphisUSA
  3. 3.Computer Science DepartmentNorthern Illinois UniversityDeKalbUSA

Personalised recommendations