An Online Environment to Compare Students’ and Expert Solutions to Ill-Structured Problems

  • Vishrant K. Gupta
  • Philippe J. GiabbanelliEmail author
  • Andrew A. Tawfik
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10925)


Practitioners often face ill-structured problems. However, it is difficult for instructors to assess their students’ work on such problems, as a broad set of solutions exist and may depend on the context. One way to assess student learning is through the evaluation of their mental models, which can be presented in the form of a causal network or ‘map’. While comparing a student’s map to an expert’s map can assist with the evaluation, this is a challenging process, in part, due to variations in language, resulting in the use of different terms for the same construct. The first step of the comparison is to address these variations by aligning as many of the students’ terms with their equivalent in the expert’s map. We present the design and implementation of a software to assist with the alignment task. The software improves on previous work by optimizing usability (e.g., minimizing the number of clicks to create an alignment) and by leveraging previous alignments to recommend new ones. In addition, alignments can be done collaboratively, as our system is available online: one instructor can invite others to edit or see the alignments. Further improvements to this system may be achieved using content-based recommender systems or natural language processing.


Community of practice Mental models Network comparison Recommender system 


  1. 1.
    Jonassen, D.H.: Toward a design theory of problem solving. Educ. Technol. Res. Dev. 48(4), 63–85 (2000)CrossRefGoogle Scholar
  2. 2.
    Senocak, E.: Development of an instrument for assessing undergraduate science students’ perceptions: the problem-based learning environment inventory. J. Sci. Educ. Technol. 18(6), 560–569 (2009)CrossRefGoogle Scholar
  3. 3.
    Savin-Baden, M.: Understanding the impact of assessment on students in problem based learning. Innov. Educ. Teach. Int. 41(2), 221–233 (2004)CrossRefGoogle Scholar
  4. 4.
    Ifenthaler, D., Pirnay-Dummer, P.: Model-Based Tools for Knowledge Assessment. In: Spector, J., Merrill, M., Elen, J., Bishop, M. (eds.) Handbook of Research on Educational Communications and Technology, pp. 289–301. Springer, New York (2014). Scholar
  5. 5.
    Giabbanelli, P.J., Tawfik, A.A.: Overcoming the PBL assessment challenge: design and development of the incremental thesaurus for assessing causal maps (ITACM). Technol. Knowl. Learn. (2017).
  6. 6.
    Hmelo-Silver, C.E., Pfeffer, M.G.: Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cogn. Sci. 28(1), 127–138 (2004)CrossRefGoogle Scholar
  7. 7.
    Jacobson, M.J.: Problem solving, cognition, and complex systems: differences between experts and novices. Complexity 6(3), 41–49 (2001)CrossRefGoogle Scholar
  8. 8.
    Baur, M., Benkert, M.: Network comparison. In: Brandes, U., Erlebach, T. (eds.) Network Analysis. LNCS, vol. 3418, pp. 318–340. Springer, Heidelberg (2005). Scholar
  9. 9.
    Dickinson, P.J., Bunke, H., Dadej, A., Kraetzl, M.: On graphs with unique node labels. In: Hancock, E., Vento, M. (eds.) GbRPR 2003. LNCS, vol. 2726, pp. 13–23. Springer, Heidelberg (2003). Scholar
  10. 10.
    Spector, J.M., et al.: Technology enhanced formative assessment for 21st century learning. J. Educ. Technol. Soc. 19(3), 58–731 (2016)Google Scholar
  11. 11.
    Koller, D., Ng, A., Do, C., Chen, Z.: Retention and intention in massive open online courses: in depth. Educause Review 3(48), 62–63 (2013)Google Scholar
  12. 12.
    Hew, K., Cheung, W.: Students’ and instructors’ use of massive open onlien courses (MOOCs): motivations and challenges. Educ. Res. Rev. 12, 45–58 (2014)CrossRefGoogle Scholar
  13. 13.
    Jordan, K.: Initial trends in enrolment and completion of massive open online courses. Int. Rev. Res. Open Distrib. Learn. 15(1), 133–160 (2014)CrossRefGoogle Scholar
  14. 14.
    Sher, A.: Assessing the relationship of student-instructor and student-student interaction to student learning and satisfaction in web-based online learning environment. 8, 102–120 (2009)Google Scholar
  15. 15.
    Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 40(6), 601–618 (2010)CrossRefGoogle Scholar
  16. 16.
    Sclater, N., Peasgood, A., Mullan, J.: Learning analytics in higher education: a review of UK and international practice (2017)Google Scholar
  17. 17.
    Rossi, L.A., Gnawali, O.: Language independent analysis and classification of discussion threads in coursera MOOC forums. In: 15th IEEE International Conference on Information Reuse and Integration, pp. 654–661. IEEE (2014)Google Scholar
  18. 18.
    Liu, W., Kidziński, Ł., Dillenbourg, P.: Semiautomatic annotation of MOOC forum posts. In: Li, Y., et al. (eds.) tate-of-the-Art and Future Directions of Smart Learning. LNET, pp. 399–408. Springer, Singapore (2016). Scholar
  19. 19.
    Agrawal, A., Venkatraman, J., Leonard, S., Paepcke, A.: YouEDU: addressing confusion in MOOC discussion forums by recommending instructional video clips, Stanford InfoLab, February 2015Google Scholar
  20. 20.
    He, J., Bailey, J., Rubinstein, B.I.P., Zhang, R.: Identifying at-risk students in massive open online courses. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 1749–1755. AAAI Press (2015)Google Scholar
  21. 21.
    Jenders, M., Krestel, R., Naumann, F.: Which answer is best? Predicting accepted answers in MOOC forums. In: Proceedings of the 25th International Conference Companion on World Wide Web, pp. 679–684 (2016)Google Scholar
  22. 22.
    Kotsiantis, S.B., Pierrakeas, C.J., Pintelas, P.E.: Preventing student dropout in distance learning using machine learning techniques. In: Palade, V., Howlett, R.J., Jain, L. (eds.) KES 2003. LNCS (LNAI), vol. 2774, pp. 267–274. Springer, Heidelberg (2003). Scholar
  23. 23.
    Qiu, J., Tang, J., Liu, T.X., Gong, J., Zhang, C., Zhang, Q., Xue, Y.: Modeling and predicting learning behavior in MOOCs. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, WSDM 2016, pp. 93–102. ACM (2016)Google Scholar
  24. 24.
    Kizilcec, R.F., Piech, C., Schneider, E.: Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. ACM (2013)Google Scholar
  25. 25.
    Ferguson, R., Clow, D.: Examining engagement: analysing learner subpopulations in massive open online courses (MOOCs). In: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, LAK 2015, pp. 51–58 (2015)Google Scholar
  26. 26.
    Ezen-Can, A., Boyer, K.E., Kellogg, S., Booth, S.: Unsupervised modeling for understanding MOOC discussion forums: a learning analytics approach. In: Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, LAK 2015, pp. 146–150. ACM (2015)Google Scholar
  27. 27.
    Stahl, G., Koschmann, T., Suthers, D.: Computer-supported collaborative learning: an historical perspective. In: Cambridge Handbook of the Learning Sciences, pp. 409–426 (2006)Google Scholar
  28. 28.
    Johnson, D., Johnson, R.: Cooperation and the use of technology. In: Handbook of Research on Educational Communications and Technology, pp. 659–670. Routledge, New York (2008)Google Scholar
  29. 29.
    Goggins, S., Galyen, K., Petakovic, E., Laffey, J.: Connecting performance to social structure and pedagogy as a pathway to scaling learning analytics in MOOCs: an exploratory study. J. Comput. Assist. Learn. 32(3), 244–266 (2016)CrossRefGoogle Scholar
  30. 30.
    Gillani, N., Eynon, R.: Communication patterns in massively open online courses. Internet High. Educ. 23, 18–26 (2014)CrossRefGoogle Scholar
  31. 31.
    Tawfik, A.A., Reeves, T.D., Stich, A.E., Gill, A., Hong, C., McDade, J., Pillutla, V.S., Zhou, X., Giabbanelli, P.J.: The nature and level of learner-learner interaction in a chemistry massive open online course (MOOC). J. Comput. High. Educ. 29(3), 411–431 (2017)CrossRefGoogle Scholar
  32. 32.
    Dowell, N., Skrypnyk, O., Joksimovic, S., Graesser, A., Dawson, S., Gasevic, D., Hennis, T., de Vries, P., Kovanovic, V.: Modeling learners’ social centrality and performance through language and discourse. In: 8th International Conference on Educational Data Mining (EDM) (2015)Google Scholar
  33. 33.
    Jiang, S., Fitzhugh, S., Warschauer, M.: Social positioning and performance in MOOCs. In: Workshop on Graph-Based Educational Data Mining, p. 14 (2014)Google Scholar
  34. 34.
    Tawfik, A.A., Giabbanelli, P.J., Hogan, M., Msilu, F., Gill, A., York, C.S.: Effects of success v failure cases on learner-learner interaction. Comput. Educ. 118, 120–132 (2018)CrossRefGoogle Scholar
  35. 35.
    Dascalu, M., McNamara, D.S., Trausan-Matu, S., Allen, L.K.: Cohesion network analysis of CSCL participation. Behav. Res. Methods 50(2), 1–16 (2017)Google Scholar
  36. 36.
    Hecking, T., Ziebarth, S., Hoppe, H.U.: Analysis of dynamic resource access patterns in online courses. J. Learn. Anal. 1(2), 34–60 (2014)CrossRefGoogle Scholar
  37. 37.
    Schumacher, C., Ifenthaler, D.: Features students really expect from learning analytics. Comput. Hum. Behav. 78, 397–407 (2018)CrossRefGoogle Scholar
  38. 38.
    Ifenthaler, D.: AKOVIA: automated knowledge visualization and assessment. Technol. Knowl. Learn. 19(1–2), 241–248 (2014)CrossRefGoogle Scholar
  39. 39.
    Risch, J.S., May, R.A., Dowson, S.T., Thomas, J.J.: A virtual environment for multimedia intelligence data analysis. IEEE Comput. Graph. Appl. 16(6), 33–41 (1996)CrossRefGoogle Scholar
  40. 40.
    Cockburn, A., McKenzie, B.: Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2002, New York, pp. 203–210. ACM (2002)Google Scholar
  41. 41.
    Stasko, J., Görg, C., Liu, Z.: Jigsaw: supporting investigative analysis through interactive visualization. Inf. Vis. 7(2), 118–132 (2008)CrossRefGoogle Scholar
  42. 42.
    Aggarwal, C.C.: Content-based recommender systems. Recommender Systems, pp. 139–166. Springer, Cham (2016). Scholar
  43. 43.
    Gedikli, F., Jannach, D., Ge, M.: How should i explain? A comparison of different explanation types for recommender systems. Int. J. Hum. Comput. Stud. 72(4), 367–382 (2014)CrossRefGoogle Scholar
  44. 44.
    Sen, S., Li, T.J.J., Team, W., Hecht, B.: WikiBrain: democratizing computation on Wikipedia. In: Proceedings of the International Symposium on Open Collaboration, OpenSym 2014, pp. 27:1–27:10. ACM (2014)Google Scholar
  45. 45.
    Lavin, E.A., Giabbanelli, P.J., Stefanik, A.T., Gray, S.A., Arlinghaus, R.: Should we simulate mental models to assess whether they agree? In: Proceedings of the Annual Simulation Symposium (ANSS) at the Spring Simulation Multi-conference (2018)Google Scholar
  46. 46.
    Drasic, L., Giabbanelli, P.: Exploring the interactions between physical well-being, and obesity. Can. J. Diabetes 39, S12–S13 (2015)CrossRefGoogle Scholar
  47. 47.
    Giabbanelli, P.J., Crutzen, R.: Creating groups with similar expected behavioural response in randomized controlled trials: a fuzzy cognitive map approach. BMC Med. Res. Methodol. 14(1), 130 (2014)CrossRefGoogle Scholar
  48. 48.
    Pratt, S.F., Giabbanelli, P.J., Mercier, J.S.: Detecting unfolding crises with visual analytics and conceptual maps emerging phenomena and big data. In: IEEE International Conference on Intelligence and Security Informatics, pp. 200–205 (2013)Google Scholar
  49. 49.
    Giabbanelli, P.J.: Modelling the spatial and social dynamics of insurgency. Secur. Inform. 3(1), 2 (2014)CrossRefGoogle Scholar
  50. 50.
    Giabbanelli, P.J., Jackson, P.J.: Using visual analytics to support the integration of expert knowledge in the design of medical models and simulations. Procedia Comput. Sci. 51, 755–764 (2015). International Conference on Computational Science, ICCS 2015CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Vishrant K. Gupta
    • 1
  • Philippe J. Giabbanelli
    • 2
    Email author
  • Andrew A. Tawfik
    • 3
  1. 1.Department of Computer ScienceNorthern Illinois UniversityDeKalbUSA
  2. 2.Computer Science DepartmentFurman UniversityGreenvilleUSA
  3. 3.Department of Instructional Design and TechnologyUniversity of MemphisMemphisUSA

Personalised recommendations