Overcoming the PBL Assessment Challenge: Design and Development of the Incremental Thesaurus for Assessing Causal Maps (ITACM)

Abstract

Because of the lack of tools available to assess problem-solving skills, teachers often revert to more traditional instructional approaches (e.g. lecture-based, memorization) that fail to prepare learners for the complexity of dynamic work environments. To overcome this challenge, technology solutions are needed that accurately and efficiently assess complex problem-solving skills such as causal reasoning. Moreover, these tools must be valid and reliable so instructors can accurately assess student learning. This emergent report details the design and development of Incremental Thesaurus for Assessing Causal Maps. As will be described, the software offers three unique features: (a) analysis of causal map with little or no manipulation of the original file; (b) a growing repository of terms that supports efficient assessment and (c) ability to codify the level of concept complexity using the structure–behavior–function framework.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

References

  1. Bastien, M., Heymann, S., & Jacomy, M. (2009) Gephi: An open source software for exploring and manipulating networks. In Proceedings of the international AAAI conference on weblogs and social media.

  2. Baur, M., & Benkert, M. (2005). Network comparison. In U. Brandes & T. Erlebach (Eds.), Network analysis (pp. 318–340). Berlin: Springer.

    Google Scholar 

  3. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.

    Article  Google Scholar 

  4. Chi, M., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5(2), 121–152.

    Article  Google Scholar 

  5. Dickinson, P. J., Bunke, H., Dadej, A., & Kraetzl, M. (2003). On graphs with unique node labels. In E. Hancock & M. Vento (Eds.), Graph based representations in pattern recognition (pp. 13–23). Berlin: Springer.

    Google Scholar 

  6. Ertmer, P. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61.

    Article  Google Scholar 

  7. Ertmer, P., & Ottenbreit-Leftwich, A. (2013). Removing obstacles to the pedagogical changes required by Jonassen’s vision of authentic technology-enabled learning. Computers & Education, 64, 175–182.

    Article  Google Scholar 

  8. Eseryel, D., Ifenthaler, D., & Ge, X. (2013). Validation study of a method for assessing complex ill-structured problem solving by using causal representations. Educational Technology Research and Development, 61(3), 443–463.

    Article  Google Scholar 

  9. Feltovich, P. J., Spiro, R. J., & Coulson, R. L. (1997). Issues of expert flexibility in contexts characterized by complexity and change. In P. Feltovich, K. Fork, & R. Hoffman (Eds.), Expertise in context: Human and machine (pp. 125–146). Cambridge, MA: MIT Press.

    Google Scholar 

  10. Giabbanelli, P. J., & Baniukiewicz, M. (2018). Navigating complex systems for policymaking using simple software tools. In V. Mago, P. J. Giabbanelli, & E. Papageorgiou (Eds.), Advanced data analytics in health. New York, NY: Springer.

    Google Scholar 

  11. Hmelo-Silver, C. (2013). Creating a learning space in problem-based learning. Interdisciplinary Journal of Problem-Based Learning. doi:10.7771/1541-5015.1334.

    Article  Google Scholar 

  12. Hmelo-Silver, C. E., Marathe, S., & Liu, L. (2007). Fish swim, rocks sit, and lungs breathe: Expert-novice understanding of complex systems. Journal of the Learning Sciences, 16(3), 307–331.

    Article  Google Scholar 

  13. Hmelo-Silver, C. E., Nagarajan, A., & Day, R. S. (2002). “It’s harder than we thought it would be”: A comparative case study of expert–novice experimentation strategies. Science Education, 86(2), 219–243.

    Article  Google Scholar 

  14. Hmelo-Silver, C. E., & Pfeffer, M. G. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 28(1), 127–138.

    Article  Google Scholar 

  15. Hung, W. (2011). Theory to reality: A few issues in implementing problem-based learning. Educational Technology Research and Development, 59(4), 529–552.

    Article  Google Scholar 

  16. Hung, W. (2016). All PBL starts here: The problem. Interdisciplinary Journal of Problem-Based Learning, 10(2), 2.

    Article  Google Scholar 

  17. Ifenthaler, D., & Eseryel, D. (2013). Facilitating complex learning by mobile augmented reality learning environments. In R. Huang, Kinshuk, & J. M. Spector (Eds.), Reshaping learning (pp. 415–438). Berlin: Springer.

    Google Scholar 

  18. Ifenthaler, D., & Pirnay-Dummer, P. (2014). Model-based tools for knowledge assessment. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (4th ed., pp. 289–301). New York, NY: Springer.

    Google Scholar 

  19. Jacobson, M. J. (2001). Problem solving, cognition, and complex systems: Differences between experts and novices. Complexity, 6(3), 41–49.

    Article  Google Scholar 

  20. Jeong, A., & Lee, W. J. (2012). Developing causal understanding with causal maps: The impact of total links, temporal flow, and lateral position of outcome nodes. Educational Technology Research and Development, 60(2), 325–340.

    Article  Google Scholar 

  21. Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology, Research and Development, 48(4), 63–85.

    Article  Google Scholar 

  22. Jonassen, D. H. (2011). Learning to solve problems: A handbook for designing problem-solving learning environments (1st ed.). London: Routledge.

    Google Scholar 

  23. Jonassen, D. H., & Hung, W. (2006). Learning to troubleshoot: A new theory-based design architecture. Educational Psychology Review, 18(1), 77–114.

    Article  Google Scholar 

  24. Kolodner, J. (1991). Improving human decision making through case-based decision aiding. AI Magazine, 12(2), 52–68.

    Google Scholar 

  25. Kolodner, J., Hmelo-Silver, C., & Narayanan, N. H. (1996). Problem-based learning meets case-based reasoning. In Proceedings of the 1996 international conference on learning sciences (pp. 188–195). Evanston, Illinois: International Society of the Learning Sciences.

  26. Moon, B., Hoffman, R. R., Novak, J., & Canas, A. (2011). Applied concept mapping. Boca Raton: CRC Press.

    Google Scholar 

  27. Savin-Baden, M. (2004). Understanding the impact of assessment on students in problem-based learning. Innovations in Education and Teaching International, 41(2), 221–233.

    Article  Google Scholar 

  28. Senocak, E. (2009). Development of an instrument for assessing undergraduate science students’ perceptions: The problem-based learning environment inventory. Journal of Science Education and Technology, 18(6), 560–569.

    Article  Google Scholar 

  29. Tamim, S., & Grant, M. (2013). Definitions and uses: Case study of teachers implementing project-based learning. Interdisciplinary Journal of Problem-Based Learning. doi:10.7771/1541-5015.1323.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Andrew A. Tawfik.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Giabbanelli, P.J., Tawfik, A.A. Overcoming the PBL Assessment Challenge: Design and Development of the Incremental Thesaurus for Assessing Causal Maps (ITACM). Tech Know Learn 24, 161–168 (2019). https://doi.org/10.1007/s10758-017-9338-8

Download citation

Keywords

  • Problem-based learning
  • Assessment
  • Causal reasoning
  • Ill-structured problem solving