Thresholds of knowledge development in complex problem solving: a multiple-case study of advanced learners’ cognitive processes

  • Treavor Bogard
  • Min Liu
  • Yueh-hui Vanessa Chiang
Research Article


This multiple-case study examined how advanced learners solved a complex problem, focusing on how their frequency and application of cognitive processes contributed to differences in performance outcomes, and developing a mental model of a problem. Fifteen graduate students with backgrounds related to the problem context participated in the study. Data sources included direct observation of solution operations, participants’ think aloud and stimulated recalls as they solved the problem, as well as solution scores indicating how well each participant solved the problem. A grounded theory approach was used to analyze stimulated recall and think aloud data. A set of thirteen cognitive processes emerged in the coding and were tallied for each participant. Individual cases were then grouped into clusters that shared similar frequencies of prior knowledge activation, performance outcomes, and tool use behaviors. Each cluster was profiled from least to most successful with descriptive accounts of each cluster’s approach to solving the problem. A cross cluster analysis indicated how learners’ cognitive processes corresponded with problem solving operations that revealed thresholds of knowledge development and formed an integrated mental model of the problem. The findings suggested that mastering problem solving operations within each threshold enhanced the learners’ conceptual awareness of where to apply cognitive processes and increased the combinations of cognitive processes they activated at higher thresholds of knowledge development. The findings have implications for anticipating where novices need support within each threshold of knowledge development during complex problem solving.


Cognitive processes Cognitive tools Complex problem solving Developing expertise Mental models Problem representation 


  1. Alibali, M. W., Phillips, K. M. O., & Fischer, A. D. (2009). Learning new problem-solving strategies leads to changes in problem representation. Cognitive Development, 24(2), 89–101. doi:10.1016/j.cogdev.2008.12.005.CrossRefGoogle Scholar
  2. Anderson, J. R. (2009). Expertise. In Cognitive psychology and its implications (7th ed., pp. 242–271). New York: W. H. Freeman.Google Scholar
  3. Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.Google Scholar
  4. Azevedo, R., & Hadwin, A. (2005, November). Scaffolding self-regulated learning and metacognition: Implications for the design of computer-based scaffolds. Instructional Science, 33(5), 367–379. doi:10.1007/s11251-005-1272-9.
  5. Bannert, M., & Reimann, P. (2011). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40(1), 193–211.CrossRefGoogle Scholar
  6. Belland, B. (2010). Portraits of middle school students constructing evidence-based arguments during problem-based learning: The impact of computer-based scaffolds. Educational Technology Research and Development, 58(3), 285–309. doi:10.1007/s11423-009-9139-4.CrossRefGoogle Scholar
  7. Belland, B., Glazewski, K., & Richardson, J. (2011). Problem-based learning and argumentation: Testing a scaffolding framework to support middle school students’ creation of evidence-based arguments. Instructional Science, 39(5), 667–694. doi:10.1007/s11251-010-9148-z.CrossRefGoogle Scholar
  8. Bixler, B. A., & Land, S. M. (2010). Supporting college students’ ill-structured problem solving in a web-based learning environment. Journal of Educational Technology Systems, 39(1), 3–15. doi:10.2190/ET.39.1.b.CrossRefGoogle Scholar
  9. Bloom, B. S., Englehart, M. B., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives, the classification of educational goals: Handbook I—Cognitive Domain. New York: McKay.Google Scholar
  10. Borkowski, J. G., Johnston, M. B., & Reid, M. K. (1987). Metacognition, motivation, and controlled performance. In S. J. Ceci (Ed.), Handbook of cognitive, social, and neuropsychological aspects of learning disabilities (Vol. 2, pp. 147–174). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  11. Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How experts differ from novices. In J. D. Bransford, A. L. Brown, & R. R. Cocking (Eds.), How people learn: Brain, mind, experience, and school (pp. 31–50). Washington, D.C.: National Academy Press.Google Scholar
  12. Bransford, J. D., & Stein, B. S. (1984). The IDEAL problem solver: A guide for improving thinking, learning, and creativity. New York: W. H. Freeman.Google Scholar
  13. Calderhead, J. (1981). Stimulated recall: A method for research on reaching. British Journal of Educational Psychology, 51(2), 211–217. doi:10.1111/j.2044-8279.1981.tb02474.x.CrossRefGoogle Scholar
  14. Charness, N. (1976). Memory of chess positions: Resistance to interference. Journal of Experimental Psychology, 2(6), 641–653. doi:10.1037/0278-7393.2.6.641.Google Scholar
  15. Chase, W. G., & Ericsson, K. A. (1982). Skill and working memory. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 16, pp. 1–58). New York: Academic Press.Google Scholar
  16. Chase, W. G., & Simon, H. A. (1973). The mind’s eye in chess. In W. G. Chase (Ed.), Visual information processing (pp. 215–281). New York: Academic Press.Google Scholar
  17. Chi, M. T. H., & Bassock, M. (1991). Learning from examples vs. self-explanations. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 251–282). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  18. Chi, M. T. H., Feltovitch, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121–152. doi:10.1207/s15516709cog0502_2.CrossRefGoogle Scholar
  19. Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In R. J. Sternberg (Ed.), Advances in the psychology of human intelligence (pp. 7–75). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  20. Cho, K., & Jonassen, D. H. (2002, September). The effects of argumentation scaffolds on argumentation and problem solving. Educational Technology Research and Development, 50(3), 5–22. doi:10.1007/BF02505022.
  21. Corbin, J. M., & Strauss, A. M. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory. Los Angeles: Sage.Google Scholar
  22. Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. Los Angeles: Sage.Google Scholar
  23. de Jong, T., & Ferguson-Hessler, M. G. M. (1991). Knowledge of problem situations in physics: A comparison of good and poor novice problem solvers. Learning and Instruction, 1(4), 289–302. doi:10.1016/0959-4752(91)90010-6.CrossRefGoogle Scholar
  24. Derry, S. J. (1996). Cognitive schema theory in the constructivist debate. Educational Psychologist, 31(3), 163–174.CrossRefGoogle Scholar
  25. Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. Oxford: B. Blackwell.Google Scholar
  26. Dunkle, M. E., Schraw, G., & Bendixen, L. D. (1995). Cognitive processes in well-defined and ill-defined problem solving. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.Google Scholar
  27. Efklides, A. (2008). Metacognition: Defining its facets and levels of functioning in relation to self-regulation and co-regulation. European Psychologist, 13(4), 277–287.CrossRefGoogle Scholar
  28. Ericsson, K. A., Prietula, M. J., & Cokely, E. T. (2007). The making of an expert. Harvard Business Review, 85(7/8), 114.Google Scholar
  29. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.Google Scholar
  30. Ericsson, K. A., & Staszewsli, J. J. (1989). Skilled memory and expertise: Mechanisms of exceptional performance. In D. Klahr & K. Kotovsky (Eds.), Complex information processing: The impact of Herbert A. Simon (pp. 235–267). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  31. Flavell, J. H. (1971). Stage-related properties of cognitive development. Cognitive Psychology, 2, 421–453. doi:10.1016/0010-0285(71)90025-9.CrossRefGoogle Scholar
  32. Foshay, R., & Kirkley, J. (1998). Principles for teaching problem solving. Edina, MN: TRO Learning.Google Scholar
  33. Funke, J., & Frensch, P. A. (1995). Complex problem solving research in North America and Europe: An integrative review. Foreign Psychology, 5, 42–47.Google Scholar
  34. Gillespie, M. (2002). EFF research principle: An approach to teaching and learning that builds expertise. In Equipped for the future (Vol. 2, pp. 1–8). Washington, D.C.: National Institute for Literacy.Google Scholar
  35. Glaser, R. (1989). Expertise and learning: How do we think about instructional processes now that we have discovered knowledge structures? In D. Klahr & K. Kotovsky (Eds.), Complex information processing: The impact of Herbert A. Simon (pp. 269–282). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  36. Henning, P. H. (2004). Everyday cognition and situated learning. In D. H. Jonassen (Ed.), Handbook of research on educational communications and technology (2nd ed., pp. 143–168). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  37. Iiyoshi, T., Hannifin, M. J., & Wang, F. (2005). Cognitive tools and student-centered learning: Rethinking tools, functions, and applications. Educational Media International, 42, 281–296. doi:10.1080/09523980500161346.CrossRefGoogle Scholar
  38. Ionescu, T. (2012). Exploring the nature of cognitive flexibility. New Ideas in Psychology, 30(2), 190–200. doi:10.1016/j.newideapsych.2011.11.001.CrossRefGoogle Scholar
  39. Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  40. Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85. doi:10.1007/BF02300500.CrossRefGoogle Scholar
  41. Jonassen, D. H. (2003). Using cognitive tools to represent problems. Journal of Research on Technology in Education, 35(3), 362–381.Google Scholar
  42. Jonassen, D. H. (2005). Tools for representing problems and the knowledge required to solve them. In S. Tergan & T. Keller (Eds.), Lecture notes in computer science (Vol. 3426, pp. 93–95). Berlin: Springer.Google Scholar
  43. Jonassen, D. H. (2006). Modeling with technology: Mindtools for conceptual change. Upper Saddle River, NJ: Merrill Prentice Hall.Google Scholar
  44. Jonassen, D. H., & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 693–719). New York: Macmillan.Google Scholar
  45. Jonassen, D. H., & Strobel, J. (2006). Modeling for meaningful learning. In D. Hung & M. S. Khine (Eds.), Engaged learning with emerging technologies (pp. 1–27). Dordrecht, South Holland: Springer.CrossRefGoogle Scholar
  46. Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424. doi:10.1080/07370000802212669.CrossRefGoogle Scholar
  47. Kapur, M., & Rummel, N. (2012). Productive failure in learning from generation and invention activities. Instructional Science, 40(4), 645–650. doi:10.1007/s11251-012-9235-4.CrossRefGoogle Scholar
  48. Kim, M. K. (2012). Theoretically grounded guidelines for assessing learning progress: Cognitive changes in ill-structured complex problem-solving contexts. Educational Technology Research and Development, 60(4), 601–622. doi:10.1007/s11423-012-9247-4.CrossRefGoogle Scholar
  49. Kim, B., & Reeves, T. (2007). Reframing research on learning with technology: In search of the meaning of cognitive tools. Instructional Science, 35(3), 207–256. doi:10.1007/s11251-006-9005-2.CrossRefGoogle Scholar
  50. Kleinmuntz, D. N., & Schkade, D. A. (1993). Information displays and decision processes. Psychological Science, 4(4), 221–227. doi:10.1111/j.1467-9280.1993.tb00265.x.CrossRefGoogle Scholar
  51. Kluwe, R. H. (1995). Single case studies and models of complex problem solving. In P. A. Frensch & J. Funke (Eds.), Complex problem solving: The European perspective (pp. 269–291). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  52. Kozma, R. (2003). The material features of multiple representations and their cognitive and social affordances for science understanding. Learning and Instruction, 13(2), 205–226. doi:10.1016/S0959-4752(02)00021-X.CrossRefGoogle Scholar
  53. Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. doi:10.1207/s15430421tip4104_2.CrossRefGoogle Scholar
  54. Lajoie, S. P. (1993). Computer environments as cognitive tools for enhancing learning. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 261–288). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  55. Lajoie, S. P. (2000). Computers as cognitive tools: No more walls (Vol. 2). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  56. Lajoie, S. P. (2008). Metacognition, self-regulation, and self-regulated learning: A rose by any other name? Educational Psychology Review, 20(4), 469–475. doi:10.1007/s10648-0089088-1.CrossRefGoogle Scholar
  57. Larkin, J. H. (1983). The role of problem representation in physics. In D. Gentner & A. L. Stevens (Eds.), Mental models (pp. 75–98). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  58. Lee, H., Lim, K., & Grabowski, B. (2010). Improving self-regulation, learning strategy use, and achievement with metacognitive feedback. Educational Technology Research and Development, 58(6), 629–648.CrossRefGoogle Scholar
  59. Liu, M., & Bera, S. (2005). An analysis of cognitive tool use patterns in a hypermedia learning environment. Educational Technology Research and Development, 53(1), 5–21.CrossRefGoogle Scholar
  60. Liu, M., Bera, S., Corliss, S., Svinicki, M., & Beth, A. (2004). Understanding the connection between cognitive tool use and cognitive processes as used by sixth graders in a problem-based hypermedia learning environment. Journal of Educational Computing Research, 31(3), 309–334.CrossRefGoogle Scholar
  61. Liu, M., Horton, L., Corliss, S. B., Svinicki, M. D., Bogard, T., Kim, J., et al. (2009). Students’ problem-solving as mediated by their cognitive tool use: A study of tool use patterns. Journal of Educational Computing Research, 40(1), 111–139.CrossRefGoogle Scholar
  62. Liu, M., Horton, L., Kang, J., Kimmons, R. & Lee, J. (2013a). Using a Ludic simulation to make learning of middle school space science fun. The International Journal of Gaming and Computer-Mediated Simulations, 5(1).Google Scholar
  63. Liu, M., Yuen, T. T. Horton, L., Lee, J., Toprac, P. and Bogard, T. (2013b). Designing technology-enriched cognitive tools to support young learners’ problem solving. The International Journal of Cognitive Technology.Google Scholar
  64. Livingston, E., & Borko, H. (1989). Expert-novice differences in teaching: A cognitive analysis and implications for teacher education. Journal of Teacher Education, 40(4), 36–42. doi:10.1177/002248718904000407.CrossRefGoogle Scholar
  65. Lyle, J. (2003). Stimulated recall: A report on its use in naturalistic research. British Educational Research Journal, 29(6), 861–878. doi:10.1080/0141192032000137349.CrossRefGoogle Scholar
  66. Mayer, R. E. (1989). Models for understanding. Review of Educational Research, 59, 43–64. doi:10.3102/00346543059001043.CrossRefGoogle Scholar
  67. Mayer, R. E. (1998). Cognitive, metacognitive, and motivational aspects of problem solving. Instructional Science, 26(1–2), 49–63.CrossRefGoogle Scholar
  68. Mayer, R. E., & Wittrock, M. C. (1996). Problem-solving transfer. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 47–62). New York: Macmillan.Google Scholar
  69. Merriënboer, J., & Stoyanov, S. (2008). Learners in a changing learning landscape: Reflection from an instructional design perspective. In J. Visser, M. Visser-Valfrey, D. N. Aspin, & J. D. Chapman (Eds.), Lifelong learning book series (Vol. 12, pp. 69–90). Dordrecht, South Holland: Springer.Google Scholar
  70. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.Google Scholar
  71. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. doi:10.1037/h0043158.CrossRefGoogle Scholar
  72. Patel, V. L., & Groen, G. J. (1991). The general and specific nature of medical expertise: A critical look. In K. A. Ericsson & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 93–125). New York: Cambridge University Press.Google Scholar
  73. Pea, R. D. (1985). Beyond amplification: Using the computer to reorganize mental functioning. Educational Psychologist, 20(4), 167–182. doi:10.1207/s15326985ep2004_2.CrossRefGoogle Scholar
  74. Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review, 18(4), 315–341. doi:10.1007/s10648-006-9029-9.CrossRefGoogle Scholar
  75. Pintrich, P. R. (2000). An achievement goal theory perspective on issues in motivation terminology, theory, and research. Contemporary Educational Psychology, 25(1), 92–104. doi:10.1006/ceps.1999.1017.CrossRefGoogle Scholar
  76. Poitras, E., Lajoie, S., & Hong, Y. J. (2011). The design of technology-rich learning environments as metacognitive tools in history education. Instructional Science, 1–29.Google Scholar
  77. Prawat, R. S. (1989). Promoting access to knowledge, strategy, and disposition in students: A research synthesis. Review of Educational Research, 59, 1–41. doi:10.2307/1170445.CrossRefGoogle Scholar
  78. Puntambekar, S., & Hübscher, R. (2005). Tools for scaffolding students in a complex learning environment: What have we gained and what have we missed? Educational Psychologist, 40(1), 1–12. doi:10.1207/s15326985ep4001_1.CrossRefGoogle Scholar
  79. Ramdass, D., & Zimmerman, B. J. (2011). Developing self-regulation skills: The important role of homework. Journal of Advanced Academics, 22(2), 194–218.CrossRefGoogle Scholar
  80. Salomon, G., Perkins, D. N., & Globerson, T. (1991). Partners in cognition: Extending human intelligent technologies. Educational Researcher, 20(3), 2–9.CrossRefGoogle Scholar
  81. Savelsbergh, E. R., de Jong, T., & Ferguson-Hessler, M. G. M. (1998). Competence-related differences in problem representations. In M. van Sommeren, P. Reimann, T. de Jong, & H. Boshuizen (Eds.), The role of multiple representations in learning and problem solving (pp. 262–282). Amsterdam: Elsevier.Google Scholar
  82. Schkade, D. A., & Kleinmuntz, D. N. (1994). Information displays and choice processes: Differential effects of organization, form, and sequence. Organizational Behavior and Human Decision Processes, 57(3), 319–337. doi:10.1006/obhd.1994.1018.CrossRefGoogle Scholar
  83. Simons, K., & Klein, J. (2007, January). The impact of scaffolding and student achievement levels in a problem-based learning environment. Instructional Science, 35(1), 41–72. doi:10.1007/s11251-006-9002-5.
  84. Spector, J. M. (2008). Expertise and dynamic tasks. In H. Qudrat-Ullah, J. M. Spector, & P. I. Davidsen (Eds.), Understanding complex systems series: Complex decision making 2008 (pp. 25–37). Berlin: Springer. doi:10.1007/978-3-540-73665-3_2.
  85. Spector, J. M. (2010). Mental representations and their analysis: An epistemological perspective. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge 2010 (Part 1, pp. 27–40). New York: Springer. doi:10.1007/978-1-4419-5662-0_3.
  86. Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (1988). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains (Tech. Rep. No. 441). Urbana-Champaign, IL: University of Illinois, Center for the Study of Reading.Google Scholar
  87. Stake, R. E. (2006). Multiple case study analysis. New York: Guilford Press.Google Scholar
  88. van Gog, T., Ericsson, K., Rikers, R., & Paas, F. (2005, September). Instructional design for advanced learners: Establishing connections between the theoretical frameworks of cognitive load and deliberate practice. Educational Technology Research and Development, 53(3), 73–81. doi:10.1007/BF02504799.
  89. Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.Google Scholar
  90. Xun, G., & Land, S. (2004). A conceptual framework for scaffolding ill-structured problem-solving processes using question prompts and peer interaction. Educational Technology Research and Development, 52(2), 5–22. doi:10.1007/BF02504836.CrossRefGoogle Scholar
  91. Yin, R. K. (2003). Case study research: Design and methods (3rd ed., Vol. 5). Thousand Oaks, CA: Sage.Google Scholar
  92. Zhang, J. (1997). The nature of external representations in problem solving. Cognitive Science, 21(2), 179–217.CrossRefGoogle Scholar
  93. Zhang, J., & Norman, D. A. (1994). Representations in distributed cognitive tasks. Cognitive Science, 18(1), 87–122. doi:10.1016/0364-0213(94)90021-3.CrossRefGoogle Scholar
  94. Zimmerman, B. J., & Campillo, M. (2003). Motivating self-regulated problem solvers. In J. E. Davidson & R. Sternberg (Eds.), The nature of problem solving (pp. 233–262). New York: Cambridge University Press.Google Scholar

Copyright information

© Association for Educational Communications and Technology 2013

Authors and Affiliations

  • Treavor Bogard
    • 1
  • Min Liu
    • 2
  • Yueh-hui Vanessa Chiang
    • 3
  1. 1.School of Education and Health Sciences, University of DaytonDaytonUSA
  2. 2.The University of Texas at AustinAustinUSA
  3. 3.Dharma Drum Mountain World Center for Buddhist EducationNew Taipei CityTaiwan

Personalised recommendations