Introduction

Problem solving is an essential skill for success in the twenty-first century (Belland, 2013; Kim & Tawfik, 2021; Law et al., 2020). It has been studied extensively in mathematics (Jonassen, 2011; Wilson et al., 1993), engineering (Pappas, 2002; Pappas & Pappas, 2003), and online information searching (Walraven et al., 2008). Metacognition is regarded as a key component in problem solving (Lin et al., 1999; Tarricone, 2011), especially in solving ill-structured problems (Ge & Land, 2003; Jonassen, 2000; Shin et al., 2003). However, there is a lack of guidance for developing metacognitive skills as a means of supporting ill-structured problem solving.

The term metacognition was first introduced by John Flavell in the 1970s, and it is also called second-order cognition and cognition of cognition (Flavell, 1987; Papaleontiou-louca, 2003; Tarricone, 2011). There are three major components of metacognition, which are metacognitive knowledge, metacognitive experience, and metacognitive skills (Efklides, 2008). Metacognitive knowledge is the learner’s stored world knowledge (Flavell, 1979), knowledge about oneself and cognitive tasks (Ku & Ho, 2010). Metacognitive experience refers to the feeling and judgment towards completing tasks (Efklides, 2008; Flavell, 1979). Metacognitive skills are also called metacognitive strategies (Efklides, 2008, 2014; Veenman & Elshout, 1999), including planning, monitoring, and evaluating thinking processes (Akturk & Sahin, 2011; Ku & Ho, 2010).

Currently, many computer-based learning materials are produced using e-Learning authoring tools, such as Articulate Storyline, Adobe Captivate, and Camtasia. Functional attributes of those e-Learning authoring tools, including multiple choice questions, short answers, and auto-grading, could be utilized to create metacognitive scaffolds in e-Learning instruction. By engaging in metacognitive-based instruction, learners can acquire metacognitive skills to plan, control and monitor their learning processes.

However, the literature is sparse in studying the application of interactive media attributes in creating computer-based instruction to support metacognitive development. In consideration of the critical role that metacognition plays in enhancing learner’s problem-solving skills, there is a need to produce evidence-based computer-based learning materials to teach metacognitive skills to an increasing number of online learners (Brown et al., 1994). The shift to virtual learning during the COVID-19 pandemic underscores the growing need for online approaches to problem solving instruction.

An instructional framework that guides the design of computer-based metacognitive materials will be beneficial to instructors and practitioners when creating problem-solving instruction. In line with these goals, this study is conducted to create an interactive content design framework that serves as guidelines for instructors and practitioners to create computer-based problem-solving instruction. By embedding metacognitive scaffolds in the self-paced instruction, learners are trained to monitor and control their thinking process efficiently during problem-solving process.

Methodology

Research Purpose and Question

The focus of this study is the design of an interactive content development framework with utilizing formative design principles to teach and train metacognitive skills. The guidelines proposed in the framework consolidate prior research from metacognition, multimedia design, instructional design, interaction design, and message design. Three key components were identified from the literature and included in this framework. Those components are metacognitive strategies, instructional strategies, and media attributes. The interactive content will be developed using e-Learning authoring tools with following the guidelines proposed in the framework. The research question framed this study is:

What features would a framework have to facilitate metacognitive strategy development in computer-mediated instruction?

Study Design

This study employed a design and development research (DDR) approach with implementing formative design principle throughout the design process. DDR is defined as “the systematic study of the design, development and evaluation processes with the aim of establishing an empirical basis for the creation of instructional and non-instructional products and tools and new or enhanced models that govern their development” (Richey & Klein, 2007, p.1). There are two types of DDR, including type Ι product and tool research and type Π model research (Richey & Klein, 2005, 2007, 2014; Richey et al., 2004). To create a framework that guides the design of metacognitive-based interactive content, this study adopts type Π model research method with suggesting key steps in designing metacognitive scaffolds in ill-structured problem-solving contexts.

This study consists of four phases, which are (1) Analysis, (2) Design and Development, (3) Evaluation, and (4) Revision. A formative design approach is adopted throughout the design process. According to Glaser et al. (2017) and Hokanson and Kenny (2020), iteration is a key parameter in design research. This study implements an iterative process to continuously design, refine, and evaluate the framework based on experts’ feedback.

Study Procedure

Analysis

A systematic and thorough review and analysis of the existing literature on metacognitive strategies, instructional strategies, and interactive attributes of the e-Learning authoring tools were performed, related questions and literature sources are presented in Table 1. Publications on these topics were obtained from journal articles, dissertations, theses, and e-Learning design practices. Key search terms included “metacognition,” “metacognitive skills,” “metacognitive strategies,” “interactivity,” “computer-based instruction,” “interaction design,” “message design,” and “screen design.”

Table 1 Design elements of interactive content in web-based instruction
Full size table

Design and Development

The design and development phase consisted of the construction of components and operationalization of the design procedures of interactive content design framework. This phase was carried out based on findings from the Analysis phase where relevant research on metacognitive strategies, instructional strategies, and media attributes was identified and synthesized.

Evaluation

Expert review was adopted as an internal validation procedure (Ross et al., 2007) to evaluate the effectiveness of the developed framework. This process occurred by asking experts to evaluate components, structure, and application of the model (Ross et al., 2007). The evaluation process consisted of two phases: pilot study and formal study. Preliminary feedback was collected from two scholar-practitioners in the pilot study to improve the framework and adjust expert review invitation messages and evaluation survey. Selected experts in instructional design, problem solving, and metacognition were invited to participate in the formal phase of the evaluation.

Revision

During the revision phase, feedback from the experts was implemented to refine the framework. The modified framework is presented in the next section.

Framework for Designing Metacognitive Scaffolds in Ill-Structured Problem-Solving Instruction Using e-Learning Authoring Tools

Scholars believe that a continuum exists between ill-structured and well-structured problems, and there is no clear boundary between them (Simon, 1973; Voss, 1988). Ill-structured and well-structured problems share similar problem-solving procedures (Gick, 1986), such as represent the problem, generate solutions, present a solution, evaluation, and reflection. This framework presents detailed instruction to guide the design of metacognitive scaffolds to facilitate learners in solving ill-structured problems. Metacognitive scaffolds help learners monitor and reflect on their thinking process (Crescenzi et al., 2021; Kim & Lim, 2019; Zhou & Lam, 2019). Commonly used metacognitive scaffolds in problem solving instruction are prompting and modeling (Hollingworth & McLoughlin, 2001; Lin et al., 1999). The following sections introduce key problem-solving stages, metacognitive scaffolds, instructional strategies, examples, and interactive media types identified from existing literature.

Ill-Structured Problem-Solving Stages

Ill-structured problem-solving instruction can be created by following general ill-structured problem-solving procedures. Those stages are representing problem (Kale et al., 2018; Kim & Tawfik, 2021), generating solutions (Ge & Land, 2004; Sinnott, 1989), presenting a solution (Ge & Land, 2003, 2004; Kim & Tawfik, 2021), evaluating the solution (Ge & Land, 2003, 2004; Jonassen, 1997), and reflecting on the problem-solving process (Kauffman et al., 2008; Kim & Hannafin, 2011). Related literature that underpins each stage is listed in Table 2.

Table 2 Ill-structured problem-solving stages
Full size table

Different from well-structured problem solving, ill-structured problem-solving procedures are situated (Jonassen, 1997, 2000; Voss, 1988), dynamically intertwined, circular, and iterative (Ge et al., 2016). The initial stage of creating metacognitive-based ill-structured problem-solving instruction is to present the problem. Problems presented in the learning materials should be situated, real-world problems to foster knowledge transfer (Lin et al., 1994; Liu et al., 2012; Park & Hannafin, 1993). Once an ill-structured problem is presented, learners can follow the problem-solving process to solve the problem. The problem-solving stages include represent problem, generate solutions, present and evaluate a solution, and reflect as shown in Fig. 1.

Fig. 1
figure 1

Metacognition-based ill-structured problem-solving process

Full size image

Represent Problem

The purpose of problem representation is to understand and form the initial state of the problem (Voss, 1988), such as identifying cause(s) of the problem and setting goals. This process is complex, which requires the learner to discover missing information, differentiate relevant information from irrelevant information, and express personal opinions of the problem (Jonassen, 1997, 2000). To assist learners with developing a problem representation, metacognitive scaffolds could be created to prompt learners to identify the causes and constraints of a problem (Ge & Land, 2003; Jonassen, 1997; Voss, 1988).

Generate Solutions

Once the initial state of an ill-structured problem is constructed, the learner moves on to generate solutions. According to Shin et al. (2003), an ill-structured problem could possess more than one solution with multiple solution paths. Jonassen (1997) posited that multiple solutions resulted from different problem representations. Sinnott (1989) described this process as a creative exercise that cultivates divergent thinking. This process requires information identification, strategy selection, data collection, and action plan formulation.

Present and Evaluate Solution

Once a solution is chosen, it should be verified and evaluated (Garofalo & Lester, 1985; Kwang, 2000). Huttenlock (2007) recommended to evaluate both the process and goal completion in an ill-structured problem domain. The evaluation of goal completion focuses on checking if the learners have enough information to confirm their claims or reach the preset goals. In contrast, process evaluation emphasizes on choosing appropriate data and strategy during the process (Huttenlock, 2007). Jonassen (1997) proposed to evaluate the viability of solutions by asking questions, such as “Was the problem solved? Did it adhere to the constraints?”.

Reflect

Reflection is the last stage in ill-structured problem-solving process. During this stage, learner revisits ideas, process, adopted strategies, and solutions (Collins & Brown, 1988; Kim & Hannafin, 2011). Learning occurs when learners reflect on their own mistakes, make plans for improvement, and prepare for knowledge transfer in other contexts. Computer-based learning environment could be used as an effective medium to facilitate the reflection process (Collins & Brown, 1988).

Metacognitive Strategies and Instructional Activities

Metacognitive strategies are essential for solving ill-structured problems by monitoring and controlling cognitive processes (Brown et al., 1994; Flavell, 1987). According to Shin et al. (2003), metacognitive skills, such as plan and monitor, are strong predictors of the success in solving unfamiliar ill-structured problems. Five metacognitive strategies are presented in this framework, which are Orient (Garofalo & Lester, 1985; Meijer et al., 2006; Molenaar & Chiu, 2014), Plan (Brown, 1987; King, 1991; Pintrich, 2002), Monitor (Brown, 1987; Efklides, 2014; Zimmerman & Campillo, 2003), Evaluate (Bannert & Reimann, 2012; Schmidt & Ford, 2003; Yıldız-Feyzioğlu et al., 2013), and Reflect (Meijer et al., 2006; Molenaar & Chiu, 2014; Zimmerman & Campillo, 2003).

Orient

Garofalo and Lester (1985) defined orienting as a strategic behavior to understand and assess a problem. Meijer et al. (2006) asserted that experts spent more time than beginners on orientation activities. Orienting occurs during the problem representation stage, and orienting activities include understanding the problem (Kale et al., 2018), examining the causes of the problem (Jonassen, 1997; Kauffman et al., 2008; Voss, 1988), identifying relevant and irrelevant information (Hollingworth & McLoughlin, 2001; Swanson, 1990), as well as associated constraints, variables, and relationships (Ge & Land, 2004; Kale et al., 2018; Voss, 1988).

Orienting skills can be trained using question prompts, such as “What are the causes of the problem?” (Kauffman et al., 2008), “What information is relevant to solve this problem and what information is irrelevant?” (King, 1991), and “What are the key variables of this problem and how are they related?” (Meijer et al., 2006). More examples can be found in Table 3 in Appendix 1.

Plan

Tzohar-Rozen and Kramarski (2014) defined planning as a skill of choosing strategies and allocating resources. It is applied in problem representation and generating solutions stages. In problem representation stage, planning activities include activating prior knowledge and experience (Huttenlock, 2007; Meijer et al., 2006; Yıldız-Feyzioğlu et al., 2013), setting goals (Bannert & Reimann, 2012; Garofalo & Lester, 1985; Yıldız-Feyzioğlu et al., 2013), and formulating hypothesis (Bogard et al., 2013; Gick, 1986; Meijer et al., 2006). During the solution generation phase, planning activities contain formulating an action plan, identifying needed information, and selecting strategies (Huttenlock, 2007). Planning scaffolds can also be designed using question prompts, such as “Do you have any related experience with this kind of problem?”, “What are possible solutions of this problem?”, and “What are the steps to solve this problem?” (Shahbodin & bt Bakar, 2010). More information on planning scaffolds and examples can be found in Table 3 (in Appendix 1) and Table 4 (in Appendix 2).

Monitor

Monitor refers to one’s awareness of task comprehension and performance (Schraw, 2001; Tzohar-Rozen & Kramarski, 2014). Quintana et al. (2005) described monitoring activities as identifying tasks, evaluating work progress, and predicting outcomes. Monitoring occurs throughout the problem-solving process, which includes the monitor of cognitive process and learner emotion. Examples of monitoring prompts include “Is this problem easy or hard?” (Tzohar-Rozen & Kramarski, 2014), “Are you reaching your goals?” (Huttenlock, 2007), and “Are you using the strategy that you chose?” (Kwang, 2000). Additionally, expert modeling should be offered when learner expresses difficulties in completing tasks. Expert modeling can be presented in video demonstration or simulation learning environment. More monitoring scaffolds and examples can be found in Table 4 in Appendix 2.

Evaluate

Evaluation strategies are used in the evaluation stage of solving ill-structured problems, which serves the purpose to justify proposed solution and process (Tzohar-Rozen & Kramarski, 2014). Metacognitive scaffolds adopted in this phase are specified as the assessment of supporting information (Bulu & Pedersen, 2010) and defending of a chosen solution (Ge & Land, 2004). However, if the chosen strategy does not meet the evaluation criteria, learner needs to restart the problem-solving process. It is through evaluation that the learner refines their working process and outcomes. Example of question prompts that are used to foster metacognitive evaluation are “What are your reasons and/or arguments for proposing this solution?” (Ge & Land, 2003) and “Have you taken into account the perspectives of different stakeholders?” (Ge & Land, 2004). More strategies and examples are described in Table 5 in Appendix 3.

Reflect

Designing metacognitive scaffolds to facilitate learner reflection is the last stage in creating computer-based ill-structured problem-solving instruction. The reflect strategy helps the learner develop an understanding of oneself and related events to inform future actions (Sandars, 2009). Guided reflection and feedback could challenge learners while help them develop a deeper understanding of related content area (Sandars, 2009). The rich interactive features of multimedia-enhanced problem-solving instruction offers new means to support those reflection activities (Gordon, 1996; Liu et al., 2012). Relevant scaffolds are presented in Table 6 in Appendix 4, such as “What did you learn from this problem?”, “Can you think of other settings where those skills could be applied?”, and “What are the pros and cons of the chosen solution?” (Bulu & Pedersen, 2010; Ge & Land, 2004).

Media Attributes and Examples

Interactive features in e-Learning authoring tools can be used effectively to design metacognitive scaffolds. The rich affordability of e-Learning authoring tools allows designers, instructors, and other stakeholders to present a problem with utilizing a wide variety of media attributes, such as text, audio, video, simulation, and animation. Interactive media types presented in this framework were identified from a variety of e-Learning authoring tools, and examples were created using Adobe Captivate (see Fig. 2 as an example). Those media types can be selected from the menu bar of e-Learning authoring tools, which includes Text Box, Short Answer, True/False, Multiple Choice, Checkbox, Radio Button, Hyperlinks, Audio Input, and Animation (Articulate Storyline, 2012; Captivate User Guide, n.d.). The following section highlights utilizing those interactive attributes to create computer-based metacognitive scaffolds.

Fig. 2
figure 2

Adding slides in captivate

Full size image

Multiple Choice Questions

Multiple Choice is a commonly used feature in e-Learning authoring tools. It provides learner with pertinent feedback based on the selected answer(s). Figure 3 shows an example of monitoring scaffolds using branching function to direct the learner to different learning paths, such as “Go to the previous slide,” “Go to the next slide,” or “Open URL or file.” Much like Multiple Choice, True/False, Checkbox, and Radio Button can be used to fulfill the same purpose with assigning variables in the Advanced Action section in Adobe Captivate.

Fig. 3
figure 3

Adding advanced actions in captivate

Full size image

Short Answer Questions

Short Answer Question is regarded as an open-ended question, which prompts the learner to respond to a question briefly with putting their thoughts into words (Pappas, 2015a). A planning scaffold created using Short Answer Question and Button (e.g., Clear and Submit) in Adobe Captivate is shown in Fig. 4. Short Answer is an ideal question format to train learner’s argumentation and justification skills by asking learner to describe different solution paths and explain why they choose a certain solution.

Fig. 4
figure 4

Sample image of metacognitive scaffolds in justifying solution stage

Full size image

Text-to-Speech

Text-to-speech feature in e-Learning authoring tools allows instructors and course designers to create audio files from written texts or scripts (Fair, n.d.). Text-to-speech can be used to present a problem or elaborate on a topic in the audio format. It helps instructors and e-Learning developers who are not native English speakers to produce English audio narration. In addition, Text-to-speech is a great tool to design computer-based interactive content for visually impaired learners as well as learners who prefer audio instruction over plain texts.

Drag-and-Drop

Drag-and-drop function is widely acknowledged for creating engaging and immersive e-Learning experiences (Pappas, 2015b). It allows the learner to drag a target and drop it to a correct spot, which could be used to match, sort, and label items. Figure 5 shows an orienting scaffold created using the drag-and-drop feature in Adobe Captivate to differentiate causes and non-causes of air pollution.

Fig. 5
figure 5

Problem orienting activity designed using drag-and-drop

Full size image

Web Objects

Web objects, including AI-based feedback systems, discussion forums, and chatbots, can be added to e-Learning courseware. AI-based feedback systems can grade students’ responses using machine learning methods and provides contextualized feedback immediately (Chen, 2004; Warschauer & Grimes, 2008); Discussion forums can also be added to support peer interaction and reflective thinking (Pedro et al., 2012); The integration of a chatbot could provide timely assistance to the learner. A chatbot is also known as conversational agent (Griol et al., 2014; Kerry et al., 2008), or personal coach (Roda et al., 2001).

In addition to multiple choice, short answer, text-to-speech, drag-and-drop, notes, and web objects, other media types, including hyperlinks and highlight box (Jonassen, 2000), matching (Pappas, 2015c), and hotspot (Brooks, n.d.) are also available across e-Learning authoring tools. More complex ill-structured problem-solving learning materials and environments, such as educational games (Pappas, 2015b), can be created with utilizing and combing a variety of media types. Tables 3, 4, 5 and 6 in Appendices together present a revised framework for developing metacognitive scaffolds using e-Learning authoring tools based on experts’ feedback.

Conclusion

This study makes several contributions to the fields of instructional design, distance education, and problem-based learning. Firstly, it bridges the gap between theory and practice by proposing a theory-based framework to guide instructional design practices. Secondly, it highlights the importance of designing ill-structured problem-solving learning solutions to teach critical metacognitive strategies. Thirdly, it provides detailed instructions for practitioners to create computer-based metacognitive scaffolds using e-Learning authoring tools. Lastly, this framework adopts the type II design and development research method that produces generalizable findings (Richey & Klein, 2007, 2014). Instructors and course designers could follow the guidelines to design computer-based ill-structured problem-solving instruction across disciplines.

Expert feedback is instrumental in refining the initial framework. For instance, the expert reviewers suggested including contextualized feedback in the framework. This indicates that educators and course designers need to dive deep into the media attributes of e-Learning authoring tools and further integrate external web applications to offer contextualized feedback to learners. The external web application includes an AI-based feedback system, discussion forum where the learner can receive feedback and continue discussions with the instructor and their peers, or a chatbot that helps the learner develop self-regulation skills via conversation exchange.

In summary, the interactive features of computer-based learning environments proposed in this framework exemplify potential additional ways to leverage media affordances and prior research in support of metacognitive skill development for ill-structured problem solving.