Introduction

In recent years, a debate erupted among researchers about the question of how much guidance do students need in problem-based learning (PBL) (Hmelo-Silver et al. 2007; Kirschner et al. 2006; Simons and Klein 2007). For instance, Kirschner et al. (2006) suggest that problem-based learning (PBL) is a minimally guided approach and is less effective and efficient than instructional approaches that place a strong emphasis on guidance of the student learning process. However, there are others who argued that the PBL approach does provide extensive guidance and scaffolding to facilitate meaningful learning (Hmelo-Silver et al. 2007; Schmidt et al. 2007; Simons and Klein 2007). Several authors describing the PBL process include descriptions of additional structured educational activities and sources of guidance such as references, audiovisual aids and lectures relevant to the problem as scaffolds to enhance student learning (Ertmer and Simons 2006; Saye and Brush 2002; Simons and Klein 2007; Taylor and Miflin 2008). However, till date, there have been different opinions about how resources or scaffolds should be used in a PBL curriculum (Taylor and Miflin 2008). Some institutions believe that PBL curricula should be characterized by as few lectures as possible, whereas others believe that there should be more structure in the curricula. The objective of this study is to investigate the influence of worksheets as a tool used to scaffold students’ learning in a PBL environment.

According to Saye and Brush (2002), scaffolding can be generally classified into two groups—hard and soft scaffolds. Soft scaffolds refer to the teacher actions in response to the learner’s efforts when the learner has a specific need (Saye and Brush 2002). An example of such scaffolds would be the tutor or facilitator in PBL. Tutors play a significant role in ensuring that the students learn and progress satisfactorily in the course of solving the problem (Maudsley 1999). They should be knowledgeable and able to effectively facilitate groups of students (Greening 1998; Hmelo-Silver 2004b). Another instance of soft scaffold, which is also characteristic of PBL, would be the formation of collaborative problem-solving groups which helps to distribute the cognitive load and allow students to learn in complex domains (Hmelo-Silver 2004b; Hmelo-Silver et al. 2007; Schmidt et al. 2007). Research done on small PBL tutorial groups indicated positive cognitive effects in aspects such as activation of prior knowledge, recall of information and causal reasoning (Dolmans and Schmidt 2006; Hmelo 1998). In addition, group discussions in such PBL groups seem to have a positive influence on the students’ interest in the subject matter (Dolmans and Schmidt 2006). This rise in interest may indirectly lead to an increase in the students’ motivation level to learn. Students also need to be willing to participate in peer teaching, as well as being actively involved in the group learning process in order for effective learning (Lohfeld et al. 2005).

On the other hand, hard scaffolds are in general static supports that can be developed based on learner difficulties prior to an assigned task (Saye and Brush 2002). Such scaffolds can be provided once a task is assigned to the learner. Hard scaffolds can be in the form of computer or paper-based cognitive tools e.g. worksheets (Belland et al. 2008). For example, one way of guiding instruction can refer to the use of scaffolds like process worksheets (Merriënboer 1997). Such worksheets provide hints or descriptions of the phases one should go through when solving the problem. Students can consult the process worksheet while they are working on the learning tasks and they may use it to monitor their progress throughout the problem-solving process.

Some studies have also suggested that fading of hard scaffolds is possible once the students have gained ability in performing the assigned tasks (Belland et al. 2008; Puntambekar and Hubscher 2005). For example, novices in a PBL environment may engage in unrelated literature searches, which indirectly results in inefficient learning (Schmidt et al. 2007). Hence, it is often essential that novice students in a PBL environment are provided with some resources to scaffold their learning, as being able to successfully search for literature and other resources usually requires a certain level of prior or domain knowledge. With increasing expertise, fewer resources should be provided to the students. In this way, independent learning is encouraged while providing a form of flexible scaffolding (Schmidt et al. 2007).

On the whole, the use of scaffolds in general and PBL context has demonstrated varying degrees of impact on student learning achievements. Some studies have shown effectiveness of scaffolds in supporting student learning (Cho and Jonassen 2002; Roehler and Cantlon 1997; Simons and Klein 2007). For example, Simons and Klein (2007) examined the impact of scaffolding and student achievement levels in a PBL environment, whereby students were subjected to different scaffolding conditions. Results from this study revealed that students who were given access to scaffolds performed significantly better in the post-tests, compared to the group with no scaffolds provided. The findings indicated that scaffolds may influence student inquiry and performance in a PBL environment. However, one of the limitations for this study would be the distribution of experimental groups. Only one class was assigned to the no scaffolding condition, whereas it would have been more desirable if the sample size of two classes was used to achieve the same number of classes in each experimental condition. Therefore, the present study sought to explore how far scaffolds, in the form of structured worksheets, help students in their learning in PBL. A quasi experimental approach was chosen in which one group of students received a scaffold during PBL and another not. Differences in their learning were determined by comparing the mean scores on a concept recall test.

Method

Participants

The sample consisted of 241 participants with an average age of 18 years and who were in their second year of study in the institution. The participants were enrolled in 17 classes for a Biomedical Science-related subject (Immunology) at a polytechnic in Singapore. The response rate for this study was approximately 89%.

Educational context

In this polytechnic, the instructional method is PBL for all its modules and programs. In this approach five students work together in one team under the guidance of a teacher. Each class comprises four to five teams. A unique feature of the PBL approach used in this polytechnic is that students work on one problem during the course of each day (Alwis and O’Grady 2002) A typical day starts with the presentation of a problem. Students discuss in their teams what they know, do not know, and need to find out. In the process, students activate their prior knowledge, come up with tentative explanations for the problem, and formulate their own learning goals (Hmelo-Silver 2004a; Schmidt 1983, 1993). Subsequently, periods of self-study follow in which students individually and collaboratively try to find information to address the learning goals. At the end of the day the teams come together to present, elaborate upon, and synthesize their findings.

Materials

Problem

The problem used is a case scenario of a patient suffering from reoccurring infections due to deficiency in one of the proteins required for activating certain immune responses within the body. Some clinical data was provided in the case study for the students to infer and analyze.

Worksheet scaffolding

For the experimental group a worksheet scaffold was devised, which aimed to guide the students towards ideas to consider during the process of analyzing and approaching the task for the day. This was carried out by including hints or providing some information within the worksheet. For example in this study, students were required to find out about the role of complement proteins in the immune system. One of the questions in the worksheet prompted the students to investigate more about the different immune processes that complement proteins are involved in. For this question, students were required to fill in a table to guide them in learning more about the various processes that involve complement proteins. At the end of the question, students were then required to summarize the functions of complement proteins with the aid of diagrams and the information gained in the earlier part of the question as hints. (Refer to Appendix section for an example of the worksheet question).

Thus the worksheet is an instructional tool consisting of a series of questions and information designed to guide students to understand complex ideas as they work through it systematically. It was provided as an additional scaffold apart from the problem trigger, and students may complete it on their own or in discussion with their teammates. For the control group, the students were only provided with the problem trigger.

Pre- and post-tests

A concept recall exercise was designed to estimate the number of relevant concepts that students were able to recall before the start of the problem analysis phase (pre-test), and at the end of the reporting phase (post-test). Both tests consisted of the following instruction: “List down all the concepts that you think are relevant to today’s problem on the complement system.” (Understanding the complement system was the focus of the learning for the day). Students were instructed to only list keywords or terminologies they thought were relevant, and not write in paragraphs or sentences. They were not allowed to discuss their answers or to refer to any resources when completing the exercise. Students’ answers to the concept recall procedure were analyzed by awarding 1 point to each relevant concept given by the student. Rating was done by the first author and a colleague of similar expertise in the field of immunology. Differences in opinion were resolved by discussion.

Learning impact questionnaire

In addition to concept recall tests, a Learning Impact Questionnaire was administered to the participants in order to find out what students perceive as important factors that impact their learning in a PBL environment. The questionnaire consisted of five items measuring how certain features of the learning environment impact student learning: worksheet, problem statement, tutor, team dynamics (level of involvement within a small group of 5 students) and class dynamics (level of involvement between teams or in a larger group of about 25 students).

The following questions were included in this questionnaire: (1) “The worksheet has a strong impact on my learning”; (2) The problem statement has a strong impact on my learning”; (3) “The tutor has a strong impact on my learning”; (4) “Team dynamics have a strong impact on my learning”; and (5) “Class dynamics have a strong impact on my learning”. The items were scored on a 5-point Likert scale: 1 (Strongly disagree), 2 (disagree), 3 (neutral), 4 (agree), and 5 (strongly agree). Students were asked to answer the questionnaire based on their overall experience in a PBL environment.

Procedure

The 17 classes were randomly assigned to the treatment condition (7 classes) and the control condition (10 classes). Each class has an average of 25 students. At the beginning of the PBL day, students completed the pre concept recall test, which took about 10 min. After that they commenced with the lesson. The PBL day was exactly the same for both groups, except that the treatment group received a worksheet with the problem. They were asked to complete the worksheet during the self-study periods. For students in the treatment group, the teacher would briefly check on their progress of the worksheet during the team discussions. After both groups had completed the PBL day, they responded to the post concept recall test, which was identical to the pre-test. In addition, the participants completed the learning impact questionnaire. As students were asked to complete the questionnaire based on their overall experience in the course of study within the institution, students who were not provided a worksheet were also able to evaluate the impact of worksheets (in general) on their learning in the PBL context.

When this study was carried out, the institution had not yet formed an Institutional Review Board (IRB) through which formal ethical approval for the study could be sought. However, measures were taken to ensure that ethical standards were met. First, the problem selected for this study was one which did not normally provide a worksheet scaffold. The students in the control group (with no scaffold provided) were therefore not disadvantaged by being in the control group. Moreover at the end of the day, the worksheet was made available to all students, including those in the control group. Second, all students and tutors who participated in this study gave informed consent, and were given a choice in regards to their participation.

Statistical analysis

In order to test whether there are significant differences in terms of students learning between the treatment and control groups we conducted an analysis of variance (ANOVA). The dependent variable was post-test score and the independent variable was the condition. In addition to the F-value and p-value, we generated eta-squared as a measure of the effect-size. In order to compare the items scores of the learning impact questionnaire we also conducted an ANOVA.

Results

The results of the ANOVA revealed that the control group (N = 143) scored significantly higher on the post-test than the treatment group (N = 98) using the worksheets (F(1, 239) = 6.47, p = .01, eta-squared = .03). The mean score for the control group was M = 5.91 (SD = 3.95) and for the treatment group M = 4.76 (SD = 2.57). This outcome is rather surprising because it suggests that the worksheet had no significant influence on students’ learning during the PBL day; on the contrary, not having a worksheet seemed to result in better learning for the day. An explanation for this unexpected outcome could however be that despite the random assignment of the groups, the control group had by chance more knowledge about the topic than the treatment group. In order to test for this possibility, we conducted another ANOVA to examine whether there were initial differences on the pre-test scores between the two groups. The results of the ANOVA revealed that there were indeed significant knowledge differences in favor of the control group: F(1, 239) = 15.08, p < .01, eta-squared = .06. The mean score for the control group on the pre-test was M = 2.32 (SD = 3.36) and for the treatment group M = 1.00 (SD = 1.65). The results of the pre- and post-tests are summarized in Table 1.

Table 1 Summary of ANOVA comparing the pre-test and post-test scores of participants
Full size table

In order to statistically correct for this initial difference in pre-test scores, we conducted an analysis of covariance (ANCOVA). The covariate was the pre-test score, the independent variable as the condition and the dependent variable was the post-test score. Despite correcting for the initial differences, the ANCOVA revealed that the differences in post-test scores were not significantly different: F(1, 238) = 1.58, p = .21, eta-squared = .01. This outcome suggests that using a worksheet as a scaffold for learning had no significant effect on student learning, even after correcting for initial knowledge differences.

We next compared the scores to the items of the Learning Impact Questionnaire to examine how students perceive various aspects (scaffolds) present in a PBL classroom environment and their respective impact on learning. The results are summarized in Table 2.

Table 2 Summary of ANOVA comparing factors that students perceive as important in impacting their learning in a PBL environment
Full size table

The ANOVA revealed that there were statistically significant differences between the scores of the items in absolute sense: F(4, 218) = 39.88, p < .01, eta-squared = .16. Considering the mean values and pairwise comparisons based on the LSD, students in our sample rated the worksheets as having the lowest impact on their learning (worksheet scored lowest as compared to all other items p < .01). There are no differences in mean scores reported in Table 2 between experimental and control group, except for the worksheet (p = .013; mean score for experimental group = 3.71; mean score for control group = 3.40). Although the p-value for worksheet is not smaller than .01, it is smaller than .05 which may be considered as a statistical significant difference. The highest impact on students’ learning was related to team dynamics, which was rated higher than for all other items (p < .01). The other factors of the learning environment fell within between these two extremes. The tutor and class dynamics were rated second most important and the problem statement third.

Overall, the results demonstrate that worksheets may not have a significant influence on students’ learning in a PBL classroom. This was inferred from our post-test achievement data, and also by students’ responses to the Learning Impact Questionnaire.

General discussion

The objective of the present study was to investigate the effects of worksheet scaffolds on students’ learning achievement in a PBL environment. Based on the results obtained from this study, there was no statistically significant difference between the levels of achievement in terms of learning outcomes for both the experimental and control group. As mentioned in the introduction to this study, there is a lack of clarity on how scaffolding can be used to achieve successful teaching (Verenikina 2008) in PBL. The contribution of this study is the evidence that, at least in this PBL context, scaffolds such as worksheets may not play a significant role in enhancing students’ learning within PBL. This is supported by the results obtained from the Learning Impact Questionnaire, which revealed that the strongest factor perceived by students to impact their learning in a PBL context is the tutor followed by team and class dynamics, whilst the influence of the worksheet was rated lowest.

With reference to prior studies, the findings obtained from this study reinforced the view that soft scaffolds, such as tutoring and collaborative small group learning, are crucial for student learning in a PBL environment. Tutors should have the relevant content knowledge to guide students throughout the process of solving the problem by asking open-ended questions to facilitate them (Hmelo-Silver 2004b; Hmelo-Silver and Barrows 2008; Maudsley 1999; Yee et al. 2006). A recent study conducted by Rotgans and Schmidt (2010a) discussed that one of the options to increase interest would be to enhance the tutor’s subject-matter expertise, thereby leading to an increase in cognitive congruence. This could be done by providing modes of additional resources (e.g. briefing sessions, reading materials) for tutors to gain more in-depth knowledge of the problem (Rotgans and Schmidt 2010a). In this study we have conducted, the responses obtained from the Learning Impact Questionnaire showed that the tutor was rated as being more significant for students’ learning and not the worksheet. Based on this finding, it may be more beneficial for student learning in PBL if the tutor provides soft scaffolding like what previous studies have suggested. This study also suggests that collaborative small group learning (team dynamics) plays a significant role in enhancing student learning. Studies have indicated that collaborative groups in PBL creates an appropriate environment for students to learn the concepts by allowing them to investigate each others’ comments and encourage further discussion (Dolmans and Schmidt 2006; Hmelo-Silver 2004b; Will 1997). Overall, this seems to support the finding from our study as to why students perceive team dynamics to have a significant impact on their learning.

Limitations

In overall, the outcome of this study seems to align with previous studies pertaining to scaffolds. Schmidt et al. (1997) commented that as students gain a certain level of prior knowledge or expertise, the degree of scaffolding should be lower (i.e. fewer resources) so as to encourage independent learning. In this study, the participants were already in their second year of the diploma course that they were enrolled in. They would have some prior knowledge and experience in searching for literature and other resources during their first year, when they were still novices to PBL. This could have lead to the participants being less reliant on the worksheets to grasp the concepts related to the topic for the day. Furthermore, there are other limitations that could have influenced the student’s reliance on using the worksheet to enhance their learning.

First, students who are less motivated in learning might not necessarily attempt to use the worksheet for their learning. According to Merriënboer (1997), students can refer to the worksheet and use it to monitor their progress during the process of solving the problem. Students who are either not keen to investigate more about the problem, or are confident in their ability to search for relevant resources to solve the problem may not deem the worksheet to be essential in scaffolding their learning. Moreover, it was unfeasible to unravel differences between using a worksheet individually or as a group. Students who completed the worksheet in groups may possibly have learnt more compared to those who completed the worksheet individually. Hence, this could have lead to the high rating of the team dynamics in the learning impact survey.

Secondly, the concept recall test is based on the assumption that students build networks of concepts in the process of learning and the more students have learned about a topic, the more coherent and detailed each network would be (Glaser and Bassok 1989). Students who have learned more effectively would therefore be able to recall more concepts and also would be able to do so more easily (Collins and Quillian 1969; Rumelhart and Norman 1978). However, it is possible that though the concept recall test could provide an indication of the quality of student learning it may not thoroughly measure students’ understanding of the topic (Yew and Schmidt 2011).

Third, through the PBL process, students are required to come up with tentative theories to explain the phenomena presented in the problem. Previous studies have indicated that there is a positive effect on motivation, interest and learning when students have a choice to determine what they wish to study or learn (Deci et al. 1991). In a recent study, it was suggested that when students gain more autonomy from tutor and team members, they would exhibit more cognitive engagement in class with the task at hand especially when they are doing individual self-study (Rotgans and Schmidt 2010b). By integrating hard scaffolds such as worksheets into the PBL curriculum, this may reduce students’ feeling of choice and autonomy, which leads to less engagement and learning. Since students in a PBL environment are expected to engage in their own knowledge construction to solve the problem, there could be a possibility that worksheets and PBL are not reconcilable. As worksheets tend to impose the theories on the students, this may affect the process of the students’ knowledge construction. However, this is only a tentative explanation, as more research is needed to establish the link between autonomy reduction and hard scaffolds in PBL.

The following suggestions could be taken into consideration for further research: (1) Results from this study generally indicated that worksheets may not be that effective as a form of scaffolding to enhance student learning in PBL in this educational context. However, it remains an empirical question to see if there are any significant differences in the level of student dependence on hard scaffolds such as worksheets, as they progress from novices to individuals who have adopted relevant PBL skills gained throughout the course of study. Since this study involved participants who are in their second year of the diploma course, it is suggested that it could be helpful to conduct similar studies on students who are in their novice and final year of the course. In this way, comparisons and inferences can be made about students’ perceptions and reliance on worksheets as a scaffold in a PBL environment. (2) Most of the previous studies conducted have demonstrated a certain level of success rate of adopting PBL in the field of science, in particular medical education. Hence, it is suggested that the influence of worksheets on students’ learning in other subject fields (e.g. engineering, arts) could be further investigated. (3) Considering that the worksheet design may vary based on the curriculum to be delivered, data could be collected for a range of topics for more conclusive findings. (4) Since tutors play an important role in observing learning processes of the students throughout the lessons, it would be beneficial to examine what are the tutors’ perceptions of using worksheets as a form of scaffolding to facilitate students’ learning. (5) One of the aims of PBL is to encourage students to move away from passive learning to active engagement during the process of solving the problem (Davis and Harden 1999). Students who are generally passive learners could be relying more on the worksheet for guidance instead of being engaged in collaborative small group learning. Hence, this may indirectly affect the responses to the Learning Impact Questionnaire. More research could be conducted to investigate if there is any correlation between the learning styles and achievement level of the students.