Developing expertise in physics
Research in educational psychology (Bransford et al. 2000; Chi et al. 1981; DeGroot 1965) has long revealed that domain experts possess a variety of skills that distinguish them from novices, such as quickly extracting important information from complex situation (Chase and Simon 1973; DeGroot 1965; Egan and Schwartz 1979; Lesgold et al. 1988), organizing knowledge around important ideas or concepts (Chi et al. 1981; Wineburg 1991), and displaying great fluency in retrieval of their knowledge (Lesgold et al. 1988; Schneider and Shiffrin 1977; Simon et al. 1980).
Yet despite extensive knowledge of expert characteristics, most introductory physics courses are still largely unsatisfactory in terms of engendering physics expertise in college physics students. College physics students frequently employ heuristic methods such as equation hunting during problem solving, rather than using more expert like strategies (Tuminaro 2004; Walsh et al. 2007).
Careful research shows that a critical factor in the development of expertise in domains from music to sports to chess is the amount of time spent on so-called deliberate practice (Ericsson et al. 1993; Ericsson 2006; Ureña 2004). Deliberate practice has three defining characteristics. First of all, each practice activity must focus on a single aspect of expertise rather than playing the “full game.” For example, tennis players separately practice their serve, backhand, and drop shot. Secondly, the same practice is repeated multiple times until a certain level of mastery is achieved. Finally, during deliberate practice, the practitioner often receives instant, directed feedback on his/her progress.
The standard approach to teaching introductory physics rests largely on having students complete lots of traditional “back of the chapter” problems—essentially “full game” practice. However, solving these problems lacks all three characteristics of deliberate practice. First of all, properly solving such a problem often involves multiple (six to seven) expert skills, yet each skill is practiced only once per problem. Secondly, the process of solving one such problem is often fairly lengthy, limiting the amount of practice on any single skill. Finally, in a traditional classroom setting, the feedback on problem solving is neither instant (often delayed from 1 day to 1 week) nor directed to any particular skill (often only on the correctness of the final answer). It is known that even after solving a significant amount of those problems, students’ physics understanding is still inadequate (Kim and Pak 2002).
Designing deliberate practice activities for the development of physics expertise
Following the principles of deliberate practice, we designed and developed a number of practice activities named “deliberate practice activities” (DPAs), as an alternative to practicing on traditional problems. Each DPA is consisted of a sequence of five to six problems that are designed according to the following design principles:
Each sequence of problems is designed to focus on one documented expert skill.
Each problem can be solved in a relatively short time, involving few operations irrelevant to the targeted expert skill. This allows for adequate repetitive practice in a reasonable amount of time.
Each problem is accompanied with detailed feedback.
For example, in one DPA sequence that focuses on training the expert skill of mapping between different representations, students are given the general mathematical expression of a physics situation (such as angular momentum) and required to map each variable in the expression, such as distance r and angle α, to the corresponding distances and angles in a specific given physics situation (Fig. 1).
Using drag-and-drop problems to improve effectiveness of DPA
A critical feature of DPA problems is that they need to be highly focused on the target skill. A good metric for “focusedness” is a lower amount of extraneous cognitive load (Gerjets and Scheiter 2004; Paas et al. 2003; Sweller et al. 2011), cognitive activities irrelevant to the skill being trained, incurred during problem solving. We observe that the traditional multiple choice (MC) problem format, including dropdown lists and checkboxes, inevitably incurs significant cognitive load, especially on DPA problems. For example, when training students to identify important features in a given situation, a multiple choice format requires students to switch back and forth between the choices and the figure, incurring much cognitive load. Previous research on problem format (Chen and Chen 2009; Gillmor et al. 2015; Huang et al. 2015) showed that students perceive higher cognitive load completing traditional multiple choice questions.
To reduce the extraneous cognitive load, we utilized a new problem format provided by the edX platform: drag-and-drop problems. “Drag-and-drop” (D&D) problems, as shown in Fig. 1b, allow students to drag a visual icon from a list to a target diagram. In this example, students indicate the important feature by directly dragging the pointer to the relevant feature.
We further reduce extraneous cognitive load in the D&D format by following multimedia design principles outlined in the study by Chen and Gladding (2014) and Schnotz (2002). In short, the theories predict that extraneous cognitive load can be further reduced when verbal/symbolic icons are replaced by visual/perceptual icons. For example, as shown in Fig. 3, angular accelerations in different directions are represented by rotation icons in different directions, rather than text such as “clockwise” or “counter-clockwise.” An additional benefit of such a design is that students’ final answer is a figure representing the physics process, which is more valuable to remember than a selected item such as item A. Utilizing multiple modes of information coding has also been shown to facilitate understanding and memorization (Mayer 2001; Schnotz and Kürschner 2008).
In summary, this study is designed to answer two related research questions:
Can DPA serve as a more effective method in developing physics expertise compared with solving traditional end-of-chapter physics problems?
Is the D&D format more effective in developing individual expert skill compared to traditional multiple choice format?
Balanced experimental design and sample
The experiments were conducted in the homework and quiz of the last three required units (10, 11, and 12) of the MOOC (Fig. 2). Students were partitioned into three groups (A, B, or C). Each group received one of three different homework treatments in each unit: DPA activities in either D&D format (DPA-D&D), MC format (DPA-MC), or traditional end-of-chapter homework (TRD). The treatments were rotated between the three groups over three units. A common second homework consisting of traditional homework problems followed the experimental treatment, unavoidably covering other topics taught in the unit that are not covered in the treatment homework. Physics problem solving expertise was assessed by a common quiz given to students together with both homework assignments. The quizzes also consisted entirely of traditional problems (mostly numeric/symbolic response).
The DPA treatment for each unit consisted of four to five DPA sequences, where each sequence consisted of three to seven DPA problems. In each unit, the first three to four DPA sequences focused on training a single, basic level expert skill. The last DPA sequence trained higher level solution planning skills by asking students to identify a number of different errors in a given solution. Each DPA treatment consisted of ~20 problems in total. Among all the DPA sequences created, five of them consisted of more than four problems each (two in unit 10, two in unit 11, and one in unit 12; these are suitable for studying students’ development of each individual skill). The TRD treatment consists of a mix of traditional symbolic problems and conceptual problems, with a total of six to ten problems each unit. Detailed solution for each problem was made available to students immediately after they have finished the problem (answered correctly or used up all the attempts). Although the deliberate practice treatment contains many more problems than the traditional treatment, both treatments cover the target material and are intended to take roughly the same amount of time.
DPA problems in D&D and MC format
An edX D&D problem (Fig. 3) consists of a target figure and a list of draggable icons called “draggables.” For each draggable, the author defines a designated target area on the target figure (different target areas can overlap each other). The problem is graded as correct when all the draggables are dropped onto their designated target areas.
Multiple attempts (5–10) are allowed for each D&D problem. A detailed solution is provided for each DPA problem involved in this experiment and is made available to students immediately after they have answered correctly or used up all their attempts.
The MC version of DPA problems involved multiple choice, checkboxes, and dropdown lists and was designed to mimic their D&D counterparts as closely as possible by using the same figures, problem text, and solution. Since D&D problems often have a very large number of possible answers, the corresponding MC version is broken into two MC sub-questions graded simultaneously, to avoid presenting too many choices. This difference theoretically gives the MC group a small advantage in problem solving, as students are informed whether their answer to each sub-question is correct after each attempt. As a result, fewer attempts are allowed for MC problems than D&D problems, since the extra feedback quickly reduces the problem space.
Selecting subject population
In order to guarantee that the users we consider interacted with the treatment homework to a significant extent, we restrict our attention to those who completed at least 70 % of the treatment homework and at least 70 % of the common quiz. This cutoff leaves a total of 219 students for unit 10, 205 students for unit 11, and 280 students for unit 12 (cf. a total of 614 students accessed at least one of the treatment problems in all three units). Our results are unlikely to be sensitive to the cutoff, as the majority of students (60–70 %) either completed >90 % of both quiz and homework or <10 % of either.
Comparing DPA and TRD format on developing overall problem solving expertise
We analyzed edX log files to estimate the time students spent on the treatments. The time spent on each problem is estimated by including all the time between loading a problem to students’ last submission to the problem, with a maximum cutoff at 30 min. The median time spent on completing the treatment homework is estimated to be around 5000 s (1.4 h) following this method, with no statistically significant difference among the three different treatments nor among the three different groups found (data not shown). Note that because timing data is highly non-normal, we used the nonparametric Mann-Whitney U test to perform pairwise comparison between groups of time-on-task during the homework.
Performance of groups on quizzes with traditional problems
Students’ physics problem solving expertise was measured by their performance, i.e., percentage of first attempt correct, on traditional quiz problems. Percentage of first attempt correct was chosen because it reflects the initial performance and shows the largest differences between groups because the majority answered correctly on the first attempt.
The performance of each treatment group on common quizzes is summarized in Table 1. Simple one-way ANOVA reveals no significant difference between the three groups on any of the quizzes. In Table 2, we show the cumulative difference in average quiz performance between the three types of treatment: DPA-D&D, DPA-MC, and TRD. Since we rotate the treatment between the three groups, the overall samples were the same for the three types of treatment (except that some students did not complete all three homework assignments). The only significant difference is between the TRD homework and DPA-MC, while DPA-D&D showed no significant difference with either of the other two.
Comparing D&D to MC in developing a single skill
The drag-and-drop format did not significantly outperform the MC format when judged by the subsequent traditional problems. This led us to investigate whether the D&D format is superior to the MC format in developing an individual expert skill. We therefore looked at the five DPA sequences that focused on a single basic expert skill, restricting attention to sequences comprised of more than four DPA problems.
The first attempt correct rate on the 1st one or two problems in all five sequences is around 70 %, indicating that the majority of students have previously mastered those skills to some extent. In order to study students who actually need to learn the skills from the activity, we choose to study the 20–30 % of students who did not correctly answer the 1st one or two problems in each sequence on their 1st attempt. We used the 1st two problems for sequences 1 and 2, because they are closely related to each other by design, and should be treated as a unity. The selected population size for each sequence varies from 20 to 40 % of the population that answered >70 % of the given sequence. An example of the analysis is illustrated in Fig. 4.
We measure the change in individual skills by students’ performance on the subsequent three to four problems in each sequence. As shown in Fig. 5, on three out of five sequences, the D&D group answered significantly more questions correctly on the first attempt than the MC group, and on sequence 3, the D&D group answered more questions correct on the second attempt. Since the distributions are highly non-normal, we used Mann-Whitney U test to measure the significance of the difference. As shown in Table 3, a highly significant difference at p < 0.01 level is observed for sequence 1 on both first and second attempt correct. Significant difference at p < 0.05 level is observed for sequence 2 on first attempt correct and sequences 3 and 4 on second attempt correct. In sequence 2, each MC format question has only three choice items, whereas most questions in other sequences have five or more choice items. Therefore, the extra feedback advantage for MC problem is strongest for sequence 2, which is probably why there is no difference on the second attempt correct. (t tests were also performed on the same data with very similar results).
Excluding initial selection bias due to different problem format
Since the initial selection of subjects is based on problems in different formats, these results could arise if students selected in the drag-and-drop groups were stronger but answered the first problem incorrectly because of its unfamiliar problem format. Their subsequent performance would then simply indicate their higher intrinsic ability.
We checked for such selection bias using two different measures. First, if the D&D format selects generally stronger students, then they should have higher physics ability as measured by item response theory (IRT) (Baker 2001; Burton 2004). Secondly, if D&D selects additional stronger students due to format unfamiliarity, then the selected D&D population should be larger than the selected MC population, since random assignment assures similar numbers of weak students in each group.
As shown in Table 4, for sequences 1–3, neither IRT ability nor size of population is significantly different between the two groups at the 0.1 level. For sequences 1 and 2, the D&D population is actually smaller. On sequence 4, the D&D group is somewhat larger than the MC group, and the difference in population is significant at the 0.1 level, whereas in sequence 5, the trend is reversed. On average, there is no difference observed between the two selected populations.
Summary and discussion of learning from DPAs
D&D format superior to MC format for DPAs
The drag-and-drop (D&D) format of deliberate practice activities (DPAs) strongly outperforms the multiple choice (MC) format at teaching students to do more DPAs in that particular format. In addition, in an end-of-course survey, 47 % of students rated the D&D format as more intuitive than the MC format, and only 20 % rated the MC format as more intuitive (Chudzicki 2015).
We interpret this as showing that D&D significantly outperforms the MC format in facilitating “rapid” learning of a single expert skill. This conclusion is buttressed by our finding that both the number and the average skill level of the students selected for our study (on the basis of poor performance on the first DPA homework problems) were insignificantly different. This rules out the possibility that students were actually learning to use the D&D format, rather than the desired target skill.
We think that the D&D format is superior mainly because it reduces extraneous cognitive load and increases visual representation. However, there could be other plausible alternative explanations as well. For example, the D&D group has more attempts per problem and students receive less feedback on each attempt, which might force students to “struggle” longer and hence learn more. Also, the new format might simply be more eye-catching to students, leading to better concentration. Further research is required to distinguish between these differences. Of particular importance would be to assess the single skill that the DPAs are supposed to teach using common DPA problems, for example, by breaking down the problem into a few sub-questions.
DPAs do not outperform traditional problems as practice for doing traditional problems
DPAs, especially in MC format, are slightly less effective preparation for a quiz containing traditional questions than solving traditional problems for the same amount of time.
It is difficult to believe that our particular implementation of DPA is less effective at teaching the targeted expert skill than solving common physics problems. Therefore, it is likely that there are additional skills involved in overall problem solving expertise; there are a number of equally plausible alternative explanations for this result:
The crucial skills that we elected to write DPAs for may not be those that actually result in the students failing to answer the quiz problems correctly. Quite likely, around one fourth of our student population are teachers and might already be fluent with the fairly basic skills that we trained in DPA.
Since MOOC students have access to all the learning resources at all times, the quizzes are essentially open-book assessments. Therefore, the assessment is insensitive to any memory benefits of the intervention.
Further work in which the common quiz questions were designed to carefully probe the mastery of the skills targeted by DPAs would seem important for evaluating the outcomes of DPAs.