Keywords

1 Introduction

The ability to tackle non-routine problems – those that cannot be solved with a known method or formula and require analysis and synthesis as well as creativity [9] – is becoming increasingly important in the 21st century [5]. However, when faced with a non-routine problem, U.S. students tend to apply memorized procedures incorrectly rather than modify them or develop new solutions [8]. One possible source for this difficulty is the typical instructional focus in U.S. schools on memorization and application of routine procedures [2, 6, 7]. Such an approach makes students proficient at executing rote procedures, but it does little to help them understand the conceptual basis for the procedures or to think creatively about novel problems - both of which are essential for developing problem-solving flexibility.

An important first step in addressing this issue is to assess how students currently approach non-routine problem solving, so that we can design the appropriate learning interventions. In this work, we present an empirical cognitive task analysis where participants were asked to think aloud while solving a series of non-routine problems from discrete mathematics. We chose this domain because discrete math problems can often be tackled from multiple perspectives while not requiring any advanced background beyond the high school curriculum [3]. Based on the findings from this study, we propose our plan for developing a tutoring system for non-routine problem-solving ability. Then, we discuss the system’s broader implications and the challenges we need to address in deploying this system at scale.

2 Assessing Students’ Problem-Solving Skills

We conducted interview sessions with three students at a private university in a midwest US city. None of the students had a mathematics-related background. The participants were asked to solve three non-routine mathematics problems on paper in one hour. They were also encouraged to think aloud and write down their draft work. The three problems in our study, taken from [3], and a brief summary of their sample solutions, are as follows.

Problem 1:

In an air show there are twenty rows. The first row contains one seat, the second three seats, the third five seats, the fourth seventh seats, and so on. How many seats are there in total?

Sample solution: In the first row there is 1 seat. In the first two rows there are 1 + 3 = 4 seats. In the first three rows there are 1 + 3 + 5 = 9 seats. In the first four rows there are 1 + 3 + 5 + 7 = 16 seats. In the first five rows there are 1 + 3 + 5 + 7 + 9 = 25 seats. Based on this pattern, in the first k rows there are k2 seats. In our case, there are 20 rows and therefore 400 seats in total.

Problem 2:

Find all integers between 1 and 99 (inclusive) with all distinct digits.

Sample solution: there are 99 integers between 1 and 99 in total, and 9 of them have non-distinct digits, namely 11, 22, 33, …, 88, 99. Hence, the remaining 90 integers have distinct digits.

Problem 3:

What is the digit in the ones place of 2 57 ?

Sample solution: Looking at the sequence of powers of 2–2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, … – we see that the corresponding sequence of digits in the ones places is 2, 4, 8, 6, 2, 4, 8, 6, 2, 4, … In other words, this sequence is a cycle of length 4. Therefore the last digit of 257 is that of 253, which is that of 249, …, which is that of 21, which is 2.

We then analyzed recordings of the participants’ think-aloud and their draftwork, from which we derived the following insights:

Pattern Identification.

Participants were aware that they had to find a pattern or formula to solve the problems, because it was not feasible to directly compute the final answer. All participants were able to identify the expected pattern for each problem as outlined above, except for one student who failed to do so for Problem 1. While this participant realized that the number of seats on row k is the k-th positive odd number, this pattern alone was insufficient to solve the problem.

Visualization.

Participants tended to visualize the problem by drawing examples and making lists or tables (Fig. 1). They expressed that these visualizations were crucial in helping them identify the correct pattern and solve the problem.

Fig. 1.
figure 1

Participants’ attempts at visualizing the problem in their draftworks.

Full size image

Computation.

Participants occasionally made computational mistakes while calculating the initial sequence values, especially in Problem 3. As a consequence, they could not identify any pattern based on the wrong values, and took some time to realize the mistake. All students who corrected their mistakes were able to subsequently solve the problem.

In summary, we found that participants were aware of the idea behind identifying patterns, and they all did so via some kind of visualization. On the other hand, computational mistakes, while not directly related to our learning objectives, can be detrimental to the overall problem-solving process. From these insights, we propose the following next steps.

3 Developing a Tutoring System for Flexible Problem-Solving

Moving forward, our plan is to iteratively conduct more cognitive task analysis interviews and develop a prototype of the system. Our initial conceptualization of how the system will work is as follows. A single round of exercise in the system incorporates four learning stages, all of which are built on established learning principles: 1) Reviewing a worked example of a non-routine mathematics problem, 2) Explaining the worked example to a partner, 3) Solving a new problem which is isomorphic to the worked example problem, and 4) Explaining the isomorphic solution to a partner. Between rounds, the student can review previous solutions, look at materials related to the problem space, or practice basic math skills. This design is intended to (1) formally introduce students to a complete solution through worked examples, (2) reinforce their understanding of the worked example through self-explanation, and (3) assess students’ learning through an isomorphic problem. Our hypothesis is that through the learning system, students will get a better sense of how to approach a novel non-routine problem, so that in case they have not yet found the solution – for example, like the participant in our study who did not identify the true pattern in Problem 1 – they can still adopt a different viewpoint and explore other strategies.

We have already begun mapping the problem space by developing a non-routine problem-solving flowchart and identifying sets of potential non-routine problem solutions. Once we have tested our solution space, we will develop and pilot a low fidelity paper prototype version of the system with college students to further refine the mathematical content and identify areas for revision to the design. We are also looking at which technological features could be useful for students learning in this domain. As a first step, our system will include a canvas for students to perform their draftwork on, as well as a simple calculator interface with basic arithmetic operations to help students avoid computational mistakes. An important follow-up question is whether students’ draftwork can be analyzed to infer their thinking process, which could in turn guide the design of appropriate feedback mechanics. While this task has previously been performed manually by domain experts [1], employing a machine learning technique to automate it to some extent would greatly enhance the system’s adaptive support functionality and scalability.

4 Conclusion

This research will provide concrete, generalizable evidence about the utility and implementation of worked examples, multiple solutions, and self-explanation to promote skills in non-routine problem solving. Results will inform future tutoring system design by identifying how and when the instructional features are most beneficial for developing problem-solving skills. We also intend to have a practical impact by distributing a tutoring system that is accessible to a wide range of students, including lower-performing students who would typically not be exposed to these types of problems and strategies [1, 4]. In addition, we will provide a teacher’s guide to support educators in using the system adaptively to support their instructional goals.