MADE Ratio: Affective Multimodal Software for Mathematical Concepts

Abstract

This paper addresses the use of multiple sensory modalities and affective strategies in learning mathematics. A case study is presented where these domains are used to design a mathematical ratio system. We adopted a multimodal approach used by other researchers, and applied our proposed design methods to create the MADE Ratio system. We then recruited participants to test the usability of the system. Our findings were that the design methods were effective, but we also gained insight about how they could be improved.

1 Introduction

In this paper we focus on interaction design and evaluation for a multimodal learning system called “MADE Ratio”. The system is a case study of our MADE (Multimodal Affect for Design and Evaluation) framework. We are considering the sensory modalities, affective and cognitive strategies and trying to solve mathematical learning challenges. Using a multimodal affective learning system should increase the motivation in learning, and will help students develop better understanding of some tasks in mathematics. We first describe our framework, then the design process and the system, and last present the study and results.

2 MADE Framework

Human-computer interaction (HCI) and education can be improved with the help of affective and cognitive strategies in a multimodal user interface (UI) environment. This multimodal environment recognizes two or more combined user input modes (multiple sensory modalities) and recognition-based technologies such as audio or gesture in a coordinated manner with multimedia system output such as images, text or audio [8, 9].

We wish to support affective aspects of learning. We reshaped the three domains of Bloom’s taxonomy [1], a classification of the different kinds of objectives that educators distinguish, and created a framework called MADE. We are considering the multiple sensory and quasi-sensory modality domains to help the affective and cognitive domains (Fig. 1).

The MADE framework is based on principles for multimodal design considering emotional and cognitive aspects of learners while interacting with a multimodal educational system. While interacting with educational software, a student can employ the sensory modalities (e.g. 3D motion controller, face detection and tracking, vision or audio); they may also need to employ some quasi-sensory modalities (e.g. embodiment, persuasion or narrative) while interacting with a supportive technology [5]. We follow Kort et al.’s affective model [7]. This identifies four phases of learning and the affective character of each: encouraging exploration with positive affect, challenges and negative affect, supporting overcoming challenges and reducing the negative affect, and affirming learning so restoring positive affect.

Fig. 1.

The MADE framework

3 Design Process

Our case study was to design a system to help learn the concept of a mathematical ratio. The modality involves embodied cognition to support pedagogy, and is based on the work of Howison et al. [6].

This design should increase the proprioceptive, kinesthetic and episodic memory experience, as the controller that we use requires physical motion of the arms and the hands. The proprioceptive memory involved will aid remembering how to perform intended physical actions in the future. The learning takes place by the students carrying out a physical activity by moving their hands and body movements to interact with the UI. By using affective strategies we can increase the episodic memory, which actually happens to learners; the learner will think about the time back when they did this with their hands. The system supports discovery learning and exploration using sensory modalities.

In our system, one of the tasks is that learners have to make a requested ratio. Feedback is provided by showing the screen in different colors and giving persuasive feedback i.e. praising messages, audio, icons or images. The learners move their left and right hands in regard to a specific ratio and distance between their hands.

We used two design methodologies, Affective Personas and Affective Essential Use Cases. Affective Personas identify key aspects of the emotional state of users, and Affective Essential Use Cases describe goals for emotional aspects of user experience to be supported by the system. We described these design processes in detail elsewhere [4].

4 System

Our system uses a Leap Motion 3D Controller (https://www.leapmotion.com/), an easy to use, low-cost hand-motion tracking controller to support the software by remote manipulation interaction (see Fig. 2). The physics-based manipulation is more than just hand waving. It can be an opportunity for the mind to reflect on what the body can already do. Embodied interaction suggests that UI becomes even more visible and available for a wider range of engagements and interactions [2]. It can drive both the understanding and solving cognitive conflicts between student’s implicit assumption and her/his own observable enactment, providing experiences that are recast in terms of emerging mathematical concepts [6].

Fig. 2.

The Leap Motion Controller

Figure 3 shows a snapshot of the main screen of our system when the ratio task has been completed. It shows how a student is interacting with a Leap Motion device using his hands and moving them to indicate a proportion of 1:3. For developing the software we used JavaScript and the Raphael graphics library (http://raphaeljs.com/) as well as PHP and MySQL.

Fig. 3.

A student sitting at a desk is moving the right hand higher from the desk searching for equivalent proportions 1 to 3 or 2 to 6; a ratio of 1:3 is reached, maintaining the goal proportion and resulting in affective feedback of audio, a message, icon, image and screen color change. (Color figure online)

The main elements are as follows:

• Main window: the main part of the screen shows two circles representing each hand. The circles move as the student moves her/his hands up and down. The background color is gray when the correct ratio is not achieved (here it is green as a ratio of 1:3 has been reached).

• Left section: provides the student with affective feedback (audio, messages, icons, images, music and color), when the task is successfully done.

• Right sidebar: the student and teacher can modify the affective settings, height, and ratio. The teacher can view and change the cognitive and affective strategies in the database.

5 Study

In this study we considered the usability of our system. Below we explain the study, talk about the expectations, our participants, the equipment, the procedure, and the analysis methods. The study was approved by our university research ethics board. We were not doing formal hypothesis testing, but simply doing usability testing and comparing the results with our expectations. Our expectations were as follows:

• 1. The Leap Motion Controller is easy and engaging

• 2. Fun results from the physical interaction

• 3. The affective design supports learning

Participants

Eleven undergraduate and graduate students (3 females and 8 males) from two universities in Canada volunteered to participate in this study. They ranged in age from 21 to 60, with a mean age of 33 (S.D. = 12). All participants were able to use their hands and arms freely and had ability to work in English. All participants reported using computers daily, and most participants (85 %) reported using them for 11–20 years. All participants stated they had used pencil and paper, or mouse and keyboard, when they learned about ratios while in school. Seven participants were right-handed, three were left-handed, and one was ambidextrous.

Equipment

A Leap Motion Controller, a Dell desktop computer with a 22-in. high-resolution LCD monitor, and two speakers were used. We used an audio recorder to capture the participant’s voice comments.

Procedure

The study took thirty minutes for each participant. We taught the participants how to interact with the system using their hands before the study, for five minutes. We used a think aloud protocol, and did audio recording. The main part of the procedure had four steps:

• 1. Training phase: We had a five minute testing phase, providing a short training exercise for the user before starting with the ratio task, asking them to have their both hands to position 5, and then space left-hand and right-hand four units apart to get familiar with the system.

• 2. Testing phase: After the training phase we had a testing phase of fifteen minutes. First they had to do the default tasks and see the affective feedback we provided them, and we monitored their reactions. Next they would customize their settings, and we monitored their reactions and saw their preferences.

• 3. A Usability questionnaire that we explain in the “Quantitative Results” section, below.

• 4. An Open-ended questionnaire that we explain in the “Qualitative Results” section, below.

We collected both qualitative and quantitative data at the same stage of the study. We used thematic analysis for the qualitative aspects of the study. We collected the data from comments and answers to open ended questions, went through it all, and identified themes.

6 Quantitative Results

First we address our stated expectations:

• 1. The Leap Motion Controller is easy and engaging: Almost all participants (82 %) agreed that there is no fatigue involved while interacting with the UI.

• 2. Fun in the physical interaction: Regarding the pleasantness and how pleasant is to interact with the UI, 91 % of participants agreed that using Leap Motion Controller would be more engaging than using mouse and keyboard.

• 3. Affective design support of learning: Regarding the affective strategies 72 % of participants agreed that it would help in motivating students in learning ratio.

Fig. 4.

Usability questionnaire about the MADE Ratio system (Color figure online)

Figure 4 shows the usability questionnaire results. On the left side there are eight statements. The participants were given a questionnaire to evaluate and rate their experience and subjective impressions of the system. For each question, subjects were given a Likert scale between agree and disagree. For analysis we used R (https://www.r-project.org/). The results of this analysis are shown in Fig. 4. The responses are shown as boxplots, where the inner quartiles are shown as colored boxes, the outer quartiles with whiskers, and outliers as circles.

The first statement is about understanding the need for affective strategies in this learning system; all the participants agreed. For the second statement (Not hard to use Leap Motion Controller) mostly everyone agreed. By providing a training phase at the beginning of the study, everyone understood how the system worked. The forth statement was understanding the meaning of different affective strategies, which everyone understood. The fifth statement was about task affirmation and reflection on the UI, and everyone but one agreed. The sixth statement explore whether the system motivated the participant in learning about mathematical ratios, but not all agreed. The seventh statement was whether it is useful to use affective strategies in learning mathematics; only two of the participants were neutral but the rest agreed. The last statement was if the participant would like to learn other mathematical concepts using this software; almost everyone agreed except one participant somewhat agreeing, and another was neutral.

Overall, the rating indicated approval of the system design; however, some differences were observed among participants.

7 Qualitative Results

At first we did a pilot study with one participant to capture firsthand behaviors and interactions that might have not been noticed. We found some problems with the system, and fixed them for the real study. One issue was that the Leap Motion Controller was far from the participant, which meant the participant was not feeling comfortable and had some fatigue with his hand. Having it closer made it easier to hold up his hands, giving a finer control over the height making it easier to use with no fatigue. Another concern was the participant said that the text wording could be improved in a lot of places, and we fixed all those instances. Another issue was brought up was the audio. He said: “I’m not a fan of audio in general, but the applause seemed over the top, and the music was in the way because we were talking.” So we provided controls for the audio.

7.1 Multiple Sensory Modalities

Fun and Engaging: Almost all the participants agreed this system would be fun and engaging for children. They said the Leap Motion was easy to use and quite fun to try for the first time. One participant said: “I personally have cousins who are currently learning ratios, and believe this would be an interesting way for them to learn, that they would enjoy”. Another participant said: “I think having Leap Motion is an effective and innovative approach for kids to be engaged in learning mathematics. The device is small, light and easy to use and having a friendly interface would be successful in my opinion for learning.” Another participant said this system is great for allowing students to respond in the modality that they have the most comfort with. He said the Leap Motion is good for visualization representations for how a ratio scales. He had a suggestion that there be option to interact in a horizontal way as well, suggesting that would more directly correspond to hand positions.

Easy to Use But Could Be Improved: One participant said the Leap Motion is mostly easy to use but it is a bit finicky. Some participants mentioned there should be a physical reference such as a ruler displayed vertically. Another said it would be nice to have a ruler (scale) guide on the side as reference points to help user more accurately move up and down.

Learnability and Memorability: One participant said that the Leap Motion would help with learnability and memory, because you’ve experienced it yourself. Another participant said it was very smooth and there was no lag-time when interacting with the Leap Motion.

Control of Audio: One participant said the audio output would be suitable as long as it is controllable. Another participant mentioned: “While performing a task, if the music is playing, it might be distracting for some people, but the praise (e.g. clapping) when the job was complete can be a good thing.”

7.2 Affective Strategies

Effect on Learning: One participant said this multimodal system had benefits by providing multiple interaction modes including audio, visual and using an embodied interaction approach; the Leap Motion can make UI learnable and memorable. One participant said: “I feel proud of myself when I hear applause.” He said that the happy face is encouraging, audio is motivating, but he did not like the specific music in the background, and he said it depends on type of music. Another participant said: “The affective strategies are great for kids, and it’s very encouraging.” Another participant said: “The affective strategies like music are helpful in learning, since it challenges the students to try, and in the meantime it presents a mathematical problem like a computer game. This interactive approach is helpful since it draws students attention more than traditional learning techniques.” Most of the participants said this system could be very good for children, e.g. music, audio, color, text and using humor and funny things can be engaging for kids. One participant said: “I like the idea of giving different types of feedback and customizing that feedback.” She said that affective strategies present in this system are appropriate for kids but not for adults.

Challenges: Some participants would have liked if the system provided the learner with a score. This possible addition could be a mechanism to make the learning seem more like a game that would keep the students’ attention and challenge them. One participant suggested that there be continuity to the music during the task (e.g. each part of the task for 30 s, and eventually music finishes when the person completes the whole task and it is done. Another participant suggested that when a user wants to start using the system, it can ask the user their favorite color for the background choice. Some participants had concerns for the location of content on the screen. Some liked the affective feedback’s location on the screen, but they said it should not go on top of the learning objective. Some participants suggest the ratio can move closer to the hand circles on the main screen as maybe a better idea.

Virtual Characters: A participant mentioned adding virtual characters; an avatar of the learner. For example, there is an appeal to emotional investments of virtual characters like Pokemon, and we could provide for the student to choose an avatar that reacts to their performance, and lets them earn token currency as a reward for success – which could be spend to customize the avatar – and allow custom themes for success/fail sounds, a custom look and feel theme to the interface as personalization.

7.3 Facial Recognition

We suggested that our system might use analysis of facial expression to detect emotional state, and asked for their comments.

Useful for Teaching: The participants agreed with having a facial recognition with the system, and they said it might be a great way for adapting difficulty to success/failure e.g. frustrations vs. excitement. One participant said: “I think that facial recognition should be used, as it would allow the teacher to be able to monitor individual performance of students, to know when to modify the difficulty or focus on a skill that doesn’t seem to be understood.” This approach might be useful to help better understand the students’ moods and feelings and how they feel about solving mathematical problems. Teachers might then adapt the exercise based on the learner’s frustration with difficulty. If a student feels that they are behind, facial expressions would help teacher to adjust/modify level of difficulty. It might be interesting data to collect for a teacher to reflect back on and use in their assessment of teaching methods, and assessing their student’s performance.

Privacy Issues: At least two participants said: “Showing the student’s face in the screen would be distracting.” One of these participants mentioned people may not like capturing their face (privacy) and added: “Probably not? It would have to be done very well, so that it’s not really telling you how you feel. [There is a] danger of being creepy monitoring. How would it improve my experience? and is it going to be worth the costs?” Another participant said that she is not sure how useful it would be. Almost all the participants agreed that it would be very interesting for the teacher to user a facial recognition. Only one of the participants said: “Would there not be some privacy concerns here? I personally would not like it. Some people do not like to express or have their emotions show/being monitored. E.g. with frustration/anger/sad. Learning environment should be more visual/interactive rather than computer monitoring; the human touch is needed here”. Therefore, they felt it could be useful for the teacher to see the video, but it would be more distracting if there was a real time video on the same page, and some people might not like having their image recorded.

7.4 General Advantages

As one participant mentioned, this system has a lot of potential for some people who have problems focusing on written material (e.g. ADHD). He added that this system provides a visual, auditory and physical interaction which is the environment they learn best in. He said: “I find it very effective for my age and personality: I don’t like voice; I’m more visual.” He would like the system to somehow show the movements of his hands on the screen. Other participants agreed on the following general advantages:

• 1. This system is fun and is more engaging compared to the traditional way of learning ratios.

• 2. Customization in this system is an advantage.

• 3. The system provides a hands-on physical interaction.

• 4. The system would raise the curiosity of children in learning.

7.5 General Disadvantages

“The User has to focus on each the left and right side of the screen. It took his attention and made he look at different directions while looking at numbers and make calculations.” This was said by one participant, who suggested that more coherence on the main screen. Another participant said: “If you could combine the points of attention; you have three-points of attention: our hands, hands shown on the screen and ratios on the left, which causes three focal points. Combine them into one.” This issue was perhaps the most prominent disadvantage; the entire list was:

• 1. Potential problem of different places to focus.

• 2. More visual cues about what is going on.

• 3. Need for challenges and game-like structure.

• 4. Potential problem with motor skills hand-eye coordination.

8 Discussion

We have described our rationale for, and implementation of, the MADE Ratio system, including our design-based research approach and reporting on qualitative and quantitative studies with eleven undergraduate and graduate students who engaged in problem-solving tasks with the MADE Ratio. Our study is consistent with the work of Howison et al. [6], while they have not emphasized the multimodality aspect and affective strategies. Out tasks were performed by the participants who had prior experience with ratios and proportions, but not in a multimodal environment. They had learned mathematical concepts with pencil and paper, or using mouse and keyboard.

The math topics we considered in this system were ratios and proportions. We tried to implement a multimodal system with a super-simple affective experience to increase engagement and attention of the students. This system is customizable, meaning the teacher can select different learning objective, affective and cognitive strategies in the database for different personas. The teacher monitors the students understanding of the material and provides different learning strategies to each student. These strategies employ sensory and quasi-sensory modalities, which we explained earlier.

Reflecting on our experience, we now suggest that the system might also have elements of persuasive technology tools, as described by B.J. Fogg [3]. Specifically, the system might be seen to be using tailoring, which is persuasion through customization (the teacher is able to customize the database according the each student’s interest), as well as reduction approach to make the complex ratio task simpler (by using the Leap Motion Controller) to increase a student’s motivation to engage in the task more frequently. Fogg has proposed that tailored information is more effective than generic information in changing attitudes and behaviors; the information provided by a system will be more persuasive if it is tailored to the individual’s needs, interests, or other factors relevant to the individual. We also discussed affect and so we might be seen as employing Pathos, from Aristotle’s modes of rhetoric. Pathos represents an appeal to the emotions of the students, and elicits feelings that already reside in them.

The reflections of our participants also leads us to consider gamification in education as a potential new approach we might explore in refining the system.

9 Conclusions

The ability to communicate emotionally and cognitively plays an essential role in HCI and education. This paper focused on UI design and implementation for a multimodal system in affective education, and conducted usability testing. We presented the rationale, design, implementation, and early results from our study of a novel educational technology. The goal of this study was to determine if this system can motivate and engage students in learning ratios and proportions. The affective strategies and the Leap Motion Controller were well suited for this environment.

We validated our three expectations: The Leap Motion Controller was easy and engaging, the system did support fun through the physical interaction, and the affective design support will increase the motivation in learning and the students did better understand some tasks in mathematics.

Our next steps will consider the following potential improvements. First, the interface should show a list of the tasks for the students to attempt, rather than relying on a teacher. Second, the system should show the student their learning progress. Third, and perhaps the most important, the system could allow the teacher to monitor and see current student status and how he/she is doing, showing the affective status, and allowing changes to be made to the task list and the affective feedback strategies. We also plan to upgrade the system adding facial recognition to it to support this. We plan to do another user study to monitor the task difficulties/easiness, and do more quantitative analysis on the data we collect. We will use an updated, modified version of the questionnaire and will consider all the feedback participants provided in this study.

In future, we will see what other relevant tasks can be specified and used with this system, and if the framework and the system could also be applicable to other domains. In the design of this system, we used two design methodologies, Affective Personas and Affective Essential Use Cases. It is hoped that these methodologies may support designers in the creation of multimodal affective software for teaching environments.