Do Learners Recognize and Relate to the Emotions Displayed By Virtual Instructors?

Abstract

There has been much research on the effectiveness of animated pedagogical agents in an educational context, however there is little research about how the emotions they display contribute to a learner’s understanding of the lesson. The positivity principle suggests that learners should learn better from instructors with positive emotions compared to those with negative emotions. Additionally, the media equation theory (Reeves and Nass 1996) would suggest this principle should be true for animated instructors as well. In an experiment, students viewed a lesson on binomial probability taught by an animated instructor who was happy (positive/active), content (positive/passive), frustrated (negative/active), or bored (negative/passive). Learners were able to recognize positive from negative emotions, rated the positive instructors as better at facilitating learning, more credible, more human-like, and more engaging. Additionally, learners who saw positive instructors indicated they tried to pay attention to the lesson and enjoyed the lesson more than those who saw negative instructors. However, learners who saw positive instructors did not perform better on a delayed test than those who saw negative instructors. This suggests that learners recognize and react to the emotions of the virtual instructors, but research is needed to determine how the emotions displayed by virtual instructors can promote better learning outcomes.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

References

  1. Baylor, A. L., & Ryu, J. (2003). The API (Agent Persona Instrument) for assessing pedagogical agent persona. Technology, Instruction, Cognition & Learning, 2, 291–314.

    Google Scholar 

  2. Calvo, R. A., D’Mello, S., Gratch, J. M., & Kappas, A. (Eds.). (2015). The Oxford handbook of affective computing. New York: Oxford University Press.

    Google Scholar 

  3. Cassell, J., Sullivan, J., Churchill, E., & Prevost, S. (Eds.)., (2000). Embodied conversational agents. Cambridge: MIT Press.

  4. Derry, S. J., Sherin, M. G., & Sherin, B. L. (2014). Multimedia learning with video. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 785–812). Cambridge: Cambridge University Press.

    Google Scholar 

  5. D’Mello, S., Lehman, B., Sullins, J., Daigle, R., Combs, R., Vogt, K., Perkins, L., & Graesser, A. (2010). A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. International Conference on Intelligent Tutoring Systems (pp. 245–254).

  6. D’Mello, S. D., Lehman, B., & Graesser, A. (2011). A motivationally supportive affect-sensitive autotutor. New Perspectives on Affect and Learning Technologies, 3, 113–126.

    Article  Google Scholar 

  7. Johnson, W. L., & Lester, J. C. (2016). Face-to-face interaction with pedagogical agents, twenty years later. International Journal of Artificial Intelligence in Education, 26(1), 25–36.

    Article  Google Scholar 

  8. Johnson, W. L., Rickel, J. W., & Lester, J. C. (2000). Animated pedagogical agents: Face-to-face interaction in interactive learning environments. International Journal of Artificial intelligence in education, 11(1), 47–78.

    Google Scholar 

  9. Lawson, A. P., Mayer, R. E., Adamo-Villani, N., Benes, B., Lei, X., & Cheng, J. (2021). Recognizing the emotional state of human and virtual instructors. Computers in Human Behavior, 114https://doi.org/10.1016/j.chb.2020.106554.

  10. Loderer, K., Pekrun, R., & Lester, J. (2021). Beyond cold technology: A systematic review and meta-analysis on emotions in technology-based learning environments. Learning and Instruction, 70. https://doi.org/10.1016/j.learninstruc.2018.08.002.

  11. Mayer, R. E. (2011). Applying the science of learning. Boston: Pearson.

    Google Scholar 

  12. Mayer, R. E. (2014a). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 43–71). Cambridge: Cambridge University Press.

    Google Scholar 

  13. Mayer, R. E. (2014b). Principles based on social cues in multimedia learning: Personalization, voice, embodiment, and image principles. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 345–368). Cambridge: Cambridge University Press.

    Google Scholar 

  14. Mayer, R. E. (2020a). Multimedia learning (3rd ed.). Cambridge: Cambridge University Press.

  15. Mayer, R. E. (2020b). Searching for the role of emotions in e-learning. Learning and Instruction, 70. https://doi.org/10.1016/j.learninstruc.2019.05.010.

  16. Mayer, R. E., & Greeno, J. G. (1972). Structural differences between outcomes produced by different instructional methods. Journal of Educational Psychology, 63(2), 165–173.

    Article  Google Scholar 

  17. Mayer, R. E., & Estrella, G. (2014). Benefits of emotional design in multimedia instruction. Learning and Instruction, 33, 12–18.

    Article  Google Scholar 

  18. Mayer, R. E., Johnson, L., Shaw, E., & Sahiba, S. (2006). Constructing computer-based tutors that are socially sensitive: Politeness in educational software. International Journal of Human Computer Studies, 64, 36–42.

    Article  Google Scholar 

  19. Mayer, R. E., Fiorella, L., & Stull, A. (2020). Five ways to increase the effectiveness of instructional video. Educational Technology Research and Development, 68, 837–852. https://doi.org/10.1007/s11423-020-09749-6.

    Article  Google Scholar 

  20. Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19, 309–326.

    Article  Google Scholar 

  21. Pekrun, R., & Perry, R. P. (2014). Control-value theory of achievement emotions. In R. Pekrun & L. Linnenbrink-Garcia (Eds.), International handbook of emotions in education (pp. 120–141). New York: Taylor and Francis.

    Google Scholar 

  22. Pekrun, R., & Linnenbrink-Garcia, L. (2012). Academic emotions and student engagement. In Handbook of research on student engagement (pp. 259–282). Boston: Springer.

    Google Scholar 

  23. Plass, J. L., & Kaplan, U. (2015). Emotional design in digital media for learning. In S. Y. Tettegah & M. P. McCreery (Eds.), Emotions, Technology, and Learning (pp. 131–161). Cambridge: Academic.

  24. Plass, J. L., & Kaplan, U. (2016). Emotional design in digital media for learning. In S. Y. Tettegah & M. Gartmeier (Eds.), Emotions, technology, design, and learning (p. 131–161). Elsevier Academic Press. https://doi.org/10.1016/B978-0-12-801856-9.00007-4.

  25. Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., & Um, E. (2014). Emotional design in multimedia learning: Effects of shape and color on affect and learning. Learning and Instruction, 29, 128–140.

    Article  Google Scholar 

  26. Plass, J. L., Homer, B. D., MacNamara, A., Ober, T., Rose, M., Pawar, S., Hovey, C. M., & Olsen, A. (2020). Emotional design for digital games for learning: The affective quality of expression, color, shape, and dimensionality. Learning and Instruction, 70. https://doi.org/10.1016/j.learninstruc.2019.01.005.  

  27. Reeves, B., & Nass, C. (1996). The media equation. New York: Cambridge University Press.

    Google Scholar 

  28. Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.

    Article  Google Scholar 

  29. Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172.

    Article  Google Scholar 

  30. Um, E. R., Plass, J. L., Hayward, E. O., & Homer, B. D. (2012). Emotional design in multimedia learning. Journal of Educational Psychology, 104(2), 485–498.

    Article  Google Scholar 

  31. Wang, N., Johnson, W. L., Mayer, R. E., Rizzo, P., Shaw, E., & Collins, H. (2008). The politeness effect: Pedagogical agents and learning outcomes. International Journal of Human Computer Studies, 66, 96–112.

    Article  Google Scholar 

  32. Woolf, B. P., Arroyo, I., Cooper, D., Burleson, W., & Muldner, K. (2010). Affective tutors: Automatic detection of the response to student emotion. In R. Nkambou, R. Mizoguchi, & J. Bourdeau (Eds.), Advances in Intelligent Tutoring Systems (pp. 207–227). Berlin: Springer.

    Google Scholar 

Download references

Acknowledgements

This project was supported by Grant 1,821,833 from the National Science Foundation.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Alyssa P. Lawson.

Ethics declarations

Conflict of Interest

The authors report no conflicts of interest.

Appendix A Video Script

Appendix A Video Script

Hi everyone. Imagine that you are trying to impress your friends with your ability to predict what will happen if you roll a die a certain number of times. For example, suppose you win if you roll 5 or 6 and you lose if you roll 1, 2, 3, or 4. Let’s say you roll the die 5 times and you win 2 times and lose 3 times. What exactly is the probability of that happening? Today, I will help you understand how to answer questions like this one. This is called binomial probability.

First, you need to understand trials and outcomes. A trial is something you do. For example, you roll a die. An outcome is what happens on the trial. For instance, if you roll a die (the trial), the outcome could be that you rolled a 4.

Second, you also need to think about success and failure. A success is defined, by you, as one or more of the possible outcomes. For example, a success of rolling the die could be that you roll a number greater than 4. That means, if you roll a die, and get a 5 or 6, a success has occurred. On the other hand, a failure occurs on any trial that is not a success. So, if you defined success as rolling a number greater than 4, failure would occur if you rolled a 1, 2, 3, or 4.

Next, we should figure out the probability of success. The probability of success is the number of success outcomes divided by the total number of outcomes (including the success outcomes) if all the outcomes have an equal chance. In this case, there are 6 equally likely outcomes and 2 of them are successes, so the probability of success is 2 out of 6 or one-third. We can expect a 5 or a 6 to come up on about one-third of the times the die is rolled. The probability of success can be symbolized by the letter P.

Similarly, there is a probability of failure. This is the probability of success subtracted from 1. So, in our example, the probability of failure is 1 minus one-third which is two-thirds. The probability of failure can be symbolized as 1 minus P.

Now you know how to determine the probability of success (symbolized as P) and the probability of failure (symbolized as 1 minus P).

The next concept you need to know is sequence. A sequence is what happens when you conduct several trials, one after another, like rolling a die 5 times in a row. For each trial, we have either a success or a failure, so the sequence reports what occurred. For example, say we rolled a die 5 times in a row and rolled a 2, then a 4, then a 6, then a 2, and then a 5. The sequence would be failure, failure, success, failure, success.

A sequence, like the previous example, has a probability of occurring, which is called the joint probability of a sequence. This can be found by multiplying the probabilities of each individual event. Let’s take the previous example. We had failure, failure, success, failure, success. Now, we multiply the probability of each happening, so we get two-thirds (for failure), times two-thirds (for failure), times one-third (for success), times two-thirds (for failure), times one-third (for success). We can also write this as one-third squared times two-thirds cubed. So, the joint probability of this particular sequence occurring is 8 out of 243.

We can compute the probability for any specific sequence. So, let’s say the number of trials in a sequence can be symbolized by the letter N and the number of successes in those trials is called R and the number of failures is N minus R. To figure out the probability of any sequence, you can use the formula displayed on the screen. We multiply the probability of success (P) by itself R times, then we multiply the probability of failure (1 minus P) by itself N minus R times, and we finally multiply those two numbers together. This is called the joint probability of a sequence.

Now you know how to compute the joint probability of a sequence of successes and failures. The next step is to figure out how many different sequences (that is, patterns of successes and failures) have that same number of successes out of N trials. For example, there are three different ways that we can have 2 successes from 3 trials:

  • success, success, failure.

  • success, failure, success.

  • failure, success, success.

As you can see, in each sequence, there are 2 successes and 1 failure. The number of different sequences having R successes in N trials is called the number of combinations. In this example, there are 3 combinations for a sequence having 2 successes out of 3 trials.

The number of combinations may be simple to work out by hand when there are just a few trials, like our previous example, but what if I asked you how many different combinations can occur for 2 successes in 5 trials? In cases like this, having a formula to find the number of combinations is quite helpful. This formula is N factorial divided by R factorial times N minus R factorial. This equation includes a factorial symbol (indicated by an exclamation point). This factorial symbol means multiply the number before the exclamation mark times the number minus one, then times the number minus two, and so on down to 1. For example, 5 factorial equals 5 times 4 times 3 times 2 times 1, which equals 120.

Now, let’s finish finding the number of combinations that can occur for 2 successes in 5 trials. So, 5 factorial is equal to 120, which we just found out. Then, we divided that by 2 factorial (which is 2 times 1) times 5 − 2 factorial, or 3 factorial (which is 3 times 2 times 1). That gives us 120 divided by 12, which equals 10. This means there are 10 ways to get 2 successes in 5 trials.

Now you see how to compute the joint probability of a particular sequence that has R successes in N trials (such as failure, failure, success, failure, success) how to compute the number of combinations in which a sequence has R successes in N trials (such as 10 ways to get 2 successes out of 5 trials).

As the final step in computing binomial probability you just put those two parts together. You can figure out the probability of getting R successes out of N trials by multiplying the number of combinations for a sequence that has R successes out of N trials by the joint probability of any one of those sequences. When you do this, you are finding the probability of R successes in N trials. So, if you put that all together you get the formula on the screen. This is what we call a binomial probability.

Let’s do an example to see how well our formula for binomial probability works. Suppose I want to find the probability of rolling a die 5 times and having 5 or 6 come up exactly 2 times. In this case, we want to know the binomial probability of 2 successes in 5 trials, when the probability of success is 1/3. The binomial probability equals the number of combinations that have 2 successes in 5 trials times the joint probability of this sequence.

The number of combinations is 5 factorial divided by 2 factorial times 3 factorial, which equals 5 times 4 times 3 times 2 times 1 divided by 2 times 1 times 3 times 2 times 1 which equals 120 divided by 12 which is 10.

The joint probability of any one sequence is one-third times one-third times two-thirds times two-thirds times two-thirds, which equals 8 divided by 243.

Multiply the number of combinations times the joint probability of a sequence. We get 10 times 8 divided by 243 which equals 80 divided by 243 (or about 0.33).

This means you have about a one-third chance of rolling a die 5 times and getting 5 or 6 to come up exactly 2 times.

Now you know how to determine the probability of R successes out of N trials when the probability of success is P.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lawson, A.P., Mayer, R.E., Adamo-Villani, N. et al. Do Learners Recognize and Relate to the Emotions Displayed By Virtual Instructors?. Int J Artif Intell Educ (2021). https://doi.org/10.1007/s40593-021-00238-2

Download citation

Keywords

  • Animated pedagogical agents
  • Affective processes
  • Emotion
  • Emotional tone
  • Learning