Skip to main content

Combining Physical, Virtual, and Mental Actions and Objects

Educational Psychology Review volume 30, pages 1091–1113 (2018)Cite this article

Abstract

The grounding of cognition in embodied actions has resulted in both theoretical formulations of encoding of information and instructional applications of these encodings. Actions on objects typically occur within matched formats: physical actions on physical objects, virtual actions on virtual objects, or mental actions on mental objects. Instead, I explore the theoretical and instructional implications of a 3 × 3 taxonomy that creates nine combinations of physical, virtual, and mental actions with physical, virtual, and mental objects. Following a description of each combination, I discuss the implications of the taxonomy for generic characteristics of instruction that apply across representational formats and taxonomic differences in which format matters. I then compare the proposed taxonomy to three other recent taxonomies including a taxonomic analysis of encoding that defines encoding as a relation between two entities (an iconic, motoric, phonetic, or semantic code) in which information in the first entity is used by the second entity. A comprehensive theory of manipulatives will require specifying the interconnections among theories that range from a small grain size in which the representation of actions differs from the representation of objects to a large grain size represented in embedded/embodied theories of cognition.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  • Abrahamson, D., & Sanchez-Garcia, R. (2016). Learning is moving in new ways: the ecological dynamics of mathematics education. Journal of the Learning Sciences, 25(2), 203–239.

    Article  Google Scholar 

  • Ainsworth, S. (2006). DeFT: a conceptual framework for considering learning with multiple representations. Learning and Instruction, 16(3), 183–198.

    Article  Google Scholar 

  • Akcayir, M., & Akcayir, G. (2017). Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educational Rearch Review, 20, 1–11.

    Google Scholar 

  • Atkinson, R. K., & Renkl, A. (2007). Interactive example-based learning environments: using interactive elements to encourage effective processing of worked examples. Educational Psychology Review, 19(3), 375–386.

    Article  Google Scholar 

  • Barrouillett, P., Portart, S., & Camos, V. (2011). On the law relating processing to storage in working memory. Psychological Review, 118(2), 175–192.

    Article  Google Scholar 

  • Barsalou, L. W. (2016). On staying grounded and avoiding quixotic dead ends. Psychonomic Bulletin & Review, 23(4), 1122–1142.

    Article  Google Scholar 

  • Berthold, K., Eysink, T. H. S., & Renkl, A. (2009). Assisting self-explanation prompts are more effective than open prompts when learning with multiple representations. Instructional Science, 37(4), 345–363.

    Article  Google Scholar 

  • Bric, J. D., Lumbard, D. C., Frelich, M. J., & Gould, J. C. (2016). Current state of virtual reality simulation in robotic surgery training: a review. Surgical Endoscopy, 30(2169), 2169–2178.

    Article  Google Scholar 

  • Burgoon, E. M., Henderson, M. D., & Markman, A. B. (2013). There are many ways to see the forest for the trees: a tour guide for abstraction. Perspectives on Psychological Science, 8(5), 501–520.

    Article  Google Scholar 

  • Butcher, K. R., & Davies, S. (2015). Inference generation during online study and multimedia learning. In E. J. O’Brien, A. E. Cook, & R. F. Lorch (Eds.), Inferences during reading (pp. 321–347). New York: Cambridge University Press.

    Chapter  Google Scholar 

  • Cangelosi, A., & Invitto, S. (2017). New technologies for human robot interaction and neuroprosthetics. IEEE Computer Electronics Magazine(July 1).

  • Carbonneau, K. J., Marley, S. C., & Selig, J. P. (2013). A meta-analysis of the efficacy of teaching mathematics with concrete manipulatives. Journal of Educational Psychology, 105(2), 380–400.

    Article  Google Scholar 

  • Chaudhary, U., Birbaumer, N., & Ramos-Murguialday, A. (2016). Brain-computer interfaces for communication and rehabilitation. Nature Reviews: Neurology, 12(9), 513–525.

    Google Scholar 

  • Connors, E. C., Chrastil, E. R., Sanchez, J., & Merabet, L. B. (2014). Virtual environments for the transfer of navigation skills in the blind: a comparison of directed instruction vs. video game based learning approaches. Frontiers in Human Neuroscience, 8(223).

  • Cooper, L. A., & Shepard, R. N. (1973). Chronometric studies of the rotaton of mental images. In W. A. Chase (Ed.), Visual information processing. New York: Academic Press.

    Google Scholar 

  • Corbett, A. T., McLaughlin, M., Scarpinatto, K. C., & Hadley, W. (2000). Analyzing and generating mathematical models: an Algebra II Cognitive Tutor design study. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Intelligent tutoring systems: 5th international conference (pp. 314–323). New York: Springer.

    Chapter  Google Scholar 

  • Cumming, J., & Williams, S. E. (2014). The role of imagery in performance. In S. M. Murphy (Ed.), The Oxford handbook of sport and performance psychology (pp. 213–232). New York: Oxford University Press.

    Google Scholar 

  • Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69.

    Article  Google Scholar 

  • Dogov, I., Graves, W. J., Nearents, M. R., Schwark, J. D., & Volkman, C. B. (2014). Effects of cooperative gaming and avatar customization on subsequent spontaneous helping behavior. Computers in Human Behavior, 33, 49–55.

    Article  Google Scholar 

  • Dumontier, M., Baker, C. J., Baran, J., Callahan, A., Chepelev, L. L., Cruz-Toledo, J.,... Hoehndorf, R. (2014). The Semanticscience Integrated Ontology (SIO) for biomedical research and knowledge discovery. Journal of Biomedical Semantics, 5(4), 1–11.

  • Engelkamp, J. (1998). Memory for actions. Hove: Psychology Press.

    Google Scholar 

  • Finke, R. A. (1990). Creative imagery: discoveries and inventions in visualization. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Frank, C., Land, W. M., Poppp, C., & Schack, T. (2014). Mental representation and mental practice: experimental investigation on the functional links between motor memory and motor imagery. PLoS One, 9(4), 1–12.

    Article  Google Scholar 

  • Fyfe, E. R., McNeil, N. M., Son, J. Y., & Goldstone, R. L. (2014). Concreteness fading in mathematics and science instruction: a systematic review. Educational Psychology Review, 26(1), 9–25.

    Article  Google Scholar 

  • Glenberg, A. M., Gutierrez, T., Levin, J. R., Japuntich, S., & Kaschak, M. P. (2004). Activity and imagined activity can enhance young children’s reading comprehension. Journal of Educational Psychology, 96, 424–436.

    Article  Google Scholar 

  • Goldin-Meadow, S. (2003). Hearing gesture: how our hands help us think. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Goldstone, R. L., Marghetis, T., Weitnauer, E., Ottmar, E. R., & Landy, D. (2017). Adapting perception, action, and technology for mathematical reasoning. Current Directions in Psychological Science, 26(5), 434–441.

    Article  Google Scholar 

  • Goldstone, R. L., & Son, J. Y. (2005). The transfer of scientific principles using concrete and idealized simulations. The Journal of the Learning Sciences, 14(1), 69–110.

    Article  Google Scholar 

  • Holmes, P. S., & Collins, D. J. (2001). The PETTLEP approach to motor imagery: a functional equivalence model for sports psychologists. Journal of Applied Sport Psychology, 13(1), 60–83.

    Article  Google Scholar 

  • Hostetter, A. B., & Alibali, M. S. (2008). Visible embodiment: gestures as simulated action. Psychonomic Bulletin & Review, 15(3), 495–514.

    Article  Google Scholar 

  • Howard, H. (2016). Architecture’s odd couple: Frank Lloyd Wright and Philip Johnson. New York: Bloomsbury Press.

    Google Scholar 

  • Hutto, D. D., Kirchhoff, M. D., & Abrahamson, A. (2015). The enactive roots of STEM: rethinking educational design in mathematics. Educational Psychology Review, 27(3), 371–389.

    Article  Google Scholar 

  • Johnson-Glenberg, M. C., Birchfield, D. A., Tolentino, L., & Koziupa, T. (2014). Collaborative embodied learning in mixed reality motion-capture environments: Two science studies. Journal of Educational Psychology, 106(1), 86–104.

    Article  Google Scholar 

  • Kalyuga, S., & Singh, A.-M. (2016). Rethinking the boundaries of cognitive load theory in complex learning. Educational Psychology Review, 28(4), 831–852.

    Article  Google Scholar 

  • Kintsch, W. (1998). Comprehension: a paradigm for cognition. Cambridge: Cambridge University Press.

    Google Scholar 

  • Kirsh, D. (2009). Problem solving and situated cognition. In P. Robbins & M. Aydede (Eds.), The Cambridge handbook of situated cognition (pp. 264–306). Cambridge: Cambridge University Press.

    Google Scholar 

  • Klahr, D., Triona, L. M., & Williams, C. (2007). Hands on what? The relative effectiveness of physical vs. virtual materials in an engineering design project by middle school children. Journal of Research in Science Teaching, 44(1), 183–203.

    Article  Google Scholar 

  • Klopfer, E., & Squire, K. (2008). Environmental detectives: the development of an augmented reality platform for environmental simulations. Educational Technology Research and Development, 56(2), 203–228.

  • Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-construction framework: bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798.

    Article  Google Scholar 

  • Kosslyn, S. M. (1981). The medium and the message in mental imagery: a theory. Psychological Review, 88(1), 46–66.

    Article  Google Scholar 

  • Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago: University of Chicago Press.

    Google Scholar 

  • Laski, E. V., Jor'dan, J. R., Daoust, C., & Murrray, A. K. (2015). What makes mathematics manipulatives effective? Lessons from cognitive science and Montessori education. SAGE Open, April–June, 1–8.

  • Leinenger, M. (2014). Phonological coding during reading. Psychological Bulletin, 140(6), 1534–1555.

    Article  Google Scholar 

  • Lillard, A. S. (2005). Montessori: the science behind the genius. Oxford: Oxford University Press.

    Google Scholar 

  • Ma, T., Li, H., Deng, L., Yang, H., Lv, X., Li, P., … Xu, P. (2017). The hybrid BCI system for movement control by combining motor imagery and moving onset visual evoked potential. Journal of Neural Engineering, 14, 1–12.

  • Marley, S. C., Szabo, S., Levin, J. R., & Glenberg, A. M. (2011). Investigation of an activity-based text-processing strategy in mixed-age child dyads. The Journal of Experimental Education, 79(3), 340–360.

    Article  Google Scholar 

  • Martin, T., & Schwartz, D. L. (2005). Physically distributed learning: adapting and reinterpreting physical environments in the development of fraction concepts. Cognitive Science, 29(4), 587–625.

    Article  Google Scholar 

  • Mayer, R. E. (2009). Multimedia learning (2nd ed.). New York: Cambridge University Press.

  • McCrudden, M. T., & Rapp, D. N. (2017). How visual displays affect cognitive processing. Educational Psychology Review, 29(3), 623–639.

    Article  Google Scholar 

  • Merabet, L. B., Connors, E. C., Halko, M. A., & Sanchez, J. (2012). Teaching the blind to find their way by playing video games. PLoS One, 7(9), e44958.

    Article  Google Scholar 

  • Moyer-Packenham, P. S. (Ed.). (2016). International perspectives on teaching and learning mathematics with virtual manipulatives. Switzerland: Springer International Publishing.

    Google Scholar 

  • Moyer-Packenham, P. S., & Bolyard, J. J. (2016). Revisiting the definition of virtual manipulatives. In P. S. Moyer-Packenham (Ed.), International perspectives on teaching and learning mathematics with virtual manipulatives (pp. 3–23). Switzerland: Springer International Publishing.

    Google Scholar 

  • Novack, M. A., Congdon, E. L., Hermani-Lopez, N., & Goldin-Meadow, S. (2014). From action to abstraction: using the hands to learn math. Psychological Science, 25(4), 903–910.

    Article  Google Scholar 

  • Novack, M. A., & Goldin-Meadow, S. (2017). Gesture as representational action: a paper about function. Psychonomic Bulletin & Review, 24(3), 652–665.

    Article  Google Scholar 

  • Pouw, W. T. J. L., van Gog, T., & Paas, F. (2014). An embedded and embodied cognitive review of instructional manipulatives. Educational Psychology Review, 26(1), 51–72.

    Article  Google Scholar 

  • Rau, M. A. (2017). Conditions for the effectiveness of multiple visual representations in enhancing STEM learning. Educational Psychology Review, 29(4), 717–761.

    Article  Google Scholar 

  • Reed, S. K. (1985). Effect of computer graphics on improving estimates to algebra word problems. Journal of Educational Psychology, 77(3), 285–298.

    Article  Google Scholar 

  • Reed, S. K. (2005). From research to practice and back: the Animation Tutor project. Educational Psychology Review, 17(1), 55–82.

    Article  Google Scholar 

  • Reed, S. K. (2006). Cognitive architectures for multimedia learning. Educational Psychologist, 41(2), 87–98.

    Article  Google Scholar 

  • Reed, S. K. (2008). Manipulating multimedia materials. In R. Zheng (Ed.), Cognitive effects of multimedia learning (pp. 51–66). New York: IGI Global.

    Google Scholar 

  • Reed, S. K. (2010). Thinking visually. New York: Taylor & Francis.

    Google Scholar 

  • Reed, S. K. (2016). A taxonomic analysis of abstraction. Perspectives on Psychological Science, 11(6), 817–837.

    Article  Google Scholar 

  • Reed, S. K. (2018). A taxonomic analysis of encoding. unpublished manuscript.

  • Reed, S. K. (in press). Modeling visuospatial reasoning. Spatial Cognition & Computation.

  • Reed, S. K., Corbett, A. T., Hoffman, B., Wagner, A., & MacClaren, B. (2013). Effect of worked examples and cognitive tutor training on constructing equations. Instructional Science, 41(1), 1–24.

    Article  Google Scholar 

  • Reed, S. K., & Ettinger, M. (1987). Usefulness of tables for solving word problems. Cognition and Instruction, 4(1), 43–59.

    Article  Google Scholar 

  • Reed, S. K., & Hoffman, B. (2010). Animation Tutor DVD Thinking Visually. New York: Taylor & Francis.

    Google Scholar 

  • Richmond, L. L., & Zachs, J. M. (2017). Constructing experience: event models from perception to action. Trends in Cognitive Sciences, 21(12), 962–980.

  • Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive Tutor: applied research in mathematics education. Psychonomic Bulletin & Review, 14(2), 249–255.

    Article  Google Scholar 

  • Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701–703.

    Article  Google Scholar 

  • Shrout, P. E., & Rodgers, J. L. (2018). Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annual Review of Psychology, 69(1), 487–510.

    Article  Google Scholar 

  • Skulmowski, A., & Rey, G. D. (2018). Embodied learning: introducing a taxonomy based on bodily engagement and task integration. Cognitive Research: Principles and Implications, 3(6), 1–10.

    Google Scholar 

  • Spataro, R., Chella, A., Allison, B., Giadina, M., Sorbello, R., Tramonte, S., et al. (2017). Reaching and grasping a glass of water by locked-in ALS patients through a BCI-controlled humanoid robot. Frontiers in Human Neuroscience, 11, 1–10.

    Article  Google Scholar 

  • Sridhar, A. N., Briggs, T. P., Kelly, J. D., & Nathan, S. (2017). Training in robotic surgery—an overview. Current Urology Reports, 18(58), 58. https://doi.org/10.1007/s11934-017-0710-y.

    Article  Google Scholar 

  • Stieff, M., & Uttal, D. (2015). How much can spatial ability improve STEM achievement? Educational Psychology Review, 27(4), 607–615.

    Article  Google Scholar 

  • Triona, L. M., & Klahr, D. (2003). Point and click or grab and heft: comparing the influence of physical and virtual instructional materials on elementary school students’ ability to design experiments. Cognition and Instruction, 21(2), 149–173.

    Article  Google Scholar 

  • Uttal, D. H., Newcombe, N. S., Meadow, N. G., Tipton, E., Hand, L. L., Alden, A. R., & Warren, C. (2013). The malleability of spatial skills: a meta-analysis of training studies. Psychological Bulletin, 139(2), 352–402.

    Article  Google Scholar 

  • van Elk, M., van Schie, H., & Bekkering, H. (2014). Action semantics: a unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge. Physics of Life Reviews, 11(2), 220–250.

    Article  Google Scholar 

  • van Gog, T., Paas, F., & Van Merrienboer, J. J. G. (2004). Process-oriented worked examples: Improving transfer performance through enhanced understanding. Instructional Science, 32, 83–98.

    Article  Google Scholar 

  • VandenBoss, G. R. (Ed.). (2006). APA dictionary of psychology. Washington, DC: American Psychological Association.

    Google Scholar 

  • Verschaffel, L., Greer, B., & de Corte, E. (Eds.). (2000). Making sense of word problems. Heereweg: Swets & Zeitlinger.

    Google Scholar 

  • Weiner, E. (2016). The geography of genius: a search for the world’s most creative places, from ancient Athens to Silicon Valley. New York: Simon & Schuster.

    Google Scholar 

  • Wu, H.-K., Lee, S. W.-Y., Chang, H.-Y., & Liang, J.-C. (2013). Current status, opportunities and challenges of aumented reality in education. Computers & Education, 62, 41–49.

    Article  Google Scholar 

  • Zwaan, R. A. (2016). Situation models, mental simulations, and abstract concepts in discourse comprehension. Psychonomic Bulletin & Review, 23(4), 1028–1034.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Psychology and CRMSE, San Diego State University, San Diego, CA, USA

    Stephen K. Reed

  2. Department of Psychology, University of California, San Diego, La Jolla, CA, 92093, USA

    Stephen K. Reed

Authors
  1. Stephen K. Reed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen K. Reed.

Ethics declarations

Conflicts of Interest

The work on this manuscript occurred while the author was a visiting scholar at the University of California, San Diego. The author has no conflict of interest regarding the views expressed in this article.

Appendix

Appendix

Theoretical claims of Engelkamp’s (1998) multimodal theory

  1. 1.

    Recall of observed actions should differ from that of performed actions because different systems are involved in encoding (p. 45). Observations encode visual information about movement but performance encodes motor information (p. 37).

  2. 2.

    Although both sensory and motor processes exert a positive influence on retention, each of these influences should be independent of each other (p. 38).

  3. 3.

    Planning an action should lead to poorer recall than performing an action because the performance includes planning (p. 46). Only part of the motor information should be available when the action has been planned but not yet performed (p. 37).

  4. 4.

    Responses can be controlled either from the conceptual system or directly from the particular input system without going through the conceptual system (p. 35).

  5. 5.

    Encoding of relational information occurs only in the conceptual system through a process of spreading activation (p. 40).

  6. 6.

    Performing an action makes it difficult to form new associations in the conceptual system because performing an action forces concentration on information relevant to the action (p. 41).

From Reed (2008). Page numbers are from Engelkamp’s (1998) book Memory for Actions.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Reed, S.K. Combining Physical, Virtual, and Mental Actions and Objects. Educ Psychol Rev 30, 1091–1113 (2018). https://doi.org/10.1007/s10648-018-9441-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10648-018-9441-y

Keywords

  • Embodied cognition
  • Situated cognition
  • Manipulatives
  • Physical actions and objects
  • Virtual actions and objects
  • Mental actions and objects