Skip to main content

Multimodal Learning

Part of the SpringerBriefs in Statistics book series (BRIEFSSTATIST)

Abstract

The widespread use of technology in the digital age continually shapes how individuals consume knowledge and learn. In the digital age, ideas are shared and represented in multiple formats and through the integration of multiple modes. Technological advances, coupled with considerations of the changing needs of today’s learners, call for exploring new directions for multimodal teaching and learning. Yet, society’s increasing reliance on, and use of, technologies for communication and learning has introduced expanded forms of meaning-making. Information and communication technologies (ICTs) and the online networks that are facilitated by their use encourage educators to transform the way education is delivered. Learning environments are in need of becoming transformed so students are able to use immersive technologies to expand their learning opportunities. This chapter explores emerging trends and pedagogies in multimodal learning that seek to take advantage of the digital tools, texts, and learning approaches that are continually shaping the ways learning occurs inside and outside of higher education.

This chapter is outlined to highlight what is found in the literature on multimodal instruction, what findings were realized at eXploring the Future of Innovative Learning Environments (X-FILEs) workshops, and lastly how multimodal instruction can be used to transform the classroom of the future. Throughout this chapter, readers will get to know a student of the future, Juan Delgado. He attends a 4-year university in Dallas, Texas, and is majoring in Mechanical Engineering taking his Introduction to the Fundamentals of Science course. Each aspect of the learning process as it relates to multimodal instruction in 2023 is outlined through the experiences of Juan to situate the impact to learners.

1 Emerging Trends and Pedagogies

The ubiquitous use of digital technologies is continually reshaping the ways individuals access information, share ideas, and communicate with one another. Doing so requires the nimble use of skills, strategies, and mindsets to navigate, communicate, and collaborate online and across multiple contexts (Leu et al. 2017). These changes have profoundly affected instructional choices in education. In today’s education landscape , three key motivations challenge traditional notions of teaching and learning and set forth a strong case for multimodal learning as a critical pedagogy:

  1. 1.

    There exists a proliferation of information in several modes: gestures, visuals, haptics, auditory productions, text-based information, and multimedia. Representing information through different modes, and/or using a combination of modes, can create multiple access points for learning (Bezemer and Kress 2016; Matusiak 2013; Nouri 2018; Sankey et al. 2010).

  2. 2.

    There is an increased opportunity for individuals, including workers and learners, to represent and communicate information using different modes (Blum and Barger 2018; Matusiak 2013).

  3. 3.

    Students are increasingly diverse (National Center for Education Statistics n.d.).

The academy has long relied upon instructional approaches that favor text-based learning. These approaches are sometimes referred to as factory models of teaching because they include top-down management and favor outcomes designed to meet societal needs and rely on age-based classrooms (Bezemer and Kress 2016; Matusiak 2013; Phuong et al. 2017; Sankey et al. 2010). Researchers posit that presenting information in this one-size-fits-all way does not accommodate the widest possible range of learners’ needs (Gee 1996; Phuong et al. 2017). As technologies continue to shape the ways individuals engage in learning, exchange ideas in networks, and communicate in multiple forms, educators must expand their ideas of how, where, under what conditions, and which tools people use to learn in a digital age. The changing nature of student demographics and shifts in the technological landscape call into question traditional ways of teaching and learning .

Technological advances, coupled with considerations of the changing needs of today’s learners, call for exploring new directions for multimodal teaching and learning. Multimodal learning (MML) can be defined as “learning environments [that] allow instructional elements to be presented in more than one sensory mode (visual, aural, written)” (Sankey et al. 2010: 853). Multimodality looks at the many different modes that people use to communicate with each other and to express themselves. Multimodal learning is relevant as increases in technological tools and associated access to multimedia composing software have led to the ease of use of many modes in presenting, representing, and responding to information.

MML calls for sensemaking where learners take in information, process, and make personal sense of ideas to form deep learning patterns (Bezemer and Kress 2016; Moreno and Mayer 2007). MML expands the range of choices available in a learning environment so that learners can co-construct learning through their preferred mode while also being challenged by integrating the use of other modes (Nouri 2018; Phuong et al. 2017; Sankey et al. 2010). MML requires a high level of agency (self-discipline) by learners, who must have the metacognition necessary to understand how they learn and also when to challenge themselves to learn in ways that lie outside their preferred modes. Such sophisticated awareness requires a student to explore their metacognition and requires instructors to explore how assessments are administered and leveraged (Bezemer and Kress 2016; Moreno and Mayer 2007; Phuong et al. 2017; Sankey et al. 2010). MML promotes a greater emphasis on encouraging student voices during the learning process and calls for educators to listen to students’ preferences. As such, multiple modes should be used to shape the content, response options, as well as the delivery of instruction (Nouri 2018; Phuong et al. 2017).

As a powerful means to customize and adjust learning strategies to reach diverse learners, MML leverages our technology-rich world, digital tools, and networks and can be used to address Universal Design for Learning (UDL) as well. UDL offers three basic principles that encourage the design of supportive learning environments. These principles call for instructional designs that build in multiple means and modes of (1) representation (offering flexible ways to present what we teach and learn), (2) action and expression (offering flexible options for how we learn and express what we know), and (3) engagement (offering flexible options for generating and sustaining motivation to learn) (Rose and Meyer 2002). Encouraging learners to compose ideas using different modes encourages connections to concepts and illustrates ways to employ the principles of UDL. Moreover, deep learning happens when students are given the opportunity to participate in a range of cognitive and social learning activities that are responsive to their preferences and needs or modes (Bezemer and Kress 2016; Moreno and Mayer 2007). Most importantly, MML serves as well-researched pedagogy that sets a platform for educators needing to shift their emphasis from the delivery of instruction to the engagement of their learner (Bezemer and Kress 2016; Tonsing-Meyer 2013). In this way, MML can be an equalizer for diverse students because the use of different modes encourages diverse ways of communicating. MML can be inclusive of multilingual students, English language learners, and those with learning differences and moreover can set the stage for an engaging, rich, and creative learning experience.

1.1 Aspects of Teaching and Learning Covered in the MML Literature

An MML literature survey was conducted in conjunction with eXploring the Future of Innovative Learning Environments (X-FILEs) project. Several areas of teaching and learning were examined to create a framework to conceptualize new forms of learning in the next five years, in other words in 2026. The areas of teaching and learning spanned (a) content presentation, (b) interactions and discussions, (c) learner activities, (d) assessment and evaluation, and (e) co-curricular learning and activities. The following is an overview of literature pertaining to each of these areas.

1.1.1 Content Presentation

Course content can be presented and delivered in multiple modes, such as visuals, auditory productions, text, and multimedia (Bezemer and Kress 2016; Matusiak 2013; Nouri 2018; Sankey et al. 2010). Through MML, content can be curated, either in advance or dynamically, to accommodate multiple access points. Multiple modes are offered so that students can gravitate toward the modes that best align with their individual styles and needs and to encourage the complementary nature of multimodal content (Nouri 2018; Phuong et al. 2017; Sankey et al. 2010). This section assumes that content is either contained in a learning management system for purely online coursework or blended for hybrid and face-to-face instruction and that multimodal content and delivery can augment traditional lectures.

1.1.2 Interactions and Communications

The use of MML in higher education creates a foundation for flexible, open, and multidirectional interaction and discussions with students, their peers, faculty, staff, and the business and community at large. As students embark on sensemaking and co-construction of knowledge in an MML-based classroom (Bezemer and Kress 2016; Moreno and Mayer 2007), they can use a variety of modes to engage with other stakeholders to shape their understanding. Using different modes, students can spontaneously engage in parallel and complementary dialog to deepen learning, answer questions, and engage in sensemaking of the learning objectives in and around a classroom.

1.1.3 Learner Activities

Jewitt (2008) suggests that how knowledge is represented, as well as the mode and media selected, is a crucial aspect of knowledge construction that makes the form of representation integral to sensemaking and learning, more generally. In other words, the way something is represented shapes what is to be learned, the curriculum content, and how it is to be learned. In our digital world, our efforts to design instruction must be responsive to the ways we intend learners to use the new knowledge they acquire in the workplace and beyond. MML calls for educators to shift their role from learning designer to learning facilitator (Bezemer and Kress 2016; Tonsing-Meyer 2013). Because MML sets a foundation for a co-construction of knowledge between faculty and students (Bezemer and Kress 2016), learner activities are available just-in-time, based on the student’s preferred mode and personalized, drawing in the student’s preferences for learning. In addition, the use and integration of different modal representations can reinforce ideas and make learning more memorable.

1.1.4 Assessment

Course assessment strategies must also be multimodal in order to ensure objectivity. Whether formative or summative, all course assessments are designed in a modally agnostic way so that the final product can be produced in the student’s preferred mode and preferably in a way that demonstrates the intersectionality and complementary nature of different modes (Jewitt 2008; Nouri 2018; Phuong et al. 2017; Sankey et al. 2010).

1.1.5 Co-curricular

Co-curricular learning refers to activities, programs, and learning experiences that complement, in some way, what students are learning in formal classes, but also serve to expand the relevancy and contextualization of that learning in the real world. Since MML is pedagogical, it does not explicitly reference co-curricular activities or learning. Even so, opening up classroom interactions and discussions into a multimodal environment provides a foundation for co-curricular learning from stakeholders outside of the classroom, such as college and university staff, business and community leaders, friends, and family. Learning from those outside of the classroom should not be overlooked or undervalued.

1.2 Research Questions Answered

The literature on multimodal instruction also answered research questions around design for multimodal instruction and how to use multiple modes to encourage deep learning. Moreover, the literature helped to contextualize many of the early research questions and provided a democratic, pedagogical framework within which to conceptualize content presentation, interactions and discussions, learner activities, assessments and evaluations, and co-curricular instruction in 2026.

1.3 Future Research

As noted in the 3-day X-FILEs workshop, there remain areas that call for future research, specifically in the areas of how MML can impact a student’s identity development and self-initiative. Also discussed in the workshop, there remain some practical areas that would benefit from further formalization specific to how institutions of higher education support the technology needed to implement multimodal instruction . More inquiry into the efficacy of some modes over others in terms of deep and sustained learning may also be needed, perhaps as a way of creating synergy across modes as well. Workshop participants also noted the importance of developing a framework for the ethical use of MML that would suggest appropriate use and rules of engagement that support students, faculty, and staff in setting healthy boundaries for immersive learning in an information- and technology-rich society. Further research is also needed to better understand the affordances that multimodal instruction can create for students, especially as it pertains to students with disabilities. Lastly, as co-curricular learning takes hold in the use of multimodal instruction , workshop dialog indicated external validation, perhaps by higher education professionals or other community and business partners of learning credentials, will need to be investigated.

2 Use of Multimodal Learning in 2026

This chapter brings MML into focus through the work of the eXploring the Future of Innovative Learning Environments (X-FILEs) project spanning a series of focus groups, 3-day in-person workshop, and further investigation through a team of researchers, industry partners, and faculty brainstorming on how this democratic pedagogy can shape (a) course content presentation, (b) interactions and discussions within a course, (c) learner activities, (d) assessment and evaluation, and (e) co-curricular aspects of education. Each section outlines opportunities for using MML as a framework, implementation strategies, and key questions that have been answered or left open in the literature.

2.1 Content Presentation

Content presentation has progressed from an educational format in which the instructor is charged with passing on his/her information to students via lecture, recitation, and associated activities into an experience in which information is presented to learners in many ways. While lecture is still used, it is frequently supplemented with additional educational content in additional formats, both in class and away from the classroom.

2.1.1 Opportunities

Several affordances exist in providing multimodal course content, chief among them being the self-directed nature of students selecting content delivered in the most familiar or comfortable mode, which is when the deepest learning occurs (Bezemer and Kress 2016; Moreno and Mayer 2007). Moreover, traditional classroom lectures can be augmented by multimodal content and delivery in the learning management system or can provide the opportunity for classroom time to be used for group- or project-based learning. Augmenting content in a learning management system or the traditional classroom lecture can create an active learning space where students are focused on discovery, rather than consumption of information. Discussed in the X-FILEs workshop, this content created by faculty could be commoditized, whereby students could subscribe to certain modes based on their individual preferences for a small fee, rather than relying on traditional per-credit tuition models. In an online course, MML course content could also spark faculty and course designers to create and curate content in new and innovative ways. Additionally, the multimodal nature of virtual environments such as lab simulations, virtual reality, and augmented reality provide experiential opportunities that would otherwise be very difficult to provide in “real life.”

Another opportunity exists to deliver MML course content to those who require assistive technologies. Assistive technology (AT) can be defined as any item, piece of equipment, software program, or product that increases, maintains, or improves the functional capabilities of people with disabilities (Assistive Technology Industry Association 2019). The number of people who could benefit from AT is not insignificant. Currently, 15% of the world’s population has some form of disability, and 20% of those people experience significant disabilities. As the world’s population continues to grow, so too will the total population of those that experience disabilities; furthermore, this population is aging, and there is an increase in chronic health conditions (World Health Organization 2011).

In addition to reaching individuals who would normally have difficulties accessing education, using these technologies can help educators stay compliant with the law. The Rehabilitation Act of 1973 and the Americans with Disabilities Act addressed workplace support for people with disabilities and were later adapted for use in educational situations (U.S. Equal Employment Opportunity Commission n.d.; U.S. General Services Administration 2018). Providing individuals with disabilities with the opportunity to the same education and in the same time frame as students without disabilities will benefit the education provider by staying compliant with laws and regulations.

2.1.2 Challenges

Workshop participants agreed that chief challenges in designing courses and delivering them to incorporate MML are cost and speed of design. It may be cost- and time-prohibitive to create content in all learning modes. For example, one course module on calculating averages in a statistics class could contain the following content items:

  1. 1.

    A word document with the problem completed by the professor (text-based)

  2. 2.

    A video of the problem being completed as students watch and observe on a virtual whiteboard (gestures, video, and auditory)

  3. 3.

    An interactive, adaptive instructional module where a student answers a series of prompts to solve the problem, with dynamic instruction provided if a wrong answer is inputted or the student indicates they have a question

In addition to the time and cost of curating MML content, faculty may find it difficult to switch between modes while in the “live” classroom environment, suggesting more of a need to augment the lecture through MML content in a learning management system as well. Faculty and instructional designers may also find it difficult to find or create content in several modes, especially if they fall outside of their individually preferred modes.

Another challenge of using MML in course content is relying on a student’s self-awareness and self-direction to explore and represent content in their preferred mode or combination of modes. Accessing all course content in one mode, even if preferred, might not be the most effective approach as combining modes may serve as reinforcement and encourage connections. There may also be implications with respect to accessibility where some modes might be richer than others. For example, a student who only accesses materials in a preferred mode may not benefit from deeper learning utilizing content presented in different modes or ones that incorporate many modes such as virtual reality.

2.1.3 Implementation Strategies

Incorporating MML in course content and delivery of this type of instruction in “real time” by a faculty member requires a symbiotic relationship with the student so that the faculty member can match instruction to the learning needs of their students. In thinking through online courses, curating content requires examining instructional design frameworks that guide the thoughtful assembly of course content. The seminal and informative ADDIE model of design, which includes the elements of analysis, design, development, implementation, and evaluation (Molenda 2003), can still be used, perhaps with iterations derived from rapid prototyping.

2.1.4 Research Questions

Literature on MML clearly answers the research questions set forth by the X-FILEs team in this area of how to choose the best mode for a subject or student group. Through its very nature, MML provides content that addresses all modes regardless of subject matter or student group characteristics.

As Juan accesses his online class for his Introduction to the Fundamentals of Science, he is faced with many choices that he will navigate throughout the term. He has a selection of content available to him to support his first week of learning, including chapters from an online textbook, a series of videos and podcasts, and a set of keywords he can use to locate his own or additional content—he is also asked to register for his asynchronous virtual reality classroom where he can access his book, the videos, and podcasts and perform and post his research. His university has obtained a grant to fund the augmented course design, and Juan has a low-cost technology fee that he pays each term to offset the cost of the virtual classroom licensing, hosting, and support to the institution. The virtual classroom is where the majority of his course will actually take place throughout the week, though he will have several opportunities to participate in the learning “live” or through other modes as well. To provide the students with a sense of familiarity, the virtual classroom is designed just as a school is; each room corresponds to a different subject or module. Juan navigates his avatar into each room to view content and participate in class as though he was sitting in a physical classroom.

2.2 Interactions and Communications

Interactions and discussions around academic coursework have evolved from episodic, in-class lectures and peer dialog to full-scale, immersive, and 24/7 opportunities for dialog and deep learning. While in-class lecture and dialog remain central ways to build foundational learning, the co-curricular opportunities for interactions and discussions serve as a platform for more immersive learning.

Using the principles of UDL (Gordon et al. 2014) and frameworks that encourage the wide use of MML, students, peers, faculty, staff, and community/business stakeholders can engage in open and flexible and dialogic interactions and discussions. Dialogic interactions create space for shared leadership in learning in which learners and instructors become co-inquirers; in this way, a more egalitarian classroom environment can emerge, devoid of traditional teacher/student power dynamics. Together, they collaboratively engage in creating and evaluating new interpretations in order to gain insight into the world, themselves, and each other (Reznitskaya 2012). Learners can and should be viewed as active drivers of their own learning (Bezemer and Kress 2016; Sankey et al. 2010). In this way, students can choose multiple modes within which to engage in learning and receive just-in-time or real-time information to help contextualize their learning. Providing an opportunity for multidirectional modes of communication enables students to be resourceful and frees up faculty time as well.

2.2.1 Opportunities

As students shift from consuming knowledge to actively driving and constructing it, they can enhance their understanding by using different modes to represent their understanding. Interacting across modes frees students to think more creatively and integrate knowledge faster. By opening up meaning-making (Postman and Weingartner 1969), the process by which people interpret situations, events, objects, or discourses, to expand communication, MML may also empower students to generate meaning and make sense of themselves, their experiences, and their relationships through their learning. Moreover, MML affords students who may have trouble communicating in a particular mode with the opportunity to express themselves in another. In this more flexible and dynamic environment, students with disabilities can freely operate within their most capable mode. MML also provides new avenues for student collaboration, beyond the confines of the traditional classroom or learning management system.

2.2.2 Challenges

Despite the benefits and possibilities that MML can make possible, there are also challenges introduced. For example, the use of images to convey meaning requires faculty to develop visual literacies required to infer how others may interpret their creations. Not all viewers walk away with the same meaning from visuals, and thus other modes may need to be integrated to convey a common meaning. Identified in the workshop, another challenge may be introduced when initializing interactions exclusively via technology among learners who come from different cultural backgrounds. In the absence of context clues that come from facial expressions, body language, and gestures, miscommunications may occur. Pairing MML strategies with opportunities to connect through two-way videoconference technologies and to openly dialog about different communication norms across cultural groups may support overcoming these challenges. A final challenge with MML identified in the workshop centers around limiting opportunities for face-to-face communication due to technological dependency. Relying solely on technologies for bridging distances between learners may in fact create more distance and feelings of isolation. MML is intended to provide options for learning and instruction and does not intentionally aim to create distance-only options. Discussing ideas face-to-face, in combination with the use of different modes of meaning-making, can provide flexible options for learning engagement . We aim to support educators in their implementation of MML in support of designing learning interactions of all kinds, including face-to-face.

Another challenge in the use of MML in discussions and interactions is the level of ability in technology use among faculty, staff, and students. The challenge stems from the concept that today’s students are digital natives, while most of today’s teachers are digital immigrants (Bezemer and Kress 2016; Blum and Barger 2018; Kirschner and De Bruyckere 2017; Lambert and Cuper 2008; Moreno and Mayer 2007; Phuong et al. 2017; Picciano 2009; Prensky 2001; Sankey et al. 2010; Wang et al. 2014). A less rigid definition of the digital natives/immigrant is now used in which the technology exposure is more a factor of a variety of reasons, such as demographics, opportunity, social influence, cognitive knowledge, and socioeconomic status (Kesharwani 2020). A key concept to take from these studies on digital natives/immigrants is that a primary focus should be placed on reaching across the technological divide between varying technology levels when designing course content. Relatedly, it is important to think through access to technology among different members of the population and how this relates to the value of MML to discussions and interactions.

Lastly, as technology evolves, communication channels must be evaluated for privacy and security. Workshop attendees stressed that it is critically important for faculty and students to know what modes and channels may be open for public viewing. In fact, faculty professional development on tool privacy and accessibility will be important and ever-changing year to year or perhaps term to term as the speed of technological advances and tool creation continues to increase. At the same time, students need orientation to tool privacy and accessibility and experience with the types of tools the university provides, as opposed to ones they are not. There may be tools that do not meet privacy or accessibility standards that the university would not support. The choice of tools and the benefits and drawbacks, should be communicated clearly to students.

2.2.3 Implementation Strategies

To motivate students to interact beyond the traditional faculty-student relationship, faculty can provide for multimodal communication either in advance, while a course is in progress, or following learning activities to promote reflection (Dewsbury 2019). For example, a classroom meeting or online discussion question could have a companion virtual world classroom where students interact in real time through avatars or a backchannel for a face-to-face meeting where students can contextualize the lecture with peers or other educational stakeholders. Course participation could also be linked to time-limited assignments in the workplace or community in person, through videoconference, or virtual networking to encourage idea-sharing and real-world application of learned content.

2.2.4 Research Questions

While research in MML is not entirely new, research that addresses challenges identified in the workshop concerning second language learners are an important direction to pursue. Offering multimodal texts is often beneficial for second language (L2) learners as multiple modes can scaffold understanding (Plass et al. 1998; Royce 2002). This is because, as Farias and Abraham (2017) state:

multimodal texts more effectively support second language reading by providing input that caters to different learning styles and that they are familiar, authentic, and contextualized to the learners’ lives. Moreover, these texts facilitate learners’ meaningful interaction not only intratextually, by exploring the text/image semiosis, but also intertextually, by allowing readers to become literate in the different genres that are constructed multimodally.

Thus, while L2 learning is supported by the use of multimodal texts, it can also develop learners’ broader understanding of concepts. Examining identity and literacy development surrounding the creation of multimodal texts both within and beyond course objectives is an fruitful area in need of additional research. These directions can further support MML for L2 learners across fields. While MML seems to indicate there are benefits associated with the use of multiple modes of communication for students, less is understood about the benefits of students’ choices; research into this area is still needed.

In the virtual classroom, Juan can engage through phone, text, email, instant messaging, or other means with his professor, a group of scientists in industry and other colleges, as well as all his peers and advisors. Professor Aranda also actively engages in the virtual classroom and spends time cultivating personalized relationships with each student—in addition to maintaining the relationship with the community of scientists and supporting the classroom. During the Tuesday evening session, she carefully plans a short didactic lecture that is then supplemented with a series of small group virtual breakout rooms facilitated by other scientists where students and scientists work through the theory and discuss ways to demonstrate mastery of the learning objectives for the week.

2.3 Learner Activities

Learner activities can take many shapes and forms and may connect resources available through coursework to online and community resources . The choices faculty make include multimodal options that build toward applying what is learned within and beyond the classroom. However, choices often depend on individual faculty and the pedagogical disposition of a department, college, or university. In this way, learner activities can be varied and may involve multiple formats and response options that call for different ways to illustrate learning.

MML calls for educators to shift their role from lecturer to learning facilitator and designer (Bezemer and Kress 2016; Tonsing-Meyer 2013). MML encourages fluidity between accessing information using multiple modes and representing knowledge through the integration of multiple modes. This approach encourages personalization as well as creativity and calls for a co-construction of knowledge between faculty and students (Bezemer and Kress 2016). In an MML environment, learner activities can be more collaborative, designed to require real-world problem-solving, and created to apply knowledge based on a compelling purpose for learning.

2.3.1 Opportunities

In shifting the focus from lecturer to learning designer or facilitator, faculty are free to work with individual students to understand their current state of learning and to work collaboratively on learner activities that will enhance their mastery. Students are thus empowered to find the best ways to integrate the subject matter alongside faculty. Faculty can also learn from student-designed learner activities when they use modes and tools unfamiliar to the faculty.

2.3.2 Challenges

When students drive their own learning, they are able to benefit from the multiple modes that are available to them. Working between and across modes may serve to support deeper learning and improve retention of key concepts. MML requires a high degree of agency, and provides opportunities for learners to select a focus or direction for themselves. This requires that learners develop the metacognition necessary to understand how they learn and also when to challenge themselves (Bezemer and Kress 2016; Moreno and Mayer 2007; Phuong et al. 2017; Sankey et al. 2010). The time needed for faculty to engage with students on learner activity design may be prohibitive in larger courses. It also may be difficult to objectively assess learning when activities are varied across a student group.

2.3.3 Implementation Strategies

Learner activities in an MML environment are student-driven, based on preferred modes. Still, faculty could develop a range of learner activities, mapped to course learning objectives, that can be completed in several modes to enable students to select an appropriate activity. These activities may be experiential and hands-on or operate in a virtual environment. For example, an assignment either could be open-ended in terms of how the learning is demonstrated such as a document, a presentation, or a video or could have a range of possible formats the students could choose from.

2.3.4 Research Questions

Research questions in this area rest on the opportunity that exists to operationalize MML-based learner activities to promote student engagement. The MML literature clearly supports the shift from subject matter delivery to that of engagement, as well as the positive benefits of this. Workshop participants aptly pointed out that it will also be important to compare MML-based learner activities to other types of pedagogical frameworks for efficacy.

All of Juan’s learning activities are “hands-on” in the virtual classroom and include a series of science experiments that he can perform in the virtual space, at home (where he can record and post in the virtual classroom), or at a series of physical labs his university has partnered with. Throughout the course, Professor Aranda is working to ensure that the course content, learning activities, and support from other scientists and peers all work together to further his individual learning. Professor Aranda is aware that Juan’s preferred modes of learning are in visual and auditory realms so she takes time to ensure that he is challenged to incorporate writing and presentation skills, as well as to spend time in the virtual space to drive the intersectionality of multimodal learning and to create more strength in other modes of learning and information consumption.

2.4 Assessment

Traditional means of assessment and evaluation have often revolved around tests and written assignments. Over time, educators have embraced new ways of assessing competency through action projects, multimedia presentations, and other types of practical application strategies. MML can be a helpful framework in the assessment strategy in a course or classroom environment that frees up the faculty and students for more creative ways to show mastery. No matter the type of assessment, formative, summative, or performance-based, designing assessments that invite different response options may free students from a specific way of illustrating their knowledge. In this way, through MML options, learners are encouraged to produce demonstrations of their learning in modes that they have an affinity for. Moreover, providing suggestions for multiple ways to illustrate learning may open the doors to creativity and deeper levels of learning (William and Flora Hewlett Foundation 2016).

2.4.1 Opportunities

By designing assessments so that they can be completed in multiple modes, students can draw from their modally-driven learning to produce artifacts in the mode that they are most comfortable and familiar with. By reducing the cognitive load associated with using an unfamiliar or uncomfortable mode, learning and mastery can be more easily conveyed. Students could avail themselves of technological advancements to capture their learning as it evolves, ostensibly faster than higher education curricular design cycles. In addition, by opening up the classroom co-curricularly as discussed in section 2.5, assessments can be conducted by stakeholders outside of the classroom walls, providing helpful feedback from employers, community partners, and others that can help shape the student’s learning in ways that may not be within the scope of what a faculty member knows or has experienced, personally. Ultimately, the opportunity to have multiple stakeholders providing evaluation and feedback affords faculty with more data to gauge learning and to plan to reduce gaps and increase levels of mastery. With more and consistent data streams through multimodal evaluation, the possibility for machine learning to aid in faster and more efficient grading also presents itself.

2.4.2 Challenges

Students might not choose a mode that effectively demonstrates their learning, or they may select a mode that the faculty member is not familiar with. Diverse outputs may challenge the use of rubrics or programmatic assessment plans that rely upon standardized artifacts or other uniform, objective measures given to students. Moreover, it may be very time-consuming for a faculty member to grade assessments produced in many modes because mastery may “look” different based on the mode in which it was produced. Students may also choose to demonstrate their learning in a mode that lacks adequate privacy or accommodations for students with disabilities—presenting concerns around assisting their selection of modal delivery as part of the assessment directions.

2.4.3 Implementation Strategies

In higher education, a culminating project or final assignment often requires learners to synthesize, analyze, and create content using a variety of modalities. Such projects, when closely aligned with learning outcomes, support relevant and authentic assessment that leverages digital communication strategies. An emerging practical model designed specifically for multimedia instructional strategies, CASPA, contains five components: consume, analyze, scaffold, produce, and assess. This model can be used to promote curriculum-based integration of multimodal projects for assessment in higher education courses. The CASPA framework for assignment design asks students to iteratively attempt and create culminating assessments throughout a course (Blum and Barger 2018). The framework allows students to experiment with different modes as they complete a culminating assignment. Faculty feedback and direction throughout the term could be used to enable learners to choose the best modal demonstration of their learning by the time the end product is due. Faculty will design assessment artifacts that are open-ended and driven by course learning objectives. No formal description of the end products or artifacts is provided to the student, though it may be appropriate to provide options to help students choose effective outputs or ones that protect the student’s privacy and that are accessible to all students, regardless of their abilities. Faculty may also consider creating standardized metrics that would allow for cross-modal comparisons of learning products illustrating that there are multiple ways to achieve course aims.

2.4.4 Research Questions

Research questions in this area center on the affordances of some modes over others, and this remains an area for further research as identified by workshop participants; perhaps some modes are more effective than others in the demonstration of learning. Furthermore, faculty modal preference is rarely discussed in the literature. Faculty who are evaluating artifacts outside their modal preference may not be able to objectively assess them due to the cognitive load required to adapt to new or unfamiliar modes.

Juan and his peers are encouraged to locate and post content that aligns to each week’s learning objective through a framework of information validation provided by the university. Students are encouraged to comment on these new content postings as a way of sensemaking and co-construction of knowledge. Experiments are the main source of assessment and evaluation in his course, but Juan also has several opportunities to identify and capture scientific theories in his everyday life through a system of digital storytelling. This can include taking a picture and posting about an experiment, recording audio or video about seeing science in nature, or providing a summary of a scientific experience through video or a text-based summary. Professor Aranda awards points to every learning capture that he is willing to submit in the course; in fact, all of the assessments in the course are documented as merely a list of learning objectives so that each student can demonstrate mastery through modes that they have an affinity for or modes that lend themselves to the objective being assessed.

2.5 Co-curricular Activities

While the idea that learning does not always take place in the traditional classroom setting has been accepted for many years (Dewey 1997), educators and students are now embracing the concept. Co-curricular learning refers to activities, programs, and learning experiences that complement, in some way, what students are learning in formal classes , but also serve to expand the relevancy and contextualization of that learning in the real world. Co-curricular contexts enable the learner to interact with material outside the classroom in such a way that they acquire new knowledge or apply a skill from one area to a new area. Increasingly, institutions of higher education are embracing co-curricular learning and its benefits to their students (Stirling and Kerr 2015; Turrentine et al. 2012).

2.5.1 Opportunities

By broadening learning to extend beyond the academy, students can build professional networks, enhance their learning through multiple perspectives, and more quickly build workforce-ready skills. Through co-curricular access to other educational stakeholders, students can engage and consume new learning in the mode they most prefer. Students could also participate and drive change in their community by working directly with business and community leaders to couple their learning with practical applications.

These co-curricular activities could take many forms, and in one scenario, educational stakeholders such as a venerable scientist or engineer would be purposefully contracted by the institution to participate in the virtual world associated with the course to:

  1. 1.

    Host weekly informational sessions via web conferences, virtual world environments, or even via chat

  2. 2.

    Make himself/herself available for text- or video-based question/answer sessions on the week’s subject matter from the classroom

  3. 3.

    Use virtualized laboratories for student observation

  4. 4.

    Provide feedback and insight into informal or formal assessment of the subject matter

Each scenario further extends the student’s learning, infused with the relevancy and currency of a working professional in their field of study.

2.5.2 Challenges

Many institutions of higher education do not have formal connections with the private sector that underrepresented students’ need to benefit from co-curricular learning. Faculty may also be challenged with assessing which learning should happen outside the classroom and what should be inside the classroom (face to face or online), may struggle with how to integrate or sequence this type of learning with classroom-based learning. Federal credit hour definitions may also limit co-curricular learning or make it difficult to “count” it, so it would be additive, perhaps contributing to cognitive overload for students. Validation of co-curricular learning may be difficult to assess as well.

2.5.3 Implementation Strategies

Faculty can design course learning objectives tied to business and community interaction, where the choice of modalities would be driven by a student’s individual preferences. For example, a course learning objective could read “Students will consult with a scientist to discuss what types of careers a science graduate can pursue upon graduation .” Faculty could design course content and set the stage for interactions and discussions that engage educational stakeholders from outside the walls of the classroom.

2.5.4 Research Questions

Research questions in this area center on the benefits and consequences of students’ learning outside of the classroom. MML literature is seemingly quiet in this area, though it was a theme discussed in the workshop. There does remain a risk to the fidelity, meaning the degree to which application of MML can be replicated, across iterations of co-curricular learning programming. As co-curricular learning increases, capturing the outcomes and related learning that stems from MML will need to be developed and well understood by faculty and students as well.

Each Tuesday night, Juan’s whole class, his professor, and a group of scientists from all over the world gather to sit in the virtual classroom to discuss vital scientific theory that will enable him to be more successful when he is attempting his experiments. These same scientists are working with Juan and Professor Aranda throughout the term to customize the learning and to be sure Juan is engaged in the most up-to-date practice. Juan is developing a strong network of future colleagues as he interacts with these external educational stakeholders.

3 Conclusion

While there are clear opportunities and affordances offered through the use of MML in the classroom, there are also inherent limitations to MML, particularly related to cognitive load. Too many modes and too much information can overwhelm learners (Moreno and Mayer 2007; Sweller et al. 1998). Furthermore, fundamentally shifting traditional teaching would require intentional, responsive, and on-going professional development so that teachers can develop multimodal literacies. One must also consider the cost and complexity (Bezemer and Kress 2016) of curating learning environments so that learners can choose from all of the varied types of modes. Though challenges do exist in the use of MML, the opportunities and affordances that exist through incorporating facets of MML in classroom teaching and learning far outweigh the challenges. Moreover, as student demographics continue to evolve into more and more diversity and technology continues to reshape learning interactions, instruction will inherently take on a multimodal nature.

In the end, Juan finishes his Introduction to the Fundamentals of Science course with a deep understanding of scientific theory and its application in the real world. He also leaves the course with a newly formed group of mentors which includes Professor Aranda and the scientists who supported the course. He has also formed deep bonds with his peers and plans to connect with them in future courses. Lastly, he leaves the course with a new sense of how improving his writing and presentation skills can provide value not only in deepening his learning but in how he presents himself to the world outside of the university.

References

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Bettyjo Bouchey .

Editor information

Editors and Affiliations

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and Permissions

Copyright information

© 2021 The Author(s)

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Bouchey, B., Castek, J., Thygeson, J. (2021). Multimodal Learning. In: Ryoo, J., Winkelmann, K. (eds) Innovative Learning Environments in STEM Higher Education. SpringerBriefs in Statistics. Springer, Cham. https://doi.org/10.1007/978-3-030-58948-6_3

Download citation