This chapter brings MML into focus through the work of the eXploring the Future of Innovative Learning Environments (X-FILEs) project spanning a series of focus groups, 3-day in-person workshop, and further investigation through a team of researchers, industry partners, and faculty brainstorming on how this democratic pedagogy can shape (a) course content presentation, (b) interactions and discussions within a course, (c) learner activities, (d) assessment and evaluation, and (e) co-curricular aspects of education. Each section outlines opportunities for using MML as a framework, implementation strategies, and key questions that have been answered or left open in the literature.
2.1 Content Presentation
Content presentation has progressed from an educational format in which the instructor is charged with passing on his/her information to students via lecture, recitation, and associated activities into an experience in which information is presented to learners in many ways. While lecture is still used, it is frequently supplemented with additional educational content in additional formats, both in class and away from the classroom.
exist in providing multimodal course content, chief among them being the self-directed nature of students selecting content delivered in the most familiar or comfortable mode, which is when the deepest learning occurs (Bezemer and Kress 2016; Moreno and Mayer 2007). Moreover, traditional classroom lectures can be augmented by multimodal content and delivery in the learning management system or can provide the opportunity for classroom time to be used for group- or project-based learning. Augmenting content in a learning management system or the traditional classroom lecture can create an active learning space where students are focused on discovery, rather than consumption of information. Discussed in the X-FILEs workshop, this content created by faculty could be commoditized, whereby students could subscribe to certain modes based on their individual preferences for a small fee, rather than relying on traditional per-credit tuition models. In an online course, MML course content
could also spark faculty and course designers to create and curate content in new and innovative ways. Additionally, the multimodal nature of virtual environments such as lab simulations, virtual reality, and augmented reality provide experiential opportunities that would otherwise be very difficult to provide in “real life.”
Another opportunity exists to deliver MML course content to those who require assistive technologies. Assistive technology (AT)
can be defined as any item, piece of equipment, software program, or product that increases, maintains, or improves the functional capabilities of people with disabilities (Assistive Technology Industry Association 2019). The number of people who could benefit from AT is not insignificant. Currently, 15% of the world’s population has some form of disability, and 20% of those people experience significant disabilities. As the world’s population continues to grow, so too will the total population of those that experience disabilities; furthermore, this population is aging, and there is an increase in chronic health conditions (World Health Organization 2011).
In addition to reaching individuals who would normally have difficulties accessing education, using these technologies can help educators stay compliant with the law. The Rehabilitation Act of 1973 and the Americans with Disabilities Act addressed workplace support for people with disabilities and were later adapted for use in educational situations (U.S. Equal Employment Opportunity Commission n.d.; U.S. General Services Administration 2018). Providing individuals with disabilities with the opportunity to the same education and in the same time frame as students without disabilities
will benefit the education provider by staying compliant with laws and regulations.
participants agreed that chief challenges in designing courses and delivering them to incorporate MML are cost and speed of design. It may be cost- and time-prohibitive to create content in all learning modes. For example, one course module on calculating averages in a statistics class could contain the following content items:
A word document with the problem completed by the professor (text-based)
A video of the problem being completed as students watch and observe on a virtual whiteboard (gestures, video, and auditory)
An interactive, adaptive instructional module where a student answers a series of prompts to solve the problem, with dynamic instruction provided if a wrong answer is inputted or the student indicates they have a question
In addition to the time and cost of curating MML content, faculty may find it difficult to switch between modes while in the “live” classroom environment, suggesting more of a need to augment the lecture through MML content in a learning management system as well. Faculty and instructional designers may also find it difficult to find or create content in several modes, especially if they fall outside of their individually preferred modes.
Another challenge of using MML
in course content is relying on a student’s self-awareness and self-direction to explore and represent content in their preferred mode or combination of modes. Accessing all course content in one mode, even if preferred, might not be the most effective approach as combining modes may serve as reinforcement and encourage connections. There may also be implications with respect to accessibility where some modes might be richer than others. For example, a student who only accesses materials in a preferred mode
may not benefit from deeper learning utilizing content presented in different modes or ones that incorporate many modes such as virtual reality.
2.1.3 Implementation Strategies
Incorporating MML in course content and delivery of this type of instruction in “real time” by a faculty member requires a symbiotic relationship with the student so that the faculty member can match instruction to the learning needs of their students. In thinking through online courses, curating content requires examining instructional design frameworks that guide the thoughtful assembly of course content. The seminal and informative ADDIE model of design, which includes the elements of analysis, design, development, implementation, and evaluation (Molenda 2003), can still be used, perhaps with iterations derived from rapid prototyping.
2.1.4 Research Questions
Literature on MML clearly answers the research questions
set forth by the X-FILEs team in this area of how to choose the best mode for a subject or student group. Through its very nature, MML provides content that addresses all modes regardless of subject matter or student group characteristics.
As Juan accesses his online class for his Introduction to the Fundamentals of Science, he is faced with many choices that he will navigate throughout the term. He has a selection of content available to him to support his first week of learning, including chapters from an online textbook, a series of videos and podcasts, and a set of keywords he can use to locate his own or additional content—he is also asked to register for his asynchronous virtual reality classroom where he can access his book, the videos, and podcasts and perform and post his research. His university has obtained a grant to fund the augmented course design, and Juan has a low-cost technology fee that he pays each term to offset the cost of the virtual classroom licensing, hosting, and support to the institution. The virtual classroom is where the majority of his course will actually take place throughout the week, though he will have several opportunities to participate in the learning “live” or through other modes as well. To provide the students with a sense of familiarity, the virtual classroom is designed just as a school is; each room corresponds to a different subject or module. Juan navigates his avatar into each room to view content and participate in class as though he was sitting in a physical classroom.
2.2 Interactions and Communications
Interactions and discussions around academic coursework have evolved from episodic, in-class lectures and peer dialog to full-scale, immersive, and 24/7 opportunities for dialog and deep learning. While in-class lecture
and dialog remain central ways to build foundational learning, the co-curricular opportunities for interactions and discussions serve as a platform for more immersive learning.
Using the principles
of UDL (Gordon et al. 2014) and frameworks that encourage the wide use of MML, students, peers, faculty, staff, and community/business stakeholders can engage in open and flexible and dialogic interactions
and discussions. Dialogic interactions
create space for shared leadership in learning in which learners and instructors become co-inquirers; in this way, a more egalitarian classroom environment can emerge, devoid of traditional teacher/student power dynamics. Together, they collaboratively engage in creating and evaluating new interpretations in order to gain insight into the world, themselves, and each other (Reznitskaya 2012). Learners
can and should be viewed as active drivers of their own learning (Bezemer and Kress 2016; Sankey et al. 2010). In this way, students can choose multiple modes within which to engage in learning and receive just-in-time or real-time information to help contextualize their learning. Providing an opportunity for multidirectional modes of communication enables students to be resourceful and frees up faculty time as well.
shift from consuming knowledge to actively driving and constructing it, they can enhance their understanding by using different modes to represent their understanding. Interacting across modes frees students to think more creatively and integrate knowledge faster. By opening up meaning-making (Postman and Weingartner 1969), the process by which people interpret situations, events, objects, or discourses, to expand communication, MML may also empower students to generate meaning and make sense of themselves, their experiences, and their relationships through their learning. Moreover, MML affords students who may have trouble communicating in a particular mode with the opportunity to express themselves in another. In this more flexible and dynamic environment, students with disabilities can freely operate within their most capable mode. MML also provides new avenues for student collaboration, beyond the confines of the traditional classroom or learning management system.
Despite the benefits and possibilities that MML can make possible, there are also challenges
introduced. For example, the use of images to convey meaning requires faculty to develop visual literacies required to infer how others may interpret their creations. Not all viewers walk away with the same meaning from visuals, and thus other modes may need to be integrated to convey a common meaning. Identified in the workshop, another challenge may be introduced
when initializing interactions exclusively via technology among learners who come from different cultural backgrounds. In the absence of context clues that come from facial expressions, body language, and gestures, miscommunications may occur. Pairing MML strategies with opportunities to connect through two-way videoconference technologies and to openly dialog about different communication norms across cultural groups may support overcoming these challenges. A final challenge with MML identified in the workshop centers around limiting opportunities for face-to-face communication due to technological dependency. Relying solely on technologies for bridging distances between learners may in fact create more distance and feelings of isolation. MML is intended to provide options for learning and instruction and does not intentionally aim to create distance-only options. Discussing ideas face-to-face, in combination with the use of different modes of meaning-making, can provide flexible options for learning engagement
. We aim to support educators in their implementation of MML in support of designing learning interactions of all kinds, including face-to-face.
Another challenge in the use of MML in discussions and interactions is the level of ability in technology use among faculty, staff, and students. The challenge stems from the concept that today’s students are digital natives, while most of today’s teachers are digital immigrants (Bezemer and Kress 2016; Blum and Barger 2018; Kirschner and De Bruyckere 2017; Lambert and Cuper 2008; Moreno and Mayer 2007; Phuong et al. 2017; Picciano 2009; Prensky 2001; Sankey et al. 2010; Wang et al. 2014). A less rigid definition of the digital natives/immigrant is now used in which the technology exposure is more a factor of a variety of reasons, such as demographics, opportunity, social influence, cognitive knowledge, and socioeconomic status (Kesharwani 2020). A key concept to take from these studies on digital natives/immigrants is that a primary focus should be placed on reaching across the technological divide between varying technology levels when designing course content. Relatedly, it is important to think through access to technology among different members of the population and how this relates to the value of MML to discussions and interactions.
Lastly, as technology evolves, communication channels must be evaluated for privacy and security. Workshop attendees stressed that it is critically important for faculty and students to know what modes and channels may be open for public viewing. In fact, faculty professional development on tool privacy and accessibility will be important and ever-changing year to year or perhaps term to term as the speed of technological advances and tool creation continues to increase. At the same time, students need orientation to tool privacy and accessibility and experience with the types of tools the university provides, as opposed to ones they are not. There may be tools that do not meet privacy or accessibility standards that the university would not support. The choice of tools and the benefits and drawbacks, should be communicated clearly to students.
2.2.3 Implementation Strategies
To motivate students to interact
beyond the traditional faculty-student relationship, faculty can provide for multimodal communication either in advance, while a course is in progress, or following learning activities to promote reflection (Dewsbury 2019). For example, a classroom meeting or online discussion question could have a companion virtual world classroom where students interact in real time through avatars or a backchannel for a face-to-face meeting where students
can contextualize the lecture with peers or other educational stakeholders. Course participation could also be linked to time-limited assignments in the workplace or community in person, through videoconference, or virtual networking to encourage idea-sharing and real-world application of learned content.
2.2.4 Research Questions
While research in MML
is not entirely new, research that addresses challenges identified in the workshop concerning second language learners are an important direction to pursue. Offering multimodal texts is often beneficial for second language (L2) learners as multiple modes can scaffold understanding (Plass et al. 1998; Royce 2002). This is because, as Farias and Abraham (2017) state:
multimodal texts more effectively support second language reading by providing input that caters to different learning styles and that they are familiar, authentic, and contextualized to the learners’ lives. Moreover, these texts facilitate learners’ meaningful interaction not only intratextually, by exploring the text/image semiosis, but also intertextually, by allowing readers to become literate in the different genres that are constructed multimodally.
Thus, while L2 learning is supported by the use of multimodal texts, it can also develop learners’ broader understanding of concepts. Examining identity and literacy development surrounding the creation of multimodal texts both within and beyond course objectives is an fruitful area in need of additional research. These directions can further support MML for L2 learners across fields. While MML seems to indicate there are benefits associated with the use of multiple modes of communication for students, less is understood about
the benefits of students’ choices; research into this area is still needed.
In the virtual classroom, Juan can engage through phone, text, email, instant messaging, or other means with his professor, a group of scientists in industry and other colleges, as well as all his peers and advisors. Professor Aranda also actively engages in the virtual classroom and spends time cultivating personalized relationships with each student—in addition to maintaining the relationship with the community of scientists and supporting the classroom. During the Tuesday evening session, she carefully plans a short didactic lecture that is then supplemented with a series of small group virtual breakout rooms facilitated by other scientists where students and scientists work through the theory and discuss ways to demonstrate mastery of the learning objectives for the week.
2.3 Learner Activities
Learner activities can take many shapes and forms and may connect resources available through coursework to online and community resources
. The choices faculty make include multimodal options that build toward applying what is learned within and beyond the classroom. However, choices often depend on individual faculty and the pedagogical disposition of a department, college, or university. In this way, learner activities can be varied and may involve multiple formats and response options that call for different ways to illustrate learning.
MML calls for educators to shift their role from lecturer to learning facilitator
(Bezemer and Kress 2016; Tonsing-Meyer 2013). MML encourages fluidity between accessing information using multiple modes and representing knowledge through the integration of multiple modes. This approach encourages personalization as well as creativity and calls for a co-construction of knowledge between faculty and students (Bezemer and Kress 2016). In an MML environment, learner activities can be more collaborative, designed to require real-world problem-solving, and created to apply knowledge based on a compelling purpose for learning.
In shifting the focus
from lecturer to learning designer or facilitator, faculty are free to work with individual students to understand their current state of learning and to work collaboratively on learner activities that will enhance their mastery. Students are thus empowered to find the best ways to integrate the subject matter alongside faculty. Faculty can also learn from student-designed learner activities when they use modes and tools unfamiliar to the faculty.
drive their own learning, they are able to benefit from the multiple modes that are available to them. Working between and across modes may serve to support deeper learning and improve retention of key concepts. MML requires a high degree of agency, and provides opportunities for learners to select a focus or direction for themselves. This requires that learners develop the metacognition necessary to understand how they learn and also when to challenge themselves (Bezemer and Kress 2016; Moreno and Mayer 2007; Phuong et al. 2017; Sankey et al. 2010). The time needed for faculty to engage with students on learner activity design may be prohibitive in larger courses. It also may be difficult to objectively assess learning when activities are varied across a student group.
2.3.3 Implementation Strategies
Learner activities in an MML
environment are student-driven, based on preferred modes. Still, faculty could develop a range of learner activities, mapped to course learning objectives, that can be completed in several modes to enable students to select an appropriate activity. These activities may be experiential
and hands-on or operate in a virtual environment. For example, an assignment either could be open-ended in terms of how the learning is demonstrated such as a document, a presentation, or a video or could have a range of possible formats the students could choose from.
2.3.4 Research Questions
in this area rest on the opportunity that exists to operationalize MML-based learner activities to promote student engagement. The MML literature clearly supports the shift from subject matter delivery to that of engagement, as well as the positive benefits of this. Workshop participants aptly pointed out that it will also be important to compare MML-based learner activities to other types of pedagogical frameworks for efficacy.
All of Juan’s learning activities are “hands-on” in the virtual classroom and include a series of science experiments that he can perform in the virtual space, at home (where he can record and post in the virtual classroom), or at a series of physical labs his university has partnered with. Throughout the course, Professor Aranda is working to ensure that the course content, learning activities, and support from other scientists and peers all work together to further his individual learning. Professor Aranda is aware that Juan’s preferred modes of learning are in visual and auditory realms so she takes time to ensure that he is challenged to incorporate writing and presentation skills, as well as to spend time in the virtual space
to drive the intersectionality of multimodal learning and to create more strength in other modes of learning and information consumption.
Traditional means of assessment and evaluation have often revolved around tests and written assignments. Over time, educators have embraced new ways of assessing competency through action projects, multimedia presentations, and other types of practical application strategies. MML can be a helpful framework in the assessment strategy in a course or classroom environment
that frees up the faculty and students for more creative ways to show mastery. No matter the type of assessment, formative, summative, or performance-based, designing assessments that invite different response options may free students from a specific way of illustrating their knowledge. In this way, through MML options, learners are encouraged to produce demonstrations of their learning in modes that they have an affinity for. Moreover, providing suggestions for multiple ways to illustrate learning may open the doors to creativity and deeper levels of learning (William and Flora Hewlett Foundation 2016).
assessments so that they can be completed in multiple modes, students can draw from their modally-driven learning to produce artifacts in the mode that they are most comfortable and familiar with. By reducing the cognitive load associated with using an unfamiliar or uncomfortable mode, learning and mastery can be more easily conveyed. Students could avail themselves of technological advancements to capture their learning as it evolves, ostensibly faster than higher education curricular design cycles. In addition, by opening up the classroom co-curricularly as discussed in section 2.5, assessments can be conducted by stakeholders outside of the classroom walls, providing helpful feedback from employers, community partners, and others that can help shape the student’s learning in ways that may not be within the scope of what a faculty member knows or has experienced, personally. Ultimately, the opportunity to have multiple stakeholders providing evaluation and feedback affords faculty with more data to gauge learning and to plan to reduce gaps and increase levels of mastery. With more and consistent data streams through multimodal
evaluation, the possibility for machine learning to aid in faster and more efficient grading also presents itself.
might not choose a mode that effectively demonstrates their learning, or they may select a mode that the faculty member is not familiar with. Diverse outputs may challenge the use of rubrics or programmatic assessment plans that rely upon standardized artifacts or other uniform, objective measures given to students. Moreover, it may be very time-consuming for a faculty member to grade assessments produced in many modes because mastery may “look” different based on the mode in which it was produced. Students may also choose to demonstrate their learning in a mode that lacks adequate privacy or accommodations for students with disabilities—presenting concerns around assisting their selection of modal delivery as part of the assessment directions.
2.4.3 Implementation Strategies
In higher education, a culminating
project or final assignment often requires learners to synthesize, analyze, and create content using a variety of modalities. Such projects, when closely aligned with learning outcomes, support relevant and authentic assessment that leverages digital communication strategies. An emerging practical model designed specifically for multimedia instructional strategies, CASPA, contains five components: consume, analyze, scaffold, produce, and assess. This model can be used to promote curriculum-based integration of multimodal projects for assessment in higher education courses. The CASPA framework for assignment design asks students to iteratively attempt and create culminating assessments throughout a course (Blum and Barger 2018). The framework allows students to experiment with different modes as they complete a culminating assignment. Faculty feedback and direction throughout the term could be used to enable learners to choose the best modal demonstration of their learning by the time the end product is due. Faculty will design assessment artifacts that are open-ended and driven by course learning objectives. No formal description of the end products or artifacts is provided to the student, though it may be appropriate to provide options to help students choose effective outputs
or ones that protect the student’s privacy and that are accessible to all students, regardless of their abilities. Faculty may also consider creating standardized metrics that would allow for cross-modal comparisons of learning products illustrating that there are multiple ways to achieve course aims.
2.4.4 Research Questions
in this area center on the affordances of some modes over others, and this remains an area for further research as identified by workshop participants; perhaps some modes are more effective than others in the demonstration of learning. Furthermore, faculty modal preference is rarely discussed in the literature. Faculty who are evaluating artifacts outside their modal preference may not be able to objectively assess them due to the cognitive load required to adapt to new or unfamiliar modes.
Juan and his peers are encouraged to locate and post content that aligns to each week’s learning objective through a framework of information validation provided by the university. Students are encouraged to comment on these new content postings as a way of sensemaking and co-construction of knowledge. Experiments are the main source of assessment and evaluation in his course, but Juan also has several opportunities to identify and capture scientific theories in his everyday life through a system of digital storytelling. This can include taking a picture and posting about an experiment, recording audio or video about seeing science in nature, or providing a summary of a scientific experience through video or a text-based summary. Professor Aranda awards points to every learning capture that he is willing to submit in the course; in fact, all of the assessments in the course are documented as merely a list of learning objectives so that each student can demonstrate mastery
through modes that they have an affinity for or modes that lend themselves to the objective being assessed.
2.5 Co-curricular Activities
While the idea that learning does not always take place in the traditional classroom setting has been accepted for many years (Dewey 1997), educators and students are now embracing the concept. Co-curricular learning refers to activities, programs, and learning experiences that complement, in some way, what students are learning in formal classes
, but also serve to expand the relevancy and contextualization of that learning in the real world. Co-curricular contexts enable the learner to interact with material outside the classroom in such a way that they acquire new knowledge or apply a skill from one area to a new area. Increasingly, institutions of higher education are embracing co-curricular learning and its benefits to their students (Stirling and Kerr 2015; Turrentine et al. 2012).
learning to extend beyond the academy, students can build professional networks, enhance their learning through multiple perspectives, and more quickly build workforce-ready skills. Through co-curricular access to other educational stakeholders, students can engage and consume new learning in the mode they most prefer. Students could also participate and drive change in their community by working directly with business and community leaders to couple their learning with practical applications.
These co-curricular activities could take many forms, and in one scenario, educational stakeholders such as a venerable scientist or engineer would be purposefully contracted by the institution to participate in the virtual world associated with the course to:
Host weekly informational sessions via web conferences, virtual world environments, or even via chat
Make himself/herself available for text- or video-based question/answer sessions on the week’s subject matter from the classroom
Use virtualized laboratories for student observation
Provide feedback and insight into informal or formal assessment of the subject matter
Each scenario further extends
the student’s learning, infused with the relevancy and currency of a working professional in their field of study.
of higher education do not have formal connections with the private sector that underrepresented students’ need to benefit from co-curricular learning. Faculty may also be challenged with assessing which learning should happen outside the classroom and what should be inside the classroom (face to face or online), may struggle with how to integrate or sequence this type of learning with classroom-based learning. Federal credit hour definitions may also limit co-curricular learning or make it difficult to “count” it, so it would be additive, perhaps contributing to cognitive overload for students. Validation of co-curricular learning may be difficult to assess as well.
2.5.3 Implementation Strategies
Faculty can design course
learning objectives tied to business and community interaction, where the choice of modalities would be driven by a student’s individual preferences. For example, a course learning objective could read “Students will consult with a scientist to discuss what types of careers a science graduate can pursue upon graduation
.” Faculty could design course content and set the stage for interactions and discussions that engage educational stakeholders from outside the walls of the classroom.
2.5.4 Research Questions
in this area center on the benefits and consequences of students’ learning outside of the classroom. MML literature is seemingly quiet in this area, though it was a theme discussed in the workshop. There does remain a risk to the fidelity, meaning the degree to which application of MML can be replicated, across iterations of co-curricular learning programming. As co-curricular learning increases, capturing the outcomes and related learning that stems from MML will need to be developed and well understood by faculty and students as well.
Each Tuesday night, Juan’s whole class, his professor, and a group of scientists from all over the world gather to sit in the virtual classroom to discuss vital scientific theory that will enable him to be more successful when he is attempting his experiments. These same scientists are working with Juan and Professor Aranda throughout the term to customize the learning and to be sure Juan is engaged in the most up-to-date practice. Juan is developing a strong network of future colleagues as he interacts with these external educational stakeholders.