Sensor-Based Analytics in Education: Lessons Learned from Research in Multimodal Learning Analytics

Abstract

The use of sensors to support learning research and practice is not new, whether in the context of wearable technology, context-aware technology, ubiquitous systems or else. Nevertheless, the proliferation of sensing technology has driven the field of learning technology in the development of tools and methods that can generate and leverage sensor-based analytics (SBA) to support complex learning processes. SBA fulfills the vision of integrating multiple sources of information, coming from different channels to strengthen learning systems’ desired features (e.g., adaptation, affect detection) and augment learners’ abilities (e.g., through embodied interaction and cognition). In addition, it offers a promising avenue for improving the research measurements in the field. In this chapter, the authors present how SBA has advanced learning technology through the lenses of their offered qualities, indented objectives and inevitable challenges. Through three case study examples, we showcase how those advancements are reflected in contemporary Multimodal Learning Analytics (MMLA) research. The chapter is concluded with a discussion on the role of SBA and a future research agenda that depicts how the lessons learned from the encountered challenges of MMLA can help us further improve the adoption of SBA for learning technology research and practice.

Appendix: Categorization of Challenges and Opportunities, Across the Three Case Studies

	Methodological	Practical	Ethical
Case study 1: Challenges	The trade-off between accuracy and authenticity. This is a controlled (lab) study. Lab studies tend to focus on the “accuracy” of the data collections and accept the low ecology of the results (authenticity)	Less impactful on the practical adoption. There are concerns about how such treatment can occur in real life settings, making it less impactful on the adoption in practice	Less focus on potential ethical and privacy challenges. The researchers follow clinical research standards and procedures, with time-tested protocols to ensure research ethics. Yet, no focus on larger ethical implications of the used technology
Case study 1: Opportunities	Proxies and Markers. Controlled environments can help us to develop powerful computational models that can be used in the “out of lab” contexts, and further validated to account for the lack of authenticity	First step: Controlled environment studies can generate significant research insights and can support the discussions with regard to adoption by contributing to various practical challenges observed	Do we really need sensor data? Controlled studies can help us to investigate the extent to which we can use mainstream LA as proxies, this allows us to develop models that operate in a sensor-free manner with very high performance (that sometimes even outperform sensor-based models), but optimize when sensor data are available
Case study 2: Challenges	Complexity of the measurements and their interpretations. One of the most important challenges connects the measurement of the multifaceted notion of the “learning experience”. Learning experience can not be portrayed by one index and its monitoring has to be context specific	The pendulum of universality and context. The development of “universal” markers to indicate the domain specific goals someone should have is a common approach in other disciplines (e.g., universality of VO2max and HR and context dependent on resting, exercising), in learning this is still a challenge we see in our studies, but shy away from developing further	The paradox of participants’ high-interest in the technology and low-interest in data privacy. Employing wearable sensor data is probably a new experience for many of the participants and the researchers have to show patience answering all questions. An interesting aspect is that a considerable amount of time was needed to explain the details about the wearable devices, but fewer to no questions were made regarding privacy issues with the collected data. (all information was given in the consent form)

1. (continued)

	Methodological	Practical	Ethical
Case study 2: Opportunities	Safeguarding learners’ SBA. Methodologies and practices to safeguard SBA (that allow us to safeguard SBA at least at the same level we do for mainstream log data) will allow us to boost use and adoption		Working towards a transparent “risk-benefit” assessment on the use of SBA. Employing SBA in education has inherent ethical consequences, however, they arise from both using, but also denying the use of SBA (if there is a proven benefit). (e.g., how ethical is it to deny the use of SBA for learning, if their significant advantage for learners is evidenced?)
	Move to non-invasive/data-sensitive SBA (as much as possible). Affective information coming from cameras, EEG and other invasive/data-sensitive streams, can be transferred through proxies to modalities such as EDA, HR. The use of these modalities is not without challenges and controversy, but the risk is reduced while the benefit stays at the same levels.
	Development of group-level SBA: Defining SBA at the group level would help in developing group-level affordances and capabilities to support collaborative learning and awareness (e.g., team engagement or alignment in this case study)
Case study 3: Challenges	Development of functionalities that leverage SBA (as we do with mainstream analytics). Development of models and ontologies of meaningful SBA and design of interfaces and implementation of system logic that utilises and accounts for SBA (e.g., feedback and adaptive support)	Space matters more for SBA. Having a properly designed room, or “spot”, makes it much easier for researchers and stakeholders to utilize SBA. However, allocation of such space is rarely possible in most educational contexts	Significant concerns among stakeholders, even when no personal data are involved. Utilizing SBA can lead to significant concerns among stakeholders. Even when SBA are employed without capturing any SBA (e.g., used for interaction only), this might serve as a barrier for future advancements and adoption of SBA in education
Case study 3: Opportunities	Leveraging on the various capacities of sensors. Sensors should not be used only as an interaction modality or as an extra measurement, but needs to enrich systems functionalities Development of SBA support for near real-time use. Designing and developing affordances that leverage SBA’s temporality in a momentary and continuous manner (e.g., agents, dashboards, reports to instructor), has been identified from both researchers’ observations and participants comments		Transparent, trustworthy and accountable SBA. Future research needs to devise frameworks and guidelines for SBA in education that will allow researchers and practitioners to employ SBA in a transparent, trustworthy and accountable manner; and result in the support and adoption from the stakeholders