Recent technological advances enabled scholars to measure HRV data with the required field accuracy using cost-efficient wearable devices (Züger et al. 2018) opening up new avenues for the field of learning analytics. While such wearables are easy to apply, they also pose additional challenges to the targeted field infrastructure often requiring a dedicated device to store the recorded data. This storage, however, must be secure, reliable, and especially for remote setups, independent of the participant’s existing IT infrastructure with remote support capabilities and yet easy to use. To gain additional insight into these challenges, we developed a physiological computing infrastructure called SEN-Pi, based on a Raspberry Pi mini-computer and a Polar H10 chest belt, and field-tested it with medical students from the EDU Institute of Higher Education to capture their learning journey during remote learning.
In the following section, we discuss the contributions of our article, the lessons learned from our field study with suggestions for future research, and the limitations of our approach.
First, our study is among the first to collect different types of data via a physiological computing infrastructure in remote setups, contributing to the goal of using physiological field data to enable machine learning classifications of flow or related constructs in the near future (see Gap 1, Related Work). Laboratory experiments remain of undisputed importance and are especially well suited for achieving high internal validity because they keep extraneous variables in the form of noise, lighting, and temperature stable (Mitchell and Jolley 2010). However, the conditions generated in laboratory experiments, as emphasized by researchers (Fairclough 2008; Loewe and Nadj 2020), correspond only to a certain extent to the conditions of a real world environment, which justifies the need for our SEN-Pi infrastructure.
Second, being in flow can lead to increased individuals’ well-being, motivation, and performance (Fullagar and Kelloway 2009) and therefore seems to be a desirable state for students during remote learning. For this reason, our work lays the groundwork to the goal of developing interventions that could support students’ flow during remote learning, for instance, by monitoring flow in (near) real-time using individuals’ HRV and/or browsing behavior data (see Gap 2, Related Work).
Third, our SEN-Pi infrastructure employs accurate ECG-based sensors to record physiological data. In contrast, most previous studies relied on PPG-based sensors (see Gap 3, Related Work), which have undesirable characteristics such as sensor artefacts caused by excessive motion. ECG devices provide more accurate readings and are less sensitive to motion artifacts (Khandoker et al. 2011). Therefore, the inclusion of ECG-based sensors in our infrastructure allows us to measure accurate data that will form the basis for the development of future classifiers using machine learning.
Fourth, SEN-Pi provides an infrastructure for field experiments that is platform-independent and can be widely deployed at participants’ homes or workplaces (see Gap 4, Related Work). Because it is a complete system, it does not impose requirements on an existing IT infrastructure, but remains cost-efficient. Thus, our physiological computing infrastructure supports future research in large-scale field experiments, which seems important since larger sample sizes control for the risk of reporting a false-negative finding (Biau et al. 2008). In addition, large data sets open up the possibility of using deep learning algorithms in the future, which has led to improvements in other complex task including ECG and EEG analysis in recent years (Rim et al. 2020).
Fifth, our physiological computing infrastructure provides research teams with full access to their collected data without the need for proprietary middleware (see Gap 5, Related Work). Other systems that have proprietary access may incur additional cost and/or effort to access the raw data using a subscription model. Also, automatic uploading of data to proprietary cloud storage does not always comply with legal regulations, such as the European Union’s General Data Protection Regulation. Our solution circumvents these issues while still providing full access to all stored data.
Sixth, the SEN-Pi infrastructure makes the collected data transparent to study participants (see Gap 6, Related Work). In particular, individuals can view their personal data in (near) real-time on a web dashboard. Thus, participants not only retain control over their personal data, but can also gain new insights and perform their own analyses with their data.
Lessons Learned and Suggestions for Future Research
The logic of when and how a notification should appear was an important aspect of our ESM browser extension. We decided to rely on the notification module offered by Google Chrome.Footnote 6 This module uses the notification center available in the OS to send a notification. However, during the first two days of the preparation phase, we noticed that some participants did not submit a completed flow survey. Among the participants, several different OS were used in different versions. While the majority were using MS Windows 10, some participants were using outdated versions. With these outdated versions, notifications were sent correctly, but the notification center attenuated the notifications by default, so the user could not see them unless they directly accessed the center or changed the default value in the settings.
For this reason, we deployed an update for the ESM browser extension on the third day of the preparation phase. This update solved the problem, as it no longer relied on the notification center available in the OS, but used ordinary browser alerts to interrupt participants. This technical problem confirmed the need for an infrastructure that is independent of participants’ IT systems and the importance of conducting a preparation phase to detect such technical problems in advance.
Since EDU medical students typically attend their online courses in a block format and an exam had to be taken after each online block, the participants’ schedules were already busy. Therefore, in order to conduct the study in this context at all, it was important to offer a balance between mandatory online appointments for the field study and flexible appointments as needed. In consultation with the EDU Institute of Higher Education, we offered four sessions: two online briefing sessions to promote the study, one kick-off session to install the software with the students, and one debriefing session to explain the results of the field study. In addition, if a participant experienced problems during the field study, they could connect the SEN-Pi to the Internet so that an experimenter could securely access the SEN-Pi remotely and offer assistance. The experimenter support was offered during regular working hours.
In the future, it would be interesting (if organizationally possible) to also offer a mandatory exchange session after the end of each study week, in order to get an immediate impression of all participants from the respective week.
Next, we want to discuss the incentive structure of the field study conducted. As an incentive to participate in the study, students were able to view their own physiological data via a personal web dashboard as well as download the data as MS Excel files for further analysis. For each pseudonym, an analysis of the collected data was also performed by the experimenters after the end of the study and stored in a secure database. Only the owners of the respective pseudonym could access the corresponding analysis. In the two briefing sessions, the students reacted very positively when these incentives were presented and declared their willingness to voluntarily participate in the study under these conditions. We explain this willingness by the fact that the topic “human physiology” is an inherently medical one and that a basic affinity of medical students was served by the study. Second, the prospect of not only receiving a one-time, personalized analysis at the end of the study, but also being provided with a personalized web dashboard with established visualizations from the medical context (e.g., the Poincare chart) in near real-time over the entire study weeks seemed to offer another explanation.
However, user behavior only partially coincided with students’ voluntary willingness to participate in the study, as ultimately only three of seven participants actually recorded HRV data. For future field studies in such setups we therefore suggest to consider additional incentives such as study credits or financial compensation.
Finally, we would like to discuss the importance of data privacy and data protection in our solution. In our requirements elicitation process, we found that HRV data required special attention in this regard. Therefore, we opted for a secure hardware storage solution in the form of the Raspberry Pi. The data was only stored locally in encrypted form. Unauthorized access via the Internet was thus made impossible. As mentioned earlier, all recorded data was made transparent to each participant in near real-time via a locally served, personalized web dashboard. All recorded data could be downloaded and viewed by participants, who retained control of their data at all times. Moreover, by using randomly assigned pseudonyms, no one, including the experimenters, was able to match the recorded data to a participant’s identity.
Nevertheless, software-based storage solutions such as cloud storage could be considered for the future. In doing so, important issues such as the location of the server, data access by the host or other third parties, and participant data sovereignty need to be discussed and evaluated. In terms of data privacy and data protection, we therefore see the current hardware storage solution offered by the SEN-Pi as an advantage.
We are aware that our approach comes with limitations. In particular, with 8 participants, we expected to collect at least 160 completed flow surveys (5 working days * 8 participants * 4 completed surveys per day), with up to 480 in an absolute best-case scenario (10 working days * 8 participants * 6 completed surveys per day). Our assumptions were consistent with the amount of data required to train a machine learning classifier according to our previous research outcomes (Rissler et al. 2020). Although we did not meet our goals for data collection, we see value in communicating our findings for conducting remote field studies since current flow research has mostly neglected requirements for field infrastructures in remote setups, especially for physiological computing (see Sect. 3).
Next, our current web dashboard was also designed as an incentive for medical students to participate in our field study. Although the charts displayed were selected for their ease of reading and explanatory screens were provided, they represent more specialized medical insights. For students in other disciplines, the dashboard could benefit from additional or different (simplified) metrics that are more appropriate for the targeted user group. Similarly, incentives for users to participate would also need to be adjusted in other contexts, such as the workplace.
Finally, we did not ask our participants to indicate their age, as it would have been possible to identify each participant by age given the limited sample size. In general, participants’ ages fall within the 20–30 age range typical for college students. However, we are confident that our infrastructure is suitable for very mixed age groups undertaking remote learning, as long as individuals are comfortable using personal computers and the Internet.