Towards a Deeper Understanding of the Behavioural Implications of Bidirectional Activity-Based Ambient Displays in Ambient Assisted Living Environments

Abstract

In this chapter, we investigate the extent to which the real-time bidirectional exchange of activity information can influence context-awareness, social presence, social connectedness, and importantly interpersonal activity synchrony in mediated ambient assisted living (AAL) environments. Additionally, we describe the design, development, and assessment of a bidirectional ambient display platform to support real-time activity awareness and social connectedness in mediated AAL contexts. In a semi-controlled study, we evaluate a conglomerate of activity-based lighting displays, to determine the effects of real-time bidirectional deployment on behaviour and social connectedness. Exploiting everyday objects, human activity levels are projected with a Philips Hue lamp, LED wallet, and LED walking cane, which render this information based on predefined patterns of light. Results from the current study show tendencies toward (1) an increase in implicit social interactions (e.g., the sense of experienced social presence and connectedness), (2) more positive social behaviours between the elderly and their caregivers in mediated AAL contexts, and (3) sporadic moments of interpersonal activity synchrony however, further investigation is necessary to determine the extent of this variable in mediated AAL contexts.

The work conducted in this chapter was carried out while the first author was a candidate in the Joint Doctorate on Interactive and Cognitive Environments (ICE) at the Eindhoven University of Technology and the University of Genova, which resulted in the following dissertation [22].

1 Introduction

In the 21st century, ageing populations around the world are increasing dramatically. Nowadays, most persons can expect to live until they are 60 and beyond, which according to the World Health Organization¹, is a ‘first time occurrence’ in mankind’s history. Population ageing presents critical challenges, which include but are not limited to the following (i) frailty, (ii) physical disabilities, (iii) cognitive and cardiovascular diseases as well as (iv) vulnerabilities to social isolation and loneliness. Despite these difficulties, many older adults are insistent on striving to maintain their autonomy and quality of life [27]. Therefore, social engagement and enhancing the quality of life of older adults are of a high priority on the political agendas of many ageing societies including Europe and Asia. Notably, this problem space presents challenges and opportunities for designing and developing technology-rich environments capable of supporting healthy and active ageing as demonstrated by the researchers in [10, 35, 61].

Ambient Assisted Living (AAL)² encompasses a broad range of Information and Communications Technologies (ICT) for enhancing functional independence, social interaction, and the overall quality of life among older adults. Currently, most AAL interventions are geared toward safety and ambulatory monitoring for emergency detection. Such systems are mostly driven by Ambient Intelligence (AmI) and can be seamlessly interwoven into the existing life patterns of older adults. In fact, ambient Intelligence (AmI), aspires to detect people’s state and adaptively respond to their needs and behaviours through the integration of ubiquitous technologies in their environment [82]. Drawing from disciplines such as artificial intelligence, human computer interaction, pervasive/ubiquitous computing, and computer networks, AmI systems can sense, reason, and adapt to offer personalized services based on the user’s context, intentions, and emotions [1, 16]. In this way, such systems, also known as context-aware systems [66], can be integrated into AAL environments to provide better care and support for the elderly living independently.

Weiser’s vision for ubiquitous computing is described in his seminal work entitled The Computer for the 21st Century [86]. In his narrative, Weiser envisaged a world where technology would silently reside in the background or periphery of the user’s attention and is available at a glance when needed. Consequently, the allocation of minimum attentional resources would enable peripheral interaction with a system as suggested in the following statement. “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it” [86, p. 1]. To this end, ubiquitous computing aims to enable calm technology [87], whereby information is transported easily between the center and periphery of attention. To achieve this, ambient displays, a sub-discipline of AmI, generally refer to systems intended for portraying various types of context information, e.g., weather, stock prices, or the presence or activities of others in the periphery of the users’ attention [62].

Within the AAL domain, some studies [11, 15, 18, 19, 55, 58, 65, 69, 84] have demonstrated benefits associated with aesthetically pleasing and informative ambient displays to raise context awareness and strengthen social interaction. Also, previous works including our very own [21, 25, 27] have demonstrated that physical activity information equally shared between two remote users can provide a sense of peripheral presence and interpersonal awareness; thus stimulating positive social behaviours in AAL contexts. However, these behavioural implications following the receipt of real-time activity cues through bidirectional activity-based ambient displays in remote AAL contexts have not been dealt with in depth. Therefore, this chapter further investigates the behavioural implications of real-time bidirectional activity-based ambient displays in mediated AAL contexts and is foreseen to provide further insights into the potential benefits and usage possibilities of bidirectional activity-based ambient displays. Consequently, this can inform the design decisions regarding the functionality, adoption, and acceptance within mediated AAL environments.

In the remaining sections of this chapter, we will discuss the following. First, we will review the literature on social well-being and its related measures. Then, we describe our design rationale and provide an overview of our system. After that, we present a user study describing our evaluation process, and later we expound upon our findings on the effect of the system on social connectedness, social presence, and interpersonal activity synchrony. Ultimately, we make our conclusions and discuss our plans for future work.

2 Social Well-Being and Related Measures

To begin with, it is necessary to understand the notion of social well-being as a critical aspect of ‘ageing in place’. Social well-being has become a central topic in gerontological research [44, 74] and is defined by the authors in [47] as “the appraisal of one’s circumstance and functioning in society” (p. 122). According to Abraham Maslow’s hierarchy of needs, love and a sense of belonging are vital for human functioning, which transcends to the primal need for intimacy, family, and friendship [53].

In Maslow’s hierarchy, once physiological and safety needs are met then a person can strive to satisfy the need for love and belonging, which is essential to fulfil esteem needs and if possible attain a state of self-actualization. Therefore, sociality is crucial for well-being, as human beings are naturally driven by an inherent desire to belong and maintain strong and lasting bonds [5]. Accordingly, this need is satisfied through regular and positive interactions with long-term social contacts [5].

Throughout the past decades, several researchers in psychology and social sciences have documented substantial empirical evidence on the impact of social relationships on promoting health, longevity, and optimal physical functioning in older adults [50, 89]. In particular, socially active senior citizens are often physically and mentally healthier when compared to those who are socially isolated [17, 88]. However, the absence of close family ties and fulfilling social relationships may cause undesirable implications such as loneliness and depression in older adults [72].

With the onset of better employment or educational opportunities, geographical distance between family members has become a primary barrier for effective communication and the provision of care for older adults [7]. Essentially, while living apart, it is crucial to stay connected and keep abreast of each others’ activities. Although the proliferation of computer-mediated technologies such as instant messaging, free or relatively cheap Voice over IP calls, and email can augment communication, such technologies are sometimes intrusive and require more attentional resources for communication. As such, this chapter explores the concept of social connectedness through peripheral technology designed to facilitate real-time activity awareness and improve interaction between the elderly and their caregivers in mediated environments. To reduce disturbances in daily life activities, we believe that an indirect means of awareness of each other’s context and activities can sustain close connections and reduce the risks of social isolation and loneliness among older adults.

2.1 Social Connectedness

The generally accepted use of the term social connectedness usually refers to a sense of “belongingness and relatedness between people” [81, p. 1]. Also, Van Bel et al. discuss the importance of understanding the temporal aspects of belongingness, which can be experienced on two levels, i.e., the (i) ‘momentary’ or (ii) ‘continuous’ feeling of connectedness. However, the authors in [80] gave precedence to the long-term experience, which is more distinctive in relatively stable interpersonal relationships; whereas the short-term experience of connectedness can be influenced by a person’s current emotion, their present assessment of their sense of belongingness or their interactions with another individual. Other factors such as age, context, gender, personality traits, culture, individual preferences, and previous relationship experience can also affect how people experience social connectedness [34].

Altogether, a sense of belonging appears to be embodied in the concept of social connectedness, such that an increase in social connectedness can lead to the positive feeling of having enough social contacts and also, support the personal assessment of being a valued member of a group. To determine a person’s social connectedness with others, Van Bel et al. suggest the following five dimensions [81].

• 1. Relationship Salience – The continued sensation of presence and togetherness with another despite being in different locations.

• 2. Contact quality – The subjective assessment of the quality of interaction with others in a person’s social network.

• 3. Shared understanding – having common interests, ideologies, and perspectives with people in one’s social network.

• 4. Knowing each others’ experiences – becoming emotionally aware of each other’s subjective feelings along with recognizing and understanding the counterpart’s experience and how they think.

• 5. Feelings of closeness – examines the intensity of the attachment with one person against all other relationships. Also, assesses the quality of communication and emphasizes confidentiality and openness in relationships.

Awareness systems build on the construct of connectedness oriented communication, which is closely aligned with the exchange of affective and relational information aimed at maintaining relationships and promoting a strong sense of connectedness [52]. Basically, social connectedness assesses the emotional experience of belongingness and can be measured qualitatively by determining heightened feelings of closeness, commonalities between relational partners, and the mutual expression of feelings and thoughts [81]. The construct can be approached quantitatively by assessing how one perceives their social situation (i.e., social appraisal) and their personal evaluation of relationship salience (i.e., the presence of another) [81].

While the notion of social connectedness is difficult to measure, the design community has noticed its relevance to tailor novel socially aware technologies to facilitate a sense of belonging in mediated environments [84]. However, there are other applicable measurements (e.g., social presence) related to this phenomenon that will now be addressed.

2.2 Social Presence

Despite many attempts to define social presence, the scientific community has not yet reached a consensus on its definition. A more concrete view is formulated by Biocca et al. in [8], where they define social presence as a “sense of being with another in a mediated environment” (p. 10), not only replicating face-to-face interactions but also considering the mediated experience of human and non-human intelligence (e.g., artificial intelligence). This shorthand definition further elaborates on the “moment-to-moment awareness of co-presence of a mediated body and the sense of accessibility of the other being’s psychological, emotional, and intentional states” [8, p. 10]. Therefore, social presence is categorized into three distinct levels as explicated by Biocca et al. below [8].

• 1. Level one (the perceptual level) – one becomes aware of the co-presence of the mediated other.

• 2. Level two (the subjective level) – is comprised of four dimensions describing the perceived accessibility of the mediated other’s:

  • attentional engagement

  • emotional state

  • comprehension

  • behavioural interaction

• 3. Level three (the intersubjective level) – assesses the degree of symmetry or correlation between one’s own feeling of social presence and their impressions of the mediated other’s psychological sense of social presence. It goes further to examine concepts such as interdependent actions e.g., reciprocity/motor mimicry in mediated environments, which is closely related to the notion of interpersonal activity synchrony [14], a concept generally known to foster socially cohesive behaviours in relationships, a focal point to be investigated in this chapter.

2.3 Coordinated Actions – Interpersonal Activity Synchrony

For many years, coordinated actions have been considered to enhance relationships and are deemed as an essential component of social behaviour and interactions [4, 6, 12,13,14]. In addition, scholars such as [4, 12] suggest a possible link between perception and behaviour such that automatic mimicry can be evoked by the mere perception of an interaction partner’s behaviour. In this chapter, interpersonal activity synchrony is investigated through a set of analogous and sometimes overlapping terms namely (i) behavioural coordination, (ii) coordinated action, (iii) motor coordination/synchrony, and (iv) emotion contagion.

Coordinated behaviour has been shown in a variety of contexts such as parent-infant bonding [14], teacher-student interactions [6], and intimate relationships [45], such that coordinated action, i.e., interpersonal activity synchrony is regarded as an indicator of social interaction. In particular, previous studies have examined this construct with reference to the synchronization of bodily actions such as oscillations of rhythmic limb [68] and lower leg [67] movements. Likewise, some scholars have found evidence of interpersonal motor coordination while two people either (i) walked side-by-side [79, 90] or (ii) swayed side-by-side in rocking chairs [64]. Added to motor synchrony, other studies have investigated coordinated behavioural markers in terms of the mimicry of conversations, collective musical behaviour, dancing, laughter, facial expression, and emotions [13, 33]. Altogether, these indicators can be combined under one umbrella term, emotion contagion, which is defined as follows. “The tendency to automatically mimic and synchronize facial expressions, vocalizations, postures, and movements with those of another person’s and, consequently, to converge emotionally” [40, p. 5].

A key problem with much of the literature examining behavioural coordination is that they tend to focus on face-to-face interactions with very little studies conducted in mediated environments. While we wholeheartedly agree that face-to-face interaction is perhaps one of the most active forms of interpersonal interaction [59, 70], given its offerings of immediate feedback, engagement, and interpretation of non-verbal communication cues among others, we also believe that there is a need to explore other types of interaction, especially for enabling peripheral interaction in AAL. As mentioned earlier, Biocca et al. highlighted interdependent actions as a critical determinant of social presence in mediated environments [8]. Thus, in an attempt to facilitate coordinated behaviour in mediated AAL environments, this chapter evaluates the extent to which the system can trigger or influence interpersonal activity synchrony.

2.4 Interpersonal Synchrony – Computational Methods in the Field

Very few studies [30, 33, 43] address the issue of interpersonal synchrony in mediated environments. Thus, to gain a deeper understanding of this social phenomenon we had to review studies demonstrating synchrony in both real life and mediated contexts. From the literature reviewed, e.g., [30, 43, 64, 79] we can infer the following indicators of synchrony.

• co-action

• coordination

• mimicry

• emotion contagion

So, how do we compute interpersonal activity synchrony in mediated AAL environments? Findings from different studies suggest that activity synchrony is determined by calculating the autocorrelation [76] or Pearson correlation [78] of the linear coupling of activity patterns. Also, researchers such as Haken et al. have considered an in-phase approach to synchrony such that motor signals are homologous and in synchrony [37]. Concerning mediated environments, scholars such as those in [30, 43] suggest cross-correlation measures for computing physiological linkage – a related measure of emotion contagion. Moreover, Biocca et al. conferred in their model of social presence that the degree of symmetry or correlation is a measure of social presence [8].

Although correlation measures are critical for calculating interpersonal synchrony, there are other mathematical constructs to consider. For example, Hove and Risen discussed the necessity of imposing a temporal lag (lasting a couple of seconds) following the reference behaviour in the cross-correlation calculation so that mimicry and by extension synchrony can be determined [42].

Considering the previously explored computational methods for evaluating interpersonal activity synchrony, we will employ cross-correlation measures for assessing this phenomenon in this chapter. Furthermore, we will impose a lag to compute this cross-correlation. More details on our evaluation and data analytical methods will be described later in this chapter. We will now provide an brief overview of our bidirectional activity-based system and subsequently discuss our methodology.

3 System Overview

3.1 Design Rationale

As mentioned earlier, our bidirectional activity-based implementation is an ambient lighting system that detects human activities and provides visual feedback through a LED cane, LED wallet, and Philips hue light orbs to create a sense of awareness and social connectedness between older adults and their caregivers. We were guided by the following design heuristics obtained through a thorough review of the literature [51, 54, 83], interviews with design experts, and our own findings from previous research [23, 26, 28] using ambient displays.

• The system should be practical, not distracting, portable, perceptible, comfortable, meaningful, reliable, subtle, discrete, aesthetically pleasing, accessible and safe.

• The system should accommodate the vision and motor impairments of the elderly population and should appeal to the intrinsic motivation to share knowledge.

• The system should support ease of use, affordance and learnability bearing in mind that the elderly are susceptible to cognitive impairments, which affects their attention and memory.

• The system should support the elderly’s autonomy and should seamlessly fit into their existing lifestyle patterns.

Motivated by the central goal of designing usable, acceptable, and accessible products for the elderly and their caregiver counterparts we sought to determine appropriate everyday objects for conveying activity information that would meet our design criteria. This was done over the course of several brainstorming sessions with experts in the field, designers, and prospective users. Notably, to provide an “always connected” service, we were interested in complementing our already existing Hue lighting system with portable ambient lighting devices. After much deliberation and reference to the Smart Cane System designed by [49], we decided that the LED cane and wallet were most suited for conveying activity information while simultaneously adhering to the design heuristics.

3.2 System Components

The entire system is composed of 5 major subsystems as illustrated in Fig. 1. A remote server subsystem resides in the central part of the system and is responsible for classifying human activities and relaying detected activities to other subsystems. A LED and Hue subsystem are located on each side of the remote server subsystem, respectively. Each LED subsystem consists of a waist-mounted smartphone, an Espressif (ESP) microcontroller with Wi-Fi capability, and an LED ring or strip. The waist-mounted phone is equipped with an accelerometer and a gyroscope for measuring the proper acceleration and orientation of the body, respectively (cf. [25]). A custom built Android application i.e., the LED controller application (app.), collects the accelerometer and gyroscope readings (sensor data) at a frequency of 50 Hz (cf. [20]) and sends it to the remote server subsystem for classification. The Android application maintains two socket connections to the central remote server, one for sending sensor data to the server for classification and the other for receiving the classified activities of the counterpart. Subsequently, the classified activities received are mapped to activity levels and then transformed to lighting property encodings, which is later broadcasted to the led strip/ring via the ESP microcontroller Wi-Fi module. To achieve this, the waist mounted phone requires a 3G/4G internet connection by which data is streamed to the remote server and a portable Wi-Fi hotspot to provide an internet connection to the ESP Wi-Fi module.

Fig. 1.

adapted from [24].

An overview of the bidirectional activity-based system

Besides, the Hue subsystem consists of a mobile phone with Wi-Fi internet connection and a Philips Hue bridge and bulb. Another custom-built Android application (i.e., the Hue controller), maintains a single socket connection to the central server subsystem for receiving the classified activities of the partner. The Hue controller then relays this information to the hue bulbs as light property encodings via the hue bridge. The Hue subsystems are deployed indoors to convey bidirectional activity information while users are situated in the comfort of their homes while the LED devices are carried when users are outdoors. This enables an “always connected” system to users. Please refer to [24] for more details on our real-time activity-based bidirectional framework.

4 Methodology

In a semi-controlled study, we evaluated a conglomerate of activity-based lighting displays designed in [21, 24, 26], to determine the effects of bidirectional deployment on behaviour and social connectedness. Our experimental approach can be described in three main stages, which are listed below.

• 1. The Pre-trial – Following the design and development of our real-time bidirectional activity-based implementation in [24], we conducted two practice sessions with a prospective caregiver and the elderly stakeholders to identify system glitches and obtain technical insights and practical recommendations for system deployment and improvement.

• 2. The Real Deployment – Following system adjustments, our bidirectional activity-based system was deployed in semi-controlled mediated environments to evaluate the effects on synchronized activities, context-awareness, social connectedness and social presence, information clarity, attentional engagement, and the users’ willingness to adopt the system.

• 3. Post Deployment Interview – We conducted a series of in-depth interviews to determine the participants’ experiences and acceptance of our activity-based system and how it affected their behaviour.

Ekman et al. maintain that synchrony is inherently activated by the degree to which people are exposed to the same stimulus [30]. The authors further highlight a study by Hasson et al. [39] whereby participants were exposed to an identical visual stimulus (i.e., a movie scene) to incite synchronized cortical activity. Accordingly, influencing our study design decision to expose half of our participants to the same stimulus (i.e., scripted activities of an actor) to induce interpersonal activity synchrony. Inspired by the previous studies on interpersonal synchrony [14, 57] and physiological linkage [30, 43] to enhance interpersonal connectedness we assume the relevance of these constructs to provide social support in AAL environments. As such, we defined the following research questions.

• To what extent does activity awareness through a bidirectional activity-based system impact the synchronization of the counterpart’s activity level with that of the caregiver?

• How does the activity level of an actor (caregiver) modulate the activity levels of their counterpart?

• What are the implications of the bidirectional activity-based system on

  • social connectedness,

  • social presence,

  • context-awareness,

  • information clarity,

  • attentional engagement and,

  • the users’ willingness to adopt the system?

4.1 Participants

Participants were recruited through personal networks and referrals from a retired professor, and engineer in the Netherlands. Notably, both the retired professor and the engineer acted as proxies to represent prospective elderly recruits. Thus, before experimentation all system requirements, designs, prototypes, and the study design were repeatedly cross-validated with these proxies. This was done as a measure to guarantee system functionality, user comfort, and privacy so that they could proceed with the recruitment. Overall, twenty-four persons (twelve pairs) participated in the study.

Table 1. Demographic characteristics of participants

The following are criteria for the inclusion and exclusion of participants in this study.

• Equal numbers of younger adults and elderly participations are essential for this study.

• Prospective younger adults should be over 18 years while prospective older adults had to be over 65 years of age.

• All prospective older adults should be relatively healthy with no history of chronic, motor, or mental diseases.

• All prospective older adults should live independently and demonstrate the ability to execute their ADLs on their own.

• Equal numbers of male and female participations are valuable for this study.

Each participant was assigned to one of two distinct user groups: (i) caregiver – who is expected to execute a series of scripted activities while simultaneously maintaining awareness of their counterpart through the proposed bidirectional activity-based system and (ii) the counterpart – who upon receiving the caregivers’ activities via the ambient display is expected to carry out their activities at their own free will. In this study, an elderly participant could serve as a caregiver, which was determined by the preliminary results in [25], showing evidence of elderly persons caring for their fellow elderly loved ones. The participant demographics are presented in Table 1. To preserve anonymity, caregivers are indicated by letters A–L and their respective counterparts are disguised using letters M–X, and not names.

Participants ranged in age from 21–75 (mean age = 47.8 and standard deviation = 20.8). In addition, we noticed that our sample size was comprised of the relatively ‘young elderly’. Participants were from different cultural backgrounds. In particular, the sample was dominated by the Dutch ($58\%$), followed by the Chinese ($17\%$), the Malaysians $13\%$, and a few ($4\%$ each) Ghanaian, Iranian, and Tanzanian participants. All participants except one pair were somewhat familiar with each other. For example, most elderly participants were members of clubs and societies for retired professionals in the Netherlands, while others were neighbours, friends, colleagues, or relatives. In addition, all participants were educated having attained either secondary diplomas, bachelor, master, or doctoral degrees. No participant reported ill health. The experiment was conducted in English and Dutch to facilitate the Dutch speaking participants. Participants received information of the protocol and provided their written, informed consent according to the Central Committee on Research Involving Human Subject³.

4.2 Experiment Set-Up

The experiment was conducted in two separate living labs at the Eindhoven University of Technology (Tu/e). These rooms were each equipped with the following items: a sofa, dining table and chairs, books, map of the building, notebook and pen, music for relaxing, coffee table, computers with WiFi connection, dumbbells and exercise videos, refreshments, newspapers, games (puzzles, bowling, and diabolo), Philips Hue light Orbs, which formed part of the room design, Philips Hue bridge, smartphone (with the custom-built Hue controller app cf. Fig. 1), and a portable LED ambient display (cane for the counterpart and wallet for the caregiver). Figure 2 demonstrates the set-up of the rooms before and after the ambient displays were deployed while Fig. 3 depicts sample game and exercise items in the rooms.

Fig. 2.

Snapshots of the experiment set-up pre- and post deployment of the ambient displays.

Fig. 3.

An illustration of the sample games and exercises available in the rooms.

Adhering to the protocol for activity detection described in [20, 24, 25], our hybrid SVM-HMM HAR model deployed in a central server subsystem, was used to detect six basic activities (standing, sitting, walking, walking upstairs and downstairs, and laying) from data received via a waist-mounted smartphone equipped with accelerometer and gyroscope sensors and an internet connection. Activities classified are saved on the server before they are sent to the Hue and LED controller subsystems. These controller subsystems are responsible for abstracting the detected activities into activity levels and mapping them to coloured lighting encodings and finally transmitting them to the ambient display components of the bidirectional system. The ambient display components of the system are the Hue light orbs, NeoPixel LEDs fitted on a wallet and a cane as illustrated in Fig. 4.

Fig. 4.

A pictographic representation of the activity-based ambient display components captured during experimentation. (Color figure online)

The displays render red coloured lighting for high activity levels (walking, walking upstairs and downstairs), green for passive activity levels (standing and walking), and blue coloured lighting for resting activity level (laying).

4.3 Evaluation Measures

• Social Connectedness – Participants rated their perceptions of their feelings of relational closeness toward their counterpart using the IOS scale [3].

• Social Presence – Participants evaluated their sense of co-presence, perceived attentional engagement, and their perception of behavioural interdependence using an adapted version of the Networked Minds Social Presence Inventory developed by [38].

• Interpersonal Activity Synchrony – Using cross-correlation measures we calculated the extent of coordinated actions between the caregiver and their counterpart in mediated AAL contexts.

• Willingness to Adopt – Using a scale of 1–10 with 10 being the most willing, participants were asked to describe their willingness to adopt bidirectional activity-based ambient displays in their own homes.

• Post-test Interview Questions – Participants gave their qualitative input on context-awareness, system relevance and usability, aesthetics, adoption, and evaluated their experience with and without the ambient displays.

4.4 Experiment Protocol

We employed a repeated measures design [32], with one independent variable namely the interaction style (with activity-based ambient light and with white light). There were two experimental conditions having two interaction styles each lasting for 30 min each.

• With activity-based ambient light – such that there is a bidirectional exchange of activity level information between the caregiver–counterpart pair using smart objects such as the Philips Hue, a LED cane, and wallet. This is the intervention condition.

• With white light – such that there is no exchange of activity information between caregiver and their counterpart. This is the control condition.

In both conditions, the caregiver followed a script and performed a similar sequence of activities. To minimize carry-over and order effects, we counterbalanced interaction styles using an AB-BA format [32]. There were two experimenters to facilitate this study. The dependent variables examined include (i) the synchrony of activity levels – interpersonal activity synchrony (on the part of the counterpart), (ii) context-awareness, (iii) social connectedness, (iv) social presence (behavioural interdependence i.e., the counterpart’s synchronized actions with the caregiver), (v) information clarity, (vi) attentional engagement, and (vii) system adoption.

Prior to the experiment, the experimenters ensured that the server was properly communicating with all subsystems. Thereafter, a meet and greet session was held with each caregiver–counterpart pair. The experimenters elaborated on the experimental details such as the significance of the light encodings, experimental conditions, measurement instruments, ambient displays, and moderated the signing of the informed consent forms. Each caregiver–counterpart pair was then fitted with the waist-mounted smartphone.

Subsequently, both the caregiver and their counterpart were placed in two separated living labs. Note that upon arrival, participants were orientated with their environment and told that they were not limited to remain indoors during each condition. In particular, caregivers were encouraged to follow a script comprising of five activities each lasting six minutes. Caregivers were also advised to execute the activities in sequential order. An example of the scripted sequence of activities is given below.

• 1. Read book or the newspaper or browse the internet

• 2. Do some physical exercise

• 3. Do some mental activity e.g., puzzle

• 4. Take a stroll

• 5. Lie on the couch

In contrast, the counterparts were not expected to follow a script. Instead, they were given a deck of activity cards (see Fig. 5 indicating the types of activities they could perform within the experiment), bearing in mind that there were no restrictions in the order or time spent in a particular activity. Additionally, counterparts were instructed to record the sequence of activities performed and the time spent in each activity in the notebook provided. This was done to establish the ground truth in a minimally invasive way.

Fig. 5.

A snapshot describing the possible activities, which could be performed in the experiment.

After the experiment preliminaries were completed, in a pre-test participants ranked their assessment of relationship closeness with their counterpart. Each experimental condition lasted for 30 min. Also, at the end of each experimental condition all participants completed a post-test ranking their interpersonal closeness with the IOS scale. Following the completion of both experimental conditions, participants ranked their experience of social presence using an adapted version of the social presence questionnaire [38] and thereafter participated in a post-evaluation interview, which was audio-taped. Interviews conducted in Dutch were facilitated and translated with the assistance a native Dutch speaker.

5 Quantitative Results

The results from both interactions styles, i.e., (i) with activity-based ambient light and (ii) with white light were analysed using the R Project for Statistical Computing. The analytical methods and research outcomes are presented and discussed below.

5.1 Clarity of Perceived Bidirectional Activity Levels

From the shorthand definition of social presence [8], it can be inferred that an understanding of a mediated body’s intentional states is an important pre-requisite for promulgating social presence in mediated environments. Figure 6 shows a scatter plot of the clarity of the information perceived in both interaction styles.

Fig. 6.

Scatter plot portraying the clarity of perceived bidirectional activity levels.

Noteworthy differences were found in the reports of information clarity with respect to the perception of activity levels in the activity-based ambient light interaction and that of white light. Statistically, a one-way ANOVA with repeated measures gave $F(1,23)=70$ and $p=1.97\text {e}{-08}$. Furthermore, by computing the $\eta _{p}^{2}$ (partial eta squared) measure, we obtained an effect size of 0.75, which is substantial according to the recommendations for the magnitude of effect sizes by [56]. From the results, we can infer that the information portrayed in the “activity-based ambient light” interaction was clear and meaningful. However, this will be confirmed later by the qualitative results.

Fig. 7.

Scatter plot of the estimated attentional resources utilized per interaction style.

5.2 Perceived Attentional Engagement

From our study findings in [21, 25, 26] that the overuse of attentional resources was a marked limitation in both studies. A remarkable result to emerge from the data is that there were fewer accounts of attentional burden during system deployment. Figure 7 provides an overview of the subjective estimates of attentional resources utilized in both interaction styles.

The scatter plots illustrate almost similar distributions between the “with white light” and the “with activity-based ambient light” interaction styles with no statistically significant difference ($p=0.195$) between them. Our findings appear to be well supported by the participants’ qualitative accounts of multi-tasking only taking occasional glances at their partner’s activities to avoid distraction and concentrate on their primary tasks.

5.3 Relationship Closeness Pre- and Post Interaction Styles

As discussed in our review of social connectedness, Van Bel et al. highlighted the feeling of closeness as a dimension of social connectedness [81]. Consequently, this measure was computed to determine the implications on interpersonal closeness with and without the activity-based ambient display.

A one-way analysis of variance (ANOVA) with repeated measures was calculated, which revealed a statistically significant difference between the self-reported IOS pre- and post- experiments with $F(2, 46)=16.25$ and $p=4.58\text {e}{-}06$. In addition, by computing the $\eta _{p}^{2}$ measure yielded an effect size of 0.41, which is reasonably large according to the recommendations for the magnitude of effect sizes by [56]. Figure 8 portrays a box plot of the perceived relationship closeness pre- and post- interaction styles.

Fig. 8.

Box plot showing IOS estimation pre- and post- interaction styles.

From Fig. 8, it is apparent that the mean IOS depreciates during the white light interaction in which there was no exchange of activity information between interaction partners. A pairwise comparison revealed a statistically significant difference in relationship closeness before stimulus exposure and following the interaction with activity-based ambient light resulting in a p-value of 0.00251. Comparing the IOS ratings before exposure and post the interaction with white light did not reveal a statistical difference ($p=0.0568$).

5.4 Estimation of Co-presence

The findings from the study in [25], point to the likelihood of experienced social presence – the feeling of being with mediated the other [8]. As we sought to validate this finding, participants gave their estimations of perceived co-presence in each interaction style. By deploying a one-way ANOVA with repeated measures we obtained a statistically significant result with $F(1,23)=26.74$ and $p=3.05\text {e}{-}05$.

Fig. 9.

Scatter plot illustrating the extent of co-presence between participant pairs.

Moreover, using $\eta _{p}^{2}$ we obtained an effect size of 0.54, which is relatively large according to the rules of thumb on the magnitude of effect sizes by [56]. From Fig. 9, it is apparent that there were more reports of experienced co-presence in the “activity-based ambient light” interaction when compared to the interaction “with white light”. This finding reinforces the usefulness of bidirectional activity-based displays for stimulating social presence.

5.5 The Extent of the Caregivers’ Influence on the Counterparts’ Activity Levels

Behavioural interdependence is underlined as an important dimension of social presence [8]. Thus, self-reports of interdependent actions could complement the cross-correlation analysis on sensed activity data. Recall that this measure was only ranked by the counterparts as caregivers were expected to strictly follow the activity script. A one-way ANOVA with repeated measures revealed a statistically significant difference between the reported influence with $F(1,11) = 10.24$ and $p = 0.00845$. Also, by calculating $\eta _{p}^{2}$ the results show an effect size of 0.48, which is large enough according to the rules of thumb on the magnitude of effect sizes by [56]. Figure 10 demonstrates the degree of symmetry of the counterparts’ activity levels with that of the caregiver.

Fig. 10.

Scatter plot showing the extent of the caregivers’ influence on the counterparts’ activity levels.

Overall, counterparts reported that they were more motivated to coordinate their activity levels with that of their caregivers while interacting with the activity-based ambient light in comparison to their interaction with white light. This confirms our assumption that a stimulus is necessary to create awareness and prompt a behavioural change to act upon the information received in mediated environments. In the case of the “with white light” interaction, the activity information was unknown, and hence there was no interaction.

5.6 System Adoption

Following system deployment, we wanted to determine the number of participants who were interested in adopting the system in the long-term. Logically, system adoption was only computed for the “with activity-based ambient light” interaction style. In this case, both the caregivers and their counterparts stated their perceptions on future system adoption. Their subjective attitudes toward adoption are depicted in Fig. 11. The findings suggest that participants were moderately inclined toward system adoption in the long-run. Additional insights are further implied in the qualitative analysis.

Fig. 11.

Scatter plot representing the subjective ratings on system adoption.

5.7 Towards Interpersonal Activity Synchrony – The Caregiver’s Influence on Their Counterpart’s Activity Levels

To analyse interpersonal activity synchrony, we calculated the sample cross-correlation coefficient (CCF) [71] between activity levels of caregivers and their counterparts for every 6-min interval that the caregivers remained in an activity level specified by the script. Due to time constraints, the script specified 5 activities to be performed within a 30-min interval. Therefore, activity levels were distributed equally in 6-min intervals. Note that resting, passive, and active activity levels were assigned the following values 0, 1, and 2, respectively.

As described in the system architecture of the bidirectional ambient display platform (cf. [24]), the server detected a maximum of two activities for every 5 s worth of data from the waist-mounted smartphone. This implies that a minimum of 2.5 s of sensor data was required in order to detect an activity. This introduced a minimum lag of 2.5 s ($1\ lag\ unit$) and a maximum lag between 5 s ($2\ lag\ units$) and 7.5 s ($3\ lag\ units$) for an activity to be collected, detected, and transmitted to a participant. The sample CCF of time-series variables x and y, representing both the caregiver’s and their counterpart’s activity levels respectively, at time t, given a lag $\tau$ was calculated as follows: Given a sample cross-covariance,

$$ \sigma _{xy}(\tau ) = \frac{\sum _{t=1}^{n-\tau }(x_{t+\tau }-\bar{x})(y_{t} - \bar{y})}{n} $$

the sample cross-correlation (CCF) is given by:

$$ \rho _{xy}(\tau ) = \frac{\sigma _{xy}(\tau )}{\sqrt{\sigma _{x}(0)\sigma _{y}(0)}} $$

where n is number of activity levels detected within a 6-min interval and $\bar{x}$ and $\bar{y}$ are the means of the activity levels of a participant pair (i.e., elderly – caregiver) within a 6-min interval. With negative lags, the caregiver is made to lead their counterpart to serve as a reference for analysing activity synchrony of the counterpart. The sample (CCF) was calculated for each 6-min interval. Thereafter, the mean of the sample CCFs with lags

$$ -1\le \tau \le -3 $$

were estimated for each interval.

Fig. 12.

Mean sample CCF for the activity-based interaction.

From the analysis shown in Fig. 12, we found no statistically significant pattern of activity synchrony between caregiver and their counterparts in the “with activity-based ambient light” interaction style. In some instances, we observed a significant positive correlation (indicating interpersonal activity synchrony) as in the case of the participant pair KW, but there was no other significant positive or negative correlations among the remaining cases. Consequently, these findings need to be interpreted with caution as we are unable to make any significant assertions regarding interpersonal activity synchrony on the basis that there was also no consistent sample CCFs within and between interaction styles.

Table 2. Percentage of time spent in activity levels

Notwithstanding the lack of synchrony, we observed that in the activity-based ambient light interaction, counterparts were in most cases as equally or more active than their caregivers whilst counterparts were frequently observed to be less active than their caregivers during the interaction with white light. Table 2 portrays the percentages of time partners spent in each activity level per interaction style. While Figs. 13 and 14 demonstrate the mean activity levels with white light and with activity-based ambient light.

Fig. 13.

Mean activity levels in the interaction with white light.

Fig. 14.

Mean activity levels in the interaction with activity-based ambient light.

This finding together with estimations of the influence of the receipt of caregivers’ activity information on their counterparts points to the possibility of interpersonal activity synchrony in long-term deployments of the system.

5.8 Discussion

In this experiment, we aspired to investigate how the exchange of activity information between two user groups (caregivers and their counterparts) would affect the following: interpersonal synchrony on the part of the counterpart, interpersonal relationship closeness, co-presence, behavioural interdependence, information clarity, attentional engagement, system adoption, and interpersonal activity synchrony. The results have further strengthened our confidence that the “with activity-based ambient light” interaction style was clearly more effective for affecting the social connectedness experience in the case of our experiment. The subjects reported increased sensations of relational closeness, co-presence, co-action with their caregiver partner in the case of the elderly, and information clarity during their interactions with the activity-based ambient display. Moreover, the idea of usable everyday objects for the bidirectional exchange of activity information was supported by a large number of participants. In addition, the findings on attentional allocation are in accordance with our intended goal to facilitate perception at a glance thereby facilitating divided attention [46]. Regrettably, evidence of interpersonal activity synchrony was significantly weaker than anticipated in this short-term deployment. However, it is worthwhile to note that the assessment of interpersonal activity synchrony in mediated environments is not trivial. In fact, in [63] Rashidi et al. reminds us of the difficulty in measuring ADLs based on the assertion that the sequence and the way in which activities are performed may vary across individuals. From this claim, it is clear that this assumption not only holds true for the recognition of ADLs in general, but also for computing interpersonal activity synchrony using peripheral displays in AAL environments. Likewise, [30] also articulated their uncertainties regarding the extent to which synchrony can occur in mediated environments. Although there were spontaneous instances of interpersonal activity synchrony as clarified in our qualitative analysis, we believe that 30 min was not enough to significantly affect interpersonal activity synchrony in mediated domains. Further work will focus on longer deployments to estimate the effects on interpersonal activity synchrony in mediated AAL environments. We will now present the qualitative findings, which was done to ascertain a subjective viewpoint and corroborate our findings on the aforementioned dependent variables ((i) interpersonal activity synchrony (on the part of the counterpart), (ii) context-awareness, (iii) social connectedness, (iv) social presence (behavioural interdependence i.e., the counterpart’s synchronized actions with the caregiver), (v) information clarity, (vi) attentional engagement, and (vii) system adoption) examined in the quantitative research.

6 Qualitative Results

Our analytical approach bears close resemblance to the procedure proposed by [73] such that interview transcripts were analysed and the findings were discussed and validated with a professional care support worker. Important ideas and suggestions provided by the domain expert were taken into account during the discussions. Exploiting the thematic analysis [9] approach, two hundred and ninety-four statements were examined to identify major themes and sub-themes related to the users’ impressions on perception, usability, system adoption, interpersonal activity synchrony, and envisioned system benefits among others. These themes and sub-themes are now discussed.

6.1 Perceived Usefulness of Bidirectional Activity-Based Displays for Promoting Context-Awareness

The participant majority praised the system for its ability to raise context-awareness. In particular, most interviewees reported on the system’s ability to trigger context-awareness owing to the following properties.

• peripheral features enabling divided attention

• information clarity

• respects privacy and dignity rights

• simplicity and effortlessness

• portability and multifunctional everyday objects

• implicit communication channel

From the quotations below, further insight can be gleaned on the implications of bidirectional for promoting social connectedness.

“I only looked at the light for a few minutes, and then I just started focusing on what I was doing while taking occasional glances.” – Q

“It is rather clear what he is doing.” – D

“I don’t feel like someone is looking around for me. It is simple.” – W

“The ability to carry the wallet around is good so you can see the activities of your partner whilst you are outside and when you in the house you can see the lamp.” – A

“The cane is so cool, I can carry it around, and it serves two purposes one as a light and the other to access information anywhere. ” – M

“We sit in the same office and to know what he is doing I would have to look at him, and then I can see that he is working on his computer. Now, the lamp is a bit more discrete and is a good indicator of his activities.” – F

“It would be nice to see what they are doing without FaceTime or taking too much time to feel their existence.” – C

6.2 Uncertainty

Although a large number of participants acknowledged the potential benefits of the system, yet, there were a few elderly participants who consistently expressed uncertainty regarding ambient technologies for social connectedness. During the investigation, it was apparent that one participant appeared to be technology-illiterate e.g., he expressed his disdain for assistive devices due to technical inexperience). In addition, the significance of culture on adoption played an important role in the level of uncertainty and ultimately another participant’s disapproval of peripheral technologies.

“I never use a computer, in fact, I don’t know how to use it. I don’t even have a smartphone. I know there are technologies to call the doctor if you need help but I would never use them, I would rather use the telephone.” – X

It is also evident that culture played a significant role in the adoption of peripheral technologies. In fact, some Dutch participants reflected response patterns that were highly individualistic⁴ in nature.

“I don’t need to know what another person is doing every moment of the day.” – X

“Okay, I see different lights showing me my partner’s activities. Then, I didn’t know what to do with it. What are the implications? Why do I need it? If my mother were alive then maybe when she was ill it would have been useful, but I don’t need it now to keep in touch with my friend.” – W

“It is not so important for me maybe there are positive effects but not for me. Most of the time, my wife and I we leave each other free, so I don’t need it.” – V

Moreover, another respondent perceived that the system could easily disappear in the background, which he thought to be negative especially for both context-awareness and social connectedness.

“I saw different lights, but it could be the same as having the television on and it fades in the background. If there are changes, then I wouldn’t notice them and I wouldn’t feel anything.” – U

6.3 Role of Perceptual Processes on the Experience of Social Connectedness and Context-Awareness

Based on the responses we can infer a possible link between context-awareness and social connectedness, which is exemplified by Mr. A’s comment.

“Perceiving your partner’s activity information makes you feel like you know their daily routines so you can form a mental pattern of what they do overtime and that can make you feel connected.” – A

Also, from Mr. A’s statement, we can deduce the relevance of cognitive processes discussed in [21] (e.g., attention, perception, pattern recognition, and memory) as key concepts essential for facilitating context-awareness and social connectedness. Furthermore, it is imperative that the activity information received from the display is aligned with the user’s mental model of their counterpart’s activities. This is reflective of top-down processing as discussed by the authors in [31].

Moreover, from Mr. M’s comment, we reckon the significance of habituation (cf. [36, 85]) for social interaction in mediated environments. This is reflected in the following quotation.

“I think if I get accustomed to observing someone else’s activity then over time I would feel even more connected.” – M

6.4 Interactivity and Social Influence

The opportunity to exchange activity information without communication media such as Skype, FaceTime, or text messaging was highly valued among younger participants. A possible explanation for their acceptance can be attributed to multiple references to separation by geographical distance from their parents. In general, a great deal of social presence was experienced between younger interaction partners coupled with sporadic occurrences of interpersonal activity synchrony between them. Furthermore, respondents elaborated on the potential social influences of bidirectional activity-based ambient systems and highlighted the effects on engagement by virtue of the cryptic nature of the display.

Social Presence. Like the respondents in [25], most younger participants were very passionate about the system’s indirect influence on social presence and by extension social connectedness. Example statements are given below.

“I liked the fact that although we were in different places, I still felt like she was quite close to me. I knew what she was doing and I was wondering what she thought about my activities. I am quite anxious for us to discuss our activities later.” – B

“I feel like she is somehow with me indirectly.” – O

“With ambient light even though I was alone, I didn’t feel alone. I think this will be useful for lonely people.” – I

Fig. 15.

Pictorial representation of interpersonal activity synchrony, the counterpart is observed in a resting state while is his caregiver is also in a resting state as depicted by the blue light. This snapshot was captured during the experiment. (Color figure online)

Interpersonal Activity Synchrony. Most of the younger participants were captivated by the possibility of synchronizing their activities with their partner. Furthermore, the participant majority suggested that the exchange of activity levels could create intimacy and increase social interaction. This is encapsulated in the statement below.

“It is nice to see what the other person is doing and that perhaps you can do the same things together to form some kind of bond.” – R

An interesting observation reflecting interpersonal activity synchrony of two interacting partners is demonstrated below.

In one instance, the caregiver stated the following. “There were times I had the feeling that he was doing what I was doing because when I was doing physical exercise, his light was also red.” – G

While her counterpart mentioned,

“I had the impression that she was mirroring me especially when I was resting she was resting.” – S

This interaction is evidenced in Fig. 15.

Social Influence: Persuasion Versus Peer Pressure. Although synchrony appears to be intriguing, some participants argued that it could potentially have positive and negative effects on social interaction. Positive influences include the system’s functional role in persuading its users to engage in the same activities. An example statement is given below.

“When I saw that my partner was active it made me feel like I should have been active as well. Also, while I was exercising and she was relaxing I felt like I wanted to relax as well.” – C

On the other hand, a few participants stated that the system appeared to have adverse consequences resulting in social pressure to prevent embarrassment. In one instance, a participant mentioned that she was uncertain as to whether or not she should coordinate her activities with her partner. This is shown below.

“There was a moment when I was sitting because I already finished exercising and I was going to read, but then she was engaged in a physical activity maybe exercising. I didn’t want her to feel like I wasn’t doing anything. I felt a bit embarrassed. She was doing something productive, and I was just there sitting. That’s not good for my reputation.” – O

While another respondent was bothered by the system’s persuasive effects as an implicit trigger point for stress.

“For me, my mom always wants me to exercise and also my dad is trying to lose weight. So, if we are both home and my dad is exercising, then it could influence me to exercise also. But this could be silently stressful because I can see my father is exercising and I am either sleeping or eating a hamburger or watching TV. Then, I could feel a bit stressed.” – N

Mysterious Engagements. As highlighted in [25], some participants expressed a liking for the system’s mysterious effects, which prompted them to mentally decrypt the exact nature of their partner’s activities. This is reflected in the following statement.

“Sometimes, I was guessing what my partner was doing. In some instances, I knew she was doing some kind of mental activity but I didn’t know exactly what she was doing. I would say it was a bit mysterious.” – Q

The respondent further argued that the system’s mysterious effects could stimulate communication through other communication media.

“For example, if I am alone at home and I am trying to figure out what my partner is doing based on the information received. Then, I may initiate further communication by calling her on Skype to determine what exactly she is doing.” – Q

6.5 Relevance to the Frail Elderly – I’m Still Young I Don’t Need It Now

Like the elderly respondents in [23], most elderly participants in this study commented on the relevance of the context-awareness systems for the frail elderly. These comments illustrate the tendency among our older participants to still feel young inside [2] by articulating their independence and stating how they demystified ageist stereotypes, e.g., ill-health, cognitive decline, feeling sad or lonely, and the lack of vigour or vitality discussed by the authors in [77]. Example statements are presented below.

“My wife and I are very active, so we don’t need it now maybe when we are older.” – D

“I am alright, I am very capable of taking care of myself at home.” – H

6.6 Risks and Emergency Management

As pointed out earlier, the majority of our younger participants were excited about the social connectedness benefits of the system. However, some participants were more focussed on the context-awareness features mainly for its potential in supporting the safety and monitoring of their elderly loved ones. One elderly participant was readily accepting of such systems because of her husband’s current battle with dementia. As Mrs. T reflected on her husband’s dementia, she stated the following.

“With lighting colour changes, I can easily observe my husband’s activities while he is in another room without being present with him all the time.” – T

Also, others mentioned the need for such systems for anomaly detection to identify irregular movement patterns of their elderly loved ones. These accounts are discussed below.

“If my relative is sick then is important to know if she is not moving at all.” – I

“I want to know if something goes wrong” – N

Although the bidirectional activity-based ambient displays were designed to provide context-awareness and enhance social connectedness, some participants suggested that it is still necessary to provide emergency detection capabilities to complement the existing system. Additionally, a few participants suggested the need for an alarm feature for notifying caregivers in the event of an emergency.

“How can you distinguish between a person sleeping or an accident where someone has fallen on the floor? I think there is need of an extra indication for falling.” – D

Also, Mr. K suggested that by introducing additional physiological measurements such as heart rate along with an alarm system could assist professional care workers.

“A supervision system for a nurse monitoring several people could detect heart rhythm and send an alarm if something is wrong.” – K

Furthermore, Mrs. T pointed out that an alarm system could assist with the monitoring of her husband with dementia who tends to wander off outdoors.

“What if my husband wakes up from his sleep and starts moving? What if he wanders off outdoors? Maybe the system could signal an alarm once the front door is opened or illuminate all the colours at once to indicate some form of danger.” – T

6.7 Design Suggestions

Overall, the design suggestions include ideas to offer more subtlety, humanize the display, improve aesthetics, battery life and sensor comfort, the addition of ancillary features such as vibration and sound, reduced sensitivity, and an extension of the system’s scope.

Support Invisible Design. Going back to Weiser’s vision of calm technology [86] (“those that disappear [...] They weave themselves into the fabric of everyday life until they are indistinguishable from it” (p. 1). A few participants made recommendations to improve the subtlety of the design. The following comments suggest how this can be achieved through simplicity, smaller LEDs, and reduced brightness for portable displays.

“Although the wallet is useful and attractive the light is quite obvious. Let’s say you have to pay with the wallet then everyone says hey it’s Christmas time! Therefore, a much simpler LED would be sufficient.” – K

“Is it necessary to have such a long stick to receive information? Is it possible to have something smaller? I think that would be better.” – W

“The cane’s LED could become irritating. Maybe, it’s because I really don’t like LED strips it’s a personal thing.” – P

“The light on the wallet is very strong maybe something less bright and smaller.” – D

More Explicit Communication Features. Although most participants were enthralled by the implicit communication characteristics of the light, there were two exceptions. In fact, these participants expressed interest in more explicit interpersonal communication features. This is apparent in the quotations below.

“When you are in the same room with a person, and you feel like you want to talk you can just talk to them. But in two different rooms, you cannot talk to the lights.” – G

“Maybe, we can interact not only by changing activity states with the lights but also exchanging messages saying now let’s get active.” – C

Improve the Battery Life. Interestingly, one participant observed the battery limitations of the LED wallet. This is depicted below.

“I think it’s a good system however the lifespan of the LED battery is short.” – A

Recommendations for maximizing the performance of the battery life (e.g., exploiting devices that work at 1.8 V) were discussed in (IWANN).

Colour. Like the experimental results in [21, 23, 25, 26], various participants desired the freedom of colour choice based on personal preference. Moreover, a few respondents were more in favour of exploiting green for resting and blue for passive. While other younger participants were cognizant of the implicit association of red with danger and a few expressed disturbance and restlessness with the colour red. As such, warmer colours such as orange were proposed as a replacement for red. Example statements are highlighted below.

“Intuitively, I would use green for a state of calmness and blue for mental activity.” – R

“For physical activity, I would use orange or yellow, something warm.” – C

Position of the Smartphone. Even though all older participants expressed their satisfaction with the waist-mounted smartphone, there were a few younger participants who expressed their discomfort. In hindsight, these participants expressed discomfort during physical activities and one participant described her overall experience with the smartphone sensor as “burdensome”. To rid themselves of the excess baggage, they proposed the following.

“The smartphone was a bit heavy. If it’s on my personal smartphone it’s okay, but if I have to carry an extra smartphone it might be too much.” – F

“The smartphone could be in the pocket to prevent discomfort during exercise.” – B

Vibration/Sound Effects. Although most participants were pleased with the peripheral nature of the system, a few were critical on the system’s ability to sustain awareness during high periods of concentration. Accordingly, they prescribed additional sound or vibration effects to alert the user’s attention and in some cases minimize the cognitive load. These recommendations are illustrated below.

“Maybe, add some vibration because when we are doing a mental activity we tend to focus and vibration could make us more alert.” – E

“Maybe, I would add sound effects so that I wouldn’t have to always look at the light.” – L

Exploitation of Additional Everyday Objects. Although almost all the informants were positive toward our design choice of exploiting a cane and wallet, there were two respondents who suggested other everyday objects such as an ambient smartphone or an ambient id/key card. Their propositions are encapsulated within the following comments.

“You can use something that’s more portable something like a mobile phone. Maybe you can use the Philips Ambilight TV as a reference.” – R

“In the context of a caregiver, I wouldn’t check my wallet all the time. They always carry an ID or a key card so some indication on those objects could be better.” – C

Expanding the System Scope. A few participants were desirous of knowing the strength of the activity level, which could be illustrated with additional colours or changes in light intensity.

“I would increase the brightness based on the intensity of the activity.” – W

However, one participant articulated her preference for only two activity levels namely (i) active or (ii) inactive to reduce any misconceptions of an intermediate activity level. Recall that a similar abstraction is implemented in [25]. Her citation is recorded below.

“Sometimes I forgot the meaning of the green and wondered whether they were engaged in mental activities or not. I think it would be better to have active or inactive states.” – C

Remarkably, the temporal nature of activity information (cf. [21, 26]) was reiterated by a young male informant when he voiced the following.

“It would be nice if I could see a summary of the data so I can see what happened in the past.” – S

Moreover, one responded urged for an expansion of the system to support self-tracking.

“It’s an interesting concept. However, I am more interested in knowing how I react when I am reading or sleeping or exercising. This would give me personal biofeedback.” – B

6.8 Design Considerations for Bidirectional Ambient Displays for AAL

There were some key factors that emerged during the discussions with our participants, which include the following.

• Privacy and Ethics

• Context of Use

• Spatial Position

• Aesthetics

Privacy and Ethics. Generally participants were satisfied with the level of privacy offered by the system. Example accounts are given below.

“You have a feeling of connectivity indicating what the partner is doing without disturbing him with camera supervision. So, everyone is free to do what he or she wants while there is still a feeling that there is life, to say the least.” – K

“It gives a good indication of what the other is doing. It is simple, and there is a certain privacy it provides. You don’t feel observed.” – P

Still, a few participants were fundamentally concerned with the potential privacy risks of ambient technologies. For example, Mrs. H remarked on the ‘big brother is watching you’ effect of the deployment of context-aware technologies in AAL environments.

“I won’t like it if I lost my independence and someone can see if I feel okay or not. I would like to maintain my privacy as it’s my right not to be okay. Someone else doesn’t need to know. For me, it would feel like a ‘big brother is watching me’. No, I wouldn’t want to be constantly monitored so that someone can see how I feel. No, I don’t like that.” – H

Moreover, even though some participants were well aware of the privacy risks they were more willing to trade privacy for security. For example, Ms. O argued in the following statement.

“It’s kind of uncomfortable for me to know that my mother always knows what I am doing right now, but for both of us to determine if we are in a ‘safe’ state then this system is very good. We are two far away [...] I want to her to know that I am okay.” – O

Context and Purpose. From the commentaries, we observed that a few younger adults highlighted that the context and purpose of the system could affect adoption. Importantly, one young person stated that the system was only relevant for context-awareness only if her elderly relative was ill. Otherwise, it could be distracting.

“It depends on the situation if my relative is sick, then I will use it. But if I don’t need to know what she is doing then it would be disturbing for my own life. So, the purpose is important.” – I

With reference to situational context, another young person mentioned its relevance only in the home.

“Also, context is important if I am at home and they are at home then possibly it is okay. If I am at work and they are at work, then I don’t need to know what they are doing. What’s important is that they are okay.” – N

However, in the home context, the user further expressed privacy concerns in the following statement.

“The thing is sometimes I sleep late and I wouldn’t want them to know that. In truth, there are some things that I need to hide. I wouldn’t want them to call and say why are you sleeping so late?” – N

To address privacy and situational context concerns, one participant suggested a service upon request functionality to maintain the right to control, access, and disseminate activity information at his convenience.

“I would use the lamp when it’s a service on request so I should be able to control the functionality. It’s a personal system so it should be visible to others only if I want to show them.” – S

Reverting to N’s reference on the importance of situational context, she also mentioned that consideration should be given to the time zones of two interacting partners for successful adoption.

“For me, I need to consider the time zones because sometimes when they are sleeping I am active and vice versa. Sometimes it would be disturbing for them.” – N

Thus, by extension, we believe that the time-zones can affect the degree of synchrony between two interaction partners.

Spatial Position and the Stability of Social Bonds. From the remarks, we see that spatial position can change how the information is perceived and the degree of experienced social connectedness.

“In a real life situation, the positioning of the light in the room would be extremely important.” – P

“I didn’t really feel the connection with the light maybe because of the location of the lamp.” – E

Besides, both P and E shared similar perspectives that perception and social connectedness are not only determined by the spatial position but also the stability of the emotional connection, which serves as a motive for observing the display consequently affecting how deeply the information is processed.

“In fact, I think the real connection outside the experiment will influence the results. If I don’t have a good relationship with the partner, then I won’t feel anything.” – E

“If there is an emotional connection between the person in the other room or the person that you are taking care of. Then, there is a positive motivation to look at the lights.” – P

Aesthetics. In a general sense, aesthetics was a major perceived benefit of the installation of the ambient displays. Thus, in designing bidirectional ambient technologies consideration must be given to the aesthetic needs of the participants. In retrospect, the participants postulated that the light’s aesthetic properties created a pleasant atmosphere, fostered creative thinking through its mysterious effects, and led to elements of surprise, and more fun and playful interactions. Example remarks are demonstrated below. Figure 16 demonstrates a participant’s interaction with the cane.

Fig. 16.

Photo demonstrating a participant’s interaction with the cane.

“I think the cane is an eye-catcher for the elderly. I think it’s is nice and I like the fact that it surprises me.” – S

“It was very fun and playful! You can use it for special activities in the home.” – T

Also, C contends the prescriptive interpretation of Sullivan’s notion of ‘form follows function’ [75] as she suggests that form is attuned to function in the statement below.

“It indicates the partner’s activities and these colours add a certain ambiance to the room.” – C

However, Lidwell et al. [48] assert that the prescriptive interpretation of ‘form follows function’ “aesthetic considerations in design should be secondary to functional considerations” (p. 106).

7 General Discussion

Overall, our participants identified several aspects that they found positive about the bidirectional activity-based ambient displays. Most participants could multi-task, feel a sense of their partner’s presence, access the activity information any and everywhere, understand the information received, enjoy an implicitly shared interaction, coordinate their activities to some extent, and maintain their privacy. Altogether, we can deduce from our findings that the process of experienced context-awareness and social connectedness among our participants included five phases: (i) visual perception, (ii) attention, (iii) memory, (iv) curiosity, and (v) habituation. Subsequently, the bidirectional exchange of activity information may consciously or unconsciously affect behavioural responses as depicted by the periodic accounts of interpersonal activity synchrony within this study. These irregular instances of coordinated actions could spark interest for further inquiry on the possibility of interpersonal activity synchrony in mediated AAL environments.

On the negative side, a few persons desired increased sensor comfort, more discreet portable displays while some felt that ambient technologies were an invasion of their personal privacy. To address privacy concerns, one informant suggested the addition of a “service upon request feature.” Likewise, Hoof et al. [41] recommended that the user has complete control over his information collected and distributed in smart home environments.

On a different note, the most striking result to emerge from the discussion was the consistent reference to safety and monitoring systems. In fact, this was not surprising as the sense of safety and security in AAL environments has been a recurring theme throughout this doctoral research. A possible interpretation for this recurrence can be found in Maslow’s hierarchy of needs, such that safety and family security needs precede the need for love and belonging [53]. Accordingly, we can infer that once our participants can guarantee the safety of their loved ones, then they can proceed to other forms of interaction to create a sense of belongingness in mediated AAL environments. As such, our design challenge has now become greater given the system scope has stretched beyond the main goal of promoting social connectedness through bidirectional ambient displays.

Going back to Mr. A’s statement regarding a mental pattern of the partner’s routine activities, it is clear that participants refer to their mental model as a reference for understanding their partner’s activities. Consequently, this raises the challenge of designing peripheral technologies, which are coherent with the user’s mental model. Norman suggests that misfortune could arise if the ‘system image’ is incoherent with the user’s conceptual model [60]. Thus, the information portrayed should match with the user’s ideology of their partner’s activities. To address this, one could deploy highly accurate machine learning classification algorithms. However, system trust is critical for determining the match between the information presented and the user’s conceptual model. Also, if there is no system trust then challenges with learnability and usability could emerge.

From our findings, technical literacy and cultural values can shape the users’ experience of interacting with the system. Recall that our bidirectional activity-based system exploits ambient technologies and IoT to create awareness and maintain social connectedness between two interaction partners in AAL. Thus, Demiris et al. [29] highlight that inadequate technical literacy could impede the process “because the discussion of security and privacy concerns or issues of accuracy and reliability of sensor systems or other computing applications often require basic understanding of networking and data transfer” (p. 110). Thus, driving the need for technological literacy interventions in AAL.

8 Conclusion and Limitations

To strengthen our assessment of the behavioural implications of bidirectional activity-based displays, this chapter provides a background on interpersonal activity synchrony. Based on the knowledge acquired from prior works, it was possible to evaluate interpersonal activity synchrony by computing the cross-correlation coefficient of the counterpart’s activity levels with that of their caregiver’s. The results of a semi-controlled study suggest higher incidents of subjective interpersonal relationship closeness, experienced social presence, behavioural interdependence (for the counterpart only), information clarity, and the participants’ willingness to adopt the technology, while utilizing minimum attentional resources with the activity-based ambient light interaction style. However, there was hardly any occurrence of interpersonal activity synchrony by using the cross-correlation approach. Nonetheless, during the post-trial interview, a few participants reported sporadic moments of synchrony during their interaction with the activity-based ambient light. Furthermore, in the said interaction style counterparts demonstrated increased tendencies to remain active in contrast to their interaction with white light.

It is plausible that some limitations could have influenced the results of this study. To begin with, we acknowledge convenience sampling as a constraint of this work. Accordingly, the findings are not entirely representative of all users within AAL community. To heighten the interest in our system, one option for future work is to specify the inclusion criteria only for the frail elderly, e.g., those with (Parkinson’s disease, Alzheimer’s disease, or even users with epilepsy). This we know would reduce the population of our study. On the other hand, it could increase the interest in our system.

We are aware that a larger data stream of activity data is necessary to better estimate interpersonal activity synchrony in mediated environments. This can be achieved by increasing the number of participants and deploying a significantly longer experiment in the users’ natural environments.

Unfortunately, the self-awareness of the wearable smart-phone sensor from [25] is still an open problem that will be addressed in future work. Notably, if our algorithms were independent to orientation and location, it could be one of the best contributions in the field of activity recognition for AAL. There are some attempts, but are very limited.