1 Introduction

According to a recent report by the UK office of communications ([29] p. 39), short-form media (images and short video clips) are increasingly being consumed on small-screened devices (tablets/mobile phones), while long-form media (long videos and films) are still mainly consumed on large-screened devices (laptops/TVs, etc.). Hou et al. [20] noted that this consumer preference to utilise larger screens for long-form media may be as a result of larger screens being more immersive than smaller screens [20]. There is still, however, the need to improve the suitability of both large- and smaller-screened devices for electronic visual media consumption (long- and short-form media). For example, with the opening of Europe’s first 4D cinema in Milton Keynes, UK, in February 2015, for example, big-screened cinemas have continued to explore new ways of increasing media immersion by utilising more of the human sensory receptors [25]. Research has also been ongoing into how smaller-screened devices can be made more suitable for the consumption of media, since a larger screen is not always available or suitable for use (e.g. when users are mobile) [27]. Augmenting images presented on these devices (large- and smaller-screened devices) with thermal stimuli, which have been shown to be an affective feedback channel, has the potential to enhance media viewing experience. Previous research has shown thermal stimuli as having the ability to influence affect in images [16, 27, 28], it is, however, not known whether it maintains this affective attributes when used to augment different-sized images presented on different devices.

Recent research on the subjective interpretation of different thermal stimuli combinations (amount of change, rate of change, etc.) has shown thermal stimulation as having the inherent ability to evoke affect when presented on its own [19, 33]. Using thermal stimuli for media (sounds and images) augmentation has also been shown to have a significant effect on emotions [1, 2, 16, 27, 28]. Augmenting images on smaller-screened mobile devices with thermal stimuli could therefore potentially improve how engaging they are, thus resulting in an improved media experience.

Existing research has mainly examined the effect different thermal stimuli combinations have on images with specific emotional properties. Some of the possibilities that can be afforded by using thermal stimuli to augment media include the potential for movie producers to enhance or reduce affect in media through thermal stimuli augmentation [1]. To the best of our knowledge, no research has been conducted to investigate whether there is a difference in the effect thermal stimulation has on images displayed in different sizes. Therefore, it is not known whether the affective attributes of thermal stimuli on images documented in previous work [1, 16, 28] will hold true for images of different display sizes, presented on different device types.

This paper presents two user studies that extends previous research [1, 16, 28] by addressing this research gap. This is achieved by investigating the effect different thermal stimuli parameters have on the emotional perception of different-sized images viewed on different device types. This is with the aim of determining whether thermal stimulation maintains its affective attributes when used to augment different-sized images presented on different devices. In study 1, we examined the effect thermal stimuli have on the emotional perception of images of different sizes. Images were displayed in small, medium and large sizes, and we compared differences in the effect thermal augmentation had on each size. In study 2, we investigated the role presentation device (tablet and mobile phone) plays in the emotional perception of images presented on them. The images augmented with thermal stimuli in user studies 1 and 2 were chosen from the International Affective Picture System (IAPS) image library [26]. The images spanned different emotion categories (e.g. high arousal/high valence) and were rated using affective rating scales.

In the next two sections, we review related work and provide an overview of the evaluation tools and equipment used for the study. We then present the procedure for study 1 and results obtained. This is followed by a description of user study 2 and results. Finally we present a discussion of our findings and some conclusions.

2 Related research

Thermal stimulation in HCI is made up of different properties such as intensity and direction of change, which all contribute to how it is perceived by people. Research in this field has frequently examined the use of thermal stimuli as a feedback channel [35, 37, 39] and the emotive properties of thermal when presented on its own [33, 38] or when used to augment media [1, 2, 16, 27, 28].

2.1 Thermal stimuli properties that impact perception

Thermal stimuli parameters that have an effect perception are stimuli intensity, stimuli direction of change and stimuli rate of change (ROC). Research by Halvey et al. [17] showed that thermal stimuli detections within the range of 26 and 38 °C increased as stimuli intensity increased, with higher stimuli intensities rated as being less comfortable. According to Wilson et al. [39], people are able to detect distinct thermal stimuli changes within the temperature range of 26 and 38 °C. For thermal stimuli within that range to be perceivable, it needs to change from a neutral point of 32 ± 4 °C at a minimum rate of change of 3 °C/s. Jones et al. [21] found faster ROCs to be perceptually different from slower ROCs, with 1 °C/s ROCs shown to be perceivable and 3 °C/s ROCs shown to have better detection rates [39]. Akazue et al. [1], however, found that a faster ROC of 3 °C/s has similar effect on the emotional perception of images as a slower ROC of 1 °C/s. Investigations conducted by Wilson et al. [39] on how well users can detect hot or cold stimuli found that both warming and cooling temperatures were perceptually different with warm stimuli perceived as less comfortable and more intense than cool stimuli.

2.2 External factors that affect thermal stimuli perception

In addition to thermal stimuli parameters capable of affecting thermal stimuli perception, there are external factors, which also contribute to how well thermal stimuli are perceived. Extreme ambient temperature and humidity conditions (humidity <30% and >90%) have been shown to have a significant effect on detection rate, time to detection and subjective perceived comfort of thermal stimuli [18, 36]. According to [19], in the presence of clothing, perceived thermal stimuli feel more comfortable than when applied directly to the skin; however, higher thermal stimuli intensities are needed for detection to take place. Existing skin temperature of a perceiver and thermal stimuli delivery location are other external factors capable of affecting perception. Existing skin temperature is generally determined by the position; the skin temperature falls within the neutral zone (~28 and 40 °C) [21]. As a result of the variance in possible existing skin temperature, there is the need to adapt the skin to a neutral stimuli before thermal stimuli presentation, in order to maximise perception [24]. Salminem et al. [33] used an adapted neutral starting temperature by first measuring the skin temperature of participants with an IR device before presenting them with thermal stimulation. Their procedure, however, involved participants removing their hand from one device to the other, during which time the resting temperature of the skin can change due to the effect of ambient temperature and humidity [18]. According to [19, 39], the thenar eminence (the bulbous part of the palm, directly below the thumb) is the optimal location for thermal stimuli delivery. Other body locations such as the waist or non-glabrous arm locations are also suitable, however, with lower detection rates. Kenshalo [23] and Gray et al. [15] in their investigation of thermal perception discovered that thermal stimuli are perceivable by both young and older humans, with perception only reducing with age (between ages 55 and 84) when thermal stimulation is presented to the feet. For individuals with higher than normal stimuli thresholds, Gray et al. [15] showed that they compensate by having a relatively increased neural activity, thus balancing out their perception of thermal stimuli.

2.3 The modulating effect of thermal stimuli on emotions

Previous work investigating the affective attribute of thermal stimuli direction of change showed it has the ability to affect subjective ratings of pleasantness [1, 22, 34, 38], arousal and dominance [33]. Research by Wilson et al. [38] investigating the mapping of thermal stimuli to models of emotion showed that emotions can be conveyed through thermal stimulation. The thermal stimuli delivery location of participants in their study was adapted to a neutral temperature of 30 °C for 30 s, and the different thermal combinations used were presented for 10 s. They found warm stimuli were perceived as more pleasant/positive than cool stimuli and altering the rate or amount of temperature change affected both valence and arousal [38]. Salminem et al. [33] in a different study that presented dynamic thermal stimuli to perceivers for 8 s showed that direction of change has little to no effect on ratings of pleasantness (valence) and approachability. Their results showed that warm stimuli only significantly differed from cool, for ratings of arousal and dominance. Also, warm stimuli were rated as being more arousing and dominant than cool stimuli.

When thermal stimulation is used to augment physical artefacts, it has the ability to enhance interactions. Gooch et al. [14] showed that thermal stimuli increased feelings of social presence in remote interactions when embedded in a physical artefact. Using thermal stimuli to augment media presents other interesting affective possibilities. According to Nakashige et al. [28], images of warm food accompanied with warm stimuli felt more delicious to participants in their study. Results by Halvey et al. [16] also showed that warm stimuli elevated feelings of arousal and valence in music and images. While this is a good indication of the ability of thermal stimuli to modulate affect, the effect reported in their study was limited to specific images. Akiyama et al. [2] found that dynamic thermal stimuli augmentation has the potential to enhance emotional experience while listening to music. Only one song was used in their study; therefore, a larger study would be needed to determine the effect thermal stimulation has on music.

In a more recent study, Akazue et al. [1] showed that the affective attributes of thermal stimuli were very effective in modulating the amount of affect evoked by images presented on a laptop screen. They also found that depending on when thermal augmentation is presented (before or during image presentation), it can either be used to create anticipation or enhance the inherent emotion an image is capable of evoking. With the increasing popularity of media consumption on mobile devices, augmenting media consumed on them with thermal stimuli has the potential to create a more enjoyable media experience. However, little work has been done to investigate whether the affective attributes of thermal stimulations on media hold true for images viewed on mobile devices. Lochtefeld et al. [27] evaluated the effect of ambient feedback (ambient light and thermal stimulation) on images viewed on a tablet and showed that thermal stimuli contributed to enriching the user experience for mobile media consumption. Although their findings [27] highlight the affective possibility that can be afforded by augmenting images on a mobile device with thermal stimuli, the effect they reported was not entirely due to thermal augmentation. Also the handpicked images they augmented had no pre-identified emotional property.

2.4 Evaluating emotional responses

The Russell circumplex [31] and vector [8, 9] models of emotion are tools that have very commonly been used to evaluate emotional experiences [1, 4, 10, 16, 38]. They have both been used as a tool for mapping the emotions that can be evoked by sounds [6, 11], images [26] and varying combinations of thermal stimuli [38]. Variations of both the Russell circumplex and vector models called the emotion wheel [5] and self-assessment manikin (SAM) [10], respectively, were used as assessment tools in our studies (see Figs. 1, 2). According to the vector model, valence, arousal and dominance emotional dimensions are not continuously related. Research by Wilson et al. [38] showed that varying thermal stimuli combinations are mainly interpreted as representing calm, pleasant and excited, unpleasant emotions on the circumplex model. Their study was designed to understand the range of emotions that might be conveyed through thermal feedback presented on its own. Participants were presented with randomly selected thermal stimuli combinations and asked to provide valence and arousal ratings on vector rating scales. The scales were 7-point sliders with two anchor labels at either end (low/high for arousal, unpleasant/pleasant for valence), and no numerical values. The subjective ratings obtained using the vector scales were then mapped to the circumplex model to understand the full range of emotions thermal feedback presented on its own can evoke. Their results showed that though vector model better explains how emotions relate to perceived valence and arousal, it does not entirely do so [38]. Therefore, in the two user studies we decided to use assessment tools based on both the Russell Circumplex and vector models to evaluate emotional responses to augmented images.

Fig. 1
figure 1

Emotion wheel

Full size image
Fig. 2
figure 2

Self-assesment manikin (SAM)

Full size image

The emotion wheel [5], derived from Russell Circumplex model, is a 0°–360° cyclic ordinal scale made up of 4 quadrants. The horizontal axis ranges from low to high valence (left to right) and the vertical axis ranges from low to high arousal (bottom to top). Baillie et al. [5] found that the emotion wheel was both easy to use and very efficient in capturing users’ emotions. The other assessment tool we used was the SAM [10] which is based on the vector model of emotion. The self-assessment manikin scale is a 9-point Likert scale with human-like Fig. at each point, which helps minimise the confusion a participant may face while trying to figure out the meaning of each of the emotional dimensions being rated [10]. A ‘9’ on the valence and arousal scales corresponded to ‘negative valence’ and ‘calm’, respectively, while on the dominance scale it corresponded to ‘dominant’. This tool was used by IAPS (see Sect.  4) to evaluate the emotional perception of images contained in their library. The valence scale ranged from positive valence to negative valence; arousal from excited to calm; dominance from dominated to dominant.

3 Equipment

Two 2-cm2 peltier modules attached to heat sinks (see A and B in Fig. 3) were used as the thermal stimuli source in the two user studies we conducted (see Fig. 3).

Fig. 3
figure 3

Peltier stimulators (A), heat sink (B), microcontroller board (C), USB connection (D)

Full size image

Similar peltier stimulators have been used in previous studies investigating thermal stimuli perception [18, 38, 39]. Each peltier stimulator allows heating and cooling and is controlled via USB connected to a host PC. Both peltier stimulators allowed temperatures to be set anywhere within the range of −20 to 45 °C accurate to 0.1 °C.

4 Image categories

The International Affective Picture System (IAPS) [26] was used as the image source for the two user studies we conducted. The image library contains over 1000 colour images designed to evoke a wide range of emotions in a perceiver. The emotionally stimulating images in the library have been used in several studies of emotion and attention [1, 12, 16, 32]. Images in the library have dimensions of 1024 × 768 pixels; as a result they are displayed in landscape mode on the image presentation devices used in the 2 user studies we conducted. Each image in the IAPS library has ratings of valence, arousal and dominance, which can be used to categorise each image into different emotion categories as illustrated in Fig. 4. Highly erotic images contained in the original image library were omitted from this study as we could not know beforehand if they could embarrass or upset a participant therefore for ethical reasons we omitted these images. The same approach was adopted by Halvey et al. [16] and Akazue et al. [1]. As part of our license agreement to use the IAPS collections, we are prohibited from reproducing any image from the collection in this paper. As a result, replica images are used in this paper for illustration purposes (see Fig. 4).

Fig. 4
figure 4

IAPS image emotion categories with representative images obtained from the Internet

Full size image

Similar to the image categorisation used by Akazue et al. [1], images contained in the IAPS library were divided into 9 categories based on their valence and arousal ratings. One of the categories (low arousal, low valence) was excluded, as it contained no images (see Fig. 4). The category with the second lowest number of images (low arousal, high valence) had 11 images.

5 Study 1: the effect of thermal stimuli parameters on the emotional perception of different-sized images

The aim of the first study was to investigate the effect of thermal stimuli on the emotional perception of images displayed in different sizes. Since, electronic visual media is either consumed on large screens (laptops/TVs) or on smaller screens (tablet/phone) [29], we wanted to examine whether thermal stimulation maintains its ability to influence affect when used to augment images displayed in smaller sizes. The research questions we attempted to answer in this first user study are as follows:

RQ 1: What effect does thermal stimuli augmentation have on the emotional perception of images presented in different display sizes?

  1. 1.1.

    Can thermal stimuli be used to enhance or reduce affect in images presented in medium and small display sizes?

  2. 1.2.

    On which display size does thermal stimuli have the highest effect in increasing or reducing emotions in images?

  3. 1.3.

    What is the effect of individual thermal stimuli parameters in increasing or decreasing emotions in images viewed on different display sizes?

Previous work investigating the ability of thermal stimuli to evoke emotions in images and the effect of different thermal stimuli presentation techniques, displayed images on a laptop. We decided to adopt the same procedure in this study in order for us to be able to compare some of our findings with previous work. Dimensions of the display sizes we used corresponded to the full screen displays of a 15.6-inch laptop, 10.1-inch tablet and a 5.1-inch mobile phone, respectively, because these are the primary devices for accessing online content, including electronic visual media [30] (see Fig. 5). Large screen display had a dimension of 1366 × 768 pixels; medium display 821 × 513 pixels and small display 427 × 240 pixels. As shown in Fig. 5, all the different-sized augmented images were presented to participants on a 15.6-inch Toshiba laptop screen.

Fig. 5
figure 5

Comparison of the size difference of the images evaluated

Full size image

Similar to the procedure of [1, 16], thermal stimuli combinations were presented to the thenar eminence (the bulbous part of the palm directly below the thumb) which has been demonstrated to be the optimal hand location for thermal stimuli perception [33, 36, 39]. Other factors that influenced the choice of thermal stimuli delivery location were comfort and safety. The thenar eminence allows perceivers to easily place their hands on the thermal stimuli source and quickly remove their hand if they feel the need to. A similar approach was followed by [9–12, 31]. Each image augmented with thermal stimuli was rated on the emotion wheel [5], and in three emotional dimensions, namely valence, arousal and dominance, using the self-assessment manikin (SAM) [10] rating scale (see Sect. 2.4). Valence in the context of this study referred to the amount of pleasure/displeasure felt by a participant upon perceiving an image; arousal referred to the level of excitement/calmness, while dominance referred to how dominated or dominant participants felt upon perceiving an image.

5.1 Thermal stimuli combinations

The combinations of thermal stimuli parameters used in this study (discussed in Sect. 2.1 ‘Thermal Stimuli Properties that affects Perception’) are:

  • One neutral stimulus of 32 °C, which lies well within the defined neutral zone for the skin, at which there is no discernible thermal sensation [1, 17, 39].

  • Two directions of changes, following the approach of [1, 16, 38]. Temperatures changed from neutral, resulting in warming and cooling.

  • Two stimuli intensities of 3 and 6 °C changing from neutral, which has been shown to have significant effects on emotions evoked by images [1], giving a total of 4 intensities (26, 29, 35 and 38 °C).

  • One rate of change (ROC) of 3 °C/s, which has been shown to have similar effect on the emotional perception of images as 1 °C/s ROC [1].

The combination of parameters resulted in 5 thermal stimuli combinations, namely neutral stimuli of 32 °C (no change), warming stimuli of 35 and 38 °C, with ROCs of 3 °C/s, and cooling stimuli of 29 and 26 °C, with ROCs of 3 °C/s each. The skin of participants was adapted to the temperature prior to each thermal stimuli delivery.

5.2 Image selection

A total of 120 images were randomly selected from 8 of the image emotion areas, combined with thermal stimuli and presented to participants in different sizes.

In each experiment, a unique image was selected from each emotion category for augmentation and presentation except for image category 2, which contained only 11 images. In image category 2, an image can be selected more than once for presentation in each experiment run, but not more than twice. Each image selected from each of the 8 image emotion categories was combined with each of the 5 thermal stimuli combinations, and presented in small, medium and large display sizes (3 sizes). Presentation of the different augmented image combinations was random. A total of 120 images (15 from each category, 5 thermal stimuli × 3 image sizes) were selected from the 8 image emotion categories for augmentation and presentation.

Different-sized augmented images were displayed to participants in 4 sets of 30 images each. The decision was to divide images into sets in order to enable the participants to take a break between sets. Perception and evaluation of the images in each set lasted for approximately 12 min. Before the commencement of the study, 5 min was used to familiarise participants with what to expect from the study. A total of 6 augmented images were presented to participants in a demo. A random image augmented with 26 and 38 °C and displayed in small, medium and large sizes was presented to each participant. Total duration of the study was approximately 1 h 30 min.

5.3 Image presentation

Participants sat at a desk, which had a laptop, and peltier stimulators placed beside the keyboard facing up. At the beginning of the evaluation, the neutral stimulus was presented to the thenar eminence of participants for 1 min in order for adaptation to occur. After the adaptation, all 120 augmented images were presented in a random order, with participants taking a 5-min break after every 30 augmented images perceived. There was a 1-min adaptation time before participants resumed the evaluation after every break. Presentation and duration of display of each augmented image followed the same procedure used by Akazue et al. [1] and IAPS [26]. Each augmented image was presented on screen for 6 s, after which SAM and the emotion wheel rating scales were displayed on screen for 15 s each. While users were providing ratings on screen, thermal stimuli returned back to neutral for adaptation to occur, before the next augmented image was presented. The same 30 s adaptation time was used by [1, 18, 19, 38, 39]. Using the same evaluation timing as IAPS enabled a direct comparison of results obtained from this study with findings published by IAPS.

5.4 Participants

A total of 18 (9 males and 9 females) participants were recruited from university and personal contacts for this study. Their ages ranged from 19 to 47, with a mean age of 28. Before the commencement of the study, participants were presented with information about the study and familiarised with the procedure for the study. Once participants felt confident enough to proceed with the experiment, the study began (Fig. 6).

Fig. 6
figure 6

Example setup for participant taking part in study 1

Full size image

5.5 Variables

This study had a total of 4 independent variables: direction of change, amount of change, image category and display size. Four dependent variables (valence rating, arousal rating, dominance rating and emotion wheel rating) were analysed. Since the images used for the study already had valence, arousal and dominance ratings, the existing ratings of the images from the IAPS were compared with the ratings obtained from our 2 user studies. This comparison generated valence difference, arousal difference and dominance difference dependent variables, giving us a final total of 7 dependent variables.

6 Study 1 results

As data were not normally distributed, Friedman tests and Wilcoxon Sign Rank tests were used for analysis.

6.1 Difference in the subjective ratings for display sizes

Friedman tests of augmented images (including images augmented with the neutral stimuli) showed no effect for display size. Thermal stimuli had similar effects on all display sizes for ratings of valence (X2(2) = 0.022, p = 0.989), arousal (X2(2) = 0.018, p = 0.991) and dominance (X2(2) = 0.261, p = 0.019). A significant difference was shown for emotion wheel ratings (X2(2) = 7.976, p < 0.001), which led to a Wilcoxon pairwise comparison. This, however, showed no significant difference in the emotion wheel ratings provided for medium and large display: (Z = −2.077, p = 0.038), small and large display: (Z = −0.945, p = 0.345) and small and medium display: (Z = −0.766, p = 0.444).

6.1.1 Difference in the emotional perception of un-augmented images presented in the 3 display sizes

The absence of a significant effect for augmented images displayed in different sizes led us to analyse un-augmented images (images presented with the neutral stimuli) on their own. Friedman tests showed that un-augmented images displayed in the 3 different sizes significantly differed from each other. A significant difference was shown for valence (X2 (2) = 80.505, p < 0.001), arousal (X2 (2) = 15.724, p < 0.001) and dominance (X2(2) = 29.738, p < 0.001).

As shown in Table 1, for valence, medium (Z = −7.227, p < 0.001)- and large-sized images (Z = −3.016, p < 0.001) felt more pleasant than small-sized images, while medium images also significantly felt more pleasant than large-sized images (Z = −6.388, p < 0.001). For arousal, small-sized images (Z = −3.295, p = 0.001) and medium-sized images (Z = −4.045, p < 0.001) felt more arousing than large-sized images. For dominance, medium-sized images felt more dominant than small-sized images (Z = −6.040, p < 0.001) and large-sized images (Z = −3.558, p < 0.001). No significant difference was found for the emotion wheel.

Table 1 Mean, median in parenthesis and standard deviations of responses relative to amount of thermal stimuli change
Full size table

6.1.2 Difference in the emotional perception of augmented images presented in the 3 display sizes

Friedman tests of augmented images (excluding images presented with the neutral stimuli) showed no significant difference for arousal (X2 (2) = 1.372, p = 0.504), dominance (X2 (2) = 3.471, p < 0.176) and the emotion wheel (X2 (2) = 80.505, p < 0.001). A significant difference was found for feelings of valence (X2 (2) = 16.657, p < 0.001). Augmented images presented on small displays felt more pleasant than those presented on medium-sized displays (Z = −4.600, p < 0.001), see Table 2.

Table 2 Mean, median in parenthesis and standard deviations ratings of responses relative to the effect 3 °C intensities have on image sizes
Full size table

This is an interesting finding, especially since analysis of un-augmented images showed that images displayed on the 3 display sizes perceptually differed from each other. According to the results, the presence of thermal stimuli augmentation seemed to eliminate the difference in the emotional perception of images displayed in different sizes. This led us to conduct further analysis to identify the effect individual thermal stimuli parameters has on each display size and compare the difference in the effect thermal stimuli parameters has on the 3 display sizes.

6.2 Direction of change

A Wilcoxon pairwise comparison of the effect direction of change had on augmented images presented on all the display sizes showed a significant difference for ratings of arousal (Z = −3.195, p = 0.001) and the emotion wheel (Z = −2.411, p = 0.016).

Warming temperatures were perceived to be more arousing than cooling temperatures (see Table 3), while modal analysis of the emotion wheel ratings showed that cooling temperature evoked happiness, and warming temperatures were perceived as exciting. No significant difference was shown for ratings of valence and dominance.

Table 3 Mean, median in parenthesis and standard deviations of responses relative to thermal stimuli direction of change
Full size table

6.2.1 Effect of warming and cooling temperatures on individual display sizes

Table 4 shows the effect direction of change had on small-sized images. Friedman tests showed a significant difference for valence (X2 (2) = 72.28, p < 0.001), arousal (X2 (2) = 6.05, p = 0.04), dominance (X2 (2) = 41.14, p < 0.001) and the emotion wheel (X2 (2) = 17.84, p < 0.001). This led to a Wilcoxon pairwise comparison of the groups contained in each of the dependent variables (p = 0.0167 with a Bonferroni correction). Warming (Z = −4.621, p < 0.001) and (Z = −7.715, p < 0.001) cooling temperatures both increased valence, with cooling temperatures increasing valence more than warming temperatures (Z = −6.264, p < 0.001).

Table 4 Mean, median in parenthesis and standard deviations of responses for medium display relative to thermal stimuli direction of change
Full size table

For arousal, warming temperatures made the images feel more arousing than cooling temperatures (Z = −3.05, p = 0.002). For dominance, cooling temperatures increased feelings of dominance (Z = −6.014 p < 0.001), and images augmented with cooling temperatures made participants feel more dominant than images augmented with warm temperatures (Z = 6.113, p < 0.001). For the emotion wheel, modal analysis showed that images augmented with cooling temperatures evoked happiness, un-augmented images were perceived as alarming, while warming temperatures evoked fear.

Table 5 shows the effect direction of change had on medium-sized images. Friedman tests found a significant difference for valence (X2 (2) = 18.891, p < 0.001), arousal (X2 (2) = 9.362, p = 0.009) and dominance (X2 (2) = 12.656, p = 0.002). This led to a Wilcoxon pairwise comparison of the groups contained in each of the dependent variables (p = 0.0167 with a Bonferroni correction). Cooling (Z = −3.967, p < 0.001) and warming (Z = −4.650, p < 0.001) temperatures both reduced valence in medium-sized images. For arousal, cooling temperatures made the images feel more arousing than warming temperatures (Z = −4.268, p < 0.001). For dominance, both cooling (Z = −4.628 p < 0.001) and warming (Z = −3.203, p = 0.001) temperatures reduced feelings of dominance in medium-sized images.

Table 5 Mean, median in parenthesis and standard deviations of responses for medium display relative to thermal stimuli direction of change
Full size table

Table 6 shows the effect direction of change had on large-sized images. Friedman tests showed a significant difference for valence (X2 (2) = 33.329, p < 0.001), arousal (X2 (2) = 32.667, p < 0.001) and the emotion wheel (X2 (2) = 15.29, p < 0.001). This led to a Wilcoxon pairwise comparison of the groups contained in each of the dependent variables. Warming (Z = −5.121, p < 0.001) temperatures increased valence and felt significantly more pleasant than cooling temperatures (Z = −6.404, p < 0.001).

Table 6 Mean, median in parenthesis and standard deviations of responses for large display relative to thermal stimuli direction of change
Full size table

For arousal, warming temperatures increased arousal (Z = −5.616, p < 0.001) and made the images feel significantly more arousing than cooling temperatures (Z = −6.604, p < 0.001). For the emotion wheel, modal analysis showed that cooling temperatures had a calming effect, warming temperatures evoked excitement, and Un-augmented images were perceived as sad.

6.2.2 Comparison of the difference in the effect warming and cooling stimuli had on the 3 display sizes

A Friedman’s test showed a significant difference in the effect direction of change had on the different-sized images for valence (X2(2) = 121.7, p < 0.001), arousal (X2(2) = 53.64, p < 0.001), dominance (X2(2) = 49.307, p < 0.001) and the emotion wheel (X2(2) = 73.088, p < 0.001). Cooling temperatures increased valence in small- and medium-sized images more than large-sized images (see Fig. 7). Cooling temperatures increased arousal in small-sized images more than medium and large images. Cooling temperatures increased feelings of dominance in small images more than medium images, which in turn felt more dominant than large images.

Fig. 7
figure 7

Mean ratings of responses relative to the effect cooling intensities have on image display sizes

Full size image

Friedman tests showed a significant difference in the effect warming temperatures had on different-sized images for valence (X2(2) = 27.573, p < 0.001), arousal (X2(2) = 37.481, p < 0.001), dominance (X2(2) = 8.417, p = 0.015) and the emotion wheel (X2(2) = 17.179, p < 0.001). Warming temperatures increased valence in large images more than medium and small images (see Fig. 8). Large images felt more arousing than medium and small images, while small images felt more arousing than medium images. Warming temperatures increased dominance in large images more than small images.

Fig. 8
figure 8

Mean ratings of responses relative to the effect warming intensities have on image display sizes

Full size image

6.3 Amount of change (stimuli intensity)

Table 7 shows the impact of stimuli intensity on images presented on all the display sizes. Friedman tests showed no significant difference for arousal, while a significant difference for ratings of valence (X2(2) = 42.803, p < 0.001), dominance (X2(2) = 20.58, p < 0.001) and the emotion wheel (X2(2) = 17.123, p < 0.001). This led to a Wilcoxon pairwise comparison of the groups contained in each of the independent variables (p = 0.0167 with a Bonferroni correction). For valence, in comparison with images augmented with the neutral stimuli, 6 °C intensities increased valence (Z = −5.370, p < 0.001), while 3 °C intensities reduced valence in images (Z = −3.467, p = 0.001). For dominance, in comparison with images augmented with the neutral stimuli 6 °C intensities reduced arousal in the images presented to participants (Z = −4.939, p < 0.001). Wilcoxon pairwise comparison of the emotion wheel ratings showed no significant difference between any of the groups.

Table 7 Mean, median in parenthesis and standard deviations of responses relative to thermal stimuli amount of change
Full size table

6.3.1 Effect of amount of change on individual display sizes

Table 8 shows the impact of stimuli intensity on small display sizes. Friedman tests found a significant difference for valence (X2 (2) = 33.09, p < 0.001), arousal (X2 (2) = 48.47, p < 0.001), dominance (X2 (2) = 12.66, p = 0.002) and the emotion wheel (X2 (2) = 36.67, p < 0.001).

Table 8 Mean, median in parenthesis and standard deviations of responses for small display relative to amount of thermal stimuli change
Full size table

This led to a Wilcoxon pairwise comparison of the groups contained in each of these independent variables (p = 0.0167 with a Bonferroni correction). For valence, 3 °C changes (Z = −8.321, p < 0.001) and 6 °C changes (Z = −7.421, p < 0.001) both increased valence in small-sized images in comparison with no change, with 6 °C intensities feeling significantly more pleasant than 3 °C intensities (Z = −4.511, p < 0.001). For arousal, 3 °C changes reduced arousal in images compared to when there was no change in temperature (Z = −2.667, p = 0.008). 3 °C changes also significantly reduced arousal in comparison with 6 °C changes (Z = −7.179, p < 0.001). For dominance, 3 °C changes (Z = −6.660, p < 0.001) and 6 °C changes (Z = −5.521, p < 0.001) both increased feelings of dominance in comparison with when there was no thermal stimuli augmentation. 3 °C changes also felt significantly more arousing than 6 °C changes (Z = −2.427, p = 0.015). Modal analysis of the emotion wheel showed that un-augmented images were perceived as alarming; 3 °C intensities had a calming effect, while 6 °C intensities were perceived as exciting.

Table 9 shows the impact of stimuli intensity for medium display sizes. Using Friedman tests, a significant difference was found for valence (X2 (2) = 33.090, p < 0.001), arousal (X2 (2) = 48.465, p < 0.001), dominance (X2 (2) = 12.656, p = 0.002) and the emotion wheel (X2 (2) = 36.668, p < 0.001). For valence, 6 °C changes reduced valence in images compared to when there was no change in temperature (Z = −5.470, p < 0.001). 3 °C changes also significantly increased valence in comparison with 6 °C intensities (Z = −10.109, p < 0.001).

Table 9 Mean, median in parenthesis and standard deviations of responses for medium display relative to amount of thermal stimuli change
Full size table

For arousal, in comparison with when there was no change in temperature, 3 °C intensities reduced arousal (Z = −4.908, p < 0.001), while 6 °C intensities increased feelings of arousal (Z = −3.229, p = 0.001). 6 °C intensities also felt significantly more arousing than 3 °C intensities (Z = −4.204, p < 0.001). For dominance, 3 °C changes (Z = −3.790, p < 0.001) and 6 °C changes (Z = −3.251, p = 0.001) both decreased feelings of dominance in comparison with when there was no thermal stimuli augmentation. 3 °C changes also significantly increased dominance than 6 °C changes (Z = −4.663, p < 0.001). Modal analysis of the emotion wheel showed that un-augmented images and images augmented with 3 °C intensities both had a calming effect on perceivers.

Table 10 shows the impact of stimuli intensity on large display sizes. After Friedman tests, we found a significant difference for valence (X2 (2) = 121.790, p < 0.001), dominance (X2 (2) = 63.674, p < 0.001) and the emotion wheel (X2 (2) = 40.115, p < 0.001). For valence, 3 °C intensities (Z = −4.429, p < 0.001) and 6 °C intensities (Z = −7.365, p < 0.001) both increased valence in comparison with when there was no thermal stimuli augmentation. 6 °C intensities also significantly increased valence in large-sized images more than 3 °C intensities. For dominance, 3 °C intensities reduced dominance in comparison with when there was no thermal augmentation (Z = −3.981, p < 0.001), while 6 °C changes (Z = −6.129, p < 0.001) increased dominance. 6 °C changes also significantly increased dominance than 3 °C changes (Z = −8.724, p < 0.001). Modal analysis of the emotion wheel showed that un-augmented images were perceived as sad, 3 °C intensities evoked fear, while 6 °C intensities had a calming effect on the perceivers.

Table 10 Mean, median in parenthesis and standard deviations of responses for large display relative to amount of thermal stimuli change
Full size table

6.3.2 Comparison of the difference in the effect 3 and 6 °C intensities had on the 3 display sizes

A Friedman test showed a significant difference in the effect 3 °C intensities had on the different screen sizes for valence (X2(2) = 56.26, p < 0.001), arousal (X2(2) = 33.79, p < 0.001), dominance (X2(2) = 37.94, p < 0.001) and the emotion wheel (X2(2) = 28.39, p < 0.001). A Wilcoxon pairwise comparison showed that 3 °C intensities increased valence in all image display sizes in comparison with no change (see Fig. 9).

Fig. 9
figure 9

Mean ratings of responses relative to the effect 3 °C intensities have on image display sizes

Full size image

Before augmentation large-sized images felt more pleasant than small-sized images, after augmentation small-sized images felt more pleasant than large-sized images. 3°C also increased valence in small-sized images more than large-sized images. Large- and medium-sized images augmented with 3 °C intensities felt more arousing than small-sized images. 3°C intensities increased dominance in medium- and small-sized images more than large images. Based on results obtained from a Friedman’s test, 6 °C intensities also significantly affected feelings of valence (X2(2) = 137.708, p < 0.001), arousal (X2(2) = 34.859, p < 0.001), dominance (X2(2) = 64.191, p < 0.001) and the emotion wheel (X2(2) = 37.831, p < 0.001) in the different-sized images presented (see Fig. 10). 6 °C intensity significantly increased valence in all image sizes.

Fig. 10
figure 10

Mean ratings of responses relative to the effect 6 °C intensities have on image display sizes

Full size image

A comparison of the different-sized images before augmentation showed that medium-sized images felt most pleasant, followed by large images, while small images felt least pleasant. A comparison of how pleasant the different-sized images felt augmentation showed that large-sized images felt most pleasant, followed by small-sized images, while medium-sized images felt least pleasant. 6 °C intensity had the highest effect in increasing valence in large images, followed by small, then medium images. Small images felt more arousing than medium and large images. With 6 °C intensities, large images felt most dominant followed by small images, then medium images.

6.4 Image category

The image category variable is made up of 8 groups. A Bonferroni correction was made for pairwise comparisons with p = 0.0018. Analysis of valence, arousal and dominance dependent variables showed that images in each of the 8 different image emotion categories maintained their distinct emotive properties after thermal stimuli augmentation. Therefore, the valence, arousal and dominance ratings we obtained for augmented images were compared with the original valence, arousal and dominance ratings before thermal stimuli augmentation, as recorded by the IAPS, in order for us to understand the effect of thermal augmentation on the different image emotion categories. This generated the valence difference, arousal difference and dominance difference variables, which are analysed below. Detailed analysis is presented in `Appendix 1'. Results showed that consistent with findings by Akazue et al. [1], thermal augmentation increased valence/arousal in low valence/low arousal images while reducing valence/arousal in high valence/high arousal images. Thermal stimuli also increased dominance in all image emotion categories.

6.5 Result summary

This study examined the affective influence of thermal stimuli augmentation on images displayed in different sizes. Results indicate that thermal stimuli augmentation was effective in modulating the amount of affect evoked by the different-sized images presented (Addressing RQ 1.1). Image category analysis showed that thermal augmentation increased valence/arousal in low valence/low arousal images while reducing valence/arousal in high valence/high arousal images. Thermal stimuli also increased dominance in all image emotion categories. These findings are consistent with results obtained by Akazue et al. [1].

Addressing RQ 1.2, thermal stimuli augmentation was shown to be most effective in increasing emotions in small-sized images. Addressing RQ 1.3, 3 and 6 °C warming and cooling intensities were both shown to significantly increase valence, and 3, 6 °C cooling intensities increased dominance in small-sized images. 6 °C cooling intensities also increased arousal. Thermal stimuli had the highest effect in reducing affect in medium-sized images. 3 and 6 °C cooling and warming intensities both reduced valence and dominance in medium-sized images, while 3 °C warming temperatures reduced arousal. Thermal stimuli had the least effect in modulating affect in large-sized images. 3 and 6 °C warming intensities both increased valence and were perceived to be significantly more pleasant than cooling intensities. Stimuli intensity had no significant effect on arousal, while 3 °C intensities reduced dominance with no significant difference between warming and cooling.

Based on the results obtained from this study, we wanted to determine whether thermal stimulation maintains its ability to modulate affect in images when presented on devices with screen sizes corresponding to medium and small displays (tablet and mobile phone). This led us to conduct a second user study investigating the role of presentation device in the emotional perception of augmented images.

7 Study 2: the effect of presentation device in the emotional perception of images

Study 1 allowed us to investigate the difference in the emotional perception of different-sized images augmented with thermal stimuli. However, there are other factors, which might influence how different-sized images augmented with thermal stimuli are perceived (such as the effect of presentation device). Thus, this second user study was designed to examine whether presentation device plays any role in the emotional perception of augmented images presented on actual mobile devices (tablet and mobile phone). The research questions the study aimed to answer are as follows:

RQ 2: What effect does thermal stimuli augmentation have on the emotional perception of images viewed on mobile devices (tablets and mobile phones)?

  1. 2.1.

    Can thermal stimuli augmentation be used to enhance or reduce the media experience on mobile devices?

  2. 2.2.

    What is the effect of presentation device on the emotional perception of augmented images?

  3. 2.3.

    On which device type does thermal stimuli augmentation have the highest effect in increasing or reducing affect in images?

  4. 2.4.

    What is the effect of individual thermal stimuli parameters in increasing or decreasing emotions in mobile devices?

In this study, augmented images were presented on a Samsung Galaxy S6 mobile phone and a Motorola Xoom 2 tablet device. Augmented images were rated orally on the SAM and emotion wheel scales. Peltier devices attached to heat sinks were placed at the back of each presentation device using adhesives (see Figs. 11, 12). A similar method was used by [38] to attach peltier devices to a mobile phone. This user study was conducted in two phases. One phase (referred to as the tablet setup) was conducted on a Motorola Xoom 2 android tablet device (10.1-in display), while the other phase (referred to as the mobile phone setup) was conducted on a Samsung Galaxy S6 mobile phone device (5.1-inch display). All participants took part in both phases. The order in which these phases were presented was counter-balanced between participants. Since tablets are very often operated while being held in the hands, with the fingers placed at the back of the device, all thermal stimuli combinations in the tablet setup were presented to the fingers (see Fig. 11). Although Wilson et al. [39] found the fingers had a lower detection rate in comparison with other hand locations, presenting thermal augmentation to any other hand location may not have allowed for natural interaction with the tablet and mobile phone devices. In the mobile phone setup, participants were requested to assume a comfortable posture and hold the device in landscape orientation, with their hands in contact with the peltier plates attached to the back of the device (see Fig. 12). The portion of their hand, which came in contact with the device, was recorded. Due to the original dimensions of the IAPS images, images were viewed in landscape mode on the mobile and the tablet. Presenting images in landscape also encouraged two-handed use of both devices (tablet and phone). Augmented images were displayed for 6 s, after which the temperature returned to neutral for adaptation to occur, and a blank white page was presented on the screen of the mobile device for 30 s. During the 30-s adaptation after each augmented image presentation, participants were requested to provide oral ratings on the SAM and emotion wheel scales printed out on paper. Ratings were provided orally as against electronically because participants were required to have their two hands in direct contact with the thermal stimuli source throughout each phase/setup, in order to maximise thermal stimuli perception [39]. At the end of the evaluation in each setup, and the entire evaluation there was an interview on the experience of participants in each phase and the study in general.

Fig. 11
figure 11

Typical tablet setup for participant taking part in study 2

Full size image
Fig. 12
figure 12

Typical mobile phone setup for participant taking part in study 2

Full size image

7.1 Image selection and presentation

In order to ensure that an image, which has already being presented on the tablet device, is not presented again on the mobile device, images contained in each of the image emotion categories were divided into two collections (see Fig. 13). Images in each emotion category were randomly divided into 2, and placed in collection 1 and 2. One collection was used as the image source for the tablet device, while the other collection was used as the image source for the mobile phone. Images were randomly selected for presentation on each device from the image collection assigned to the device.

Fig. 13
figure 13

IAPS images divided into two collections across the different image emotion categories

Full size image

7.2 Tablet setup

An image was randomly selected from each of the 8 emotion categories (see Fig. 4), combined with each of the 5 thermal stimuli combinations (26, 29, 32, 35 and 38 °C) and presented to perceivers. Presentation and evaluation time of images followed the same pattern used in study 1. Images were displayed for 6 s after which a blank white page was displayed for 30 s. The self-assessment manikin and emotion wheel rating scales printed on paper were placed in front of participants, for them to orally select ratings that best describe how the perceived image made them feel.

The researcher administering the experiment recorded participant’s oral selections electronically against the rated augmented image. Microsoft reaction cards used by Lochtefeld et al. [13] were also considered as a tool for collecting subjective feedback; however, this tool was designed for measuring desirability as against emotional reactions [7]. A total of 40 augmented images were presented to perceivers in this setup.

7.3 Mobile phone setup

Similar to the tablet setup, an image was randomly selected from each of the 8 emotion categories (see Fig. 4), combined with each of the 5 thermal stimuli combinations (26, 29, 32, 35 and 38 °C) and presented to perceivers.

Participants also perceived a total of 40 augmented images in this phase. Presentation and evaluation of augmented images followed the same procedure detailed in ‘Phase 1’ above.

7.4 Data gathering

Data collection followed the same pattern used in study 1. Participants were allotted 15 s to provide valence, arousal and dominance ratings on the SAM scale and another 15 s to provide ratings on the emotion wheel. Participants were also interviewed at the end of each setup/phase, and at the end of the entire evaluation. The interview questions presented at the end of each setup were as follows:

  1. 1.

    What kind of effect do you feel warm stimuli had on the images viewed on this device?

  2. 2.

    In which image category was this effect most pronounced?

  3. 3.

    What kind of effect do you think cool stimuli had on images viewed on this device?

  4. 4.

    In which image category was this effect most pronounced?

The above questions were designed to enable participants to provide information about the effects they perceived thermal augmentation had on the different images viewed in each setup. The second interview, which was conducted at the end of the entire evaluation, was designed to examine whether participants had a preference for any of the presentation devices used and why. Interview questions presented at the end of the evaluation are as follows:

  1. 1.

    Which device did you prefer viewing images augmented with thermal stimuli on and why?

  2. 2.

    Was there any difference in how you felt when viewing augmented images on the phone and on the tablet?

  3. 3.

    If yes, what was this difference?

  4. 4.

    Is there any particular image category more suitable for viewing on either the mobile phone or tablet?

7.5 Participants

Eleven participants (6 males, 5 females) between the ages of 18 and 44 with a mean age of 30 were recruited from personal and university contacts, for this study.

7.6 Variables

This study had a total of 4 independent variables: presentation device, direction of change, amount of change and image category. Dependent variables were valence, arousal, dominance and emotion wheel ratings. As in study 1, valence difference, arousal difference and dominance difference were also analysed for the image category independent variable.

8 Study 2 results

As data were not normally distributed, Friedman tests and Wilcoxon Sign Rank tests were used for analysis.

8.1 Comparisons of subjective ratings provided for device type

Wilcoxon comparison of augmented images (including images presented with neutral stimuli) displayed on the tablet and mobile phone showed that the thermal stimuli had similar effects for ratings of valence (Z = −0.287, p = 0.774), arousal (Z = −0.255, p = 0.799), dominance (Z = −0.3868, p = 0.385) and the emotion wheel (Z = −0.177, p = 0.860).

Wilcoxon comparison of un-augmented images (images augmented with the neutral stimuli) showed there was still no significant difference between images viewed on the two devices for rating of valence (Z = −1.249, p = 0.212), arousal (Z = −0.531, p = 0.596), dominance (Z = −1.397, p = 0.162) and the emotion wheel (Z = −1.025, p = 0.306). Analysis of augmented images (excluding images presented with the neutral stimuli) also showed that augmented images presented on that tablet were perceptually similar to those presented on the phone for ratings of valence (Z = −0.925, p = 0.355), arousal (Z = −0.087, p = 0.931), dominance (Z = −0.594, p = 0.552) and the emotion wheel (Z = −0.522, p = 0.601).

8.2 Amount of change (stimuli intensity)

Friedman tests showed no significant difference between the 3 groups contained in this variable for ratings of valence (X2(2) = 1.162, p = 0.559), arousal (X2(2) = 0.186, p = 0.911), dominance (X2(2) = 1.540, p = 0.463) and the emotion wheel (X2(2) = 3.629, p = 0.163).

8.2.1 Effect of amount of change on images displayed on the mobile phone

Amount of change had no significant effect on images displayed on the mobile phone for ratings of valence (X2(2) = 1.226, p = 0.542), arousal (X2(2) = 0.393, p = 0.822), dominance (X2(2) = 1.497, p = 0.473) and the emotion wheel (X2(2) = 2.761, p = 0.251).

8.2.2 Effect of amount of change on images displayed on the tablet device

Amount of change had no significant effect on images displayed on the mobile phone for ratings of valence (X2(2) = 1.108, p = 0.575), arousal (X2(2) = 0.007, p = 0.996), dominance (X2(2) = 0.208, p = 0.901) and the emotion wheel (X2(2) = 2.953, p = 0.228).

8.3 Direction of change

Wilcoxon pairwise comparison showed that there was no significant difference between the effect warming and cooling temperatures had on images presented on both the mobile phone and tablet. Direction of change had similar effects for feelings of valence (Z = −0.438, p = 0.661), arousal (Z = −0.506, p = 0.613), dominance (Z = −1.085, p = 0.278) and the emotion wheel (Z = −0.526, p = 0.599).

8.3.1 Effect of direction of change on images displayed on the mobile phone

Direction of change had no significant effect on images displayed on the mobile phone for ratings of valence (Z = −0.182, p = 0.856), arousal (Z = −1.263, p = 0.206), dominance (Z = −1.429, p = 0.153) and the emotion wheel (Z = −1.578, p = 0.114).

8.3.2 Effect of direction of change on images displayed on the tablet device

Direction of change had no significant effect on images displayed on the mobile phone for ratings of valence (Z = −0.290, p = 0.772), arousal (Z = −0.183, p = 0.855), dominance (Z = −0.150, p = 0.881) and the emotion wheel (Z = −1.050, p = 0.294).

8.4 Image category

The image category variable is made up of 8 groups. A Bonferroni correction was made for pairwise comparisons with p = 0.0018. Analysis of valence, arousal and dominance dependent variables showed that images in each of the 8 different image emotion categories maintained their distinct emotive properties after thermal stimuli augmentation. Detailed analysis is presented in `Appendix 2'.

8.5 Qualitative results

An adapted version of constant comparative method [3, 13] was used to analyse the interview data. Incidents in the qualitative data obtained were first coded into multiple categories, after which categories were integrated, and related categories combined to form higher-level concepts. The coded data were then processed to obtain the findings discussed in this section. According to Baillie [3], this technique helps ensure that findings can be attributed directly to the words and comments of the participants. Generated codes were formed into the following main themes:

8.5.1 Effect of thermal augmentation on images presented on mobile devices

Eight out of 11 participants felt thermal augmentation did have a significant effect on how they perceived the pictures displayed on the tablet and mobile phone. Images augmented with different thermal stimuli combinations were perceived to be more enjoyable than those augmented with the neutral stimuli combination. P8 commented that ‘I enjoyed viewing the pictures with different temperature more than when presented on their own’. According to P1 thermal augmentation tended to enhance the inherent emotions in the images presented—‘Temperature enhanced the message of the images’. This effect of thermal augmentation enhancing emotions was more pronounced for certain image types more than others. P9 mentioned that for some of the images presented, ‘the emotion from the image was more dominant’. This was also supported by comments by P2 who noted that ‘I don’t think the temperature had much of an effect on pictures of people skiing and rafting because they felt nice on their own and I like to do those things’. This suggests that emotions evoked by images with high or low arousal and valence, tended to override the effect of thermal augmentation on how those images were perceived. Where thermal stimuli augmentation does have an effect, it amplified the original emotion the images were capable of evoking. This is consistent with results obtained by Akazue et al. [1] who found that presenting thermal augmentation while an image is being displayed amplified the inherent emotion an image is capable of evoking.

8.5.2 Difference in the effect of thermal augmentation on images presented on the tablet and mobile phone devices

Six out of 11 participants felt the effect thermal stimuli augmentation had on the tablet varied from the effect it had on the phone. P1 stated that ‘I found the effect of temperature enhancing the emotions in the images much more effective on the mobile than on the tablet, maybe because it is a smaller screen’. This differs from the result we obtained from the quantitative data. Interview results showed that warm stimuli when presented with low valence images generally enhanced negative emotions on the mobile phone, while cool stimuli enhanced positive emotions irrespective of the image type. Warm stimuli presented with high valence pictures on the mobile device, however, increased valence in such images. P3 mentioned that ‘When I saw a flower and felt warm temperature I felt happy, but when I saw something terrible and it felt warm it made the picture feel worse. I also felt happy when I felt the cool temperature regardless of the picture because I like cool temperatures’. Warming stimuli had a different effect on the tablet device. Warm stimuli reduced valence in all image types, including high valence images, while cool stimuli increased valence in all image categories. This contrasts with quantitative results, which showed that direction of change had no significant effect on ratings of valence, arousal and dominance. P7 mentioned that ‘There were some pictures I felt differently about when I started to feel the warm temperature. Picture of leaves, which was calming, felt worse when I felt the warm temperature. The cool temperature made the pictures feel better’.

8.5.3 Effect of presentation device on the emotional perception of augmented images

Eight out of 11 participants preferred viewing augmented images on the tablet to the phone. Participants that preferred the tablet did so mainly because it had a bigger screen and images displayed were larger than those displayed on the mobile device. P11: ‘I prefer the tablet because the pictures were bigger’. P7 mentioned this bigger screen will be preferred for viewing nicer images (images with high valence), while low valence images will be more suited to smaller display sizes (mobile phone): P11: ‘I would have preferred to see the nicer pictures on the tablet and the worse ones on the mobile phone’. According to the results obtained, the display size of the presentation devices strongly influences the effect presentation device has on the different augmented images presented on them. This contrasts with the results obtained from the quantitative analysis. This qualitative result suggests that devices with larger displays enhance the inherent emotions in augmented images, while smaller-screened devices reduce the emotions augmented images are able to evoke. With this finding, interaction designers can make low valence augmented images feel worse by increasing the display size, or make them feel better by reducing the display size. High valence augmented images can also be made more pleasant by increasing display size or less pleasant by reducing display size.

8.6 Results summary

This study investigated the role of image presentation device on the emotional perception of images augmented with thermal stimuli. Results obtained from the quantitative analysis showed that thermal stimuli augmentation and the presentation devices used in this study (tablet and mobile phone) had no significant effect on the emotional perception of images presented on them. Analysis of the qualitative data, however, yielded different results. Addressing RQ 2.1, interview responses indicated that augmented images viewed on the mobile phone was perceptually different to augmented images viewed on the tablet. Addressing RQ 2.2, the display size of the presentation devices strongly influenced the effect it had on the augmented images. The tablet screen with a larger screen amplified the inherent emotions in augmented images presented on them while the smaller-sized mobile phone reduced the emotions augmented images are capable of evoking (RQ 2.3). Addressing RQ 2.4, augmented images also felt more enjoyable than un-augmented ones, with warm stimuli shown to reduce valence, while cool stimuli increased valence in all image emotion categories.

9 Discussion

Thermal stimuli augmentation maintains its affective influence when used to augment images displayed in medium and small sizes and images presented on tablet and mobile phones (addressing RQ 1.1, 2.1): Analysis of the image category variable in study 1 showed that thermal stimuli increased valence in low valence images, while reducing valence in high valence images (see `Appendix 1'). This is consistent with results obtained by Akazue et al. [1]. Thermal stimuli direction of change and amount of change both significantly influenced the amount of affect evoked by images presented on medium and small display screens. 3 °C intensities increased arousal in medium-sized screens, while warming and cooling 6 °C intensities reduced valence, arousal and dominance. In small image displays, 3 and 6 °C warming and cooling intensities both increased feelings of valence. These results indicate that thermal augmentation is effective for modulating affect in small- and medium-sized images. In study 2, thermal stimuli augmentation was also shown to be effective for influencing affect in images presented on a mobile phone and tablet. In contrast to study 1, analysis of image category in study 2 showed that thermal stimuli increased valence in neutral-high valence images, while reducing valence in low valence images (see `Appendix 2').

Image display size seems to have a stronger influence on the emotional perception of augmented images than presentation device (addressing RQ 2.2): Comparison of the quantitative results obtained in studies 1 and 2 shows a difference in the amount of affective influence image display size and presentation device have on images augmented with thermal stimuli. In study 1, image display size was found to have a significant influence on the emotional perception of images augmented with thermal stimuli. Augmented images displayed in different sizes were found to be perceptually different. While in study 2, quantitative results showed presentation device had no significant influence on the emotional perception of images augmented with thermal stimuli. Qualitative results, however, showed that images presented on mobile devices were perceived to be more enjoyable than images with no thermal augmentation. Eight out of eleven participants preferred viewing augmented images on the tablet than the phone, mainly because of the larger-sized images displayed on the tablet. This effect shown by the qualitative results may have been as a result of the difference in the display size dimensions of the mobile phone and tablet, as against the presentation devices themselves, since study 1 showed augmented images viewed in different sizes were perceptually different. Furthermore, comparison of the affective influence of thermal stimuli augmentation on the different-sized images in study 1 showed that thermal stimuli had the least effect in modulating affect in large-sized images. As a result, the variance shown between results obtained in studies 1 and 2 could be because large-sized un-augmented images tend to be inherently more immersive than small-sized un-augmented images. Variance in the sensitivity of the thermal stimuli delivery location could have also given rise to thermal stimuli having a stronger effect in study 1, than in study 2. Findings discussed in the rest of this section are quantitative and qualitative results obtained from both study 1 and study 2.

Thermal augmentation is most effective in increasing affect in small-sized images and images presented on mobile phones (addressing RQ 1.2, 2.3): Comparison of the affective influence of thermal stimuli augmentation on the different-sized images in study 1 showed that thermal stimuli augmentation was most effective for increasing affect in small-sized images, reducing emotions in medium-sized images, and had the least effect on large-sized images. This could have been because large-sized un-augmented images tend to be inherently more immersive than smaller-sized un-augmented images, thus resulting in thermal stimuli augmentation having a stronger affective influence on smaller-sized images typically presented on mobile phones than on larger-sized images with higher inherent emotive attributes. In study 2, quantitative results showed thermal augmentation to be more effective in modulating affect in images displayed on a mobile phone, than those displayed on a tablet. Although the stronger effect of thermal stimuli on the mobile phone could have been as a result of its smaller screen, which reduces the amount of visual information available to a perceiver.

Different thermal stimuli parameters are effective in modulating emotions in images displayed in different sizes on different device types (addressing RQ 1.3, 2.4): In study 1, 6 °C and 3 °C intensities increased feelings of valence and dominance in small-sized images. 3°C intensities also reduced arousal, and 6 °C were perceived to be more arousing than 3 °C intensities. For medium-sized images, 3 °C intensities increased feelings of arousal and felt more pleasant than 6 °C, while 6 °C intensities reduced feelings of arousal and valence. Both 3 and 6 °C intensities reduced feelings of dominance in medium-sized images. 3 and 6 °C intensities increased valence in large-sized images. 6°C intensities increased dominance in large-sized images, while 3 °C intensities reduced feelings of dominance. Amount of change had no significant effect on arousal for large-sized images. Cooling and warming temperatures both increased pleasantness in small-sized images, while cooling temperatures increased feelings of dominance in perceivers more than warming temperatures. Warming temperatures also felt more arousing than cooling temperatures for small-sized pictures. For medium-sized images, both warming and cooling reduced feelings of dominance and how pleasant the images felt. Cooling temperatures felt more arousing than warming temperatures. For large-sized images, warming temperatures increased how pleasant and how arousing the images were perceived. Warming temperatures were also perceived to be more pleasant and arousing than cooling temperatures. Direction of change had no statistically significant effect on the feeling of dominance in large-sized images.

According to these results, in the design of media experiences, 3 and 6 °C warming and cooling thermal stimuli intensities can be used to significantly increase valence and feelings of dominance in small-sized images. 3°C warming intensities can also be used to reduce how arousing small-sized images are perceived. 6 °C cooling thermal stimuli intensities can be used to reduce valence, arousal and feelings of dominance in small-sized images. When designing affective interactions for mobile phones, warm stimuli may be suitable for reducing valence in low valence images and increasing valence in high valence images. Augmenting images on a tablet with warm stimuli may also be effective for reducing valence in all images types. In order to increase valence in all picture types, irrespective of the presentation device, results show cool stimuli would be most suitable.

10 Conclusion

This paper presented the first known investigation of the effect thermal stimuli augmentation has on different-sized images, and images presented on different presentation devices. Augmenting small image displays with 3 and 6 °C warming and cooling intensities, changing from a neutral point of 32 °C was found to be very effective in increasing feelings of valence and dominance in all image emotion categories. 3 °C cooling intensities increased valence and arousal in medium-sized images, while 6 °C warming and cooling intensities reduced valence and dominance. Although the majority of the participants preferred viewing augmented images on the tablet (medium-sized display) because of the bigger screen, thermal augmentation seemed to have a stronger effect in modulating emotions in the images displayed on the phone (small-sized display).

These results act as a foundation upon which systems that harness thermal stimuli properties for the purpose of improving user experiences with media viewed on different display sizes can be built. Some of its potential applications in media include: providing interaction designers with the ability to influence emotions on different types of media presented in specific display sizes, e.g., designing thermal augmentation for images shared on social media platforms designed to run on specific devices (Desktop, Tablet, Phone); providing interaction designers with the ability to increase the suitability of smaller displays for image viewing by using thermal stimuli augmentation to compensate for lack of screen space and enabling the creation of more affective media viewing experiences designed for specific devices types and sizes. Results show that with thermal stimuli augmentation, emotions in small-, medium- and large-sized images can be influenced and smaller-sized images can be made perceptually similar to larger-sized images.