1 Introduction

Equipment of office environments consumes a large amount of electrical energy to maintain building operation and to power office appliances. While many facility managers have started campaigns to motivate energy consumption awareness of office workers, manually operating equipment to minimise energy needs is a burden for users. Shared devices and the anonymous office environment itself lead to unresolved responsibilities and low incentives for users to operate equipment efficiently. Automatic device control could address the issue. However, to date, devices and offices often lack appropriate sensors that indicate usage. Consequently, equipment is operated based on average expected need, rather than the actual office worker behaviour. Computer screens are an example of a widely used office appliance that contributes to energy consumption in offices. Screens are often computer-controlled by monitoring user input and suspending the screen to standby mode, after a constant timeout without input. The US Department of Energy recommends a timeout of 20 min (European Commission 2002), as screens may be used even in phases without entry. Since sensors that could provide accurate user presence information are lacking, the timeout setting provides a rough balance between energy consumption and user comfort. Considering the extensive deployment of computer screens in typical office environments, solutions that gain even small energy saving benefits per device would be beneficial. Alternative screen control strategies, i.e., using sensors to detect presence and desk activity could prove more effective than the computer-controlled operation, hence saving energy while maintaining personal comfort.

Earlier work using a proximity-controlled screen management showed that 43 % of electric energy could be saved, compared to a standard 20-min screen timeout (Jaramillo and Amft 2013). To measure user-screen proximity, two ultrasound range (USR) sensors were horizontally attached to office computer screens. The USR sensors’ distance readings were used to recognise desk activities, including ScreenWork, DeskWork, and Away. Hence, ScreenWork required users to be detected at short distance to both sensors at the screen. The energy savings resulted from two effects: (1) switching screens off at shorter timeouts (below 1 min) compared to the computer-controlled operation, and (2) interrupting the screen power supply, rather than switching to standby mode. We found that activity recognition performance and user presence detection speed to switch a screen on when the office worker intended to use it, were critical system design objectives. Minimal delay to switch screens on is vital acceptance criteria affecting user comfort. Minimising screen operating time even during short phases, e.g., when users turn away from the screen is key to energy saving. In addition, the study showed that reflections from other objects in the vicinity, including chairs, and the variation of user distances could not be handled with a recognition and control approach that is fixed at system design time. A dynamically adaptive control approach could help to find and maintain the balance between user comfort and energy needs and ensure that commissioning effort does not increase due to the new technology.

In this paper we propose and evaluate an adaptive proximity controller (APC) system for managing office screens power that is sensitive—to rapidly switch screens on—and aims at minimal detection error when switching off. The APC system uses implicit user feedback to rate screen switch-off operations made, and adjusts further system behaviour. Our approach to use implicit feedback is based on observations during preliminary studies, where users intuitively responded by moving head and upper body when their computer screen was switched off unexpectedly. User movements are captured as increase in proximity variations and subsequently interpreted as negative feedback by our APC system. Conversely, low variation in proximity measurements following a switch-off operation can be handled as positive (confirmatory) feedback for the previous decision. Decision confidence is modelled as time delay to switch off the screen after detecting that the screen was not used, screen time off (STOFF). Increasing STOFF yields longer time delays before the screen is switched off and reduces chances of interrupting screen use. However, increasing STOFF also increases screen operating time and hence energy consumption. We further employ an active probing mechanism by sporadically reducing STOFF after detecting that the user left the screen.

In particular, this paper provides the following main contributions:

  1. 1.

    We present our APC system architecture and approach to implicit feedback control using proximity variance. The proximity variance was estimated from the user’s body movement in front of screens. We describe the system implementation and user implicit feedback control handling.

  2. 2.

    We present a multi-day intervention study using our APC system with 12 participants in their regular office environment. We analyse the systems’ recognition performance to switch on/off screens and illustrate how the implicit feedback and probing improves system decisions.

  3. 3.

    We assess user comfort from responses provided by study participants. Furthermore, we analyse the energy savings achieved by the APC system.

2 Related work

Office activities and occupant behaviour in private homes and buildings have been often considered to lower energy consumption or to improve user comfort. Recent surveys included aspects of occupant behaviour, modern sensing technologies, and sensor data analysis for intelligent buildings (Williams et al. 2012; Nguyen and Aiello 2013). In particular, Williams et al. (2012) provided a comprehensive meta-analysis on energy savings by means of different lighting controls, assessing important perspectives related to user preferences. Various related contributions focused on activity detection algorithm performance and on lowering energy consumption, both mostly based on computer simulations. User comfort remains a critical design aspect that is often insufficiently integrated with system recognition and energy consumption performance optimisations. In particular, we found no adaptive control approach that evaluates the dynamic balance between energy consumption and user comfort as the APC system that we propose in this work.

Multi-modal sensing approaches using computer-mediated communication technologies (Begole et al. 2003), soft sensing (Ghai et al. 2012) and opportunistic sensing (Tarzia et al. 2009) have been considered to analyse occupant behaviour in buildings. In this direction, energy saving potential was estimated in simulations of activities in office environments. Recently, our group reported actual benefits of opportunistic sensing in office environments (Milenkovic and Amft 2013a, b). However, approaches for controlling appliances and sensor pattern adaptation are required.

Passive infrared (PIR) sensing has been frequently used and deployed for energy saving applications, e.g., to control overhead lighting in offices. Energy savings of 32 % were estimated from a partially automated lighting system based on occupancy sensors and user dimming-controls (Galasiu and Newsham 2009). In another work, overall savings of 70 % were found for a lighting system, controlled by the combination of occupancy sensors, daylight harvesting with light sensors, and individual dimming controls per user (Galasiu et al. 2007). More recently, Wahl et al. (2012b) proposed a green autonomous self sustaining node for counting people in office environments based on PIR sensors. Results from occupant behaviour simulations using this solution confirmed accuracy of the algorithmic approach (Wahl et al. 2012a). PIR-based approaches typically suffer from sensitivity to ambient light conditions and remain limited to the detection of movement mostly. To this end, actual presence estimation can bring advantages in energy saving, as confirmed in our preliminary investigation using USR sensors (Jaramillo and Amft 2013).

Computer vision techniques have been widely used in estimating user activity in buildings, e.g. (Moore et al. 1999; Ayers and Shah 2001; Wojek et al. 2006). An energy saving application using face recognition to control home TV brightness showed a reduction on energy consumption of ∼30 % (Ariizumi et al. 2008). Samsung introduced a camera-based comfort feature for personal devices, such as smart phones and tablets, termed “Smart stay” (Schwartz 2012) where the devices’ frontal camera is used to determine whether the user is looking at the screen. The feature enables users to follow displayed information, while not touching the screen that would otherwise time out. However, due to privacy and security considerations, cameras are frequently banned from office desks. Moreover, changes in lighting conditions and user-screen distance information, as it is used for the APC system in our present work, would create additional challenges when applying a vision-based detection and control. Moreover, our APC system considers user movement as implicit feedback to dynamically adapt its operation. In contrast, webcams typically capture the head region only, thus may provide limited information on upper body motion.

Ultrasound sensing approaches have been proposed for context analysis in office environments before. Ultrasonic frequencies were transmitted and received through office devices, such as computer speakers and microphones to infer user attention state (Tarzia et al. 2009). Ultrasonic sensor arrays were used for localisation and tracking of multiple occupants with the aim of deploying lighting control applications (Caicedo and Ashish 2012). Finally, ultrasound sensors were placed in ceiling mounts to detect person falls on the floor and trigger an alarm system in assistive environments (Shah et al. 2011).

In general, proximity measurements can be obtained using different sensing principles, including ultrasound and infrared (IR) modalities. USR sensors typically provide wider fields of view than IR sensors of about 45°. For our APC system, the wider field of view of USR sensors permit the system to detect users earlier when approaching the screen and thus to turn it on instantly. Furthermore, USR sensors are not affected by differences in ambient light conditions as common IR sensors. Given the low cost and minimal energy requirements of USRs, our APC system relies on USR sensors to estimate user-screen proximity.

Adaptation mechanisms to control appliances or office screens for user comfort are not yet commonly deployed in offices. An attempt towards the adaptation of overhead lighting in office rooms to support individual and group activities has been studied by Magielse et al. (2011). To maximise user comfort, lights were actuated according to relevant activities, such as reading, presentations, and meetings. A multi-modal sensor network and a decision tree algorithm were used for activity detection. In order to test the feasibility of this technology, a qualitative user study was carried out, resulting in important insights for future system developments. Our present work specifically focuses on maintaining user comfort while reducing energy consumption. To this end, we assessed user comfort by extending the established Standard Usability Scale (SUS) questionnaire (Sauro and Lewis 2012).

3 APC system overview

Our APC system architecture comprises two stages: (1) detecting desk activities using USRs, and (2) dynamically adapting the proximity-based control during system runtime. In this section, an overview of the APC architecture is provided and the implicit user feedback mechanism is introduced.

3.1 Detecting desk activities

User behaviour at a desk is captured in a categorical variable describing the desk activities. We used two USR sensors mounted at the top or bottom of a computer screen to obtain user-screen proximity estimates as independent variable input. Based on the proximity estimates, three desk activities are recognised: (1) working in front of the computer screen (ScreenWork), (2) working at the desk but not in front of the computer screen (DeskWork), and (3) being away from the desk (Away). We considered that users did not require the computer screen to be powered during DeskWork and Away conditions.

An efficient detection of desk activities can be achieved by analysing the USR sensors’ field of view: when the computer screen is adjusted to the user, USRs typically provide reflection measurements with similar proximity estimates during ScreenWork, whereas Away could be described by the absence of reflections. For DeskWork, we considered that users move from the front of the screen to either side of the desk, e.g., to perform paper work, making phone calls, drinking coffee, etc. User behaviour during DeskWork results in dissimilar proximity estimates of the two USRs located at either side of the screen. In addition, we used raw proximity measurements to detect movement of approaching users, as detailed in Sect. 3.2 below. Figure 1 illustrates the workplace configuration for a deployed APC system.

Fig. 1
figure 1

Configuration of an office workplace, as considered for our APC system. Three desk activities are recognised from user-screen proximity estimates derived by two USR sensors attached to the screen: (1) ScreenWork, (2) DeskWork, and (3) Away, but only ScreenWork requires the computer screen to operate. Proximity variance due to upper body motion is subsequently used as implicit user feedback to evaluate screen switch-off operations

Full size image

3.2 Activity-based screen control

Based on recognised activities per office desk, the Control component provides screen switching commands. The computer screen is switched on or off using a remotely controllable power plug that connects screen and power outlet. Hence, the APC system operates independent of the exact computer hard- and software used at each desk, as long as the computer screen has a separate power interface. Finally, the Adaptation component is used to dynamically adapt Control component parameters that influence the switching operation. The APC architecture is illustrated in Fig. 2.

Fig. 2
figure 2

APC system architecture overview. Based on USR proximity estimates, desk activities are recognised. The identified activity is used by the Control component to generate screen on/off commands. Screen commands are sent to a power switch actuator that interrupts the screen’s power interface. Control parameters are dynamically adapted using implicit user feedback in the Adaptation component via the STOFF variable

Full size image

We denote the screen time on (STON) as the time required for the screen to turn on after user movement in front of the screen was detected, or ScreenWork was recognised. To achieve a low STON, i.e., to quickly detect user movement, the APC system monitors changes in raw proximity measurements. Upon detecting an approaching user via reduced proximity readings, the screen-on command would be sent. Subsequently, the activity classification output is used to maintain the screen on/off state value.

We consider STOFF as dependent variable and system parameter to trade-off between energy saving and user comfort. STOFF is the delay time before a control operation (switch-off) is sent to the power switch. Ideally, the screen switch-off operation should occur immediately after transferring from ScreenWork to another activity state, thus maximise energy savings. However, due to the noisy USR measurements (Jaramillo and Amft 2013), larger STOFF is likely to increase user comfort. We control STOFF by dynamically adjusting the delay time window ω k (see Sect. 3.3 for details).

Figure 3 illustrates the control parameters and their timing behaviour for an example scenario involving a ScreenWork episode with respect to ground truth (GT). With reference to the notation introduced in Fig. 3, we expect the time delay before t 2 to be very short: t 2 − t 1 ≤ 0.5 s. Since an approaching user is likely to use the screen, t 3 − t 1 ≤ ∼3 s. The delay time window ω k controls STOFF to dynamically change the delay time of the screen-off control command after the activity Away or DeskWork was detected. The time difference t 6 − t 5 (observation window O k ) was kept constant. The switch-on operation was subsequently evaluated by measuring the parameter GSTON, hence the time between screen-on control and ScreenWork according to GT (t 3 − t 2). The switch-off operation was subsequently evaluated by measuring the parameter GSTOFF (t 5 − t 4).

Fig. 3
figure 3

Action timing of the APC system for an example ScreenWork episode. Upon detecting user movement at time t 1, the computer screen is switched on (t 2). ScreenWork will be reported at t ′3 only, due to recognition delay after the ground truth (GT) label at t 3. Following the end of the screen work episode at t 4, another activity will be classified at t ′4 (here Away). The timing parameters are further detailed in the main text

Full size image

3.3 Control parameter adaptation

The Adaptation component adjusts the decision time window ω k based on implicit positive and negative feedback information. We interpret upper body motion as implicit negative response when it occurs immediately following a switch-off operation. We denote negative feedback as increased proximity variance within observation window O k . Upon negative feedback, the Adaptation component aims at correcting erroneous operations by increasing ω k at a fixed rate. When ω k is large, the screen would not switch off during short activity periods of Away or DeskWork. Thus, perceived screen flickering is reduced, which, in turn, improves user comfort.

Positive feedback is triggered when in the observation window with duration O k , no user movement is detected following a control operation. The positive feedback is modelled as confidence based on the count of positive proximity sample observations at ω k . The frequency of positive feedback is considered as confidence in the current delay window size ω k . Once the delay window confidence exceeds a fixed threshold, ω k is updated to a smaller value randomly chosen in the interval given by the previous and the current ω k value. These parameters are formally described in Sect. 4. Figure 3 illustrates the timing behaviour of the control parameters. The observation window O k is used as interval for capturing implicit feedback.

4 APC system implementation

After pre-processing raw proximity data into distance estimates, features were extracted from the proximity values. User movement was detected and desk activities were classified using these features. Subsequently, Control and Adaptation components used the movement detection and activity recognition results to operate the screen and dynamically adapt control parameters. This section details all processing steps.

4.1 Pre-processing and feature extraction

We derived the following features: proximity per sensor, proximity variance per sensor, sum of the variances of sensors, object motion per sensor, and difference of motion between sensors. Figure 4 illustrates the APC system processing and actuation implementation.

Fig. 4
figure 4

Illustration of the APC system components, functions, and symbols used. Based on sensor data pre-processing (P), feature extraction (F), and activity classification (C), raw proximity measurements were converted into activity states (a k ). The Adaptation component consists of motion detector (MD) and delay adapter (DA) functions that process implicit user feedback from the raw proximity measurements. The control decision (CD) receives activity states (a k ) and the motion feature (m k ). Finally, the edge delayer (ED) outputs the switching operation (o k ) based on the adaption parameter (ω k )

Full size image

Proximity variance features This feature group helps identifying changes in activity and can be used to detect objects, such as chairs that exhibit a proximity variance close to zero. During ScreenWork, proximity variation greater than zero is likely due to user movements. We further consider the sum of the proximity variances of both USR sensors to support recognising ScreenWork.

Motion features This feature group consists of a binary movement/no movement detector, based on proximity sample differences. For ScreenWork, we typically expect small differences between proximity samples compared to DeskWork. By computing the difference between left and right sensor motion, a relative motion measure was obtained, indicating whether the user is present at either side of the screen.

4.2 Activity classification

The computed features were used in different combinations with threshold classifiers to recognise three activity states: DeskWork (difference of motion between sensors), Away (sum of proximity variances, raw proximity data), and ScreenWork (sum of proximity variances with the rest of possible states). The classifier output was provided to the Control component that translated the activity states into switch-on and switch-off commands of the screen power controller. For the online system, all stages were implemented using the Context Recognition and Network Toolbox (CRNT) (Bannach et al. 2008).

4.3 Adaptation component

The Adaptation component provides Motion detector and Delay adapter functions. Using the Motion detector, feedback feature m k was derived using a movement detector on the raw proximity measurements. Equation 1 details the decision rule, where d l and d r are the raw proximity measurements of the left and right USR sensors and k is the discrete time index.

$$m_{k} = \big( (d_{l_{k}} - d_{l_{k-1}}) > Th \big) \mid \big( (d_{r_{k}} - d_{r_{k-1}}) > Th \big).$$
(1)

The Delay adapter provides the dynamic system behaviour based on ω k , described in Sect. 3. Equation 2 shows the formal definition of all parameters considered for the adaptation. Here, we denote negative observations (N o ), positive observations (P o ), and the delay window confidence (Co). P o_counter and N o_counter count positive and negative observations respectively. Equation 3 describes how ω k is updated. Here, Step denotes the relative change of ω k and rand is an uniform random number with 0 ≤ rand ≤ 1.

$$\begin{aligned} k_{0} & = k {\quad {\rm if}\ } o_{k-1} = 1 \quad{\rm and}\quad o_{k} = 0, \\ N_{o} & = o_{k} = 0 \quad{\rm and}\quad m_{k} = 1 \quad{\rm and}\quad k_{0} < k < k_{0} + O_{k}, \\ P_{o} & = o_{k} = 0 \quad{\rm and}\quad m_{k} = 0 \quad{\rm and}\quad k \geq k_{0} + O_{k}, \\ Co & = \frac{P_{o_{counter}}}{P_{o_{counter}}+N_{o_{counter}}}, \end{aligned}$$
(2)
$$\begin{aligned} \omega_{k_{new}} =\left\{ \begin{array}{*{20}l}\omega_{k} + Step & {\quad {\rm if}}\quad N_{o} \\ \omega_{k} - rand\cdot(\omega_{k} - \omega_{k_{old}}) + \omega_{k_{old}} & {\quad {\rm if}}\quad Co \geq Co_{threshold} \quad{\rm and}\quad \omega_{k} > \omega_{t_{old}}\\ \omega_{k} - rand\cdot(\omega_{k} - \omega_{k_{min}}) + \omega_{k_{min}} & {\quad {\rm if}}\quad Co \geq Co_{threshold} \quad{\rm and}\quad \omega_{k} < \omega_{t_{old}}\\ \omega_{k} & \quad \hbox{otherwise.} \end{array}\right. \end{aligned}$$
(3)

4.4 Control component

The APC system uses a feed-forward control mechanism, where each system component depends on the preceding one as illustrated in Fig. 4. The control command o k indicates the binary switch-on and switch-off operations, depending on to the recognised user behaviour.

A control decision is obtained as logic or between the activity classification output a k and the feedback feature m k in the Control decision CD function block. The logic combination allows the controller to rapidly issue screen-on operations (when user movement occurs), as it does not rely on a k , but rather on intermediate motion features.

The Edge delayer responds instantly to changes into the activity state ScreenWork, but delays the transition into other states by the delay window time ω k . Equation 5 describes the implementation of the filter functionality, where Eq. 4 shows k 0, indicating the time when transitions from ScreenWork occur. As a result, the screen-on time is maintained for to prevent flickering effects.

$$k_{0} = k, {\quad {\rm if}}\; x_{k-1} = 1 \;{\rm and}\; x_{k} = 0,$$
(4)
$$\begin{aligned} o_{k} =\left\{ \begin{array}{*{20}l} 1 & {\quad {\rm if}}\quad x_{k} = 1 \\ 1 & {\quad {\rm if}}\quad x_{k} = 0 {\; {\rm and}\quad } k_{0} < k < k_{0}+\omega_{k} \\ 0 & {\quad {\rm otherwise}} \\ \end{array} \right. \end{aligned} .$$
(5)

5 Intervention study methodology

We evaluated the APC system in an intervention study deployed in a real-life office environment with 12 participants. We evaluated the system recognition performance, investigated potential energy savings, and assessed user comfort parameters. Results of the analyses are reported in Sects. 6 and 7. In this section, the intervention study deployment is described, including the annotation method and the user evaluation techniques.

5.1 Study design

Participants We selected a floor of the Electrical Engineering faculty building at TU Eindhoven. This floor comprises shared and individual offices, all occupied by students and academic staff of the faculty. Other rooms, atypical for an office building due to the temporary presence of occupants, such as students room, pantry area and meeting rooms were excluded. From the individuals initially interviewed, 12 participants from mixed office types, accepted to participate in our intervention study. Most of the rejections were obtained due to absence and holidays. The final set of participants were aged between 24 and 45 years and had a technical or administrative background. Most of the participants were PhD students. Upon signing the informed consent form, participants were included in the study.

Procedure Participants were introduced to the study goals and the APC system through personal appointments and participant information sheets. As part of the instructions, we asked the users to perform daily routines as they usually do. In addition, we asked them to note their office activities as frequently as possible, and report specifically on ScreenWork, DeskWork and Away. Participants were provided with a web-based annotation tool and we sent them regular email reminders to complete annotations.

The study was divided into two phases: a baseline (BS) recording of 11 days was followed by the intervention study (IS) recording during the next 8 days. During BS, screens were not controlled. BS was used to collect information about the typical energy consumption from all participants. During IS, the APC system was used to control screens according to recognised user activities and implicit feedback, as described in Sect. 4.

Installations The APC system was installed in all 12 workplaces of participants. The workplaces were distributed across nine different offices. We used two USR sensors from Davantech model SRF08 (Robot Electronics 2011a, b) and a plug-in power meter from Plugwise also known as a Circle (Plugwise 2012). USR sensors were mounted on top or bottom corners of the computer screens, angled such that they faced the user in ergonomic screen working conditions. The sensors covered a field of view of approximately 45° in the horizontal plane. Ranging was set to measure distances below 150 cm for both sensors. We obtained distance measurements from both sensors at a rate of 2 Hz. A window size of 2 s was used for feature extraction. Both USR sensors were interfaced to a gateway computer, via commercially available USB-I2C modules (Robot Electronics 2011a, b). The USRs consumed a peak-power of ∼1.375 W during initialisation and typically ∼55 mW during operation (ranging mode).

Instantaneous power consumption was continuously measured per computer screen using Plugwise Circle networks, sampled at a rate of 1–1.5 sa/min per plug, depending on the number of devices per network. Circles were interfaced to the gateway computer via ZigBee. The actuation of computer screens was controlled using the APC system during the intervention study as described in Sect. 4.

For the study implementation, CRNT (Bannach et al. 2008) was extended to perform the following processing functions: reading USR sensors and plug devices of each workplace, extract features, classify activities, and implement all adaptation of control components functions. We installed CRNT instances on the gateway computers per office room (totally 9 instances). All gateways were monitored via network connection from a central server.

5.2 Estimating ground truth (GT)

We obtained annotations of ScreenWork, DeskWork and Away activities from the study participants via the web-based annotation tool, as described above. However, annotations for the entire APC intervention were not sufficiently detailed and accurate. In a post-processing step, we defined rules for an automatic labelling procedure based on the USR sensor data. The automatic labelling was subsequently applied to the full dataset to derive GT. Furthermore, we derived GT for four randomly selected full-day recordings based on user annotations and by additionally reviewing USR sensor data. The manually derived GT from these reference days was then compared to the automatic labelling annotations. Using a sample-by-sample analysis, we found a correspondence of 83 % between both methods. We considered the correspondence sufficient for the subsequent performance analyses and thus used the automatic labelling output as GT for all recordings (see Sect. 8 for more details).

Automatic labelling was implemented by low-pass filtering the raw proximity data to eliminate proximity fluctuations. ScreenWork was automatically labelled when both proximity sensor signals were greater than zero and when the proximity variance exceeded a threshold. Similarly, DeskWork was selected if only one of the proximity sensor signals was above zero (user at one side of the desk) and the variance feature exceeded the threshold. This threshold was empirically estimated and set to 1,600. The labels for both ScreenWork and DeskWork activities were then elongated to an episode of at least 30 and 10 s, respectively. Overlaps were subsequently resolved with precedence for DeskWork. Finally, Away was chosen for all remaining samples that had not been assigned to another class. The automatic labelling parameters were found through manual optimisation and performance analysis, compared to the manually derived GT.

5.3 System operation performance evaluation

To assess the APC operation performance, we analysed screen-on and screen-off operations performed by the system compared to the GT reference. We account screen-on and screen-off operations according to Eq. 6. Following the timings illustrated in Fig. 3, a screen-on operation (t 2) was considered as correct event (E on ) when it occurred before the start of a screen work period in GT (t 3), i.e., while a user approaches the screen. Conversely, a screen-off operation (t 4′) was considered a correct event (E off ), if it occurred after the end of an actual screen work period. The time measures GSTON (t 3 − t 2) and GSTOFF (t 4 − t ′4 ) were used to verify the timing behaviour.

$$E_{on,i} = \left\{ \begin{array}{*{20}l} 1 & {\quad {\rm if}}\quad t_2 \leq t_3 \\ 0 & {\quad {\rm otherwise}} \end{array}\right., \quad E_{off,i}=\left\{\begin{array}{ll} 1 & {\quad {\rm if}}\quad t_4 \geq t_4'\\ 0 & {\quad {\rm otherwise}.} \end{array}\right.$$
(6)

Furthermore, we defined the event recall (A e ) as the ratio between the sum of correctly operated screen-on and screen-off events (E on , E off ), and the total amount of events (T e ), according to Eq. 7.

$$A_e = \sum_{i=i}^{T_e} \big\{ E_{on}, E_{off} \big\} / T_e.$$
(7)

5.4 Energy consumption evaluation

In order to calculate an average screen power consumption P, we used power samples P k measured in Watt using the plug-in power meter as described in Eq. 8. Screen energy consumption per day was derived by considering the screen use time H in hours, where N is the total amount of power measurement samples.

$$P=\frac{1}{N} \sum\limits_{k=1}^{N}P_k [W],\quad E_d=\frac{1}{\text{1,000}} P \cdot H [Wh].$$
(8)

From Eq. 8, we can see that the typical average power varies according to the screen brand, while screen use time per day depends on the user presence. Therefore screen use time was not comparable across participants. To compare energy consumption during BS and IS recordings, we first normalised the energy measurements by presence time to compute the energy consumption per day E d using Eq. 8. The average consumption over study days was used separately for BS and IS, hence E b d and E i d . The comparison of BS and IS per participant shows the amount of energy saved per participant during the APC intervention, when compared to the BS recording session.

E d was used to compute the daily energy consumption for both BS and IS days, E BS d and E IS d , using Eq. 9.

$$\begin{aligned} E^{BS}_d= \frac{1}{B} \sum \limits_{d=1}^{B}E^{b}_d [{\rm kWh}] , \quad& E^{IS}_d=\frac{1}{I} \sum \limits_{d=1}^{I}E^{i}_d [{\rm kWh}], \end{aligned}$$
(9)

where B are the total BS days and I are the total IS days.

5.5 User comfort evaluation

In order to assess user comfort with the APC system, we designed pre-study and post-study questionnaires. The pre-study questionnaire aimed at assessing user habits about office screen control. The participant responses were gathered through interviews of about 5 min duration. The starting item was whether the participant controls his/her office screen, comprising the following three choices: manually, via computer software, or no control. Afterwards, the questionnaire provided a section of follow-up questions aiming at identifying specific details about each control type, such as: “I manually turn off the screen during lunch periods”, “I use the energy preferences of my computer software and have set a sleep time of 10 min”. In addition, we provided the option to note more details about the control strategy.

The post-study survey was provided to the participants at the end of the study and aimed at identifying relevant user comfort parameters. The questionnaire was presented to participants via a web interface, including 17 questions (see Table 1) with a scoring scale between 1 and 5, where 1 meant Strongly disagree and 5 meant Strongly agree. The first ten questions were taken from the SUS (Sauro and Lewis 2012), that according to the ISO standard of 1998, defines usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”. Additionally, we evaluated different system characteristics by designing seven further questions focused on system response time and efficiency of the adaptive control mechanism. For all questions, we assigned positive and negative tones resulting in comfort-related (C) and discomfort-related (D) subgroups (see Table 1). Furthermore, we included an open section to add remarks and suggestions for future implementations.

Table 1 Post-study questionnaire used to assess user comfort
Full size table

6 System operation performance results

A set of operation performance metrics were used to evaluate the APC system. We measured the detection algorithm accuracy against GT. Then, we performed a comparative analysis across study participants to analyse adaptation performance and control parameters for the implicit feedback. Finally, using an event-based analysis, we evaluated the recall for screen-on and screen-off system operations.

6.1 Online recognition algorithm performance

A sample-by-sample accuracy analysis of the recognition vs. GT for detecting Away episodes showed an overall per-class accuracy of 98 %, representing a 10 % improvement over the previous implementation (Jaramillo and Amft 2013).

The accuracy improvement stems from the feature combination selected for the APC system design presented in this work. In particular, the proximity variance showed to be effective in differentiating between users in front of the screen and common office objects, such as chairs.

6.2 Analysis of delay window time ω k

In our adaptation approach, we used the delay window time parameter ω k to denote the STOFF delay, as described in Sect. 4. Figure 5 shows an example of the ω k size variation over the IS recording days. For almost all participants ω k was bounded below 60 s. Figure 5 also shows that the APC system was able to adjust ω k based on implicit positive and negative feedback. As expected, during the initial days of the intervention the system executed more adaptations than towards the end. It is noticeable that already at day 4 of the intervention, the slope of ω k was reduced, indicating a steady operation state for most participants. Although from days 5 to 6 more further increases in ω k were observed, the reduction on day 8 for participant 2 indicates that suitable values were found at ω k  ≈ 30 s.

Fig. 5
figure 5

Examples of the temporal size variation and adaptation of the delay window time parameter ω k for three participants across the full intervention study duration. During the first IS days, more adaptations were performed than during the final days. A reduction of ω k indicate the effect of a positive implicit feedback for day 8 of participant 2

Full size image

6.3 Analysis of the observation window parameter O k

For all the analyses, the observation window O k was set sufficiently large to capture implicit user feedback. Based on previous experience, we chose O k  = 5 s. We analysed the distribution of the time to feedback per participant. On average, time to feedback was below 2.5 s, suggesting that O k was chosen bigger than necessary. If O k could be reduced, positive feedback would be obtained faster and therefore energy saving would be larger than in the configuration used during the IS. Figure 6 shows the average time to feedback for all study participants.

Fig. 6
figure 6

Average time to feedback for all study participants. Observation window O k was chosen based on previous experience, O k  = 5 s

Full size image

Furthermore, using O k and the feedback received from users during the intervention study, we calculated estimates for the number of corrections executed by the Adaptation component in order to investigate potential timing optimisations. Specifically, we used the control signal recorded during the APC intervention to simulate feedback within an O k  = 5 s, i.e., the number of corrections the system would perform. The underlying assumption is that after every control operation for turning screens off, the observation window O k would be maintained to capture implicit user feedback, i.e., a motion in front of the screen. Figure 7 shows the actual feedback instances received during the APC intervention. For participant 7, a large number of corrections were observed. From the daily recording reports we confirmed that most hardware problems occurred for this participant. In comparison, when the feedback instances due to faulty hardware were removed (estimated corrections for O w  = 5 s), the correction count was reduced. For participants 8 and 12, estimated corrections were larger than actual corrections made, suggesting that the adaptation system was able to correctly switch on the screen using feature information, rather than based on reacting to implicit user feedback only.

Fig. 7
figure 7

Corrections executed by the APC system for each participant during the intervention and corrections simulated. Corrections were simulated for O k  = 5 and 2.5 s, based on actual system operations recorded during the APC intervention. The comparison suggests a decreased number of corrections for smaller O k , thus indicating that further increases of energy savings are feasible

Full size image

In a post-study analysis, we evaluated O k  = 2.5 s. From Fig. 7, it can be seen that the number of system corrections would fall below 100 corrections (equivalent to the maximum ω k  = 60 s), thus allowing the APC system to turn off the screen faster and thus improving energy savings. One important case to further analyse is participant 4, for whom, even after the simulated reduction of O k , corrections exceed 100. This suggests that other system parameters may require adjustment.

6.4 Event-based performance of screen-on and screen-off

GSTON and GSTOFF variables were used to verify timing behaviour of screen-on and screen-off control operations. The analysis was performed by comparing timestamps of control operations recorded during the APC intervention against and that of the corresponding GT event. Figure 8 shows the estimated mean and variance of GSTON and GSTOFF parameters per user. For most cases, the screen-on was realised at least 15 s before the actual start of ScreenWork activity. This indicates that the system was able to anticipate the screen use. Subsequently, screen-off events occurred on average 20 s after the actual end of ScreenWork.

Fig. 8
figure 8

Average GSTON and GSTOFF per participant. GSTON and GSTOFF were measured as time differences between control operations and the GT event. Following the APC design requirements to increase user comfort, negative GSTON were desired, as it indicates that a control operation was executed before the GT event, i.e., the screen turns on before the user starts ScreenWork. Conversely, positive GSTOFF were desired, as it indicates that a control operation was delayed beyond the GT event, i.e., the screen turns off after the user finishes ScreenWork

Full size image

We calculated the recall of screen-on and screen-off control operations according to Sect. 4. We analysed screen-on and screen-off control operations separately. We accounted for 9,130 correct screen-on events out of 9,346 total instances. Conversely, 8,860 correct screen-off events out of 9,346 were counted. The recall results of the analysis are shown in Fig. 9. An event-based performance of at least 90 % was observed for screen-on events. The lower performance of screen-off events can be associated to the adaptation mechanism, which incurs errors especially at the beginning of the recording session and during the observation window size probing.

Fig. 9
figure 9

Event-based performance for screen-on and screen-off control operations. The recall of screen-on and screen-off control events was obtained from the count of correct events (E on , E off ) normalised by the total event count Te as defined in Sect. 3. Overall, the system responded to screen-on and screen-off events with a recall above 90 %

Full size image

7 Intervention evaluation results

In this section we present the energy saving analysis by comparing BS and IS study phases. Furthermore, we describe findings related to user comfort and the system usability assessment.

7.1 Comparative analysis of energy savings

With E BS d and E IS d , the energy saved across all study participants and conditions, i.e., BS and IS were determined. As Fig. 10 and Table 2 show, 21 % of the electrical energy was saved on average. The per-participant results show a considerable variability in the energy savings. This effect is further discussed in Sect. 8.

Fig. 10
figure 10

Overall energy savings achieved per participant over the 8 IS days, compared to BS measurements over 11 days. The average energy savings achieved are shown as ‘Overall APC energy savings’

Full size image
Table 2 Energy consumption during BS and IS phases and savings achieved for all 12 participants
Full size table

In previous work (Jaramillo and Amft 2013), it was found that a high user activity variation enables the system to save more energy by switching off the office screens for short periods of unused time. The APC system proposed in this study introduces a new saving condition based on the delay window time ω k . ω k varies across participants and can be adapted from a minimum of 2 s up to 60 s, as described in Sect. 4.

Due to hardware errors during the study, some instantaneous power consumption measurements were lost (see Sect. 8 for a more detailed). Our energy saving analysis per participant considers power measurements with respect to actual presence time of each participant, as listed in Table 2.

7.2 User evaluation results

The implementation of the pre-study questionnaire showed that around 56 % of the participants were used to manually control their screens. Most participants turned the screen off when going home at the end of the day. Others were used to turn off their screens during lunch breaks and meetings. Only a few participants were used to turn off screens when doing other desk-related activities, other than computer work. Nearly 28 % of the interviewed participants did not control their screens actively, but rather unplug the screen from the laptop and leave the screen to go on standby mode. Three out of 12 participants said they use power management preferences provided by the computer operating system and adjusted timeouts times to 10 or 2 min.

User comfort was evaluated using a post-study questionnaire. Several analyses were carried out in order to (1) identify relevant comfort parameters using the individual question’s scores and (2) analyse identified important relations, i.e, to overall energy savings. For the following analysis, we used the averaged question scores from subgroups of positive and negative tone questions, that we regard as comfort- and discomfort-related questions.

In the analysis of individual questions, we found that 67 % of the participants perceived the APC system as easy to use. In fact, 83 % of the users answered that the system operation was easy to learn, 75 % said that no previous knowledge of tools was required and 75 % of the participants commented that they did not required any support from technicians. Regarding the system response time, participants answers showed a normal distribution centred around the agreement values. We observed that system efficiency was difficult to evaluate for participants, as 33 % reported that the system was not efficient, 50 % agreed that it was efficient, and 17 % remained neutral about this characteristic.

Participants commented during and after the study on the noise produced by the USR measurement operation. We found that 33 % of the participants who rated the system to enhance workplace comfort or said that the system was lean to use, positively perceived the system as not too noisy. By contrast, the same number of participants felt the system was very noisy and rated the enhancement of the workplace comfort or “lean to use the system” negatively. Overall, the user evaluation showed that it offers good efficiency and response time. The participants seemed to accept the APC system as usable technology, even though, the system can be still improved towards comfort enhancement.

We assessed the comfort/discomfort-related subgroup of questions. Both indicated neutral responses, with 58 and 42 % of the participants respectively. We observed that comfort-related items were answered by 33 % of the participants in a positive way, with respect to only 8.3 % of negative responses. Discomfort-related items were answered by 50 % of the participants in a negative way. This analysis confirms a correct interpretation of the user perspective, regarding discomfort-related questions, while still some confusion exists regarding the interpretation of comfort-related questions. Figure 11 shows the subgroup scores with respect to the actual energy savings made. Comfort-related scores remain close to neutral independent of the energy saving, potentially decreasing for largest overall energy savings. These findings confirm that our objective to balance user comfort and energy saving by means of an adaptive control approach was achieved.

Fig. 11
figure 11

Comfort- and discomfort-related subgroup scores with respect to overall energy savings per participant. The neutral comfort ratings confirm that the APC system balances user comfort and energy saving by means of an adaptive control approach

Full size image

8 Discussion

Our APC system aims at supporting office workers in controlling computer screens. Our results confirm that balancing energy consumption and user comfort is feasible, where the probing re-calibrated ω k using forward corrections based on implicit user feedback. While in earlier work (Jaramillo and Amft 2013), energy savings above 40 % were obtained, the APC system presented in this work achieved 21 % only. However, the adaptation mechanism used in the present study showed larger user acceptance, which seems essential for an actual deployment.

The present study confirmed that the energy saving potential depends on personal habits, such as the frequency of interrupts during screen work. To this end, the APC system uses implicit user feedback to continuously re-calibrate control parameters. Hence the APC system balances between energy consumption and user comfort. The adaptive control approach used in this work could also ease system commissioning, as key system parameters do not need manual calibration per desk or user. Furthermore, the intervention study results hint at additional energy saving potential: shorter observation window settings for O k could be sufficient to capture implicit feedback, hence the time to switch-off the computer screen could be reduced.

Assessing user comfort and energy savings in a real-life intervention study raised methodological challenges. In this section, we discuss the APC design, deployment, and limitations of the present work.

8.1 USR sensors to detect desk activities

Various modalities could be considered for recognising desk-related activities. Advanced computer vision methods exist that could be applied with commonly available webcams. However, the field of view of webcams is typically limited to the head region. Repositioning the camera conflicts with its primary use and thus limits detection options. Furthermore, delays in switching screens on would negatively affect user comfort. As a consequence, cameras may require a similar two-stage detection approach as presented in this work, to identify approaching users rapidly.

It can be expected that sensor device costs will continue to drop. The configurable USR sensor device used in our study could be replaced by a cheaper model that has lower depth resolution, since only user-screen distances below 1 m are relevant for the proximity-based control. To this end, USRs can compete with accurate IR sensors. However, for IR sensors, measurement accuracy would be affected by the reduced field of view, varying lighting conditions, etc.

8.2 Proximity-based features

Any office screen model could be used for the automatic control. In earlier work (Jaramillo and Amft 2013), only proximity estimates were thresholded to determine activities that required ergonomic arrangements to achieve sufficient accuracy. In the APC system presented in this work, a combination of variance and motion features was used that showed constant results across participants with different habits and sitting positions in front of the office screen. Therefore, neither different ergonomic positions nor office screens affected our findings.

8.3 Intervention study installation

Hardware failures during the intervention study increased user discomfort, hence negatively affecting the user’s perception of the APC system. Most errors could be attributed to the prototypical state of the APC system installation. For example, we observed sporadic errors in acquiring proximity and power consumption measurements due to temporary wireless connection losses.

For non-recoverable errors, the APC system was restarted or the failing equipment was replaced by the study managers. The failures had an effect on the system’s control parameter adaptation. We implemented a recovery function, such that every time the APC system was initialising, the last adaptation parameters settings were recovered. Furthermore, we observed that erroneous proximity measurements of the USR sensors were compensated by the APC system adaptation. In Sect. 6 we detailed how this effect was investigated.

Other issues included malfunctioning of the power meters, resulting in loss of power measurements. Measurement losses could explain some of the variation in energy savings between participants. Moreover, a low activity variation for some participants could explain variations in energy savings, i.e. for participants that changed activities frequently, larger energy savings can be expected. However, the impact of activity variations could not be confirmed from an analysis of GT labels.

The APC system used proximity measurements from USR sensors that emit ultrasound waves at ∼40 kHz. During ranging, the sensors exhibit an audible sound, i.e. low-volume “clicks”. While the USR sampling rate was configured to a minimum frequency in order to recognise desk-related activities, some participants felt disturbed by the sounds. When asked, participants expressed that they perceived the sound as unusual and hence provided a lower rating on workplace comfort and for the item “lean to use the system”. However, we regard our results not as conclusive regarding user distraction by USRs or the applicability of USR sensors for the APC system. In further studies the sound effect could be studied and optimised with users. Alternative proximity measurement approaches could be considered, if the distracting effect would persist.

8.4 Sample size and ground truth trade-off

High-quality user annotations from which GT could be derived were difficult to obtain with the present study design. While we developed a web-based annotation tool for this study, participants were not able to record the exact moments in which a ScreenWork activity started and ended due to inherent delays before they can use the computer screen. Furthermore, self-reported user annotations may not be sufficient as we observed that participants forgot to annotate activities while being involved in their daily routines. As a consequence, we believe that our post-processing was useful to revise user annotations and incorporate expert knowledge as procedure to minimise errors in processing the large data amounts. A correspondence of 83 % between annotations obtained by the automatic labelling and the manual labelling was hence regarded as sufficient for our performance analyses.

The APC system study aimed at analysing the feasibility of the proposed implicit user feedback and obtaining insight on the options to balance energy consumption and user comfort. To implement the intervention study, we interviewed all ∼30 occupants of the office spaces at the considered building floor, except for undergraduate student rooms, as described in Sect. 5. A key exclusion criteria for our study was if occupants could not be present during the considered recording period. Consequently, participant sample included in the study resulted from the limitations in continuous availability of the considered population. In order to extend the sample, a longer study period would be needed for baseline and intervention phases. At the same time, system installation and activity annotation would raise efforts for participants too. We believe that the results obtained with our present study of 11 days for baseline and 8 days for intervention phases and totally 12 participants with varying screen types, habits, and ergonomic conditions, the benefit of further investigations on the APC system approach can be warranted.

9 Conclusions

In this paper, we proposed an online recognition algorithm and APC system which does not require user-specific preliminary training. Thus, our approach permits to use an activity-pattern based recognition and control system without specific commissioning of essential design parameters. The combination of the proximity features showed recognition accuracies above 90 %. A novel implicit user feedback approach was introduced in order to correct erroneous control operations and dynamically adjust the control parameter delay window time ω k . The delay window provided an effective means to reduce screen flickering, which supports a comfortable system operation.

We deployed the APC system in a real-life intervention study over 8 days, after an 11 days baseline and included 12 participants. The overall energy consumption was reduced during the APC intervention by up to 21 %, when compared to the baseline measurements where no control of the screens was implemented.

Using an extended version of the SUS questionnaire, we further evaluated user comfort of the APC system. The positive and negative tone of the questions was confirmed and a subgroup analysis of comfort- and discomfort-related questions showed the the APC system can effectively balance energy consumption with user comfort.