Keywords

1 Introduction

User modeling is a core area of research, which aims at endowing machines with intelligent mechanisms to understand and model human-like behavior. It finds vast applications in areas such as passive surveillance, health monitoring wearables, domotics, fitness applications, wireless sensor networks, and robotics to name a few [1].

In this work, we focus on a target group which consists of older adults (age 55+) that are still active in a computerized work environment. In general for this age group, virtual assistance systems [2] can play an important role in their expectations for an active professional future, since typically they find it difficult to adapt their capabilities so as to cope with the most recent software changes, interface layouts, and computer work paradigms used in their work institutions. The CogniWin EU AAL project [3] aims at providing a personalization framework for motivating older adults to stay for longer active in the workplace. Thus, a virtual assistance system with learning and adaptation skills according to the user specificities is proposed, which also provides well-being guidance. Assistance to the older adults is delivered by analyzing multi-sensorial behavioral data provided at the low-level by an eye tracking system, and an intelligent computer mouse.

In this article, we will turn our attention to the intelligent computer mouse component of CogniWinFootnote 1, denoted as CogniMouse. Just like any other computer mouse, this human interface device enables the user to navigate within the operating system, and more importantly trigger computer actions as a result of physically clicking the mouse buttons. However, unlike other commercial mice, it encompasses several sensors that are additionally used to perceive human actions while working on a computer. Early detection of patterns and deviations from standard behavior is the main motivation of our work, as it enables to prevent further cognitive degradation in older adults by designing proper user intervention when abnormal episodes occur.

Besides developing the appropriate software drivers to extract the low-level sensor data, we propose a multi-sensor fusion process to detect whether the user is feeling frustrated while working on a computer with CogniMouse. Understanding that a user is frustrated is of key importance, as it enables a virtual assistance system to act accordingly to safeguard the user, e.g. by suggesting to take a break, or assisting the user in a computer task that he/she cannot complete by him/herself. Having this in mind, we not only provide a definition of user frustration, but also formulate a probabilistic algorithm that is based on a multivariate Bayesian Classifier [4], which takes processed sensor measurements and historical data as inputs, and continuously provides the likeliness of a user to feel frustrated.

In addition to describing the method to implicitly identify user frustration while using CogniMouse to interact with a software application, we plan a future validation study of the approach with a large group of active older adults in an end-user organization. Furthermore, we present a use case scenario to highlight the practical advantages to the user when running the proposed approach.

The article is organized as follows: in Sect. 2 we overview seminal work on frustration measuring and exploiting various inputs to examine user behavior. Afterwards we present CogniMouse, the multi-sensor computer mouse developed in the scope of the CogniWin project in Sect. 3, and describe how we identify frustrated behavior events while the user interacts with the system. We then reveal our planned method of study so as to validate the sensor fusion classifier and present a use case scenario in Sect. 4. Finally the article ends with conclusions and a discussion of future research directions in Sect. 5.

2 Related Work

In the context of information technology and services, the literature reveals several works for detecting user frustration based on implicit user modeling approaches. Typically, frustration is detected and represented as a binary phenomenon (e.g., frustrated vs. normal state) [7, 10, 11], while more recent works represent frustration in multiple levels [14] given that individuals might react differently (lower to higher levels) depending on the context of use (e.g., an individual experiencing a slow internet connection while booking a flight ticket online vs. an individual browsing the Web over coffee). Popular examples of user frustration detection include approaches based on facial analysis [6], pressure sensors embedded in computer mice [7], behavioral observations and user system interaction analysis [8, 9], speech patterns [10], and electrocardiograph data of individuals [11]. Other approaches include automated methods based on fusing multiple channels of affect-related information such as video camera, pressure sensing chair, pressure mouse and skin conductance measurement [12], and studies that compared frustration levels extracted through eye-tracking, mouse tracking tasks and electroencephalograph signals [13]. More recent works have focused on detecting user frustration in mobile contexts. Taylor et al. [14] developed models that predict multiple levels of user frustration through physiological signals retrieved from external devices (armband, wireless heart rate detector, finger clip skin conductivity sensor). A user study was conducted in a mobile setting aiming to evaluate the accuracy of the developed models by triggering user frustration through system response delays. Results revealed that the models of physiological responses classified five levels of frustration with over 80 % accuracy. Gao et al. [15] presented an alternative approach in the context of mobile devices, avoiding complications of external sensors, and instead leveraging how users touch and use mobile screen devices.

figure a

Considering the definition of Frustration presented above, in the next section we present an innovative probabilistic classification algorithm that receives real-time signals and physiological measurement streams given by an in-house instrumented computer mouse, named CogniMouse, so as to identify frustration events of older adults at work.

Fig. 1.
figure 1

Mouse sketch design.

Full size image

3 CogniMouse

CogniMouse is an advanced human interface device built around a standard computer mouse. Having the project’s goals in mind [3], the sensors chosen when designing and assembling the device were: galvanic skin response sensor, temperature sensor, inertial measurement unit (IMU), grip/pressure sensor, and heart rate sensor.

The main innovative goal behind this smart device is to detect older adults’ hesitation [16], emotional states (e.g., anxiety) and frustration occurring whilst performing tasks in a personal computer. Hence, its user-friendly and familiar design illustrated in Fig. 1, has been selected to ensure the comfort of older adults, making them more likely to accept CogniMouse when carrying out their work. In fact, introductory interviews show that the design choice is particularly important, as users feel more comfortable with designs that are similar to the mice they normally work with. Furthermore, despite the suite of sensors included, the mouse has shown to be completely unobtrusive, and users frequently face it as a standard computer mouse.

Supporting Windows 7, 8 and 10, the software was developed in C# object-oriented programming language under the Sharp Develop integrated development environment (IDE). The C# language has been chosen for two main reasons: (i) it provides a good trade-off between efficiency, security and robustness and; (ii) it provides a number of libraries for communication with hardware components (e.g., human-interface devices and sensors). Being able to run in the background without interfering with user actions, the software acquires multi-sensory information in real time. Besides this, the classification software leverages valuable information such as mouse motion and click streams through low-level Operating System (OS) calls, a priori knowledge of the user health profile and history of sensor data. From the combination of all these components, it is possible to assess the user’s conditions. More particularly, we focus on the detection of symptoms associated to frustration episodes, such as: lack of control, agitation, general unhappiness, aggressiveness, etc. Furthermore, a Web-based graphical user interface has been designed and developed that illustrates in real-time the raw signals of the computer mouse, the user’s current state (e.g., increased frustration) and historical data about prior user states (Fig. 2).

Fig. 2.
figure 2

Web-based graphical user interface of CogniMouse illustrating the raw signals of the mouse and the user’s current and prior states.

Full size image

A classification algorithm grounded on probabilistic theory is currently under testing. The algorithm continuously provides a level of certainty at which the user might be experiencing frustration. A probabilistic solution has been utilized due to the recognized results in the state of the art on classification algorithms from multiple sources with bounded levels of uncertainty [17].

In particular, we employ a Bayesian-based formalism inspired on conditional probability distributions to solve the problem due to its flexibility of incorporating new variables/inputs. The inputs used for the classification algorithm are: (i) grip force; (ii) acceleration vector and; (iii) click stream frequency. We use numerical series of past measurements of frustration with recursively decreasing weights to model the input prior distribution. Likelihood functions for each input have been derived heuristically by defining increasing influence to high deviations or abnormal input levels, considering the user typical health parameters. An independent frustration measurement is obtained by applying Bayes formula at each step considering prior and likelihood distributions, paired with a normalization factor that scales the result to a [0, 1] interval:

$$\begin{aligned} P (Frus|Grip,Acc,Click) = \frac{P(Frus) \cdot P(Grip|Frus) \cdot P(Acc|Frus) \cdot P(Click|Frus)}{P(Grip) \cdot P(Acc) \cdot P(Click)} \end{aligned}$$
(1)

The formula is applied whenever the user is working with CogniMouse. This is given by a skin conductance threshold, which allows the software to know when the hand of the user is touching the mouse surface, via the galvanic skin response sensor. The high-level classification approach is illustrated in Fig. 3.

Fig. 3.
figure 3

High-level data flow diagram of CogniMouse.

Full size image

In the next section, we reveal our plans to validate the method and present a use case scenario to highlight the importance of detecting user frustration.

4 Planned Evaluation and Use Case Scenario

Method of Study. A validation of the algorithm proposed is currently planned with a large group of active older adults in an end-user organization. The aim of the planned studies is threefold: (i) study the relation among user frustration and physiological signals extracted from the developed computer mouse; (ii) investigate the detection accuracy of triggering events of frustration; and (iii) adjust and improve the approach developed in an iterative mode.

The study’s sample will consist of around 45 older adults (age 55+) that work in computerized environments. Following existing experimental approaches [13, 14, 16], we planned several controlled laboratory sessions in which participants will perform a series of tasks (e.g., change a setting on their desktop-based email client). While performing the tasks, the computer mouse will record the motion stream data and the users’ physiological signals through the embedded sensors. We will follow the think-aloud protocol during the studies, utilizing screen capturing software and audio data for post-analysis by user experience experts. During user interactions, frustration events will be triggered by employing delays in the system’s response [14] as well as providing tasks with different levels of difficulty (some tasks will be very hard, or impossible to complete). Frustration triggering events will be analyzed and cross-verified by annotations from psychologists and experts which will be witnessing the trials attentively. Finally, a post-study survey will be conducted based on the Rosenzweig Frustration Test (RFT) [18] to profile the users regarding their tolerance to frustration, and the NASA Task Load Index (NASA-TLX) [19] to assess the users’ perceived workload and frustration aiming to triangulate the results.

Use Case Scenario. Susan is a 58 year old employee at Zuyderland Hoogstaete in Sittard, Holland. She is part of the entertainment staff of the extramural care department in the elderly home care facilities. She is an experienced worker who began her career 35 years ago in another regional elderly home institution. She has been working as a member of the entertainment staff for the last 6 years, and she is considered a very kind employee, being especially good in organizing activities for the elderly. In the last year, Susan has been experiencing some problems with her memory, which according to her doctor, is normal for her age. Besides that, she still feels young and active, and her intention is to stay for a few additional years working in the company in a paid mode.

Susan has basic computer knowledge and skills, and has been able to manage her computerized work at the institution so far. However, she feels that her younger colleagues can work more easily with the utilized software and sometimes she feels that she cannot handle new technological challenges that arise. At home, her children help her to do essential things e.g. online banking. At work, Susan often asks younger colleagues to help her with the computer tasks, such as writing the daily reports of the elderly.

The management of Zuyderland Hoogstaete recently decided to buy a new activity registration system called Citard Active. All seniors in Susan’s department are going to use this application. The application, whose front-end is accessed on a tablet, smartphone or all-in-one PC make it possible for elderly to subscribe for activities, invite friends, has a notification reminder and provides accessory advice. All members of the entertainment staff have a key role in providing the system with relevant information about their elderly via an authentication and secure management system at the back-end.

With the arrival of Citard Active a few more responsibilities and work are expected from Susan. As a consequence, she is getting more and more insecure and nervous and she is even considering an early retirement. Susan experiences a lot of anxiety when working with Citard Active, because it makes her feel tired and she often gets frustrated from her non-productive way of using it and difficulty in the adaptation to the new system. She also finds it hard to read all the information on the screen and distinguish the different buttons. After 2 weeks working with the system, Susan hardly manages to login. Entering new activities for the elderly is very complex for Susan as well as other functionalities of the system. Therefore, she constantly needs help from a colleague.

Her director recognized her concerns and provides her with an innovative computer mouse called CogniMouse that will assist her to adapt her process’ operations to accommodate the changes that the new software caused. She was told that the mouse will monitor her computer tasks’ activities and adapt the new workload to her performance in order to avoid overloading her and all the frustration and performance loss that could be generated. One day, Susan was having a hard time in changing the properties of an activity that was previously created in the system. After a few failed attempts, she felt agitated and overwhelmed. CogniMouse detected her frustration by analyzing the data collected from the embedded sensors, and the contextual data from the operating system. Accordingly, the system presents a graphical help wizard of how to further proceed by guiding Susan’s mouse actions to the graphical system area that contains the next step of her process. Susan felt really happy for the assistance provided. Later on, in that same day when Susan was having a hard time creating a different activity type from the ones that she usually inserts, the system detected her frustration and advised her to stretch her legs and take a short break to clear her mind. Susan was very surprised to see that the system helps her as she feels that CogniMouse is refreshing her memory and assisting her to complete the tasks. Thus, she feels much more motivated at work. Moreover, when facing negative emotions she gets relieved by inspecting her health parameters on the screen so as to assess her own state. The quality of work and job satisfaction increases a lot and Susan can now manage to work for many years as an appreciated member of the entertainment staff.

5 Conclusion and Future Work

In this paper, building upon a classical definition of frustration, we have described a methodology to implicitly detect users’ frustration in real-time while interacting with a computer mouse in the work environment. For this purpose, an existing off-the-shelf computer mouse was redesigned and developed, embedding physiological sensors for measuring in real-time the users’ frustration levels. A user study is planned in the short-term future for the validation of the approach and a use case scenario has been designed to demonstrate the impact that our solution will potentially have in the effectiveness and motivation of older adults in the work environment.

In the future, we intend to combine information extracted from CogniMouse with additional sensors such as an eye tracker, and contextualize the mouse data so that the system provides adaptive support taking the user’s task into account when frustration is identified. Moreover, this work also opens interesting prospects to similar classification methods for detection of user states in the future, such as stress and anxiety, tiredness or boredom while using the system.