Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Virtual assistance systems can play an important role for increasing the productivity in professional work environments where employees are engaged in complex computerized tasks, which require an intensive cognitive effort and high levels of concentration. Virtual assistance providing easy to understand and self-adaptive guidance is paramount to increase the user’s efficiency when interacting with computers. For older adults, aged 55 and above with expectations for an active professional future, virtual assistance can provide even greater benefits [1].

Typically, when working in highly computerized environments, members of this age group are required to attain new knowledge and adapt their capabilities to cope with fast software changes in their organizations. This often results in hesitating behavior and anxiety, which can cause long-term discomfort and frustration at the workplace [2].

CogniWin proposes to continuously track different biometric measurements, aiming to implicitly extract information about the user physiological status, such as arousal states in stressfull conditions, while interacting with the computer. Advanced monitoring is achieved combining data input from three devices: An instrumented intelligent mouse, an eye tracker, and the Microsoft Kinect One for Windows.

Besides acquiring real-time data from different sources, the system captures information about the actions that are being performed by the user, and provides a repository to store important user-related contextual information obtained a priori, such as health profile and user capabilities. Thus, the system is able to transmit knowledge for both existing and new employees and automatically adapt and improve by capturing, saving and learning daily activities, as well as contextualizing them.

Multi-sensor data fusion will drive advanced behavior analysis to aid in real-time user interaction analysis and recognition of abnormal behaviors. This will in turn trigger adequate personalized well-being advice actions, assisting the user to quickly achieve predetermined goals and improve performance.

In order to address CogniWin’s objectives, the project tackles technological innovation beyond the state-of-the-art in the following areas: (i) Development of an affordable intelligent mouse integrating multiple sensors to measure physiological parameters and corresponding software to analyze the collected data, in order to provide personalized support to the user; (ii) Design and development of a user profile based on specific metrics of cognitive processing parameters, which greatly influence the execution of computerized tasks; (iii) Design and development of adaptive and natural user interfaces based on user context and on their related cognitive model; (iv) Development of innovative context-aware ICT services for active and adaptive on-demand support and access to integrated information, guidance and well-being advice, promoting increased performance and problem-solving abilities to older adults with a minimum of support from other people.

2 The CogniWin Architecture

The architecture can be broken down in four separate layers: hardware, which is closely connected to the lower level components, including device drivers and the database. The higher level components, used to analyze and store data and finally user interface components, for user interaction.

The hardware and lower level components layer includes: (i) Knowledge repository and database; (ii) Eye tracker; (iii) Intelligent mouse; (iv) Microsoft Kinect One for Windows. The higher level components include: (i) Behavioral analysis; (ii) Contextual recorder; (iii) Data fusion; (iv) Cognitive model; (v) Application model. Finally, the user interface components include: (i) Well-being advisor; (ii) Personal learning assistant. Figure 1 shows the overall logical architecture of the CogniWin system.

Fig. 1.
figure 1

The overall CogniWin architecture.

Full size image

2.1 The CogniWin Components

The hardware integration and lower level software components comprise the basic layer of the architecture. Lower level software components are usually drivers (i.e. intelligent mouse, eye tracker, Kinect One) of the hardware components which are connected to the PC. The database is the central place of data storage in the CogniWin system. A special part of the database is the knowledge base which contains and manages all the information about the individual users performance. This component receives inputs from the contextual recorder and the data fusion components. This last one combines data from the eye tracker, the intelligent mouse and the Microsoft Kinect One. These two components are in the higher software layer.

Microsoft Outlook [3] and Citard Active [4] (running in Google Chrome) are the client applications currently supported by the application module. As the CogniWin system should be usable with many more applications, each one will need a custom application model. This model contains information about all possible tasks as well as interaction points (buttons) and metadata, such as the application name and version. The contextual recorder is able to detect the active window and store the corresponding application name and version. By searching through this metadata, the personal learning assistant is then able to find a suitable application model.

Fig. 2.
figure 2

CogniWin hardware components.

Full size image

2.2 The Technology Used

The CogniWin system runs on a workstation, where all data recording and processing as well as user input and output takes place. For the pre-trials, standard office Personal Computers were selected by the end-users partners to perform preliminarily testing with the system. This real world scenario intends to demonstrate that the CogniWin system functions properly on existing computers without the need to buy new expensive hardware. The Workstation requires at least two free USB 3.0 ports to connect the eye tracker and the Kinect One for Windows and one USB 2.0 port for the intelligent mouse. It should be a 64-bit machine with a minimum of 4GB of RAM. Since the algorithms used by the eye tracker and the Kinect One are computationally intensive, the workstation should at least have a mid-range processor.

The Eye Tracker (Fig. 2a) is a product of “THEEYETRIBE” [5]. It has a small form factor (\(20 \times 1.9 \times 1.9\) cm) and can be used together with desktops, laptops and tablets. It does not need a dedicated power supply, and it can even be used on the go. The tracking works by illuminating the face of the user with infrared LEDs and capturing video footage of it. The video is analyzed with eye tracking algorithms which extract gaze related data. This technology is suitable for most environments, but is best used indoors without direct sunlight on the device. It supports screen sizes of up to \(24\,''\) and has two sampling rates: 30 Hz and 60 Hz.

The Intelligent Mouse (Fig. 2b) is developed in the scope of the CogniWin project. It embeds sensors to measure the galvanic skin response (GSR), grip force, heart rate, temperature, and trembling. It can be used to detect user stress, anxiety, fatigue, lack of confidence and more, [6–8].

The Microsoft Kinect One (Fig. 2c) is a depth sensor device developed by Microsoft, which has been intensively used in research lately, due to its use in innovative natural human computing solutions across a variety of industries. It encompasses depth sensing technology based in time-of-flight (TOF) of infrared light, a built-in color camera, an infrared (IR) emitter, and a microphone array, enabling it to sense the location and movements of people as well as their voices.

The CogniWin system supports the Microsoft Windows Operating System. A relational database management system is used in connection with the knowledge repository as data storage. The knowledge repository works as an abstraction layer, decoupling the database from all other components. This enables the use of other languages than SQL to store data, since only the knowledge repository deals with the database itself. CogniWin also makes use of RabbitMQ 3.42, which is a message broker enabling communication between the different components. For example the eye tracker component will send its data asynchronously via RabbitMQ to the data fusion component, which will process it and send it to the knowledge repository for permanent storage which can be then requested by the well-being advisor.

Fig. 3.
figure 3

Behavior analysis component overview.

Full size image

3 User Behavior Analysis

In order to provide personalized support to the user, the CogniWin system must be able to perceive and contextualize the user behavior. Having this in mind, several cues are extracted from the sensors and devices at the lower level of the architecture, which are combined with user’s task performance data and prior knowledge to assess the user’s state and trigger assistance (cf. Fig. 3).

When using the intelligent mouse endowed with an advanced embedded sensor suite, the system will continuously monitor and measure physiological parameters [9], such as the galvanic skin response and sweat of the hand of the user, gripping force, heart rate, temperature, and measurements of shaking or hand trembling. Figure 2b illustrates the first prototype of the intelligent mouse based on a common model from Microsoft. Additional measurements are provided via the contactless eye-tracker, namely, eye gaze point, pupil size, blinking rate, fixation rate and duration, saccades rate and duration, and maximum velocity. Moreover, the Microsoft Kinect One [10] can eventually be leveraged to extract other information, e.g. user’s posture, facial expressions, gesture interpretation, and voice tone.

As seen above, user-related information can be extracted using the lower level devices. However, it is also crucial to extract task-related information, therefore enabling CogniWin to contextualize the user’s task, assess its performance and infer on the user behavior. To this end, the Contextual Recorder component will be responsible to log the user’s keyboard and mouse events, and identify which task, process or services the user is running so as to determine the context according to the actions performed. This is based on log trails, screenshots, mouse motion and key-loggers.

The Knowledge Repository will collect the data extracted by the Contextual Recorder, and also retain the older adults’ know-how, as well as their health profile. This will allow the system to learn and adapt to the user specificities, as well as providing personalized assistance.

The user interaction with the system results in the combination of the outcomes of the aforementioned modules via the Sensor Fusion component. By means of advanced behavioral analysis and reasoning algorithms, considering prior health, personal and cognitive characteristics, contextual data, and online real-time monitoring of physiological parameters, several different user behaviors will be recognized. Hence, the goal of the Behavioral Analysis component is to identify the following user behaviors when performing a task: Normal state, Hesitation, Drowsiness, Vigilance, Fatigue, Cognitive overload, Stress and Anxiety, and Frustration.

Finally, after the identification of the distinct user states, other modules are responsible to trigger an adequate action to assist the user in a personalized way, as seen in the next section.

4 User Support

The features of personalized assistance and guidance in the CogniWin system are provided essentially by two components, respectively the Personal Learning Assistant (PLA) and the Well-Being Advisor (WBA).

4.1 Personal Learning Assistant (PLA)

The PLA module provides personalized tips and advice (textual, video, audio) on contextualized tasks based on the users cognitive characteristics, on the identification of the ongoing working activity and based on integrated indicators (e.g. time per task, tasks organization, level of concentration, etc.) thus aiding the user to achieve goals and improve performance. The PLA will intervene in case of detecting abnormal user performance so as to provide adaptive support and guidance services to reduce the anxiety and stress level of the user.

The PLA system addresses mistakes and hesitations of employees while performing tasks by providing them a contextual help or a step-by-step procedure for their tasks achievement. A common case is when the end users already know how to use a system but find difficulty in certain tasks. It also provides suggestions for recovering from mistakes or suggestions for performing options when hesitations are detected.

The PLA is a top layer service working in close connection with the Knowledge Repository, the Contextual Recorder and the end users’ cognitive model. It compares at runtime, users activities with their specific record of tasks and historical activities. The PLA will display a help invitation, once it detects:

  • First time Task

  • Decreased performance

  • On user request

These are taken into account in order to refine users preferences and their corresponding cognitive model for upcoming tasks.

First Time Task (FTT). Based on the past recorded events (x,y mouse position, active software, action) the system is able to detect if the user is doing the associated task for the first time or not. In the first case, the system will display a help invitation describing the action undertaken by the user and the list of associated documents, videos or pictures stored in the Knowledge Repository. The user can either choose the most suitable form of help or decline the invitation. Based on cognitive models the PLA will feature the most suitable form of help. If the user chooses another form, the preferences will be taken into account for the next PLA invitation. Since this is a FTT, no user personalized metrics are yet associated with it. The basic comparison point will be the median stored metrics (if any) resulting from different users while dealing with this task. Otherwise, the user performance will be stored as reference and associated to the task.

Example: The user places the mouse on x,y (screen spec) corresponding to the “create an automatic reply” (Task) on outlook (active window) and clicks on it (trigger). The help display window appears and invites the user to choose between a tutorial, a video or some pictures showing different scenarios. Based on the user’s cognitive model, the video will be the first link followed by the pictures and finally the tutorial.

Decreased Performance (DP). The algorithm to detect DP assumes that the user has completed the task previously and that there is a record in the database associating it to the user. In this case (DP) the time between each click will be longer than the median time stored in the database. Depending on the median value, a minimal time should be specified for each task before considering that the user needs help. For example, if it takes around 10 seconds to click on the next link after having chosen “create an automatic reply”, the help invitation will not be displayed immediately after the median time is exceeded. In order to keep it friendly, we will display it only after the median time has been exceeded by 15 seconds.

On User Request. During the user’s activity, the system is continuously monitoring the active window and the mouse position. Once the user clicks on the PLA icon, in the active task bar, the PLA will display help according to the user’s current activity. As it can be difficult to determine exactly for which tasks the help is being requested, the invitation will display all existing information associated to the active window screen.

4.2 Well-Being Advisor

The Well-Being Advisor (WBA) is a top layer component, along with the Personal Learning Assistant. It interacts with the computer users and provides personalized advice to prevent unwanted age related health situations and to alleviate eventual decrease of cognitive capabilities, thus effectively preserving and improving the users well-being.

The WBA addresses unwanted situations that are known to substantially reduce the users productivity such as tiredness, incorrect sitting position, anxiety and stress, also preventing health related work absences. The WBA makes use of the advanced monitoring sensors from the CogniWin system (e.g. the intelligent mouse and the eye tracker). These devices measure several physiological and visual parameters which in turn enables the WBA to analyze the output and provide personalized support. The WBA also takes into consideration the users health related characteristics by checking a profile previously stored in the system database.

When the user is too tired the productivity will decrease. The Well-Being Advisor will address user tiredness following two distinct approaches: promoting regular work breaks, and actively detecting when the user exhibits tiredness symptoms.

Promoting Regular Work Breaks. Tiredness symptoms can be partially mitigated by promoting work breaks at regular intervals. This is based on the idea that frequent breaks can improve mental agility [11] and prevent the user from draining all its energies. One of the most well know time management methods following this idea is the “Pomodoro Technique” [12] which uses a timer to break down work into intervals, traditionally 25 min in length, separated by short breaks. CogniWin implements this method allowing the user to configure the work and pause intervals. Furthermore, we intent to combine this static method with information from the active sensors to allow more flexible decision making. For instance, we can anticipate a short pause if the sensors detect tiredness or delay the pause if the system knows the user is concentrated and being productive.

Active Detection of Tiredness Symptoms. Using information from the eye tracker sensor it is possible to determine when the user is too tired for work or even if the user has fallen asleep. The WBA continuously monitors the eye tracker measurements, filtered by the Data fusion and Behavior Analysis components and decides when to advise the user to take a break.

Anxiety and Stress. Anxiety and stress are two well-known factors responsible for causing health and productivity degradations on workers [13]. Older users can feel more anxious or stressed when dealing with frequent software changes or when facing new computer tasks. The CogniWin system issues a warning message proposing a short break or stress reducing exercises when it detects that the user is experiencing high levels of anxiety or stress. To perform this task, CogniWin takes advantage of the physiological sensors continuously measuring the users Galvanic Skin Response (GSR) which is a direct indication for the anxiety and stress levels being experienced by the person, again filtered by the Data fusion and Behavior Analysis modules [9].

5 Evaluation

The initial system evaluation will take place in the pre-trail phase of the project, during February 2015. The Pre-trials evaluation will: (i) Provide feedback to the CogniWin developers, on topics such as the validation of the architecture and the specification of all individual modules. This will ensure that as early as possible in the project, we obtain invaluable insight about most of the concepts and usability issues, which undoubtedly can make for much more robust final prototype implementations, (ii) Provide input to the CogniWin developers regarding the user interface from the high-level modules (Personal Learning Assistant and Well-being Advisor). We will analyze the users real interactions to determine if the proposed assistance is well accepted and what adjustments will be needed, (iii) Validate and provide feedback for the evaluation metrics themselves (both automatic and user feedback). Adjustments to the evaluation metrics, methodology and tools will be made following the pre-trial results.

5.1 Pre-trials Installation

The pre-trial pilot hardware and software will be installed and evaluated at the premises of Orbis Medisch en Zorgconcern, Netherlands [14] and ArgYou AG, Switzerland [15], the two end-users organizations of the project consortium. In the case of Orbis Medisch en Zorgconcern, the pre-trial systems will be tested on two separate facilities, both situated in the city of Sittard-Geleen in the Nederlands. One of the facilities is the Orbis Medical Centre (OMC), a general regional hospital, built in 2009 and presently one of the most advanced hospitals in Europe. The other facility is the Orbis Hoogstaete, a modern elderly care center. For both, the pre-trials will be performed in conference-rooms and will run on all-in-one PCs with mouse, keyboard and the eye-tracking device. The target application is the Citard Active software, normally used by the administrative staff. Orbis participates with seven volunteer users that will have the opportunity to test the system and also fill-in the questionnaire at the end of the trial period.

The user group is composed by adults aged 55 or older, with or without light physical or cognitive age related limitations, who are engaged in computerized tasks that require an intensive cognitive effort and a high level of concentration. For the Orbis Hoogstaete facility, this specifically includes care coordinators, occupational therapists, and animation employees. For the Orbis Medical Centre the users are office administrative employees.

In the case of ArgYou AG, the pre-trial systems will be tested in the CogniWin office at ArgYou AG in Berne, Switzerland. For the pre-trial, all-in-one PCs with intelligent mouse, keyboard and eye-tracking device are available. Microsoft Outlook software will be available on each PC. All the participants of the trials are included in the following criteria: over 50 years-old and working with branch specific knowledge (tourism, banking, hotel business, food and beverage industry, insurance companies, etc.). All participants will sign an informed consent which is specified by the project’s privacy protection plan.

Table 1. Overview of the evaluation objectives.
Full size table

5.2 Evaluation Goals and Methods

After a initial preparatory phase, the CogniWin functionalities available for the Pre-trial will be activated. During this phase the CogniWin system will collect data and interact with the user to provide assistance. After the Pre-trial ends, all automatically collected data will be analyzed to calculate the objective evaluation indicators. The user will be requested to fill an online questionnaire at the end of the Pre-trial for assessing subjective evaluation aspects such as measuring the user acceptance and satisfaction with the new input devices and the CogniWin system.

Table 1 represents the evaluation goals and the corresponding indicators that will be calculated for assessing the project objectives. We divided the evaluation into subjective indicators, obtained from the user’s responses to the online questionnaire and objective indicators, calculated from the stored data. At the time of writing this article, the results of the pre-trials evaluation are not available yet, and will be the subject of deep analysis in the short-term future.

6 Conclusion and Future Work

The paper presents the first prototype implementation of the CogniWin system, an innovative virtual assistant to support and motivate older adults that work on computerized activities, to remain active and productive by providing personalized learning assistance and well-being guidance. The consortium recognizes the importance of supporting this target user group in their work environment, and the CogniWin system is an answer to such challenge, aiming to effectively enabling older adults to cope better with software changes in their workplace and organizations. The paper describes in detail the different modules of the CogniWin system, as well as the methodology developed to evaluate the first pre-trial tests performed with real end-users. It is also noteworthy that our implementation is not restricted to a generic help functionality. The help provided is adjusted to address the user’s preferences and to take into consideration previous interactive sessions, where the user successfully completed the planed tasks. Additionally, the measurement of physiological and performance parameters, when using the CogniWin system, will help the user to obtain optimized support as well as perform automated detection when unwanted situations occur. During the second project year, we will focus on enhancing the system with more functionality which has been planned in the scope of the project. This implementation especially involves the implementation of more enhanced data fusion and behavioral analysis components, as well as a cognitive model. These components will enable the system to address the person’s needs in a more optimized way. The aim is that the Cogniwin system gets self-learning functionalities to provide the user (or user groups) the best and unobtrusive support possible. In the cognitive model component we plan to group behaviors and preferences of individual users but also user groups. Depending on these specifications also new users will be supported in an optimized way. The cognitive model will take the information into account gathered from the behavior analysis as well as the contextual recorder. The first results have shown that a system like CogniWin is appreciated by the user and can be helpful to support them in their daily work with the computer. Especially when there is a difficult software systems involved. Anyway we think that it will be very important to provide the user an optimized support. Otherwise the system can easily be considered as annoying and will not be used. To be able to provide this optimized support on individual level, we first gather unobtrusive data (intelligent mouse, eye tracker, contextual recorder) and on the other side provide high self-learning capabilities of the system as well as a complex behavior and data analysis. Both facts are considered as being helpful to provide a highly accepted system which will be used and is efficiently supporting the older people at work.