ADELA: a conversational virtual assistant to prevent delirium in hospitalized older persons

Delirium is a sudden mental state that causes confusion and disorientation, affecting a person's ability to think and remember clearly. Virtual assistants are a promising alternative for non-pharmacological interventions. This research aims to present a prototype of ADELA, a conversational assistant to prevent delirium in hospitalized older persons who speak Spanish. A co-creation process with medical experts to identify requirements was carried out to later develop the assistant iteratively and evaluate it from a technical and usability perspectives; the latest using the Spanish version of the System Usability Scale (SUS) and the Chatbot Usability Questionnaire (CUQ), supplemented by qualitative data. Mean values of 75.5 and 85.94 were obtained for SUS and CUQ, respectively. The technical evaluation helped defining the minimum environmental specifications required for deployment. Obtained results imply the assistant is usable and potentially accepted by the target population; useful information was extracted for refinement. Technical evaluation showed positive results, indicating it can be used in a real clinical environment.


3
ADELA: a conversational virtual assistant to prevent delirium… 1 Background

Clinical context
Delirium is a deterioration in mental functioning that occurs within hours or days and is usually triggered by an acute medical pathology, a trauma, or drugs. It is one of the most frequent medical emergencies, with a prevalence of around 20% in patients admitted to medical services, and even higher in special services such as Orthogeriatric Units or in Intensive Care Units [1]. The prevalence of delirium in hospitalized older patients has skyrocketed 90% in some moments of COVID pandemic [2].
The positive side is that delirium is preventable in 30-40% of the cases [3], being one of the most beneficial strategies for primary prevention multicomponent non-pharmacological interventions [4].
Virtual assistants may provide cognitive stimulation, social interaction, and sensory engagement to older patients during hospital stay. This can be achieved through a variety of methods, such as personalized conversations, games, and activities that are tailored to the patient's interests and abilities. Virtual assistants can also provide reminders for medication and daily routines, as well as information about the hospital environment and procedures to reduce confusion and anxiety [5,6].
The use of a virtual assistant in the prevention phase of delirium could be particularly useful in contexts where patients are at high risk, such as cases of cognitive impairment, sensory deprivation, or a previous history of delirium [7]. This virtual assistant could also be used in combination with other preventative interventions such as early mobilization, sleep promotion, and medication management, to provide a comprehensive approach.
Smart personal assistants, such as Amazon Alexa, allow searching for information or scheduling events among other functionalities. However, the use of virtual assistants in the health field is still to be explored, so its potential acceptance by older people is uncertain.
The ADELA project, funded by Fundación MAPFRE, aims to design and develop a conversational virtual assistant to prevent delirium in older people (+ 65 years old) during hospitalization. This population is the most prone to develop delirium during hospital admission, due to age-related changes in cognition and physiology [8].

Related work
Balsa et al. [9] proposed an intelligent personal assistant (VASelfCare) to help older people coping with type 2 diabetes. The assistant is integrated in a mobile app and the interaction is done via voice or gestures. A usability evaluation of the assistant obtained a SUS score of 73.75.
Dimeff et al. [10] presented a virtual assistant (Dr. Dave) aimed at reducing unnecessary hospitalizations and suicide events in general patients. To assess 1 3 usability, authors used the Usability Satisfaction and Acceptability Questionnaire (USAQ), adapted from SUS. A promising average USAQ score of 4.4 (out of 5) was obtained.
Ireland et al. [11] proposed a conversational assistant (Harlie) to treat neurological conditions such as Parkinson's disease. Focus groups were organized to receive feedback from final users, including older people. In general, users had a positive impression, but they found some technical issues and problematic conversational responses.
Inkster et al. [12] developed a conversational assistant (Wysa) embedded in a mobile application focused on mental health for the general population. This system was tested in a study where participants showed depressive symptoms, obtaining an engagement above 70% among the users by doing an analysis through the Patient Health Questionnaire (PHQ-9) [13].
Sun [14] investigated how visual enhanced on voice user interfaces, such as conversational assistants, might mitigate usability challenged by older adults during interaction. The articled concluded that integrating visual output as a feedback mechanism facilitates interaction between older adults and assistants.
Liu et al. [15] examined older adults' preferences for intelligent virtual assistants regarding information modality and feedback. Results showed that the visual-auditory bimodality is superior to single visual modality and single auditory modality for older adults.
Markfeld et al. [16] conducted a study to examine the effect of different feedback modalities (visual and auditory) in a table setting robot assistant for elder care. The visual feedback included the use of LEDs and a screen. The combination of LED lights and verbal commands increased participants' understanding, contributing to the quality of the interaction.

Objectives
The main objective of this research work is to design and develop ADELA, a conversational virtual assistant to prevent delirium in hospitalized older persons. Specific objectives are: • To implement a co-creation process with domain experts from the clinical field to extract functionalities and operational requirements. • To perform an iterative development process along with clinical domain specialists to produce a functional prototype of ADELA. • To refine the ADELA prototype through a usability study with potential end users. This enhanced version of ADELA will be used in a clinical trial aimed at demonstrating its usefulness to prevent delirium in hospitalized older patients (recruitment to be started in December 2022).

3
ADELA: a conversational virtual assistant to prevent delirium…

Material and methods
This section presents the methodology used to conceptualize, design, develop, and refine the conversational assistant. Figure 1 shows the followed methodology, consisting in 5 phases, to achieve the final version of the conversational assistant.

Co-creation process
The purpose of this phase was to extract information from domain experts to focus and guide the overall design process of ADELA.
The co-creation process was conducted through a series of meetings and workshops, which brought together a heterogeneous group consisting of three medical specialists (geriatricians), that are considered the domain experts, and two technologists. Prior to the meetings, ideas were proposed to reach common ground. The purpose of this process was to enable the development team to obtain a more holistic view of what the system should include and how it should behave. The Delirium Prevention Protocol by the Geriatrics Service of the Getafe University Hospital and the clinical guidelines of the National Institute for Health and Care Excellence [17] were used as baseline to reach the functionalities presented in Table 1.
The Human-Centered Design approach was adopted to ensure that ADELA would be accepted and adopted. This approach involves understanding the users' needs, behaviors, and preferences through research and analysis, and then designing solutions that meet those needs. Participatory design methods, such as design workshops and usability tests, were employed to actively involve users and stakeholders throughout the design process [18].
These methods enabled the development team to gain insights into the users' mental models, expectations, and needs, and to ensure that the design of ADELA aligns with these factors. The use of participatory design is especially important in cases where the adoption and acceptance of new technologies by potential users is low, as it helps to create a usable and successful software system by perfectly adapting the design of the software to the mental model of the potential users [19].
The workshops were used to specify all the relevant aspects of the users and the system's context of use and to translate them into a system's design. These workshops involved focus groups [20] participated by the co-creation team. The second method, usability tests, was later used to evaluate the effectiveness of the first designed prototype in meeting user needs and expectations.

Functionalities
Given the set of requirements identified in the previous phase, ADELA was designed to have the functionalities described below.

Basic intents
ADELA incorporates a set of basic functionalities that are launched by the user, called intents, which were developed to follow the flow shown in Fig. 2.
To activate basic intents, it is necessary for the user to wake the assistant up by using the wake word "Adela". A similar system is used in assistants such as Amazon Alexa [21]. A pre-trained neural model known as Porcupine from PicoVoice platform [22] was used for this task.
Each basic intent has been implemented in AWS Lex, a cloud service capable of extracting a user's intent from a recording. Additional cloud services such as AWS Lambda (to process intent responses) and AWS Polly (text to human speech) were used to complete this functionality.
The assistant incorporates 25 intents in total such as asking for the time, weather information and start a memory game, among others.

Reminders
ADELA provides reminders throughout the day, keeping users active with different activities and helping them remember things. Eight types of reminders, described in Table 1, have been included. The list of reminders is stored in an AWS DynamoDB cloud service database.

Playing relaxing music
According to Table 1, ADELA offers the possibility to play relaxing music during the user's preparation for sleep, to create an appropriate atmosphere. To this end, several relaxing songs have been incorporated into the assistant.

Phone calls
ADELA can receive phone calls to facilitate communication between patients and their relatives, thanks to the integration of the Simcom SIM868 module [23]. This module works like a mobile phone.
A whitelist with relatives' phone numbers is stored within AWS DynamoDB service to avoid nuisance calls.

Lighting control
According to Table 1, ADELA must be able to control the illumination of the room where it is located to help patients maintain circadian rhythms. The lighting control has been implemented using a table lamp incorporating a smart bulb, but it is scalable to more complex settings.
The assistant automatically turns on and off the room lighting throughout the day, but it can also receive commands from the user.

Cognitive games
ADELA incorporates six different memory games to promote cognitive stimulation. Games included are explained in detail in Appendix 2. These games require cloud services such as AWS Lex, Dynamo DB, and Google Speech-to-Text.

Prototype design
The design of the working prototype was conducted partially in parallel to the functionalities phase described above. Figure 3 represents the conceptual architecture of the system, including ADELA assistant. Components and relationships are described below:

Conceptual architecture
• The ADELA virtual assistant is the central technological component. The older person communicates with the assistant using the voice. • The environment automation consists of a Wi-Fi lamp, that is remotely controlled by the assistant.

3
ADELA: a conversational virtual assistant to prevent delirium… • External servers correspond to the cloud services used to meet the assistant's needs, such as a database. • The medical professionals are responsible for creating personalized intervention plans, and adapting the assistant to the patient's needs and habits. • An alert log is maintained to be analyzed later in case it is necessary. • The relatives of the older person can communicate with him/her thanks to the built-in phone call functionality provided by the assistant. Figure 4 shows the physical prototype of the device running ADELA along with its components. The prototype includes a LED strip, controlled by an Arduino Nano, with different colors to indicate the status of the assistant (see Fig. 5). This functionality has been inspired by the work presented in [14,15] and [16]. It has been claimed that visual-auditory bimodality in assistants and combining LED lights and verbal commands improves the interaction between assistants and older people.

Usability evaluation
The main objective was to collect feedback and suggestions from potential users to improve the prototype, and thus minimize potential usability problems arising from an improbable design. This is necessary to ensure good usability, user experience and acceptance of the ADELA assistant in a real-life scenario (i.e., first in a clinical trial and later in routine care). The experiment consisted of an individual 30-min session with each participant following the script described in Appendix 1.
This evaluation was designed to be an observational study in which ten older persons (N = 10) participated. Participants, recruited at the Geriatrics Service of Getafe University Hospital, interacted with ADELA in a single working session and provided feedback. The tests were carried out in a room within the hospital dependencies replicating the same conditions of a potentially realistic scenario. The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of Getafe University Hospital (protocol code A06/22 approved 23 June 2022). Participation criteria were:  o Inability of the participant to understand and use the ADELA system.
Usability is defined as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [24][25][26]. Used assessment tools were: • System Usability Scale (SUS) [27]: short 10-item Likert questionnaire that provides a measure of people's subjective perceptions of the usability of a system. These 10 items can be evaluated from '1-fully disagree' to '5-fully-agree'. Total score ranges from 0 to 100. • Chatbot Usability Questionnaire (CUQ) [28]: Likert questionnaire consisting of 16 items aimed at measuring usability of chatbots. • Ad-hoc satisfaction questionnaire: customized questionnaire with 5 open-ended questions to collect general user feedback (see Table 2). These questions were designed to complement aspects that are not directly covered by quantitative tools such as SUS and CUQ so additional fallbacks of the evaluated prototype can be identified and thus overcome.

Technical evaluation
Every usability evaluation session was audio-recorded to extract technical conclusions and later analyze the performance of the system and understand its limitations. Data used in this analysis comes from the usability phase (N = 10).

Refinement of the assistant
This phase consisted in introducing the changes to be drawn from the evaluation of the first prototype with potential end-users. This led to a refined version of the assistant that will be deployed in a clinical trial at Getafe University Hospital.

Results and discussion
Descriptive statistics are used within this manuscript to analyze obtained results. This methodology is a useful tool for summarizing and describing data such as those collected in these studies.

Usability evaluation
Answers provided to the ad-hoc qualitative questions aimed to extract potential improvements, which were used to guide the refinement process. In general, all participants found the assistant practical and enjoyable. The main shortcoming pointed out was the difficulty in waking the assistant up by pronouncing its name ("Adela"). It was usually necessary to repeat the name of the assistant twice or more to wake it up. Regarding explanations and messages provided by the assistant, they were perceived as clear and easy to understand. It was mentioned that the speed of the assistant's speech was sometimes too fast. According to other studies where the usability of a system is measured by SUS, a system can be considered usable if its SUS score is higher than 68 [27]. In our usability study, an average score of 75 was obtained. Therefore, ADELA assistant could be considered already usable in its initial version. Figure 6 shows the percentage of responses (positive, neutral, and negative) in five categories from grouped SUS questions. Most categories were positively ADELA: a conversational virtual assistant to prevent delirium… evaluated. However, measuring the ease of learning to use ADELA showed that more than 50% of the participants expressed some difficulties.
Regarding CUQ assessment tool, other studies using it [28,29] also take as a reference a score of 68, from which a system can be considered usable [27,30,31]. In this study, an average CUQ score of 85 was obtained, so ADELA assistant can also be treated as very usable. Figure 7 shows the average score per question in CUQ questionnaire. Odd-numbered questions have a positive meaning if their score is 5. In this case, all odd-numbered questions achieved scores higher than 4, except for question 9, which assesses how well the assistant understands the user. Despite this, participants agree on the assistant's ability to handle and resolve errors (question 13).
On the other hand, even-numbered questions have a positive meaning if their score is 1. The worst results in this group, although positive, are found in questions 8 and 10. Some speeches or interactions from the assistant seems not to be sufficiently clear. Question 16, which indicates whether the assistant is complex, has a score slightly higher than 2, can be considered as an acceptable result, as older people do not see the assistant as complex to use.

Cloud response time
ADELA assistant uses several cloud services. This means that the system must have a stable and reliable internet connection to provide a feeling of seamless interaction.
The test consisted of asking ADELA a basic intent and measuring the time between the recording was sent until ADELA began to speak. This process was  Table 3.
As mentioned earlier, ADELA will be firstly deployed for a clinical trial at Getafe University Hospital, where the average connection speed varies between 10 and 20 mbps. Average response time will be around 2.7 s, considered appropriate to achieve a reasonable interaction.

Reliability of AWS Lex
AWS Lex is the cloud service used in ADELA assistant to understand users' intent from an audio recording. Therefore, the reliability of this service must be high to assist users.
Intents detection rate from AWS Lex is highly dependent on the way the user speaks, which must be strongly considered when working with older persons.
An evaluation of this service was undertaken to assess its effectiveness in identifying ten different intents with ten potential users of ADELA. Whenever no predefined intent or an erroneous intent was detected, the assistant prompted the user to repeat the question.
The percentage of misguided intents ranged from 0 to 30%. Users 1, 5, 6, 7, 9, and 10 did not exhibit any misguided intents during the observation period. Users 2 and 8 had a relatively low percentage of misguided intents (10%), while users 3 and 4 had 30% and 20% of misguided intents, respectively. The rate of misguided intents is somehow variable among users; an average intent error rate of around 8% was observed. It was also found that all users were able to complete their intents in a second attempt in case AWS Lex did not resolve it the first time. Therefore, in case of understanding errors, the assistant recovers easily.

Refinement of the assistant
The analysis of the results from the refinement phase has led to several ideas on how improve ADELA, which are described in Table 4. These improvements have allowed to build a new version of ADELA ready to be deployed in a real environment.

Conclusions
ADELA is a research project aimed at designing and developing an intelligent conversational assistant to prevent delirium in hospitalized older people. To achieve this goal, the project went through several phases until a refined system, ready to be used in a clinical environment, was released. ADELA was built through a co-creation phase in which a multidisciplinary team elicited the requirements of the assistant. Thanks to this co-creation phase, a first functional prototype of the conversational assistant was designed and built. This first prototype was exposed to a technical and usability evaluation process conducted with 10 potential end users (older adults). SUS and CUQ were used with a mean score of 75.5 and 85.94, respectively.
Changes to improve the first ADELA design included, among others, simplifying voice interaction and provide the user with simple information on how to use the system. The technical evaluation allowed identifying a minimum requirement of a 10mbps connection in the environment where the assistant is deployed.
It is important to acknowledge some limitations of this study that may have affected the interpretation of the results. First, the potential bias introduced by the population participating in the study; it is possible that those who volunteered had a higher interest or motivation to participate than others, which may have influenced the results. Additionally, it is important to note that the study was conducted with healthy older individuals, and not with those who would typically use the system in Table 4 Problems in ADELA assistant detected in refinement phase with its respective solutions

Problem Solution
User difficulties in using the ADELA assistant on first contact Carry out an introductory session where the user is taught on how to use and interact with the assistant The assistant sometimes delivers sentences that are too long and complex

Simplify communication
The assistant speaks too fast Reduce the assistant's speech rate to 85% The detection of the wake word ("Adela") is sometimes not successful The time during which the assistant is listening is too short Increase listening time by 15% compared to the original a real-world context (older persons with a condition leading to being hospitalized), which could limit the generalizability of the findings. From a usability standpoint, these limitations may result in a positive bias towards the system. Furthermore, it should be noted that some technical aspects could also present several limitations. For example, the reliability of Lex was tested with a relatively small sample size of 10 individuals, which may not be sufficient to fully evaluate its reliability in different contexts. Moreover, due to the variety of voices, accents, and speech problems, the results may not be conclusive and may require further investigation. These limitations should be considered when interpreting the results of this study.
Finally, a second version of the assistant, ready to be deployed in a real environment, was released including the improvements drawn from the evaluation phase.
Future work will comprise deploying ADELA in a real scenario to assess its clinical validity in preventing delirium throughout a clinical trial. Furthermore, new features will be added to the system to enhance its behavior. Among these functionalities is the inclusion of Natural Language Processing (NLP) with models such as GPT-3 to expand the assistant's conversation options, reduce its cloud dependency by using offline services and explore new wake words detection methods to improve user experience.

Approximate duration 30 min
Before starting the activity, the elderly user will be informed about the following aspects of the ADELA assistant: • How the assistant should be woken up. • The assistant can be asked questions both 'waking her up' and when she initiates the conversation. • Explanation of the physical exercise sheet. • The assistant can receive calls. • The assistant can be asked to play cognitive (or memory) games. • Color coding of the LEDs on the device according to their status.
The activities to be carried out in the test, divided into sections, are set out below: Section 1 Check basic interaction to activate the wizard and ask for the current time.
• Objective: To test basic user-assistant interaction, as well as the phrase used by the user to request the functionality. • Technical details: Not applicable. • Relevant information: The assistant is called 'Adela' and will be activated when you tell her your name. • Activities requested from the user: Know where I am.
• Expected outcome: The assistant will tell you where you are.
Know where your relatives are.
• Expected outcome: The assistant will tell you the location of your relatives.
Ask the assistant who he/she is and what he/she does.
• Expected outcome: The assistant briefly describes himself/herself and explains his/her capabilities.
Ask the assistant to turn up the volume.
• Expected outcome: The assistant turns up the volume.
Ask the assistant to turn down the volume.
• Expected result: The assistant turns down the volume.
Ask the assistant to play soothing music.
• Expected result: The attendant will play music.
Ask the attendant to stop the music.
• Expected result: The attendant will stop the music.
Ask the attendant to turn on the light.
• Expected result: The attendant will turn on the light.
Ask the attendant to turn off the light.
• Expected result: The attendant will turn off the light.
Section 2 Full flow in hydration reminder.
• Objective: To determine the time required by the user to complete flow. • Technical details: It will be forced in real time to launch the reminder. • Relevant information: The user will be provided with a table with a bottle of water. The wizard will prompt the user with a recommendation for water intake. • Activities requested from the user: Follow the instructions of the wizard.
• Expected result: The wizard triggers the hydration reminder, and the user takes time to drink water.
Section 3 Full flow in reminder in reminder visual/auditory aid.
• Objective: To determine the user's response time.
• Technical details: The reminder will be forced in real time.
• Relevant information: The user will be provided with a table where he/she should leave his/her glasses/hearing aids (if used). The assistant will indicate the user to put on the visual and hearing aid. • Activities requested of the user: Follow the instructions of the wizard.
• Expected outcome: The assistant launches the visual/auditory aid reminder, and the user takes time to put on their device. The assistant waits for 30 s to listen in case the user needs anything else.
Section 4 Full flow and time required per phase in physical activity performance.
• Objective: To determine the user's response time.
• Technical details: The reminder shall be forced in real time. The physical activity plan will be limited to 1 exercise. • Relevant information: The user will have an exercise sheet on a table. • Activities requested from the user: Follow the instructions of the wizard.
• Expected outcome: The assistant instructs the user to pick up the exercise sheets from his or her desk and waits 10 s. The assistant then tells the user the partial physical plan to perform. After 2 min (20 min for a complete plan), the assistant will ask the user if they have finished their activity and will say goodbye.
Section 5 Testing user-assistant interaction in the phone call function.
• Objective: To determine whether the sound quality and call acceptance and rejection mechanisms are effective. • Technical details: An evaluator's phone number will be added to the database to simulate a call with a family member. • Relevant information: When receiving a call, the assistant will ask whether to take or reject the call. At the end of the conversation, the user cannot hang up the call. The person who made the call to the attendant must hang up the call. • Activities requested from the user: The user picks up the call.
• Expected result: The attendant receives a call, and the user confirms that he/she wants to take the call. During the call both the sender and the receiver can communicate clearly and have a conversation. When the conversation is completed, the sender hangs up the call to end the process.
The user rejects the call.
• Expected result: The attendant receives a call, and the user rejects the call. The call is terminated, and the normal flow of the attendant continues.
Section 6 Evaluate flow of each cognitive game.
• Objective: To determine the level of understanding and difficulty of each game for the user. • Technical details: In the last game, it will be forced in real time to launch the reminder. • Relevant information: The user must activate the assistant by her name Adela and ask the assistant to perform a cognitive activity. • Expected result: The user indicates the category to which an object belongs from the possibilities indicated by the assistant, and the assistant corrects his/her answer.
• Expected result: The user counts the number of times a specific word appears in a list of words given by the assistant, and the assistant corrects the answer.
• Random game (triggered by the daily reminder to perform cognitive activity).
• Expected outcome: The user has to decide the appropriate word among several options with respect to the context of a sentence of the assistant, and the assistant corrects his answer.

Section 7
Checking the flow at bedtime.
• Objective: To determine the user's ability to respond to requests from the wizard and to time the flow subparts. • Technical details: The corresponding reminder will be advanced in the test. • Relevant information: You will be asked if you want to reduce the lighting in the room as well as play relaxing music during the process. • Activities requested from the user: Rest reminder.
• Expected outcome: The assistant indicates that it is time to get ready for bed. The user is asked if he/she wants to lower the lighting by 50% and play relaxing music. After 2 min (10 min in real environment), the lighting will be switched off and the music will be stopped, the user is asked if he/ she is ready to go to sleep and the system will be switched off.

3
ADELA: a conversational virtual assistant to prevent delirium… Funding Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was supported by Fundación MAPFRE. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Data availability
The data that support the findings of this study are available upon reasonable request from the corresponding author, RPR. The data are not publicly available due to privacy of research participants.

Declarations
Conflict of interest Authors have no competing, or other interests that might be perceived to influence the results and/or discussion reported in this paper.
Ethical approval All procedures performed in studies involving human participants were in accordance with the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of Getafe University Hospital (protocol code A06/22 approved 23 June 2022.
Informed consent Informed consent was obtained from all individual participants included in the study.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/.