Globally, youth and young adults aged 16–24 years experience mental health challenges at alarming rates (Ghafari et al., 2022). In a recent meta-analysis, the median age of onset across all mental disorders was between 14.5 and 18 years (Solmi et al., 2022). Over 25% of youth are experiencing mental health challenges, yet > 50% of those needing mental health support go without care (Ghafari et al., 2022; Ma et al., 2021; Racine et al., 2021). Sexual and gender minority youth (SGMY) face higher rates of mental health challenges compared to non-SGMY, including depression and anxiety, due to increased levels of stigma, discrimination, and social isolation (Rhoades et al., 2018; Steinke et al., 2017; Wilson & Cariola, 2020). In addition to poorer mental health, SGMY are at an increased risk of sexually transmitted infections (STIs) and HIV compared to non-SGMY (Rasberry et al., 2018). However, the current mental healthcare infrastructure is not built to support and effectively treat the vast number of youths experiencing mental health challenges, especially those with marginalized identities (Ghafari et al., 2022; Rasberry et al., 2018). This lack of infrastructure causes delays in accessing treatment and resources, negatively impacting physical and mental health outcomes (Ghafari et al., 2022; Rasberry et al., 2018). Accessing mental health resources is even more crucial for SGMY and those living with HIV (Remien et al., 2019).

Numerous studies show mobile health (mHealth) technology to be a helpful way to connect youth to their peers and provide health care information (Horvath et al., 2013; Kachur et al., 2013). For example, “BrdsNBz” was a text messaging line developed in response to the high rates of pregnancy and STI for youth ages 14–19 in North Carolina where youth anonymously texted and asked sexual health questions (Phillips, 2010; Willoughby, 2015). Another example is “MyPEEPS Mobile” a mobile app intervention that includes HIV prevention and sexual health education for sexual minority cisgender boys and men aged 13–18 years (Schnall et al., 2022).

Chatbots (e.g., the pop-up boxes to ask questions or receive information frequently seen on websites for online shopping, banking, and other internet-based services) are software applications that can deliver information in a conversational, human-like manner by text or voice. Chatbots fall under the umbrella of mHealth interventions, which are technology-based methods to deliver healthcare, improve patient outcomes, and connect individuals with needed services (Kang & Exworthy, 2022; Kassavou et al., 2022; Olano-Espinosa et al., 2022; Venter et al., 2018). Many organizations use custom-built chatbots to assist customers quickly instead of speaking with a human, to help consumers troubleshoot issues outside of business hours, and to provide an efficient way to get customer service support. In the health sector, the use of chatbots is rapidly growing, with applications in a variety of settings, including mental health interventions; chronic pain management; emergency department social needs screenings; and information for breast cancer patients (Bibault et al., 2019; Hauser-Ulrich et al., 2020; Kocielnik et al., 2019; Vaidyam et al., 2019). Because of the near ubiquity of smartphone use by young adults with 95% of teens having access to smartphones, chatbots are increasingly used to deliver age-tailored information and interventions on sensitive and stigmatized topics like depression and anxiety or sexual health, as in Planned Parenthood’s “Roo” chatbot (Belanger, 2019; Fitzpatrick et al., 2017; Schaeffer, 2019). Additionally, technology-based and peer navigation interventions are acceptable and effective among HIV-negative SGMY, with youth particularly appreciating technology-based options for healthcare (Baghaei et al., 2020; Horvath et al., 2013; Kachur et al., 2013). Specific to mental health service delivery, chatbots can increase access to needed services, providing users anonymity, thereby eliminating social stigma as a barrier that often prevents successful linkage to mental health services and resources (Kosyluk et al., 2021; Skjuve & Brandtzæg, 2018; Vaidyam et al., 2019).

This study developed and evaluated a pilot chatbot service navigator called Tampa Bay Area Treatment and Health Advisor (Tabatha-YYC), to improve linkage to mental health resources among youth at risk of STIs and HIV at an urban youth sexual health clinic in Tampa, Florida. Here, we report on the iterative, user-focused design process, assessing the usability and acceptability of the Tabatha-YYC chatbot.

Methods

Using a human-centered process, we designed a prototype mental health resource locator chatbot for youth and assessed it for acceptability among a sample of potential target users (Fig. 1). A Youth Advisory Board (YAB) comprised of youth and young adults aged 16–24 years who represented the target population guided the building of a chatbot with the goal of creating a mental health resource navigator that would provide users with resources based on their responses to a depression-screening questionnaire embedded within the chatbot. The YAB recommended which resources to include in the chatbot, where resources should be placed, and suggested youth-friendly language. Based on their input, Tabatha-YYC was programmed by study team members (GS and SG) using SmartBot360 software (SmartBot360, 2022). After an iterative evaluation process consisting of three rounds of feedback with the YAB and integration of feedback into the chatbot, the final Tabatha-YYC prototype was ready to be tested with youth naïve to the chatbot design. The chatbot was tested using a think-aloud protocol (in which youth spoke about their experience “in the moment” as they interacted with the chatbot for the first time), completed a demographic survey, and rated usability problems using the Health Information Technology Usability Evaluation Scale (see Measures) (Bertini et al., 2006; Schnall et al., 2018).

Fig. 1
figure 1

Timeline for Tabatha-YYC usability testing

Full size image

Study activities took place from June 2021 to May 2022. The University of South Florida’s Institutional Review Board approved all study activities, and participants provided informed consent before participation after which they received $25 for their time.

Youth Advisory Board

Sample Selection

Recruitment was done via snowball sampling from the clinic’s existing affiliated YABs and referrals by the clinic staff and YAB members. Due to clinic policies and the COVID-19 pandemic, meetings took place over Microsoft Teams video calls.

Procedures

The YAB met three times between June 2021 and October 2021. During the first meeting, study team members (GS and SG) met with the YAB to explain the study, describe chatbots, and hear recommendations from YAB members on how they would like Tabatha-YYC to be formatted, including what resources should be included and the type of language Tabatha-YYC used in its responses to users. In the second and third meetings, GS and SG shared the updated iterations of Tabatha-YYC to receive YAB member feedback on the changes made until a final version was ready to be tested for usability testing with naïve users.

Naïve Users

Sample Selection

A convenience sample of 20 youth naïve to the Tabatha-YYC design process were recruited through the Ybor Youth Clinic which serves SGMY vulnerable to STIs/HIV. Inclusion criteria were youth ages 16–24 years and able to speak and read English. Clinic staff attempted to recruit a diverse sample of participants (i.e., varying ages from racial, sexual, and gender minorities) and referred them to study staff who obtained informed consent (or assent from youth ages 16–17 years) and conducted study procedures either in person at the clinic or over Microsoft Teams, depending on the COVID-19 policies for the clinic at the time.

Procedures

The same procedures were followed for both in-person or virtual visits. First, participants opened the Tabatha-YYC chatbot on a laptop (using either their own for virtual study visits or the study laptop for in-person visits) and recorded using the screen share and record features of Microsoft Teams. The laptop camera was off during the study visit (either remote or in-person) to ensure that only the screen was recorded and not the participants face as they interacted with the chatbot. Participants used a think-aloud procedure to provide their feedback during the study visit. Think-aloud is a procedure in which participants interacted with the chatbot by responding to the prompts and provided a stream-of-consciousness discussion stating their thoughts, reactions, and interactions as they moved through the pilot Tabatha-YYC chatbot (Bertini et al., 2006).

After interacting with the chatbot, participants were interviewed using a semi-structured guide to understand their experience, thoughts on the chatbot’s usefulness and resources, and the feasibility of using the chatbot in a clinical setting (Fig. 2). Finally, participants completed a demographic survey that included a usability rating questionnaire.

Fig. 2
figure 2

User feedback semi-structured interview guide

Full size image

Instruments

Standardized measures were used to capture participant demographics, mental health history, anxiety symptoms, perceived stigma, attitudes towards treatment-seeking, label avoidance, and usability of mobile health technology. The survey included the PhenX toolkit demographic information (sex, gender, sexual orientation, race/ethnicity, and education level) and mental health history (current mental health treatment) (Hamilton et al., 2011). The Generalized Anxiety Disorder-2-item Scale was used to provide information on anxiety symptoms (Kroenke et al., 2007). The Attitudes Towards Mental Health Treatment Scale was adapted from Fischer and Turner’s (1970) Attitudes Toward Seeking Professional Psychological Help scale to update language and culturally meaningful items for attitudes towards seeking help from mental health professionals (i.e., therapists, counselors, psychiatrists, or psychologists) (Conner et al., 2018). The perceived devaluation discrimination scale was used to measure perceived stigma, the perception of stigma held within the public (Link et al., 2001). The self-stigma of seeking help was used to measure label avoidance, the construct that an individual avoids seeking out treatment due to the stigma surrounding mental illness (Vogel et al., 2006).

The Health Information Technology Usability Evaluation Scale was used to measure an individual’s assessment of the usability of mobile health technology (Schnall et al., 2018). The Health-ITUES is a 20-item scale with four subscales (impact, perceived usefulness, perceived ease of use, and user control) answered on a five-point Likert scale (Schnall et al., 2018). The Health-ITUES and its four subscales have shown good Cronbach’s alphas, with scores ranging from 0.85 to 0.92 (Schnall et al., 2018). The sociodemographic and mental health measures (perceived stigma, attitudes toward mental health treatment, and label avoidance) were not linked to users sub-scores on the Health-ITUES scale.

Data Analysis

Audio recordings were transcribed verbatim using Otter.ai and analyzed using a rapid-analysis approach following the rainbow visualization method (Hamilton & Finley, 2019; Otter.ai, 2016; Sharon, 2013). The rainbow excel visualization technique is a testing tool for usability research analysis and allows for easy visualization during rapid inductive analysis, creating clear visualizations on themes that received the most feedback from users (Sharon, 2013). An inductive, rapid analysis approach was utilized to summarize the transcript data into structured thematic categories based on the interview questions, allowing for quicker implementation and dissemination of user feedback for future development and application of Tabatha-YYC. Study team members (GS, SG, KG, JT) independently coded two transcripts utilizing inductive codes created under the structured thematic categories based on the interview guide. Next, they met to adjudicate differences in coding and refine the codebook. Then, GS exported and aggregated the coded data into their rainbow visualization excel codebook. After the final codebook was created, the coders formed dyads. One dyad (GS and SG) coded eight transcripts and the other dyad (KG, JT) coded nine transcripts. Each dyad met to adjudicate differences in coding. Finally, all four coders met to compare findings, reach consensus on coding discrepancies, and export and aggregate the coded data into a final rainbow visual codebook in excel.

Results

Youth Advisory Board

The YAB approved the final version of Tabatha-YYC that was tested by naïve users. The YAB’s feedback primarily centered on three main areas: providing appropriate resources, ensuring that Tabatha-YYC had a secure and welcoming feel for users, and making the chatbot more engaging and efficient.

The YAB members significantly contributed to the identification of the resources to be included in the chatbot relevant to youth in the Ybor Youth Clinic. While simultaneously focusing on the importance of diverse resources and coping strategies within the chatbot experience, the most prominent feedback from the YAB was to guarantee that the resources provided were free, ensuring that the strategies provided are accessible to all youth interacting with the chatbot regardless of their financial status. The YAB felt that having various resources and coping strategies readily accessible was crucial as sometimes youth receive services associated with their school and are left without resources during the summer. The YAB felt it important that the chatbot provide a resource to help fill the gaps in care that some youth experience. One YAB member commented:

“My school has a school psychologist, but in the summer, I do not have access to her.”

A consistent point of discussion during the three iterative feedback meetings with the YAB was to make sure that the chatbot felt welcoming and inclusive, providing resources inclusive of various sexual orientations and gender identities and using inclusive language throughout user interaction with Tabatha-YYC. The YAB cautioned against using too much “clinical jargon” when discussing mental health and providing resources. Figure 3 shows an example of the welcoming language the YAB wanted the chatbot to display to help cultivate a feeling of privacy and inclusivity.

Fig. 3
figure 3

Tabatha-YYC welcome message

Full size image

Naïve Users

Twenty youth participated in the usability testing, of which 75% were aged 21–24 years; see Table 1. The study participants had both pessimistic (mean score = 30.3, SD = 4.21) and beneficial (mean score = 30.3, SD = 4.40) attitudes toward treatment-seeking. The perceived devaluation and discrimination scale’s overall mean score was 39.8 (SD = 4.84) and the mean overall Self-Stigma of Seeking Help Scale score was 19.9 (SD = 6.06). The higher overall means for the perceived devaluation and discrimination scale and self-stigma of seeking help scale scores illustrate that participants indicated higher perceived stigma and greater likelihood to avoid seeking out treatment due to stigma surrounding mental illness. However, the participants overall had minimal anxiety with a mean GAD-2 of 2.6 (SD = 2.09).

Table 1 Participant Characteristics (n=20)
Full size table

Naïve-Users: Health-ITUES Usability Problems

The Health-ITUES results are listed in Table 2. Perceived ease of use was the scale that scored the best (i.e., highest score) among participants. The second-highest score was perceived usefulness. The scale with the worst (i.e., lowest) score was user-control. This scoring corroborates the qualitative feedback found during the think-aloud procedure and individual interviews with the naïve-users after their exposure to the chatbot. The overall health-ITUES score of 4.3 met the optimum cut-off point for usability identified in a previous study (Loh et al., 2022).

Table 2 Health-ITUES Scores (n=20)*
Full size table

Naïve-User: Qualitative Feedback

In addition to the survey, the inductive, rapid analysis using the rainbow analysis excel visualization was used for the qualitative analysis of the semi-structured interviews and think-aloud procedures conducted with the participants. Four recurring responses that we received during our interviews were the chatbot is engaging and accessible; usefulness of chatbots; importance of privacy; challenges in user-control and information placement.

Chatbot is Engaging and Accessible

Aligned with the high perceived ease of use score, most participants found the chatbot to be engaging and a product that could be easily accessible to the target population. Participants particularly liked being given age-specific chatbot responses and resources after the chatbot inquired about their age group (those 14–17 and those 18 years of age and older) at the beginning of chatbot interaction. Many participants also appreciated Tabatha-YYC’s tailored responses based on their feedback. One participant stated:

“I think it’s really helpful. Because it gives you everything in one spot. Um, sometimes it can be difficult to find a good counselor or like, someone who is fitting for you. And it kind of gave me like, it gave me stuff for if you’re part of the LGBTQ+ community, it gave me stuff if you’re like a college student, it gave me just regular. So, it gave me a lot of information that would normally be hard to look for…” (YYCN8, 20 years old, straight, Black female)

Usefulness of Chatbots

Continuing to corroborate the Health-ITUES scores in the qualitative analysis, every participant but one stated that they would recommend the chatbot to others or would use the chatbot themselves. Tabatha-YYC was created to be used in a clinic setting, and most participants stated that this chatbot would be useful in a clinic setting. Participants also noted that the chatbot serves as a good middle ground for users who do not necessarily want to talk to a person face-to-face about their mental health needs or who may need quick access to resources. As two participants commented:

“You know, I feel like this is something that can be easily done on a personal piece of technology that someone brings to the doctor’s office. And it’s a, I feel like, it’s also a great way to start the conversation, if somebody was having these issues. And I feel like what better time than to do it if you’re going for a doctor’s appointment?” (YYCN3, 22 years old, gay, Black male)

“Uh I thought it was really cool. Honestly. I like the fact that it does give a lot of you know, mental health resources you know, for someone who may be going through something, and you know, may need those resources. Really cool chat. I like how it responded you know, really quickly, honestly, and I thought it was it was helpful. Honestly, if I know anybody who needed that I would definitely refer them to Tabatha.” (YYCN20, straight, Black female)

One of the options participants found most useful was the variety of resources provided in the chatbot. Nineteen of the twenty participants reported that the resources provided were valuable and exceeded their expectations for a chatbot. More specifically, participants found the breathing exercises helpful and were surprised and enjoyed having a musical alternative like the Spotify playlists to help them relax. Two naïve users mentioned:

“I thought the [Spotify] playlist was very, very unique and very helpful. Um, actually wasn’t expecting that. I thought it was like a nice little touch, too… I just thought it was just going to give me like the basic bare minimum. It kind of exceeded my expectations with just like the amount of resources.” (YYCN3, 22 years old, gay, Black male)

“I guess I didn’t expect as many resources as I got. It even like tells you at the end, like you can call someone at the Youth Clinic. And you can actually speak with like a specific person to like, help you out. So, I thought that was pretty neat.” (YYCN17, 20 years old, straight, Black female)

Participants also commented that they enjoyed being exposed to new ways to help cope with stress and anxiety such as the tapping exercises. One participant commented:

“I thought it was interesting because I’ve never heard about it [tapping exercises] before and it’s definitely a unique way to go about treating anxiety.” (YYCN16, 17 years old, straight, Black female)

Importance of Privacy

The theme of privacy was recurring and was essential for participants when navigating a mental health resource chatbot. Privacy and confidentiality were also initial themes the YAB members stated would be crucial to participants. Some participants highlighted that the chatbot felt safe, and others felt that the chatbot needed to continue to reassure users about its anonymity and trustworthiness throughout the use of the chatbot. The importance of privacy and the two contrasting perspectives from participants are noted in the feedback below:

“I like how it says, “thanks for sharing.” And that, you know, the bot’s here to help connect me to the resources to the best of its ability I think that’s very secure, and it gives a feeling of security and trust, I like the wording and how it asks, or adds that on. And then the same is for the second question of if I’m willing to answer rather than just asking me the questions.” (YYCN9, 24 years old, straight, Black male)

“So just like helping these youth stay anonymous, and, you know, reassuring them that you know, aren’t gonna run back and tell their parents you know, personal matters because that, I feel like that could definitely be a barrier to some youth, you know...” (YYCN15, 20 years old, straight, White female)

Challenges in User-Control and Information Placement

One of the issues that emerged in Tabatha-YYC was the way resources were displayed. While participants liked the resources provided and the efficiency of the chatbot, many commented that the chatbot responded too quickly. This response made it challenging for them to read all the information provided and corroborates our Health-ITUES low score for user-control. Participants felt that they didn’t have control over how quickly information was provided, as one user commented:

“I think they’re a little too quick. ‘Cause like this like this whole thing popped up within a span of couple of seconds. And when the next, this one popped, when the second one popped up, this one, like, I couldn’t see the rest. I had to scroll back up and see what it said.” (YYCN2, 22 years old, gay, Asian male)

The users also struggled with the amount of information provided and where the resources were placed within the chatbot flow. Several users commented that they felt that the coping exercises and resources were ill-placed. One user stated:

“I feel like, it’s like ill-placed like, it’s like weird placement because like all I’ve said was like my initial things, like I think it should go further into the chat. Like because I feel like talking about your anxiety, like brings it out more, so then would be an appropriate time to breathe or to do a breathing exercise, I mean.” (YYCN13, 21 years old, straight, Latinx female)

Discussion

A novel mental health resource navigator “Tabatha-YYC” was feasible and highly acceptable among youth at risk of HIV/STIs. Until now, mental health chatbot research has focused primarily on the possibility of introducing chatbots in mental health care, with scant empirical evidence regarding the actual acceptability of a chatbot based on observed user interactions (Abd-Alrazaq et al., 2021; Mariamo et al., 2021). This pilot study demonstrates how supportive technology like chatbots could be programmed to deliver resources tailored to youth when human peer navigators are unavailable and/or as an adjunct to human-provided resources and highlights the importance of privacy and confidentiality of the said technology. The youth expressed their appreciation of the chatbot reminding them the interaction was confidential. The chatbot was private as it was completed on the youths’ phones, and it was a secure platform, not sharing the youth’s information to others. This focus on confidentiality and privacy is seen in the literature as demonstrated by the MyVoice national text message polls of youth ages 14–24 years which indicated that youth worry about privacy and discrimination by their providers, sometimes lying about their behavior when worried about confidentiality (Zucker et al., 2019).

In addition to the novel nature of the chatbot, this study is innovative in the mHealth field as there is limited data on chatbot acceptability and usability with youth and young adults ages 16–24 years at risk of STIs that addresses mental health concerns (Abd-Alrazaq et al., 2021). Most mHealth technology related to youth at risk of HIV and other STIs currently focuses on addressing youth HIV prevention and testing through apps; they are not addressing the mental health concerns of youth at risk of HIV and other STIs (Sanabria et al., 2021; Sullivan et al., 2015). Additionally, although mHealth applications like apps and chatbots are frequently used by youth, few mHealth interventions are designed and evaluated by youth (Zapata et al., 2015). In a recent scoping review of patients’ perspectives on mental health chatbots, thirty-seven studies were included, with only three focused on adolescents and young adults, and none focused on adolescents and young adults at risk of HIV and other STIs (Abd-Alrazaq et al., 2021).

Finally, performing usability studies and having a multi-faceted iterative design process centered on the user is vital when creating chatbots and other mHealth interventions (Jaspers, 2009; Kushniruk et al., 2017). In Ab-Alrazaq et al.’s review (2021) of mental health chatbots, only three studies were usability studies, all of which only used one method, surveys, to capture usability feedback. The framework used in this study provides a structure of future usability studies as it uses a multi-method, user-centered approach to create and evaluate the usability of the chatbot using a YAB-centered design with the Health-ITUES scales, think-aloud protocol, and semi-structured interviews.

Limitations

A limitation of this study is the diversity of our sample of users. Although we attempted to include a diverse sample with various ages and sexual minorities, we were only able to enroll one person in the study who was within the ages of 16–17 years, and 60% of our sample identified as heterosexual, limiting the generalizability of our sample to our target audience of SGMY at risk for STIs. Furthermore, our sample only included individuals local to Tampa Bay Region, limiting findings beyond this study. Additionally, it would be important to include an evaluation of the chatbot by a usability expert in future usability studies as this would permit a more comprehensive evaluation beyond what was possible in the present study (Nielson & Molich, 1990).

Conclusion

Feedback from the YAB led to the creation of the Tabatha-YYC mental health resource and service navigator chatbot prototype tested by naïve users who found the chatbot useful, engaging, and accessible as a secure resource to help connect SGMY to mental health resources. Despite challenges in user-control and information placement, participants found our chatbot acceptable and usable as a good starting point for further mental health chatbot development. Feedback from this usability testing is being used to finalize a chatbot to be later tested with a larger population. This usability testing provides important design methodology considerations and key insights into the preferences of youth at risk of STIs who are seeking mental health resources, using a YAB-centered design to help ensure acceptability of future chatbot developments.