Introduction

Healthcare services delivery was altered during the COVID-19 pandemic to meet social distancing and safety requirements (Galea et al., 2020). COVID-19’s impact was particularly concerning for services delivered in institutional settings prone to outbreaks (Druss, 2020) and programs caring for those experiencing mental/behavioral illnesses (Pfefferbaum & North, 2020) and homelessness (Tsai & Wilson, 2020).

One such program is the U.S. Department of Veterans Affairs (VA)’s Mental Health Residential Rehabilitation and Treatment Programs (RRTPs), offering 24/7 services to Veterans experiencing housing instability and/or needing mental/behavioral healthcare (Veterans Health Administration, 2019). Many of the approximately 250 RRTPs are located on VA campuses and have close living quarters, making them vulnerable during crises like COVID-19. Furthermore, RRTPs’ in-person therapies (Melemis, 2015) require substantial revisions for virtual delivery.

Lessons from COVID-19 are emerging regarding residential treatment programs’ challenges and adoption of virtual programming (Fareed & Fareed, 2021; Herrera 2021). However, RRTPs’ services cannot all be delivered virtually. For instance, RRTPs provide food, shelter, and healthcare, requiring coordination with providers, governmental entities, and community programs. Furthermore, many RRTPs emphasize employment-related aspects of recovery: Veterans have jobs at the hospital or in the community and move across VA grounds, increasing the likelihood of COVID-19 transmission.

An established framework for examining a past event (e.g., initial COVID-19 outbreak) is the World Health Organization (WHO)’s After Action Review (AAR) (WHO, 2019). AARs target (i) what was expected, (ii) what actually happened, (iii) what went well, and (iv) what could be improved and how. AARs have been used by numerous entities to systematically learn what they were trying to accomplish (e.g., responses to hurricanes (Knox, 2013) and other public health crises (Fidler, 2009; Mase et al., 2017)) and non-judgmentally cull lessons toward improving future performance.

We applied AAR to examine perceptions of staff and Veterans regarding two RRTPs’ responses to COVID-19. Per the WHO’s AAR guidance (WHO, 2019), we facilitated separate small-group discussions (resembling focus groups) with staff and Veterans to assess their perceptions. We qualitatively analyzed the discussion data to identify salient themes from participants’ experiences and their recommendations for future crises. AAR could be used for a broader review of RRTPs and serve as a guide for other programs to prepare for local, national, or global crises.

Methods

Setting and Measures

This study was VA Health Services Research and Development Service (HSR&D)-funded, and small-group discussions were conducted in November-December 2020 at two medical centers within the VA New England Healthcare System. We adapted the WHO’s AAR to examine RRTPs’ COVID-19 responses for (i) what was expected, (ii) what actually happened, (iii) what went well, and (iv) what could be improved and how. We followed the WHO’s “Debrief AAR” format of facilitator-led, small-group discussions, minimizing participant burden and being feasible with few resources. We used semi-structured questions addressing participants’ RRTP experiences regarding (i) emergency response and communication, (ii) impact on care and mental health, (iii) and care coordination, case management, and community engagement.

Sample

We used combined purposive and snowball sampling to recruit participants over one month, reviewing a study information sheet with potential participants. Verbal consent was obtained from both staff and Veteran participants. From Veteran participants, we also obtained signed HIPAA authorization forms. Fourteen frontline staff members (seven from each RRTP) were recruited and enrolled. Due to scheduling conflicts and the need to maintain staff coverage at the RRTPs, six staff from each RRTP (a total of 12) participated in two small-group discussions. Participating staff were from varied clinical and administrative positions. Six Veterans were recruited from one RRTP and four enrolled and participated in the small-group discussion. Participating Veterans had received RRTP care during the early months of the pandemic. Each discussion session lasted approximately one hour. Five program leaders across the two RRTPs reviewed our summaries of the discussions to help further contextualize our findings.

Procedures

Three small-group discussion sessions were conducted virtually via Microsoft Teams. The small-group discussions resembled focus groups, as both bring together a group of people to hold a moderated discussion of their experiences and perspectives on a pre-defined topic (Leung & Savithiri, 2009). However, while focus groups traditionally involve exploring interactions between participants (Mansell et al., 2004) by observing the group’s dynamic and body language, our small-group discussions centered around collecting and analyzing data specifically on the discussed content. Each RRTP had one staff session, while only one of the RRTPs had a Veteran session. This was because one RRTP had sharply decreased the number of resident Veterans, and we were unable to reach and recruit from that limited pool of eligible Veterans.

Small-group discussion sessions were audiotaped, transcribed, and de-identified. Sessions were co-facilitated by BK and BAP, with SKS serving as primary notetaker; all are Health Services Researchers and skilled in qualitative research. These researchers and the participants were the only individuals at the sessions. For participant-checking, we shared deidentified session notes with the participants, asking for and incorporating their feedback on any edits needed. Participating Veterans received a $50 gift card. We shared participant-checked summaries of the discussions with RRTP leadership, who provided both oral and written feedback to help contextualize our findings.

Data Analysis

We used thematic analysis (Miles & Huberman, 1994) to examine the data. One researcher created a summary for the first site, organized by a priori codes (AAR’s four questions). A second researcher reviewed the summary, identifying thematic categories as additional codes. Three researchers refined the summary and codes, then used the same process for the other site.

We used an Excel-based analysis template consisting of three columns: Code, summary, and quotes. Rows were grouped by thematic category and a priori AAR codes per category. One transcript was independently coded by two researchers; disagreements were resolved with input from another researcher. This resulted in revised code descriptions containing examples; the analysis template was accordingly updated. Three researchers coded the remaining transcripts using the same process, updating codes and re-coding as needed. They subsequently analyzed the completed templates, identifying emergent themes under each thematic category and reaching agreement around thematic findings.

The research was approved by the Institutional Review Boards of the VA Boston and Bedford Healthcare Systems.

Results

Themes

Six thematic categories emerged: (i) Overall impact of COVID-19 on RRTP operations, (ii) COVID-19 safety and risk perceptions, (iii) communication, (iv) mental health and well-being, (v) programming changes, and (vi) technology use. We describe each theme, focusing on “what was expected” and “what happened.” We share participants’ recommendations, focusing on “what went well” and “what could be improved and how.”

Participants’ Experiences (Based Primarily on Responses to “What Was Expected” and “What Happened” AAR Questions)

Overall Impact of COVID-19 on RRTP Operations

There was initially much confusion, without standard operating procedures to address the crisis. Both programs sought to remain open when many similar programs limited admissions. One Veteran explained, “… the staff did what they could …. They were just as clueless as some of us ….” Staff needed to figure out how to provide care while staying safe. As time progressed, staff and Veterans became better informed and familiar with new routines. In-person treatment groups were limited to 3–4 people. When groups were not held, one resident said, “There was a lot of time for … Veterans to like think about using [substances] ….” Other participating residents agreed, knowing of Veterans with similar experiences. Hospital leadership decided to repurpose one program’s building for non-intensive COVID-19 patients. Some Veterans felt they were losing their sense of community. One staff member reflected, “… mental health, substance abuse, PTSD … often get … the short end of the stick because they don’t have physical ailments ….”

COVID-19 Safety and Risk Perceptions

Infection Mitigation Measures. Staff described difficulty obtaining sufficient personal protective equipment (PPE) for two months. At one site, staff provided bandanas to Veterans for face coverings. Masks were distributed in mid-April but not initially enforced. Even when enforced, some Veterans were not compliant. One site could request regular professional housekeeping; however, with uneven results, staff also cleaned. One Veteran said, “A lot of people also [weren’t] taking care of their personal hygiene.” The other site did not have professional housekeeping until they were moved to a different building. Professional cleaning was deemed important for thorough sanitization.

COVID-19 Screening, Testing, and Treatment. Nurses screened residents daily. Symptomatic residents were isolated until they had test results. Even asymptomatic COVID-positive residents were isolated. Staff got tested at community pharmacies before on-site testing became available for employees. By April 2020, residents were tested every two weeks. Participants liked one program having a negative pressure unit (area with lower air pressure to prevent harmful particles from leaving the area (Al-Benna, 2021)) for COVID-positive residents not requiring hospitalization.

Communal Areas and Public Interactions. When possible, staff arranged for one resident per room and dining in shifts or using large spaces. Small groups of residents would sometimes meet in person – e.g., with the Director for COVID-19-related updates. When the weather allowed, community meetings were held outside so everyone could join. No visitors were allowed on the units and the residents were no longer given passes to leave hospital grounds. Consideration was given to transporting residents off campus to meet discharge goals (e.g., getting a license, visiting potential apartments). Programs either used several vans or limited the number of people who could ride.

Communication

Communication for Staff Coordination. Emails about changing guidelines were difficult to keep up with. Staff were uncertain about telework expectations. When scheduling Veteran appointments, staff were uncertain whether they would be virtual or in-person, often requiring additional phone calls. Hospital leadership shared COVID-related information through town halls. Over time, staff felt communication became more effective.

Communication Involving RRTP Veterans. Staff participants felt communicating uncertain information to Veterans was “more problematic than helpful.” Delivering the same message (e.g., “from … the same check sheet”) to small Veteran groups worked well. Nurses faced difficulty explaining the importance of PPE to Veterans, especially when some were receiving conflicting messages from media. Veteran participants felt that staff announced restrictions on leaving VA grounds without seeking their input. Yet, Veteran participants agreed that being cautious was better than being unsafe.

Mental Health and Well-Being

Impact of Restrictions. Some Veterans likened the restrictions to incarceration. One Veteran shared, “… you’re allowed to go to the [hospital store] or to work but back to the [RRTP]; you had to be there for every check so, or you can get discharged.” One staff participant noted, “… [the residents] really have no place to go, … even … the gym, which [are] very common with Veterans, [was] also blocked.” One Veteran participant shared having suicidal thoughts and needing help. He “was feeling lost,” until he spoke with nurses and saw the psychiatrist. Residents could not attend community recovery meetings. It is unclear if COVID-19 affected Veterans’ overall recovery trajectory. At one site, however, staff noticed a surprising reduction in relapse, which they attributed to diminished access to alcohol and other substances due to residents not being able to leave the hospital campus.

24/7 Psychoemotional Support Appreciated. Having social workers and nurses available 24/7 was appreciated by Veterans. “They’re just here … to make sure you’re okay and you’re not going to do [anything] to yourself.” Nurses felt that in the evenings, residents were more willing to share their feelings and frustrations regarding restrictions. When needed, the nurses consulted social workers for guidance and validation.

Programming Changes

Therapeutic Programming. Telehealth-based treatment for groups and individuals had mixed results. One staff member said, “… feedback I got from most Veterans: It’s hard to engage. It’s hard to have multiple people sharing ….” Participants recommended continuing in-person therapeutic group programming whenever possible. As one staff member noted, “There’s something about that connection in-person that has always … been special about residential programs ….”

Recreational Programming. Staff struggled with unforeseen limitations: “… playing a simple game, you don’t think of, but everybody’s touching different pieces ….” Therapists tried socially-distanced activities like scavenger hunts, but found low motivation among Veterans to participate.

Technology Use

Initial Transition to Virtual Programming. Nurses loaned tablets to residents for participating in appointments with clinicians, joining meetings virtually, and communicating with loved ones outside of the hospital campus. Although some staff were experienced telehealth providers, others had never provided virtual treatment. Multiple telehealth platforms being used was confusing for some Veterans, and guidance was in flux for providers on permitted platforms. One Veteran explained, “… it was horrible because it was brand new to us. I’m not that computer savvy …. I like it now. In March [2020], I was pulling my hair out of my head.” One site’s limited Wi-Fi bandwidth made video calls difficult initially.

Challenges and Benefits of Telehealth. One Veteran explained finding telehealth groups difficult: “… I can just like sit back and not participate at all, because nobody is really watching you.” Participants also found larger sessions challenging when individuals spoke over one another. Once established, telehealth was viewed positively by staff: “… once I got the hang of doing [telemedicine] … I really got in the swing of it. So, I’ll be glad to continue … work[ing] from home.”

Participants’ Recommendations (Based Primarily on Responses to “What Went Well” and “What Could Be Improved and How” AAR Questions)

Overall Impact of COVID-19 on RRTP Operations

As housing and treatment became harder to find due to the pandemic, participants recommended keeping residential treatment programs open, especially for those who may not be able to find alternative housing or treatment programs. They also recommended giving staff the opportunity to provide input on decisions that impact the RRTP.

COVID-19 Safety and Risk Perceptions

Participants recommended ensuring that PPE and professional cleaning are available. They also recommended considering both safety and comfort in community settings with shared common areas such as dining halls and bathrooms. Additional recommendations included, when possible, planning to safely house and care for COVID-19-positive residents on-site (e.g., by setting up a negative pressure unit) and having established procedures for what to do while awaiting test results.

Communication

Both staff and residents deemed communication critical and recommended that everyone needs to recognize that initial guidance may change during a crisis. They recommended staff explicitly communicate with residents the reasons for COVID-19 precautions and programming changes. They suggested holding community meetings (even if they are virtual), to allow everyone to get the same information at the same time and to ask questions. Additional recommendations included staff using email to help inform various shifts and documenting standard operating procedures, as well as having one-on-one conversations with residents to provide COVID-19 education.

Mental Health and Well-Being

Participants deemed it important to recognize that both staff and residents experience anxiety and stress. They recommended making psychoemotional support available around the clock by ensuring appropriate staffing in the evenings when residents often express their frustrations. They also recommended developing safe procedures for family visits to encourage engagement in treatment, as isolation and lockdown in the facility occasionally lead patients to discharge against medical advice. A related recommendation was to develop procedures so individuals in residential care treatment programs can go off-site for employment, medical care and other appointments, family visits, and recreation.

Programming Changes

Participants recommended maintaining the focus on treatment, even if limited to one session each in the morning and afternoon. They also recommended keeping recovery-oriented programming separate from COVID-19-related information sharing. They encouraged use of creative solutions, such as outside meetings and socially distanced recreational opportunities, to maintain the sense of community and engage residents in treatment. They also suggested giving staff the opportunity to develop and test virtually-delivered content.

Technology Use

Participants recommended ensuring that adequate Wi-Fi and devices are available to residents to keep in contact with their support networks and for telehealth. They also recommended choosing one platform for telehealth delivery and integrating technology training into RRTP orientation for both incoming Veterans and staff. An additional recommendation was to provide a mix of in-person and remote services post-pandemic to help with transitions to telehealth when crises arise.

Discussion

In this paper, we present findings from two VA RRTPs’ responses to COVID-19 using the WHO’s AAR framework. Qualitative analysis of small-group discussions with RRTP staff and Veterans led to thematic findings centered around (i) the overall impact of COVID-19 on RRTP operations, (ii) COVID-19 safety and risk perceptions, (iii) communication, (iv) mental health and well-being, (v) programming changes, and (vi) technology use. The experiences and recommendations emphasized by study participants may warrant heightened attention as RRTPs prepare for future waves of COVID-19 or another pandemic.

Study participants voiced crisis-related and programmatic suggestions during the small-group discussions. Their crisis-related suggestions included soliciting input from RRTP staff on decisions impacting the program, communicating to residents the reasons for changes being made, preparing to use telehealth by integrating technological training into program orientation for both residents and staff, and pilot-testing the virtual-format sessions with residents to obtain feedback on ways to make the sessions engaging. Their programmatic suggestions included keeping recovery-oriented programming sessions separate from discussing crisis-related matters, ensuring adequate staffing on all shifts to provide psychoemotional support to residents, developing safe procedures through which residents can engage with family and friends, and planning safe recreational activities to maintain resident well-being.

Several of the participants’ recommendations are especially notable in how they are directly linked both to specific experiences that the participants shared during the small-group discussions and to extant literature. First, the recommendations to keep the programs open and to employ recreational opportunities can help address maintaining a sense of community, which was reported by participants as having decreased amid changes caused by the pandemic. Sense of belonging and connection to the community are well understood to be protective factors that support behavioral healthcare and recovery (Hagerty et al., 1992; Hystad & Carpiano, 2012; Guthrie-Gower & Wilson-Menzfeld, 2022), which underscores the importance of these recommendations. Second, the recommendations to ensure PPE availability and to clearly communicate required safety measures to minimize confusion can help mitigate risk of infection for both staff and Veterans. Through follow-up conversations with program leadership after conducting the small-group discussions reported in this work, we learned that the program manager had started to further emphasize the reasons and details for COVID-related precautions and changes during all-resident community meetings (Kim et al., 2022). Best practices for ensuring PPE availability and mask compliance as identified in other treatment settings (Thornton et al., 2022; Datta et al., 2022) may additionally inform future actions that programs can take. Third, the recommendation to ensure that professional cleaning is available can help keep communal areas adequately sanitized and decrease the spread of infection. Although residents cleaning and managing their own shared spaces may contribute to building the aforementioned sense of community under normal circumstances, programs may benefit during future similar crises by utilizing professional cleaning services that adhere to most up-to-date guidelines for safety (Ahmad et al., 2020; Ulsenheimer et al., 2022) while engaging in other safe community-building activities (as recommended by our participants) to counterbalance the potential resulting decrease in the sense of community.

Two qualitative studies (Clair et al., 2021; Pagano et al., 2021) and two commentaries (Fareed & Fareed, 2021; Herrera 2021) previously examined COVID-19 experience in residential treatment programs. One of the two qualitative studies ascertained COVID-19’s perceived impact through two focus groups with nine Veterans at a large VA homeless RRTP in southern California (Clair et al., 2021), identifying three challenges – communication, interruption of services, and social isolation/confinement. These areas were also raised in our small-group discussions. The second qualitative study thematically analyzed COVID-19’s impact on non-VA publicly-funded residential substance use treatment programs serving a homeless population in California (Pagano et al., 2021), revealing some similarities to our work including safety, mental health and well-being of staff and clients, and telehealth. The commentaries included recommendations that parallel those expressed by our participants, such as the need for reliable internet access for telehealth and for maintaining contact with outside support networks, and employing creative programming solutions to keep residents engaged in treatment.

Many healthcare services have been actively learning from and responding to challenges posed by COVID-19 (El-Hage et al., 2021; Abram et al., 2021; Castro & Sloane, 2021). Our study contributes to this widespread effort by focusing on residential treatment programs that provide care for individuals experiencing housing instability and affected by multiple complex social determinants of health. The two RRTPs we examined needed to coordinate their COVID-19 response around the clock and across shifts, while closely monitoring residents for worsening mental health or additive behaviors. Study participants’ recommendations provide valuable findings that other programs may find applicable to their own settings during future COVID-19-related or other emergencies.

A unique contribution of our study is the application of the WHO’s AAR (WHO, 2019), which placed minimal burden on participants (especially amidst their responding to a pandemic) by limiting their involvement to small-group discussions. At the same time, it also strove to be comprehensive by adhering to AAR’s focus on constructively reviewing events to ensure identification of concrete and actionable recommendations from the participants for the future. Beyond findings from the participants (which are the focus of this paper), the debrief process per the WHO’s AAR (WHO, 2019) allowed us to also present results on best practices, challenges, and follow-up actions in an organized fashion to leadership and incorporate their feedback (Kim et al., 2022). Through these conversations, we learned about steps that had been taken since our small-group discussions to address challenges, and the leaders shared their operational perspective. Importantly, besides approximately one hour of participants’ time (and their brief participant-checking of our summarized notes), the only resource that we required was researchers who could coordinate and facilitate the small-group discussions. These discussions provided participants with a structured way to reflect on their experiences as a team, which other studies can apply.

Comparisons of our results with the two other aforementioned qualitative studies (Clair et al., 2021; Pagano et al., 2021) demonstrate the potential utility of the AAR to reveal a broad representation of experiences and recommendations from multiple perspectives – leadership, staff, and patients. The AAR format can facilitate findings to be considered for possible programmatic and care-delivery changes in public, not-for-profit, and private healthcare organizations. AARs can be a useful tool to quickly evaluate the impact of changes on the system and uncover improvements to retain when the crisis has passed.

Our participants’ recommendations relate to interpersonal connections and information sharing, such as including frontline staff in the decision-making process whenever possible, supporting mechanisms for Veteran-staff interactions, and maintaining a sense of community through family involvement and participation in external activities. These recommendations may be especially pertinent for crises such as COVID-19, which require safety measures that restrict social contact. The desire for individuals to connect during challenging experiences is well documented also for contexts beyond communicable diseases or limited social contact (Shigemoto & Kawachi, 2020; McDonald-Harker et al., 2021). For further insight into the extent to which our participants’ recommendations apply to crises beyond COVID-19, future studies can adapt our procedures to investigate RRTPs’ responses to non-COVID-19 threats, then compare resulting recommendations to our findings.

Our study has several limitations. First, we only studied two RRTPs, with staff data from the two programs and Veteran data from just one program. Thus, our findings may not be representative of non-participants and individuals at other RRTPs in different regions of the country with different operating structures. In interpreting our findings, one might exercise caution regarding the generalizability of the participants’ viewpoints, which would have required a different sampling approach that allows drawing inferences about the prevalence of identified viewpoints (Wood & Christy, 1999; Chang et al., 2009). Second, some participants were acquainted with one or more of the study team members for reasons unrelated to this study, either through having engaged together in prior studies or sharing academic affiliations. There is a possibility that this may have affected the information that those participants shared during the small-group discussions and/or the study team’s interpretation of data collected on those participants’ perceptions and experiences. However, we looked for and did not notice any differential patterns in discussion content between those and other participants. Importantly, in participant-checking our collected data, no participants raised any concerns that the aggregately presented information misrepresented their perceptions. Third, the events studied through this work unfolded at a federal agency, which may be subject to federal policy regulations during times of crisis that were not explicitly taken into account by this study in gathering and reporting on participants’ recommendations for future practice. Moreover, although this AAR- and qualitative analysis-based work is not reliant on characteristics unique to the agency, the work was made possible through federal research funding. For other programs and healthcare systems, alternative sources of funding may need to be sought prior to adapting and applying the AAR framework.

Our AAR approach helped identify concerns and suggestions to improve two RRTP’s services at the start of COVID-19 and share actionable findings with RRTP leadership to prepare for future crises. The approach can be further adapted, beyond the realms of residential treatment and VA, as a consistent framework for reviewing crisis responses across multiple healthcare services. Doing so will help identify commonalities and heterogeneities among services that operate in widely varying contexts, and lead to determining both standardized and tailored preparations that the services can make for future pandemic-related and other crises. The AAR approach can be easily applied and fielded quickly to debrief and learn from system disruptions. The structured approach can uncover processes that should be sustained or adjustments and innovations that are needed to improve care. AAR results may improve program responses in the event of another crisis, whether a pandemic, man-made crisis, or natural disaster. And review of AAR results can help organizations improve their programs even in non-crisis situations. These results add to the still small amount of literature about how public and private residential treatment programs are impacted by, and respond to, crises.