Throughout the novel coronavirus (COVID-19) pandemic, applied behavior analysis (ABA) providers have become increasingly more reliant on telehealth to provide services to patients with autism spectrum disorder (ASD). Telehealth services are health services provided by a clinician to a patient electronically, without requiring the patient to travel to the clinician (Council of Autism Service Providers, 2020). Prior to the pandemic, telehealth services used in ABA therapy were primarily asynchronous—that is to say, they were implemented outside of real-time sessions (e.g., clinicians reviewing videos, caregivers completing online training courses)—or involved the training of caregivers. With the closure of many health clinics at the start of the pandemic, many direct ABA services shifted to telehealth. Videoconferencing technology was harnessed by behavior technicians (BTs; credentialed and noncredentialed individuals providing direct therapy to patients) and Board Certified Behavior Analysts (BCBAs) to provide direct ABA therapy and supervision to their patients synchronously (i.e., in real time). Due to the rapid adoption of telehealth direct therapy, ABA clinicians are in need of resources to promote treatment integrity and quality assurance.

Telehealth service delivery models have been used for years in the ABA field, predominantly for diagnostic assessments and caregiver training. Sutherland, Trembath, and Roberts (2018) completed a systematic literature review examining 14 telehealth studies. The quality of the studies varied, along with the service provided via telehealth (e.g., diagnostic assessments, caregiver-implemented services, caregiver consultations). Sutherland et al. found the results of the studies to be largely positive and suggestive that services provided through telehealth were equivalent to services conducted in person. However, research examining telehealth direct therapy delivered to patients was notably lacking. Researchers have also assessed the feasibility of training caregivers to implement ABA using telehealth service delivery models (Akemoglu, Muharib, & Meadan, 2019; Dimian, Elmquist, Reichle, & Simacek, 2018). Akemoglu et al. (2019) also conducted a systematic literature review examining 12 telehealth studies involving the instruction of parents on teaching language and communication strategies to their children. Similar to the Sutherland et al. review, the quality of the studies reviewed varied. However, the results also suggested that parent-implemented strategies (e.g., environmental rearrangement, modeling, time delay, following the child’s lead, prompting strategies, and expansions) taught over telehealth were effective at increasing targeted children’s communication rates and social communication skills.

Although existing research supports the use of telehealth service delivery, particularly for caregiver training, there is limited literature on the use of telehealth direct therapy. A recent study by Pellegrino and DiGennaro Reed (2020) addressed this gap in the literature; two adults with intellectual and developmental disabilities were taught independent living skills to the mastery criterion using total task chain presentation and least-to-most prompting over synchronous videoconferencing telehealth sessions. Ferguson et al. (2020) also taught skills through telehealth direct therapy to six children with ASD in a small-group format. All the participants acquired their targeted lessons. Although preliminary research seems promising, more research is needed to evaluate the effectiveness of telehealth direct therapy for patients with ASD.

Despite the limited research on telehealth direct therapy, the COVID-19 pandemic has created an expanded need for telehealth service delivery models. With health organizations across the world increasingly restricting in-person contact, including closing schools, more clinicians are offering telehealth direct therapy. The pandemic also brought on changes in the landscape of ABA insurance coverage for telehealth services. Prior to the pandemic, telehealth was covered by a small minority of payors in limited contexts. Whereas Medicaid included both in-person and synchronous telehealth services within their face-to-face service requirements (Centers for Medicare and Medicaid Services, 2011), the vast majority of payors excluded services that were not conducted in person. Following the public health emergency declaration, the U.S. Department of Health and Human Services (2020b) took steps to increase the ease of telehealth service provision, leading many payors to increase telehealth service coverage. This rapid transition to telehealth service delivery models was necessary for the safety of patients, families, and clinicians. However, it is equally important to consider appropriate training and support for ABA practitioners and consumers regarding telehealth service delivery models (Council of Autism Service Providers, 2020; Cox, Plavnick, & Brodhead, 2020). Although it is critical for clinicians to use the tools at their disposal to provide intervention that promotes the safety of all involved, it is also necessary to assess the efficacy of the telehealth service delivery model being provided. As BCBAs incorporate telehealth direct therapy into their practice with patients, they are still ethically bound to ensure that they themselves and those they supervise (e.g., Registered Behavior Technicians[RBTs]) are practicing within their scope of competence (Behavior Analyst Certification Board, 2014).

Before starting telehealth direct therapy, it is critical for the supervisor to complete an assessment of appropriateness and identify potential modifications needed for the individual patient (Columbo, Wallace, & Taylor, 2020; Council of Autism Service Providers, 2020; Rodriguez, 2020). In order to guide ethically appropriate clinical recommendations, Rodriguez (2020) published a tool aimed at assisting clinicians in choosing an appropriate method of delivery of telehealth direct therapy. Rodriguez’s tool includes two parts: first, an assessment conducted over synchronous videoconferencing technology with a supervisor and, second, the selection of the corresponding treatment via the matrix provided. The matrix helps assess the patient’s and caregiver’s readiness for telehealth direct therapy by evaluating skill building or behavior intervention plan (BIP) modifications that may be required, along with any coaching or supports that may need to be implemented for success in a telehealth setting. In order to have a successful telehealth direct therapy session with a patient, there are several considerations to certify the BT, patient, and family are successful. The tool published by Rodriguez evaluates the various considerations to be taken prior to implementing telehealth direct therapy, aiding clinicians in completing their ethical responsibilities in making appropriate clinical recommendations.

Currently, there is a lack of resources available to promote treatment integrity of ongoing telehealth direct therapy. Pollard, Karimi, and Ficcaglia (2017) highlighted the need for clinicians and providers of telehealth services to establish “robust quality assurance and improvement systems” in order to improve treatment quality and provide consumer protection. The authors highlighted three key factors: targeted education and training, systems for regular review, and regular communication and review with all stakeholders in the system. Furthermore, in a review of the literature, Fryling, Wallace, and Yassine (2012) identified a correlation between treatment integrity and the effectiveness of the intervention. Thus, the evaluation of and intervention to support treatment integrity likely will impact the effectiveness of an intervention. Literature on treatment integrity and quality assurance for direct ABA services delivered via telehealth is scarce. Additionally, no treatment integrity measures have been published to assess telehealth direct therapy.

The purpose of this article is to introduce a measure, the Telehealth Therapy Treatment Integrity Measure (TTTIM), designed to assess telehealth direct therapy treatment integrity. First, a background is provided to give context to the challenges faced by ABA providers conducting telehealth direct therapy, which helped guide the development of the TTTIM. Subsequently, the design and uses of the TTTIM are described. Although the TTTIM was created to help alleviate the urgent need for treatment quality assurance brought on by the COVID-19 public health crisis, this tool is also a preliminary step toward refining the practice of telehealth direct therapy, which will likely have benefits in the field of ABA for years to come.

Background

There are many unique challenges faced by ABA providers implementing telehealth direct therapy as compared to in-person services. As is typical of in-person therapy sessions, telehealth direct therapy will vary greatly across individuals due to variations in patients’ goals, current skill sets, and presentations of challenging behaviors. In addition, there are many considerations that are unique to telehealth direct therapy, including involving the caregiver in treatment, building BT–child rapport, managing challenging behaviors, promoting engagement, managing expectations, setting treatment goals, protecting patient privacy, training BTs, and evaluating their performance. These considerations, which helped guide the development of the TTTIM, are discussed in what follows.

One significant consideration for telehealth direct therapy that varies across patients is caregiver support. Caregiver presence and participation may be required for telehealth direct therapy for some patients, as opposed to the flexible or optional participation of caregivers during in-person sessions. Thus, telehealth direct therapy varies on an additional dimension of the amount of support provided by the BT versus the caregiver. The roles vary on a continuum from the BT providing all skill acquisition and BIP interventions to the caregiver providing all skill acquisition and BIP interventions upon direction from the BT. Some patients may be able to connect to the videoconferencing software and gather any needed materials completely independently; other patients will require a caregiver to complete these tasks. Regardless of where the patient falls on this continuum, the BT is expected to lead the session with the patient and is responsible for collecting data and guiding the content/structure of the session. Additionally, even in a situation where the BT provides all interventions in the absence of caregiver support, a caregiver should be present for all telehealth direct therapy sessions as they are responsible for the physical care and safety of the patient (Council of Autism Service Providers, 2020).

Additional elements of ABA therapy may look different during telehealth direct therapy, as a result of the variations in caregiver support and the logistical challenge of providing therapy via videoconferencing technology. Although there is existing research on the positive effects of pairing when working with patients in person, no research exists for pairing via telehealth service delivery models. The telehealth direct services model places extra importance on the ability of the clinician to minimize challenging behaviors during telehealth direct therapy because the clinician alone cannot keep the patient safe during an episode of dangerous behavior and cannot provide therapy to a patient who has escaped the session by terminating the videoconferencing mode or eloping from view. There is evidence that building rapport through the use of pairing and demand reduction can modulate the averseness of task demands, with good rapport between clinician and patient resulting in a decrease in escape-maintained behavior (McLaughlin & Carr, 2005). Conversely, poor rapport may increase the averseness of demands, resulting in an increase in escape-maintained problem behavior.

Similar to the challenges in building rapport through a videoconferencing platform, engagement of the patient during telehealth direct therapy may be more challenging than engagement in person, as the BT is not physically present to provide reinforcement. Whereas a caregiver can provide physical items or actions as reinforcers during a session, patients who are able to independently access telehealth direct therapy may receive all engagement solely from the BT through the videoconferencing platform. Regardless of the amount of support provided by the caregiver during the session, the BT should identify and provide preferred items/actions to the patient through the videoconferencing platform in order to build rapport and pair themselves with reinforcement. Reinforcers that are present in the patient’s immediate, physical environment can also be provided, but those presented through the videoconferencing platform will aid in pairing. For patients for whom the caregiver is providing all interventions under the direction of the BT, all reinforcers can be provided physically by the caregiver. However, the BT should still strive to engage with the patient. Item F-5 of the RBT Task List (2nd edition) states that RBTs should “maintain client dignity” (Behavior Analyst Certification Board, 2018); although this includes several components, one piece involves the demonstration of respect for the patient, which the BT can do by engaging with the patient directly. If this is a difficult skill for the patient, engagement opportunities can be created that require a lower response effort on the part of the patient (e.g., targeting mastered skills, incorporating target skills into preferred activities). Additionally, engagement is not solely about providing reinforcement; engagement requires the clinician to use techniques to maximize attention and effort from the patient, building upon a foundation of rapport with the patient. Examples of engagement techniques include gaining the patient’s attention (e.g., putting engaging activities on the screen), providing directions, and creating learning opportunities.

Setting clear expectations is another area that may be more challenging during telehealth direct therapy than during in-person sessions. In order to set expectations during an in-person session, coordination between the BT and other adults is not typically required; additionally, the BT is able to use physical items (e.g., write out a schedule, show the toy that the patient can play with next) and whole-body cues (e.g., gesture to an item) to communicate with the patient. Both aspects may be more challenging during telehealth direct therapy. Depending on the level of caregiver support needed, the BT may need to engage in frequent coordination with the caregiver in order to plan and execute successful telehealth direct therapy. Additionally, as the BT is not able to use physical items or whole-body cues, either the caregiver needs to engage in the physical action or the BT needs to use electronic tools (e.g., screen sharing) or verbal speech to communicate expectations to the patient.

The goals for telehealth direct therapy versus in-person services may also differ. During the early days of the COVID-19 pandemic, Columbo et al. (2020) posited that if caregivers were sufficiently skilled to implement the patient’s program, maintenance should be prioritized in order to reduce errors of commission, which impact treatment outcomes. Although Columbo et al. referenced a need related specifically to the pandemic, other emergency situations may arise where a focus on maintenance would also be appropriate (e.g., temporary relocation due to a house fire, a family medical issue that precludes in-person sessions). However, outside of similar emergency situations, the focus of telehealth direct therapy would likely be on the acquisition of new skills or on the generalization of skills to the home environment without the physical presence of a BT. Thus, following the assessment of fit for telehealth direct therapy, the supervisor should consider whether the focus should be on maintenance, generalization, or acquisition of new skills. For interventions to be carried out by the caregiver, the complexity of the interventions should be considered; only those within the caregiver’s level of competence and tolerance should be included. Additionally, the supervisor may consider certain goals as more or less appropriate for the telehealth direct setting for the individual patient. For example, the supervisor may plan for initial telehealth direct therapy to focus on telehealth readiness skills, such as sustaining attention to the device, responding to demands placed through a videoconferencing platform, complying in the absence of physical prompting, and logging in to or accessing the videoconferencing platform independently (as clinically appropriate).

Although professionalism must be considered in all ABA service delivery options, there are some additional considerations in the provision of telehealth direct therapy. All Health Insurance Portability and Accountability Act (HIPAA) guidelines must be considered when providing telehealth direct therapy, including collecting data. Although the Notification of Enforcement Discretion for Telehealth Remote Communications During Covid-19 Nationwide Public Health Emergency (U.S. Department of Health and Human Services, 2020a) allows health providers to use certain videoconferencing platforms that do not meet HIPAA standards, other HIPAA standards were not relaxed. Following the COVID-19 public health emergency, HIPAA-compliant videoconferencing software should be used for all telehealth services. All available confidentiality protections should be considered, including providing patients/caregivers with unique meeting codes for each session, using passwords to protect links, and locking meetings upon the entry of all participants (Pollard et al., 2017). Additionally, all elements of the environment to be viewed by the patient should be considered, including professional dress on the part of the clinician and the appropriateness of the background that is visually present. Especially during the current pandemic, as many clinicians implement telehealth direct therapy from their homes, attention should be given to maintaining privacy, including using headphones, locking doors, and/or coordinating with family members to ensure a lack of disruption (Pollard et al., 2017).

An additional factor to consider in the implementation of quality telehealth direct therapy is the skill set of the BT. As the previous paragraphs described, there are new skills required on the part of the BT to implement effective telehealth direct therapy. If a BT was trained on and previously implemented in-person sessions, it is necessary for them to generalize skills from the in-person to the telehealth direct therapy model. As Stokes and Baer (1977) asserted in their seminal paper on generalization, it cannot be assumed that generalization will occur without a plan for the facilitation of generalization (termed “train and hope”). Therefore, it is critical to not only assess but also plan for generalization of these skills on the part of the BT. BCBAs have an ethical obligation to ensure that BTs providing telehealth direct therapy are adequately trained. In order to ensure BTs are initially trained to competency and maintain those skills (Behavior Analyst Certification Board, 2014), a method of evaluating the skill set of BTs in providing telehealth direct therapy is necessary.

The Telehealth Therapy Treatment Integrity Measure (TTTIM)

The TTTIM (see Fig. 1) is presented to fill the gap in the literature regarding telehealth direct therapy treatment integrity and quality assurance. The TTTIM is a 44-item tool designed to be completed during a telehealth direct therapy overlap. Aspects of the TTTIM are discussed in turn, including the purpose, preparation for implementation, description, scoring, and administration. Two case studies are also described.

Fig. 1
figure 1

Telehealth Therapy Treatment Integrity Measure

Full size image

Purpose of the TTTIM

The TTTIM is designed as a training tool to ensure that telehealth direct therapy is implemented appropriately. In terms of specifics, the information gathered from the TTTIM can serve several purposes. The items on the TTTIM can guide the provision of initial training to a BT in telehealth direct therapy procedures; additionally, a BT can use the TTTIM as a reminder of important elements to include or as a self-evaluation tool. During telehealth direct therapy, the information recorded on the measure can guide the supervisor in collecting baseline data on the BT’s performance, providing in-the-moment feedback to the BT, and supporting the improvement of the BT’s skill set through the use of evidence-based training such as behavioral skills training (BST; Leaf et al., 2015). An additional purpose of the TTTIM is to assist in the selection of professional goals for a BT; goals can be written from items missed on the TTTIM to focus supervisor support, feedback, and data collection. Multiple administrations of the TTTIM may be completed for a given BT to provide, along with other information, an overall measure of the BT’s performance/skill.

Preparation for Implementation of the TTTIM

The TTTIM is designed to assess a BT’s proficiency in implementing telehealth direct therapy. As the measure is intended to apply to any potential patient with whom a BT may implement telehealth direct therapy, the items are designed to be broad enough to fit the variations that occur across sessions; however, there are considerations a supervisor should make before implementing the TTTIM. Several procedures should occur before the implementation of the TTTIM, detailed in Fig. 2, which can be used as a checklist for the supervisor. First, it is recommended that the supervisor do the following for each patient: (a) assess the appropriateness of telehealth direct therapy for the patient (for examples, see Council of Autism Service Providers, 2020, and Rodriguez, 2020), (b) assess the BT’s skill set, (c) train BTs in telehealth procedures, (d) modify the patient’s program (as necessary), and (e) ensure appropriate credentials or experience for the rater. Specifically regarding supervisor modifications, the supervisor should consider defining expectations for what the BT should target (e.g., the number of learning opportunities that should be contrived, how much maintenance vs. new skill acquisition should be included) and how those skills should be targeted (e.g., the method of stimuli presentation) during telehealth direct therapy for each patient. Additionally, the supervisor may find it beneficial to use the TTTIM as a guide when making modifications to a patient’s program and defining expectations for BTs. Some of the items listed on the TTTIM are broad by design, to allow for patient-specific differences; as the BT should not be expected to identify how a skill should be implemented with a given patient, this operational definition should be provided by the supervisor. Not only is this additional programming clarity likely to improve intervention quality, but it will also allow for easier scoring of some of the TTTIM items. As the TTTIM is described next, the items that would benefit from clear expectations stated by the supervisor are discussed.

Fig. 2
figure 2

Preparation for Implementation of the Telehealth Therapy Treatment Integrity Measure (TTTIM)

Full size image

Description of the TTTIM

The TTTIM was developed with the aforementioned assumptions in mind. There are seven main sections on the measure, addressing different aspects of telehealth direct therapy: (a) caregiver engagement and support, (b) planning, (c) patient engagement, (d) downtime, (e) BIP, (f) skill acquisition, and (g) data collection. These sections were selected to address challenges in providing telehealth direct therapy based on clinical experience, as well as various factors of telehealth direct therapy quality assurance as described by Pollard et al. (2017). In each section, there are between five and eight items, detailing specific behaviors expected on the part of the BT. Items include skills a BT should have in their repertoire from in-person sessions, as well as new skills necessary to ensure appropriate patient behavior in the telehealth direct therapy setting. The BT is expected to ensure all applicable items occur in a session; however, as caregivers are involved to varying degrees as described in the previous section, the physical act of completing the item may be done by the caregiver. The items within each section are discussed in turn.

The section on caregiver engagement and support addresses the skill set of interacting with a caregiver appropriately during telehealth direct therapy. As more input from the caregiver may be needed to ensure that the patient participates in telehealth direct therapy, a greater level of interaction may be needed between the BT and the caregiver than during an in-person session. Expected behaviors on the part of a BT include “greets the caregiver,” “uses everyday language,” “is respectful/professional in all interactions,” “sets clear session expectations with the caregiver,” “provides a rationale/purpose for the lesson before running a new lesson,” “provides the caregiver with direction regarding program/BIP implementation,” “provides the caregiver with frequent reinforcement,” and “collaborates to collect accurate data.” Some of these items require generalization of skills that are also expected during an in-person session (e.g., greets the caregiver); however, other items may require a skill set that the BT has not previously used (or not used to the extent needed during telehealth direct therapy). For example, during an in-person session, a BT is generally able to see all patient behaviors without requiring feedback from a caregiver. However, in telehealth direct therapy, situations may occur that require the BT to ask for the caregiver’s assistance in collecting data. For example, the patient may fall to the floor, off screen; if head banging was a behavior in the patient’s BIP, the BT would likely need to ask the caregiver if the patient engaged in head banging when the patient moved off screen, as well as ask the caregiver to move the camera so that the BT can see the patient. Additionally, the BT would need to provide the caregiver with support in responding to the head banging per the patient’s BIP. In general, the supervisor should define in advance for the BT the degree of caregiver involvement expected in the session.

Planning involves the set of skills needed to ensure a session flows smoothly. As a BT is not present in person to ensure that items are prepared in advance, a greater degree of planning and coordination with the caregiver is likely necessary to ensure lessons are implemented effectively during a session. Expected behaviors on the part of the BT include “ensures all needed materials are present, asking caregivers for assistance as needed,” “collaborates with caregiver/patient to plan session,” “plans for meals/bathroom/transitions,” “creates an activity schedule/plan for the session,” “meets the learning opportunity goal,” and “checks in with the caregiver/patient and adjusts when appropriate.” Although setting a clear plan for a session is always important, the lack of physical presence with the patient and potential need to coordinate with a caregiver add an extra challenge. For example, if the patient moves between rooms (e.g., to the bathroom, to eat a meal, to get an item), either the patient or the caregiver must ensure that the device with the videoconferencing platform is moved as well. Furthermore, as all items in front of the patient may not be visible to the BT, the BT needs to ensure that all needed items (e.g., visual schedules, token economies) are present and available by asking the caregiver/patient to move the camera so those items are in view or by asking the caregiver/patient if the items are available. The item “meets the learning opportunity goal” requires the supervisor to set specific expectations. As there is little guidance in the literature regarding the rate of presentation of learning opportunities, an example goal is to capture learning opportunities for at least three lessons. In addition to varying across patients, this goal may change over time; during an initial telehealth direct therapy session, a supervisor may set the goal lower and then gradually increase with patient success.

Patient engagement is likely the section of the TTTIM that involves the greatest change from an in-person session. Ensuring that the patient is engaged, with the BT and/or the caregiver, throughout telehealth direct therapy requires skills that might be new for the BT. Expected behaviors on the part of the BT include “greets the patient,” “uses age-appropriate language,” “creates clear engagement expectations,” “conducts regular preference assessments,” “contrives opportunities for the patient to engage with the caregiver/BT,” “uses a variety of engagement techniques to achieve engagement,” and “effectively uses technology to build engagement.” Although a caregiver can provide toys/items to the patient during telehealth direct therapy, the BT may need to find other ways of pairing themselves with reinforcement. For example, when sharing a token board over a videoconferencing platform (e.g., using screen-sharing technology), the BT can use a picture of the patient’s favorite character as the background of the token board. Additionally, a BT can use a stuffed animal to play “hide-and-seek” with the patient through a videoconferencing platform. Games such as hangman, tic-tac-toe, and charades can be played using tools such as screen sharing. The supervisor should provide guidance regarding the technology to be used to build engagement for each patient; if a patient has skill deficits in interacting with technology, the supervisor should consider adding that component to the patient’s treatment plan. Furthermore, clear expectations should be set between the patient and the caregiver/BT (depending on who is implementing the intervention), and there may be situations where setting clear engagement expectations is more necessary in the telehealth setting. For example, a patient may benefit from the presentation of a rule to stay in front of the camera during the session or to ask before leaving the area (so that the caregiver can move the device accordingly).

Downtime addresses components to be implemented in between structured lessons. Expected behaviors on the part of the BT include “ensures engagement while inputting data, facilitated by the caregiver if needed,” “sets clear downtime expectations,” “maintains appropriate engagement with the patient,” “targets and reinforces functional language,” “contrives opportunities for natural environment teaching (NET) during every break,” and “reinforces appropriate downtime behavior.” Even though BTs are not physically present during downtime/breaks during telehealth direct therapy, they likely need to ensure that the patient remains engaged during these times. Setting clear expectations (e.g., how long breaks are) is critical to reducing any potential challenging behaviors upon transitioning to the next activity. When inputting data, the BT can either ensure that the patient is engaged in an independent leisure activity or ask the caregiver to interact with the patient. Additionally, targeting and reinforcing functional language is as critical in the telehealth direct therapy model as it is during in-person sessions, necessitating BT generalization of this skill.

BIP primarily includes skills that a BT already demonstrates during in-person sessions; however, it is necessary to ensure that these skills generalize to telehealth direct therapy. Expected behaviors on the part of the BT include “incorporates antecedents that elicit challenging behaviors only per supervisor notes,” “implements antecedent modifications,” “reinforces replacement behaviors,” “implements consequence manipulations,” and “collects data as specified in the BIP.” As a BT is not present physically in person to aid caregivers in responding to challenging behavior (e.g., response blocking, redirecting), care must be taken to ensure that caregivers are prepared before including triggers for challenging behaviors. The supervisor should clearly define for the BT if and when triggers should be included during the telehealth session (ideally after ensuring that caregivers are sufficiently trained and prepared to manage the behavior). Additionally, as telehealth direct therapy with some patients may require a large response effort on the part of the caregiver, a supervisor may consider the benefits of incorporating positive reinforcement for the caregiver; if a caregiver receives positive reinforcement (e.g., engaging positively with their child instead of managing a challenging behavior) in response to their efforts in implementing interventions during telehealth direct therapy, there will be an increase in the future frequency of the caregiver’s behavior of participating in telehealth sessions.

Skill acquisition targets the components surrounding the implementation of lessons with a patient. Although the supervisor is responsible for identifying the specific lessons to target and how they should be targeted over telehealth direct therapy, it is a BT’s responsibility to implement those lessons accurately. Expected behaviors on the part of the BT include “ensures that all stimuli/materials are prepared before starting the lesson,” “sets clear expectations prior to the lesson,” “implements the activity schedule and/or plan,” “troubleshoots implementation difficulties,” “implements the lesson per supervisor notes, reaching out to supervisor if unclear,” “prompts/fades per notes,” and “uses reinforcement per notes.” Similar to the last section, skill acquisition primarily considers the BT’s generalization of in-person skills. However, as the physical presentation of visual discriminative stimuli may be more challenging over telehealth, it is critical to ensure that this skill set generalizes. For example, if a BT typically presents pictures in person for a lesson, the supervisor needs to identify how the BT should present those pictures over telehealth and provide resources to the BT as appropriate. It is unlikely that the caregiver will have pictures available in the home, so the BT may need to present the pictures using screen sharing. Even with direct guidance from the supervisor regarding how to present stimuli, ensuring that the pictures are available and ready to go possibly involves more steps (i.e., greater response effort) than during in-person sessions. During skill acquisition, difficulties in implementation may arise; for example, a caregiver/patient may communicate that a picture shared on-screen is not big enough to see, or they may communicate that they cannot see the content at all. In these situations, the BT will need to think flexibly in order to address the issue in the moment, following up with a supervisor as needed to address long-term challenges.

Data collection addresses all behaviors surrounding the accurate gathering of data throughout the telehealth direct therapy session. As different providers of ABA use various data collection methods (e.g., pencil and paper, electronic data collection), the implementation of the items in this section will likely vary greatly. Expected behaviors on the part of the BT include “collects data immediately after an occurrence of behavior,” “spends no more than the specified time collecting data,” “notes any challenges in caregiver communication,” “notes intervention modifications used in the session,” and “communicates challenges in accurate data collection to supervisor.” If a BT is collecting data via pencil and paper, the BT will need to ensure that all items are available before starting the session so that they do not need to leave the videoconferencing platform camera view. If a BT is collecting electronic data, the BT can either ensure that they have two devices (e.g., one device for the telehealth session and one device for the data collection) or collect data with pencil and paper, transferring it to the electronic system at appropriate intervals. Although important for all sessions, it is critical that BTs write notes regarding anything specific to the telehealth session to ensure the supervisor is informed regarding any telehealth-specific challenges or successes. As the view via telehealth may not be optimal, it is critical for the BT to communicate to the supervisor any issues, including ongoing concerns with the placement of the camera on the caregiver/patient’s end and potential data discrepancies. The supervisor may need to problem-solve issues with the caregiver/patient outside of a telehealth session. Although data collection is an extremely important part of sessions, it is important for the BT to have an appropriate balance between data collection and patient engagement; thus, the BT should follow expectations set by the supervisor regarding how much time to spend on data collection.

Scoring the TTTIM

For each of the seven sections of the TTTIM, a BT receives a score of either “needs support,” “partially met,” or “met,” based on the BT’s independent completion of the items in each section. As the TTTIM is designed to provide a broad overview of the BT’s skill set, a separate score is not provided for each item. The simple scoring procedures for the TTTIM were designed to increase the accuracy of scoring. In a review of interobserver agreement, it was found that increased complexity (e.g., number of response categories) was associated with decreased interobserver agreement (Kazdin, 1977). Although interobserver agreement is not synonymous with accuracy, it is a method of confirming the reliability of the data collection measure. Measures that have lower levels of complexity are more likely to have higher levels of reliability.

Scoring for individual items should follow supervisor-specified operational definitions for the individual patient (per the guidance in the section “Preparation for Implementation of the TTTIM”). As needed, professional goals may be written for a BT to focus training and support on a specific item. Although there is little empirical guidance regarding appropriate mastery criteria, Hogan, Knez, and Kahng (2015) noted that mastery criteria for skills taught using BST are “typically strict,” and they provided the example of 90% accuracy. Additionally, McDougale, Richling, Longino, and O’Rourke (2019) conducted a review of mastery criteria in three prominent behavior-analytic journals and found that 90% was the most common criterion for skill acquisition research; however, in a survey of practitioners, 80% was more commonly used (Richling, Williams, & Carr, 2019). Given the lack of agreement and the potential variation across patients regarding critical elements, we designated mastery as demonstrating all applicable items of the TTTIM and partial mastery at the level of 80%. A score of “needs support” is provided if the BT independently completes less than 80% of the items in the section during the course of the overlap (i.e., misses two or more items in the section). A score of “partially met” is provided if the BT independently completes between 80% and 99% of the items in the section during the course of the overlap (i.e., misses one item in the section). A score of “met” is provided if the BT independently completes 100% of the items in the section during the course of the overlap (i.e., misses no items in the section). If a prompt is provided during the course of the observation for an item, then that item is not considered to have been independently completed by the BT. Any items for which the opportunity does not occur in the specific overlap are considered not applicable and are not included in the scoring of the BT during that overlap. As the scoring is a reflection of the BT’s performance in this particular session with this particular patient, items that are not applicable to that session are not scored; in order to obtain a broader picture of the BT’s skills, the TTTIM should be scored across a number of sessions and patients, ensuring that opportunities for all items were provided across sessions.

Administration of the TTTIM

The TTTIM is designed to be implemented during synchronous telehealth direct therapy or via asynchronous viewing of a recording of telehealth direct therapy. Regardless of the videoconferencing platform used (e.g., Zoom), for all telehealth direct therapy, the BT and patient/caregiver should be present in the videoconferencing platform synchronously, using video and audio, with the patient/caregiver joining the video call from their home or other location. The BT should provide the patient/caregiver with feedback if the audio/video needs adjusting (e.g., if the patient was not on the screen, if the patient was accidentally muted). For real-time scoring, the rater should be present synchronously in the videoconferencing platform; for video scoring from a recording, the rater should view the video recording at a later date. Regardless of the method of scoring, the rater should provide feedback to the BT, corresponding to the items on the TTTIM, either at the end of the overlap or at a separate time. Information from two implementations of the TTTIM is detailed in the followings sections as case studies to highlight potential strengths and weaknesses of this tool.

Case Study 1

The first case study involves the implementation of the TTTIM during a synchronous overlap with a noncredentialed BT with 3 years of clinical experience. The BT received training on telehealth direct therapy prior to implementing any telehealth direct sessions and had been implementing telehealth direct sessions for approximately 3 months. The TTTIM rater was a BCBA and the patient’s supervisor (the first author). The patient, a 9-year-old male, had received both therapy from a BT and supervision by a BCBA solely through telehealth for approximately 3 months due to the COVID-19 pandemic. Per the supervisor, the BT implemented all BIP and skill acquisition interventions, with support from the caregiver only as needed (e.g., providing redirection if the patient left the screen or was nonresponsive). The patient was a new patient for the BT, and the overlap for the case study occurred on the BT’s second session with this patient. Upon starting the overlap, the BT expressed gratitude at the occurrence of the overlap, as she was having difficulty engaging the patient over telehealth. At the outset of the overlap, the supervisor identified behaviors on which to collect interobserver agreement data; data were compared at the end of the overlap. Throughout the overlap, the supervisor collected data on the TTTIM according to the scoring notes. Upon an incorrect response from the BT for the item “maintains appropriate engagement with the patient” in the Downtime section, the supervisor used BST to teach the BT to use this skill with this patient. Following the use of BST, the BT engaged in the behavior correctly at the end of the overlap. Later in the overlap, following an incorrect response from the BT for the item “checks in with caregiver/patient and adjusts when appropriate” in the Planning section, the supervisor again used BST to teach the BT to implement this skill with the patient. Following the overlap, the supervisor provided written feedback summarizing the verbal feedback provided during the overlap, along with two new professional goals addressing the most critical areas of need for this BT with this patient (those targeted with BST during the course of the overlap). Not all items marked incorrect on the TTTIM were addressed during this overlap, in order to provide the BT with an appropriate balance of positive and constructive feedback. The scores for the BT for each section of the TTTIM were as follows: caregiver engagement and support–needs support, planning–needs support, patient engagement–needs support, downtime–needs support, BIP–needs support, skill acquisition–met, and data collection–partially met. The scores were not provided to the BT in order to focus the BT’s attention on the professional goals and were instead used only to guide the supervisor in providing support and feedback.

Case Study 2

The second case study involves the implementation of the TTTIM during an asynchronous overlap with a noncredentialed BT with 3 years of clinical experience. The original telehealth direct therapy session was conducted and recorded via a synchronous videoconferencing platform. The BT received training on telehealth direct therapy prior to implementing any telehealth direct therapy and had been implementing sessions for approximately 5 weeks. The TTTIM rater was the third author, a BCBA with no prior exposure to the BT’s therapy sessions and only one previous direct telehealth session with the patient; the BT requested feedback from this BCBA. The patient, a 6-year-old male, had received both therapy from a BT and supervision by a BCBA solely through telehealth for approximately 5 weeks due to the COVID-19 pandemic. At the time the TTTIM was implemented, the BT had completed 15 sessions with the patient. Implementing the TTTIM via video review was difficult for certain sections due to the inability to confirm or check with the BT if something was completed in the moment. Some items, such as “communicates challenges in accurate data collection to supervisor,” had to be marked as N/A until the TTTIM was reviewed with the BT, as these items could not be discerned from the video alone. Although the rating supervisor was not the patient’s or BT’s supervisor, the TTTIM proved to be a valuable tool from which to make notes of feedback for both. For example, one highlight of the session was the BT’s use of the patient’s preferred items during the session and the way the items were integrated into downtime and skill acquisition time. However, the TTTIM item “uses reinforcement per notes” served as a reminder to check the supervisor’s notes regarding telehealth-specific reinforcement and recommend to the patient’s supervisor that the notes be updated to reflect some of the items and activities that were successful in the session with the wider clinical team. This BT–patient pair was on their 15th session together, so the item regarding providing a rationale/purpose for lessons was marked as N/A as it had been covered in the earlier sessions of telehealth therapy. Other items pertaining to coordination with the caregiver were marked with a “+” because they did occur, but not to the extent they would need to if it were an earlier session. The scores for the BT for each section of the TTTIM were as follows: caregiver engagement and support–met, planning–met, patient engagement–partially met, downtime–met, BIP–met, skill acquisition–met, and data collection–met. The scores were not provided to the BT in order to focus the BT’s attention on the professional goals and were instead used only to guide the supervisor in providing support and feedback.

Discussion

The purpose of this article was to present a treatment integrity measure to assess the accurate implementation of telehealth direct therapy and guide the training of the BT during telehealth direct therapy sessions. The TTTIM looks to identify the important components of telehealth direct therapy across seven domains that are meant to provide a framework that clinicians can work from as they support skill demonstration and create professional goals for the BT providing telehealth direct therapy.

Due to the rapid shift from in-person to telehealth sessions, expectations of logistics of a telehealth direct therapy session and expectations for the level of involvement from families may have been unclear to families and practitioners. The level of experience with telehealth models that families and clinicians had before the COVID-19 pandemic likely varied as much as their understanding of the technology they were using to deliver and receive the service. With many people new to telehealth, it can make for a wide range of both models of service delivery and effectiveness of that service. The purpose of the TTTIM is to identify the components likely to increase the effectiveness of telehealth direct therapy and to provide a framework and point of reference to guide the assessment and training of practicing BTs. Although the TTTIM can guide training for new BTs, this measure assumes that the BTs conducting the telehealth direct therapy have already been trained to criterion on many in-person service delivery skills. The seven components identified can help ensure that those providing the service are still working within their scope of competence.

The case study data presented previously highlight some potential strengths and weaknesses of the TTTIM. The TTTIM serves as a tool to aid the supervisor in providing support to the BT. First and foremost, the TTTIM aids the supervisor in assessing the BT’s skill set as it relates to telehealth direct therapy for individual patients. The most valuable information resulting from scoring the TTTIM is the guidance provided by the supervisor to the BT following that assessment. As a supervisor may find a certain item on the TTTIM to be an area of weakness for an individual BT, the supervisor may choose to highlight that item as a professional goal and develop an individualized checklist of component steps for the BT. In a situation similar to Case Study 1 where the BT received low scores in a majority of sections, the supervisor should consider providing frequent support to the BT, gradually increasing expectations in terms of the specific feedback and professional goals provided until the BT meets the criteria in all sections. Additionally, the relative importance the supervisor places on individual items (and thus the presentation of specific feedback) may vary as the BT conducts more sessions with a given patient. The TTTIM was designed to be flexible, to allow administration during any telehealth direct therapy session with any patient; although this is a strength of the measure, it is critical for those interpreting the TTTIM results to be aware that the scoring of items is highly dependent on the supervisor’s guidance provided for individual patients. Additionally, as in Case Study 2, the TTTIM may serve as a framework for providing supervisors with feedback regarding the details present in the treatment plan for telehealth direct therapy sessions; a supervisor could also use the TTTIM as a self-evaluation tool. However, the TTTIM is still a new tool, and the main weakness of the TTTIM is the lack of data supporting its use. Although this is an important avenue for future research, even in the absence of such data, the TTTIM can provide valuable guidance for supervisors, BTs, and even companies regarding the expectations for telehealth direct therapy sessions. Although the TTTIM was designed for telehealth direct therapy, adapted and following a supervisor’s recommendation, it could also be used as an in-person tool to provide guidance for clinical teams. Additionally, the TTTIM, potentially in a modified form, could be used in a study to assess the efficacy of telehealth direct therapy, as a measure of treatment integrity.

Although some clinicians or patients may view the telehealth direct therapy model as a lesser service compared to in-person therapy, there are many benefits to telehealth direct therapy that may improve aspects of teaching certain skills. For example, a supervisor can quickly prompt a BT to modify a teaching approach discreetly through chat, or request that the BT target a skill whose mention might act as a prompt itself for the patient. Additionally, there is the possibility that telehealth direct therapy may increase generalization if the patient and caregiver are present in the home, with the patient engaging in skills in their natural environment. Telehealth may also allow for the targeting of skills (e.g., sleep) that may be challenging to incorporate into in-person sessions for logistical reasons. Although the focus of telehealth direct therapy is not on caregiver training and is instead on building the patient’s skills, telehealth could be a tool to increase caregiver independence, due to the lack of an in-person presence of a BT (e.g., the BT is physically unable to block a behavior, requiring the caregiver to do so).

Future iterations of the measure would likely include additional refinements, as relevant components may have been missed due to the rapid development of the measure. As future data and analyses of telehealth direct therapy models are released and key factors of successful sessions are further analyzed, updates to the TTTIM can continue to refine and improve the measure. The TTTIM was created with the intention of being a more general measure that can be applied to telehealth direct therapy for a wide range of patients, but as the telehealth direct therapy model continues to grow, research can address the need for different variations of the measure to be created to apply to the varying levels of learning targets, behaviors, and caregiver support.

With the rapid transition from in-person to telehealth sessions, many clinicians may have felt they were entering a territory where they had little to no experience. With neither previous experience to draw from nor available treatment integrity measures specific to telehealth, supervisors might have felt that telehealth would not be effective or clinicians may not have understood how to best support BTs in providing effective telehealth direct therapy. As telehealth continues to be used and clinicians proceed to innovate the telehealth direct therapy model, the TTTIM can provide a framework and structure for successful and supportive telehealth direct therapy.