Accessing and reading the scholarly literature is critical for professionals in the field of behavior analysis. As stated by Jacobson (1990), “Because the highest quality services are dynamic in nature in terms of evolving in accordance with the availability of new information [from the science], practitioners need professional development opportunities to maintain a current knowledge base” (p. 212). Like in any other vital field, relevant empirical knowledge continues to accumulate, leading to improvements in existing assessment and intervention methods, and the development of novel approaches. One example is the number of research studies focusing on developing safe and practical approaches to the functional analysis of problem behavior (e.g., Bloom et al., 2011. Hanley et al., 2014; Jessel et al., 2018; Jessel et al., 2019; Smith & Churchill, 2002; Thomason-Sassi et al., 2011). The same is true for many other areas of practice in which a lot of research relevant to practice has been published in recent years, such as verbal behavior interventions (Petursdottir, 2018; Petursdottir & Devine, 2017) or teaching prevention skills (Gunby et al., 2010; Johnson et al., 2005; Petit-Frere & Miltenberger, 2021). In these and other areas, lack of contact with the literature might result in less efficient and effective practices.

Research suggests that client outcomes are directly related to the clinical competence of the professional who provides the services (Parsons & Reid, 2011). To stay competent, it is important to stay updated on recent research developments related to one’s areas of practice. Therefore, the previous Professional and Ethical Compliance Code for Behavior Analysts from the BACB (2014) specified that “[b]ehavior analysts maintain knowledge of current scientific and professional information in their areas of practice and undertake ongoing efforts to maintain competence in the skills they use by reading the appropriate literature [emphasis added]...” (p. 4). The latest version of the BACB’s Ethics Code for Behavior Analysts (2020) states that, “[b]ehavior analysts actively engage in professional development activities to maintain and further their professional competence. Professional development activities include reading relevant literature [emphasis added]...” (p. 9). The new ethics code also highlights several areas where staying current with research developments is expected of certified professionals. For example, the code outlines four foundational principles, the last of which suggests behavior analysts should ensure their competence by “[r]emaining current and increasing their knowledge of best practices and advances in ABA. ..” (BACB, 2020, p. 4). In addition, when making ethical decisions, the code dictates that behavior analysts should consult available resources, including relevant research (BACB, 2020, p. 5).

Several publications have included discussions about the importance of accessing the literature and offer advice on how to do so (Briggs & Mitteer, 2021; Carr & Briggs, 2010; Dubuque, 2011; Gillis & Carr, 2014; Mattson, 2017; Parsons & Reid, 2011). In addition, some professional organizations provide access to relevant literature. At the time of the current study, the Association for Behavior Analysis International (ABAI) provided full access to Perspectives on Behavior Science to student, affiliate, and full members. ABAI also provides Behavior Science News updates on Twitter. The BACB provides full access to Journal of Applied Behavior Analysis, Journal of the Experimental Analysis of Behavior, Behavioral Interventions and ProQuest’s Education Collection for certified behavior analysts. The Association for Professional Behavior Analysts (APBA) also provides discount subscriptions to Child & Family Behavior Therapy, Journal of Organizational Behavior Management, and Journal of Mental Health Research in Intellectual Disabilities, as well as the research updating service Current Contents in ABA (full disclosure: the first author of the current article is affiliated with Current Contents in ABA). However, it is not well-known to what extent professional behavior analysts use these or other services to access and stay current with the literature.

In a recent study, Bains (2020) surveyed 19 behavior analysts at the Faison Center about their information seeking habits. When asked to select from a list of resources used when seeking professional information (they could select all applicable options), all of the participants (100%) indicated they sought out other people as sources of information. The second most used source for information was academic journals (selected by 95% of participants), followed by professional experience/personal knowledge (90%). The least used resource reported by this group was academic databases, selected by only 52% of respondents. Ninety-four percent of these respondents reported searching specific journals, rather than an academic database (e.g., ERIC or PsychINFO) reportedly only used by 68% of respondents. Only 26% of these behavior analysts used the research resources accessible through the BACB website when looking for information. Although satisfaction with individual sources of information was generally high (e.g., 89% were mostly or completely satisfied with academic journals as a source of information), the study did not evaluate satisfaction with overall access to information. Further, the author did not evaluate frequency of searches nor the extent to which the participants conducted successful literature searches. This study did not find a correlation between years of experience and source of information used.

Comparisons with related human services fields may be illustrative in this context. Fulcher-Rood et al. (2020) recently conducted an investigation with school-based speech-language pathologists regarding the use of evidence-based practice in their clinical work. Although the study was mostly qualitative, some quantitative findings were reported. The 20 participants reported reading on average two research-based articles per month, with a range of one to seven. The most commonly used resource for literature searches was a web search (56%), followed by keyword searches on the website of the American Speech-Language Hearing Association (52%). However, Fulcher-Rood et al. reported no specific data on the frequency of web searches. The qualitative analysis indicated that the participants valued research findings as they relate to practice, but many reported a lack of time to find, read, and think about the evidence.

Given the paucity of research in this important area, we conducted a study to evaluate the literature searches of practicing behavior analysts. The current study was informed by pilot survey data collected by the first and second authors that suggested that only 38% of BCBAs were satisfied with the research resources available to them, and access to a university library seemed to positively influence both satisfaction with available resources and frequency of contact with the literature. In follow-up interviews, a small number of BCBAs reported that time and paywalls (i.e., lack of access) were barriers to effective literature search, but lack of skill with using search terms and advanced search options were also cited as reasons for unsatisfactory search results. These preliminary results led to the current survey, which was designed to answer the following questions: (1) What are the contemporary literature search practices of practicing behavior analysts specific to conducting research reviews? (2) Do literature search practices and skills vary by level of education and years of experience? (3) Do literature search practices and skills vary by access to a university library?

Method

Participants and Procedures

The population of interest for this survey was practicing applied behavior analysts, with a specific focus on their approach to seeking out and finding research to inform their service delivery. A combination of voluntary sampling (i.e., participants received the invitation and then self-selected to participate) and snowball sampling (i.e., use of participants to recruit additional participants meeting the study criteria) was used to recruit research participants for this cross-sectional study (Remler & Van Ryzin, 2011). The research team initially recruited survey participants by contacting BACB registrants using the BACB Mass Email Campaign, with the recruitment email being distributed in October 2020. Due to lack of participation, the research team extended recruitment efforts to include social media in November 2020. Survey invitations were posted under the first author’s LinkedIn profile, Twitter account, and her company’s Facebook page. Additional survey invitations were shared via 18 ABA-focused Facebook groups (e.g., ABA Skill Share, BCBA Share, APBA’s Facebook page, ABA Marketplace) and the Teaching Behavior Analysis (TBA) listserv. Given the type of recruitment and the inability to calculate the number of potential participants who received some type of invitation to participate in the study, a response rate was not calculated. A total of 202 surveys had been received by June 2021; 180 of those surveys were complete, resulting in a completion rate of 89.1%. A survey was considered complete if the participant viewed all the questions and reached the end of the survey, even if they failed to answer some questions. Given the high completion rate, the 12 partially completed surveys were not included in data analysis, because these surveys were missing data on at least 25% of the questions and the participants did not reach the end of the survey.

Instrumentation

The survey presented 24–30 questions to participants. Some questions were follow-up questions and were only presented contingent upon particular answers to preceding questions (e.g., when presented with the question, “Are you satisfied with the research resources available to you,” the responses “no” or “somewhat” led to a follow-up question seeking clarification on the dissatisfaction expressed by the participant). Therefore, a participant may not have been presented with all 30 questions in total. Nine of the questions were demographic in nature, and the remaining questions focused on literature search procedures used by participants and confidence ratings about conducting research. A measure of reliability was calculated for those items specific to attitudes and beliefs as they were rated using a Likert-style scale, indicating a Cronbach alpha score of 0.81 (considered to be a “good” measure of reliability; Ercan et al., 2007). All statistical analyses were conducted using SPSS 26 for Mac (IBM Corp., 2019).

Demographics

Of the 180 participants who completed the survey, 165 (91.7%) indicated they are presently practicing as an applied behavior analyst. The largest group of participants had been practicing for more than 11 years (n = 89, 49.5%). Ninety-seven participants (53.9%) reported their highest level of education to be at the master’s level. One hundred eight (60.4%) reported their certification status as that of a board certified behavior analyst. A majority of participants reported their highest degree to be directly related to behavioral science (n = 98, 54.4%). Forty-four states were represented, with Texas (n = 24, 13.3%), Florida (n = 17, 9.4%), Massachusetts (n = 12, 6.7%), Arizona (n = 11, 5.6%), and Maryland (n = 11, 5.6%) representing the top five. Twenty-one (11.6%) participants indicated they were practicing outside of the United States. See Table 1 for additional information.

Table 1 Demographics
Full size table

Results

Contemporary Literature Search Practices

Participants were asked to respond to a variety of questions designed to evaluate how they search for research literature (see Table 2). Questions regarding frequency in conducting literature searches, familiarity with available resources, and reasons for conducting literature searches were presented as part of the survey. Questions were designed to allow participants to select all options that applied when seeking clarification regarding specific practices. General interest in a specific intervention or practice (n = 118, 65.6%) was identified by the greatest number of participants as a reason for conducting a literature review. Participants were asked to report the frequency with which they conducted searches (1 = less than once a month, 2 = one to three times per month, 3 = one to six times per week, 4 = once a day, and 5 = never). Prior to analysis, this variable was recoded so a response of 5 = never was weighted as 0. This recoding of the “never” responses from a 5 to a 0 helped ensure accurate reflection of the frequency of searches conducted to avoid a positive skew (i.e., if left as a 5, the response would affect the mean frequency with which searches were conducted in a “positive” manner, artificially increasing the mean frequency of which searches were conducted). The combined number of participants that reported searching once per week or more often (item selection 3 or 4) was 87 (48.3%). The primary print-based sources that the participants used to find research included textbooks (n = 105, 65.6%), printed research articles (n = 68, 37.7%), and academic journals (n = 67, 37.2%). Several on-line resources were recognized by participants as useful in accessing research. The top three online resources included an academic web search (e.g., Google Scholar, Microsoft Academic; n = 131, 72.7%), research available through a university library subscription (n = 118, 65.6%), and research available through the BACB (e.g., JABA, JEAB; n = 118, 65.6%). See Table 2 for further details. For the sake of brevity, not all responses were included in Table 2; the full results are available by request from the lead author. When asked about their familiarity with the Carr and Briggs (2010) article that outlined strategies for staying in touch with the literature, 22.2% (n = 40) of the participants reported integrating the outlined strategies into their literature searches; 42.2% (n = 76) of participants were not aware of the article. When asked about using available literature resources, more than half of the participants (n = 118, 65.6%) reported using the research resource provided by the BACB. Thirty-five participants (19.4%) were aware of the resource but had not used it (these participants may have had access to other resources, such as a university library). However, 10% (n = 18) of participants indicated they were not aware that the BACB provided access to behavior analytic research.

Table 2 Contemporary Research Practices
Full size table

When it came to satisfaction with research resources, 47.2% (n = 85) of participants indicated they were satisfied with the resources available to them, whereas 36.7% (n = 66) were somewhat satisfied and 7.8% (n = 14) were unsatisfied. Likewise, about half of the participants strongly agreed that they were able to find what they were looking for through searches of the literature (i.e., n = 113, or 62.8% of participants rated this item as an “8” or higher on a scale of 1–10). Some participants reported barriers to satisfaction: 41.7% (n = 75) indicated that a subscription or payment interfered with accessing resources, whereas 28.9% (n = 52), reported that they were not always able to find what they wanted.

When asked about how they learned to conduct literature searches, 46.5% (n = 89) of participants indicated they learned this in graduate school. Another 21.7% (n = 39) learned through trial and error or were self-taught, but only 3.9% (seven participants) reported learning literature search strategies in supervision. A little over 85% (n = 143, 86.7%) of participants used advanced search terms when conducting a literature search. Forty seven percent (n = 85, 47.2%) of participants were satisfied with the results of their search terms used. Most of the respondents (n = 161, 89.4%) skipped the question, “In the context of teaching or supervision, have you ever instructed others on conducting literature searches?”

Analysis of Group Comparisons

A series of independent samples t-test and analysis of variance (ANOVA) were conducted to determine if group differences existed based on the following dependent variables: participant perception of satisfaction with research results and satisfaction with available research resources, skill in finding research, ability to conduct a meaningful review of research, belief the research will inform their practice, confidence in finding the answer/solution sought, and frequency of conducting research reviews. Level of education and the use of university library resources were used as independent variables in the independent samples t tests (two groups), whereas an ANOVA was used to examine group differences based on years of experience (four groups). These four groups reflected a relatively equal distribution of participants and allowed for an appropriate analysis of group differences.

Level of Education

Participants were asked to identify their level of education. No participants reported a bachelor’s degree as the highest degree held, and four participants failed to provide a response and were not included in the group comparison. As such, the level of education reflected a dichotomous split, with participants reporting either a master’s degree (n = 97, 53.9%) or doctorate (n = 79, 43.9%). Because there were only two groups, an independent sample t-test was considered the most appropriate analysis (Lomax & Hahs-Vaughn, 2012). A follow up Welch t’ test was conducted specific to one dependent variable (the belief that research would inform practice) as equal variance could not be assumed and there was an unequal number of participants in each group (Lomax & Hahs-Vaughn, 2012). The n for the independent samples t-test varied slightly, because three participants with a master’s degree and eight with a doctorate degree did not provide responses to the analyzed questions. Results indicated that participants reporting their education level as a doctorate reported higher rates of satisfaction and confidence across all the tested domains and conducted reviews of the research on a more frequent basis (Table 3).

Table 3 Group Differences Based on Level of Education
Full size table

Use of University Library Resources for Online Searches

Participants were asked, “What online sources do you use to access research?” One hundred eighteen (65.6%) participants indicated they used a university library as part of their research review (Table 2). Given that most professional behavior analysts have graduated from a university and no longer have access to a university library, follow-up analyses were conducted to consider these group differences.

Similar to the analysis conducted on level of education, an independent-samples t-test comparing those using a university library resource against those who did not was completed (Table 4). Given the unequal distribution of participants, a Welch t’ test was again conducted (Lomax & Hahs-Vaughn, 2012). Findings indicate those using a university library resource conducted literature searches more frequently, reported a higher skill level in conducting searches, reported more confidence in their ability to conduct a meaningful literature search, reported more confidence that the identified research would inform their practice and lead to a desired outcome, and reported more satisfaction with the research resources available to them. Figure 1 illustrates the frequency of literature searches among these two groups.

Table 4 Group Differences: Use of University-Based Resources
Full size table
Fig. 1
figure 1

Frequency of Literature Searches and Use of University Library Resources

Full size image

Years of Experience

Groups were broken down into one of four categories based on years of experience: (a) new professionals (0–5 years of experience, n = 30), (b) mid-career professionals (6–10, n = 46), (c) seasoned professionals (11–20, n = 56); and (d) late career (21 or more, n = 33). This grouping allowed for a comparison of group means using ANOVA to analyze differences between groups across the following variables: (a) frequency in conducting searches for research literature, (b) satisfaction with results obtained through the search, (c) skill in conducting literature searches, (d) confidence in conducting the literature search, (e) perceived impact on informing personal practice, (f) confidence in finding needed answers, and (g) satisfaction with available research resources (see Table 5).

Table 5 Group Differences Based on Years of Experience: ANOVA
Full size table

The results of the ANOVA indicated statistically significant results in three areas: (a) perceived skill in conducting online literature searches, (b) the belief that conducting a literature search would help inform their practice, and (c) perceived ability to find the desired answer/solution (see Table 5). Because there were four groups involved in the comparison, a post-hoc statistical analysis applying Tukey’s honestly significant difference (HSD) was used to control for a Type I error (Nanda et al., 2021; see Table 6). New professionals (0–5 years of experience) rated themselves as less confident in their research skills when compared to both mid-career (6–11 years) and late-career professionals (21 or more years). New professionals were also less confident that their review of the research would inform their practice and less confident in their ability to find the desired answer/solution when compared against late-career professionals. No other statistically significant results were noted.

Table 6 Group Differences Based on Years of Experience: Results of Post-Hoc Tukey’s HSD
Full size table

Discussion

We implemented an online survey to evaluate the literature searches of board certified behavior analysts, with 180 participants providing responses. Approximately half of the participants reported searching the literature at least once per week, whereas only 8.9% reported searching once per month or less often. Only two participants (1.1%) indicated they never conduct literature searches. The most common conditions prompting a literature search were due to a general interest in a specific intervention or practice, and word of mouth (i.e., hearing about research from others). Supervision of BCBA trainees was also a common reason for conducting a literature search, followed closely by making efforts to improve existing behavior change plans, staff training, teaching, and learning about new or unfamiliar target behaviors or contexts. The most commonly used online resources were academic web search (Google Scholar or Microsoft Academic), university library, the BACB website, PubMed Central/NIH, and ResearchGate. The most common paper-based resource was textbooks, followed by printed research articles and academic journals.

These results differ somewhat from the results of Bains (2020), who found that the participants were more likely to search specific journals than academic databases. These differences are likely explained by the participant pool and setting of the Bains study, which included a relatively small number of BCBAs who all worked for the same organization that may have provided access to specific journals. Similar to the results of Fulcher-Rood et al. (2020) for speech-language pathologists, a web search was the most commonly used search strategy, although a higher proportion of behavior analysts reported using that strategy (72.7% vs. 56%).

Overall, these results indicate that most practicing behavior analysts conduct regular literature searches to inform their practice. However, although many governing bodies in behavior analysis suggest reading the literature to stay competent, there is currently no way of knowing how much contact with the literature is sufficient. More research should be conducted to determine if there is a dependent relation between frequency of searching the literature (or reading the literature) and client outcomes.

It is interesting that less than half of the respondents in the current study (45%) were satisfied with the research resources available to them, whereas 33.7% were somewhat satisfied. A substantial proportion (37.6%) of participants reported that subscription or payment requirements functioned as barriers to successful literature searches, whereas a smaller proportion (26.2%) reported that they were not always able to find what they wanted. Fulcher-Rood et al. (2020) reported similar concerns for the field of speech-language pathology, adding the general concern of a lack of time to search for and read research.

Level of education, years of experience, and access to university libraries were all found to affect various aspects of satisfaction with literature searches and confidence in conducting the searches. In particular, a higher level of education (i.e., doctorate degree) predicted higher satisfaction with search results and available research resources, higher confidence in their skill in reviewing and finding research, stronger belief that reviewing research informs practice and leads to solutions to clinical problems, and more frequent literature searches compared to those with a master’s degree. More experience in the field was associated with greater confidence in one’s skills in conducting online searches and ability to find solutions through the searches, as well as a stronger belief that research reviews inform these solutions. These findings are consistent with the general notion that searching, interpreting, and utilizing the literature to improve the quality of services are acquired skills that are shaped over the course of a career. However, more could be done to improve the skills of early-career behavior analysts to identify and utilize the relevant scientific literature, both during graduate classes and supervised fieldwork. This could include instruction and training on how to use deliberate strategies to stay in touch with the literature (Briggs & Mitteer, 2021; Carr & Briggs, 2010), education on the resources that are available to practicing behavior analysts (e.g., through the BACB website), and the use of efficient search strategies. It has been suggested that behavior analytic supervision should include, “modeling professional development behavior by consuming the published literature, identifying relevant articles, and analyzing those articles with the supervisee” (Sellers et al., 2016, p. 282). More research should be conducted to determine the extent to which this is practiced among supervising professionals.

Access to a university library emerged as an important variable that was associated with most measures of satisfaction and confidence. This is an important consideration, because most professionals lose access to university-based resources (most significant, licensed research databases) upon graduation. Those participants that reported using a university library were likely to report higher search frequency, higher satisfaction with obtained results and higher satisfaction with available research resources than those who do not use a university library. In addition, participants with access to a university library were likely to rate themselves as more skilled in conducting research, better able to find the answer or solution they were looking for, and more likely to report a belief that reviewing the research informs their practice. It is likely that this group of participants are employed in a role that requires frequent literature searches (e.g., professor, lecturer, researcher). Further this group is more likely to be actively involved in generating, reviewing, and disseminating research developments in the field. In addition, using a university library likely provides more true-positive search results (i.e., identification of relevant articles plus access to a full text version) than a search-engine literature search. All of these factors could lead this university-library group to rate themselves higher on the literature search questions of the survey. For the group of professionals that do not have access to a university library, strategic training on which research resources are available to them, including the resources available through the BACB may increase satisfaction ratings. In addition, strategic training in conducting literature searches within those resources may address the confidence ratings. One strategy might be the use of an access guide so professionals can stay informed of the various resources available to them. Researchers could also investigate the effects of training professionals to conduct literature searches, including information related to useful strategies and tactics for effective and efficient database use, key term use, and search limits. Research could also focus on the effects of a literature briefing service that decreases the amount of time it takes to read or look for relevant research. Perhaps clinics and organizations could train a few clinicians (e.g., “literature search committee”) to conduct literature reviews and identify the appropriate literature on behalf of the other clinicians. Leaders in evidence-based medicine acknowledge that clinicians are not likely to have more than 30 min a week to find and evaluate research (Sackett et al., 2000). It is important to ask how to make the best use of this limited time.

Some limitations to the current study should be noted. At the time of this study there were 44,025 BCBAs registered by the BACB. An acceptable response rate for this survey ranges from 10% to 50%, given the nonprobability methods of recruitment (Dillman et al., 2014). This group of 180 participants falls short of this ideal, which would be at least 4,400 BCBAs. However, the current sample size is similar to prior survey-based research with behavior analysts (Brand et al., 2020; Colombo et al., 2021; Frieder et al., 2018; Sellers et al., 2019). This survey was distributed during the COVID-19 pandemic, during which many clinics and ABA practices were busy restructuring their clinical practice to follow the guidance of health officials (see Behavior Analysis in Practice, 13(2)). BCBAs may not have had the time or the energy to respond to an online survey. In addition, many researchers shifted to online platforms during the pandemic, leading to a preponderance of survey-type research. This may have ultimately led to survey fatigue (De Koning et al., 2021). The conclusions of the current study could be strengthened through further replications and follow-up studies.

In addition, this project involved involuntary snowball sampling, a nonprobability sampling technique in which participants identify and recruit additional participants for the study. When our study was shared via social media and email, potential participants shared the survey link with their social network. Although this was a serendipitous means of spreading the word, it could potentially bias the sample size if the survey was only shared with one subgroup of the population to be studied.

This survey did not gather demographic or race information from the participants. In an effort to continue to rehabilitate our field’s cultural blindness (Ala’i, 2019), it is important to further investigate if factors such as socioeconomics, race, or ethnicity affect access to behavior analytic research. However, we did collect data on years of experience in the field, and analyses of these data suggested that the participants in this study may have had more experience, on average, in the field than the general population of BCBAs. Only 16% of participants in this study had 1 to 5 years of experience in the field, whereas available data from the BACB indicate that 50% of currently certified professionals have fewer than 5 years of experience (Sellers, 2020).

As previously reported, none of the survey questions were required. In other words, participants could proceed through the survey leaving some (or many) questions unanswered. This inconsistent denominator in our analyses resulted in more difficult calculations and some analyses that were hard to compare. For example, there were 25 participants that did not answer if they used advanced search options and 28 participants that did not answer if they were satisfied with the research resources available to them. The skill ratings questions had similar levels of missing data (28–29 missing answers).

This study represents a step toward understanding how professional behavior analysts search for and access the scholarly literature. It can be argued that this is a crucial issue, because relating applied practice back to empirically validated behavioral principles and valid and reliable research findings, “... can have the effect of making a body of technology into a discipline rather than a collection of tricks” (Baer et al., 1968). It is important to note that the current results suggest that although most professional behavior analysts make regular meaningful contact with the literature, the frequency and quality of this contact could be improved. This state of affairs seems to be partially due to lack of resources, but also due to lack of knowledge of existing resources, as well as skill deficits in conducting effective and efficient literature searches. Making research available to the general public is in line with various open-access movements that are just starting to emerge in behavior analysis (see Gilroy & Kaplan, 2019; Howard, 2019; and openbehavioralscience.org). These findings serve as a reminder to all relevant professional and scientific organizations in behavior analysis to continue efforts to ensure that professionals are able to identify and access scientific literature that is needed for high-quality service provision.