Introduction

Home visiting programs provide direct services and support to families in their homes and coordinate referrals to other community-based supports (California Evidence-Based Clearinghouse for Child Welfare, 2020). Prior research has concluded that evidence-based home visiting programs have positive impacts on the prevention of a variety of child maltreatment (abuse, neglect, or exploitation) and developmental outcomes (Duggan et al., 2004; LeCroy & Krysik, 2011; LeCroy & Lopez, 2018; Kirkland et al., 2020). Improving the reach of these types of supports and services is a high priority in rural areas such as the Ozark region of Missouri (Missouri Department of Health and Senior Services, 2020). Compared to the state rate of 42 child maltreatment incidents per 10,000 children, Ozark counties have higher rates, ranging from 45 to 62 per 10,000 children (Fostering Court Improvement, 2020). In addition, families in this region are burdened by socioeconomic hardship and high rates of preventable health outcomes, including high infant mortality (Missouri Department of Health & Senior Services, 2020). Child poverty ranges from 30 to 37% in the Ozark counties, compared to 18% in the state of Missouri (County Health Rankings & Roadmaps, 2020). Additionally, all counties in the region have higher teen birth rates (ranging from 41 to 51 per 1,000 female population aged 15–19) compared to the state rate of 26 per 1,000 (County Health Rankings & Roadmaps, 2020).

These poor maternal and child health indicators in the Ozark counties are exacerbated by challenges in obtaining access to health care and social services due to both health professional shortages and a high proportion of the population who experience cost and transportation barriers (Missouri Department of Health and Senior Services, 2016). It is in this context that Whole Kids Outreach (WKO) serves rural Missouri families through the implementation of programs addressing child well-being and reduction in infant mortality and child maltreatment, including Healthy Families America (HFA).

Healthy Families America (HFA) is a home visiting program that aims to promote positive parenting, enhance child health and development, and prevent child abuse and neglect (Healthy Families America, 2021). Working along with home visitors, also called family support workers or Outreach Specialists (OS), HFA works with families to reduce the risk of adverse childhood outcomes, with voluntary services beginning prenatally or during the post-partum period and continuing for up to 3 years, sometimes longer. These home visitors promote positive and responsive parenting, provide parenting education and emotional support, and connect families to tangible resources like medical insurance, preventive health care, and housing assistance (Daro & Harding, 1999; Kirkland et al., 2020; Green et al., 2018). Evaluations have documented that HFA is effective in reducing the risk of child maltreatment and adverse childhood outcomes (Ingemann et al., 2023). While fidelity to HFA is a guiding principle, previous WKO evaluation findings suggested some HFA requirements were not responsive to the unique cultural and geographic characteristics of it’s rural communities, thereby limiting the program’s effectiveness (Biggs & Garstka, 2017). Little is known from prior evaluation research about factors related to HFA program success and challenges specific to rural communities, nor of how community, organizational, and individual factors impact program implementation, and in turn, determine program success in the prevention of child maltreatment in rural settings (Duggan et al., 2004; DuMont et al., 2010).

Implementation science offers frameworks that can be used to guide the identification of strategies to enhance program implementation of evidence-based programs such as HFA and improve outcomes such as child maltreatment in the rural Ozark region of Missouri. The Consolidated Framework for Implementation Research (CFIR) offers an overarching typology to promote theory development and verification of key implementation factors across multiple contexts (Damschroder et al., 2009). It also helps in the identification of implementation factors and optimizes implementation strategies from the local to international context (Meshkovska et al., 2023; Chitiyo et al., 2023). As described by Keith et al. (2017), it is also helpful to systematically identify factors that may emerge at multiple levels to influence the implementation of HFA and produce actionable findings for organizations to make improvements to implementation while maintaining fidelity. There are numerous constructs nested within the five major CFIR domains to guide exploration of multi-level phenomena under study: (1) intervention characteristics, 2) inner setting, 3) outer setting, 4) individual characteristics, and 5) implementation process) (Damschroder et al., 2009).

While CFIR offers a framework for capturing important constructs, to develop strategies that are aligned with both community needs and organizational mission, it is also crucial to engage stakeholders, integrating their perspectives throughout all phases of assessment and strategy development. The concept mapping approach provides a structure to ensure participation from community members as well as WKO staff throughout the research process. Concept mapping is a structured conceptualization process and a participatory qualitative research method that yields a conceptual framework for how a group views a particular topic or aspect of a topic (Burke et al., 2005). It offers an integrated approach for participant engagement in research, via a multi-stage, structured, mixed-method process that employs both qualitative and quantitative components to gather ideas from participants, organize concepts, and, with emphasis on visual representation, facilitate discussion of results for group prioritization and planning (Caracelli & Green, 1993; Kane & Trochim, 2007).

The purpose of this study, was to apply a community collaborative approach, using concept mapping informed by the CFIR framework, to examine ways to amplify the impact of the HFA program in preventing child maltreatment in the rural, high-need regions in Missouri by identifying, understanding, and prioritizing factors affecting the implementation of HFA. This paper also highlights factors that may be modified to improve the adaptation of HFA in rural communities through subsequent strategy development to enhance HFA program impacts and subsequently improve the prevention of child maltreatment.

Methods

Using an observational implementation research design, we employed a concept mapping approach, a mixed-method research methodology to identify implementation factors and engage in a collaborative prioritization process to develop strategies to improve the implementation of HFA in rural communities (Hwang et al., 2020). This collaborative approach is applied to the collection, analysis, and interpretation of data in a phased sequence: brainstorming (generation of ideas), structuring (sorting and rating of ideas), cluster analysis, interpretation and collaborative model building, and prioritization of clusters and development of strategies to address these priorities (Kane & Trochim, 2007). Demographic information, including age, gender, ethnicity, education level, associated organization and position, and duration of work in that organization was collected from all participants. This study was conducted in accordance with prevailing ethical principles and before participant recruitment and data collection, the study phases were approved by the Saint Louis University Institutional Review Board (IRB#29,981).

Brainstorming: Generation of a List of Key Issues from Stakeholders

The purpose of the interview guide was to elicit open-ended responses with respect to consideration of facilitators and barriers to HFA program implementation both within and outside the WKO organizational environment. The CFIR-based domains of the semi-structured interview guide addressed characteristics of the HFA program, staff characteristics (perceptions of the program and interactions), inner organizational-level characteristics (communications, procedures, culture/climate), and outer organization characteristics (linkages with other agencies/organizations, community perceptions). For interviews conducted with external partners, prompts were modified accordingly (e.g., excluding questions addressing internal WKO characteristics, adding prompts for interactions between partner organizations and WKO). External partners included representatives from area elementary schools, chambers of commerce, community-based organizations, clinical providers, and county health departments.

A total of 22 individual in-person and phone interviews were conducted with stakeholders including WKO staff/ OS (n = 8) and external partner agency representatives, including members of the board of directors (n = 14). From the interview transcripts, 704 independent statements were abstracted by three researchers. These same researchers then synthesized this list into 84 unique statements after removing duplicates and overlapping statements.

Structuring (Sorting and Rating Ideas)

The list of 84 unique statements was then entered into the online concept mapping system (The Concept System® Global Max©, 2021.224.12, Concept Systems Inc, https://conceptsystemsglobal.com/index.php). Participants (10 external partners and 10 WKO staff including OS) were asked to sort statements into like groups according to their conceptual similarities and to rate the importance of each statement on a Likert scale ranging from 0 (not important) to 10 (extremely important) (see Fig. 1).

Data Analysis and Representation of Clustered Ideas

Sorting data was used to produce visual concept maps that demonstrated relationships among the statements. This step involved a sequential process of similarity matrix creation, multidimensional scaling, hierarchical cluster analysis, and cluster solution map formation (Streeter et al., 2011). The similarity matrix identified the total number of times a statement was grouped with another statement. This similarity matrix was first used to create a point map through multidimensional scaling analysis where each statement was given an (X, Y) coordinate. Using these coordinates, hierarchical cluster analysis was utilized to produce defined geometric clusters that represented major aggregated concepts.

Fig. 1
figure 1

Concept mapping process

Full size image

Interpretation and Collaborative Model Building

To determine a final cluster solution for the prioritization phase of the project, clusters were analyzed starting with a 12-cluster solution and working downwards as clusters merged. The statements in each cluster were examined to ensure that clusters that were merged contained conceptually similar concepts. The final six-cluster solution was chosen by the consensus of the authors to balance the adequate description of key issues and parsimony. The final stress value of 0.2 after 14 iterations represented the goodness of fit of the configuration. This value measures “the degree to which the distances on the map are discrepant from the values in the input similarity matrix”, with a low-stress value indicating a better overall fit of the map to the data (Kane & Trochim, 2007).

To ensure a collaborative model-building approach, this final six-cluster map was then presented to the WKO staff and external partners at two separate community forums, including those who were involved in previous phases of the project. During these in-person group meetings, participants (Group 1 n = 8; Group 2 n = 9) discussed the cluster content statements, the overall meaning of each cluster in relation to others, and the final concept map to ensure it addressed the main issues raised. These meetings also offered participants the opportunity to discuss the extent to which the six-cluster solution reflected and/or differed from their perspectives of key issues to consider in implementing HFA.

Utilization of the Findings to Prioritize and Inform Practice

During the two community forums, the nominal group technique was used to prioritize which clusters could serve as leverage points to improve the implementation. Participants were first asked to prioritize clusters based on the importance of the issue, the potential to create change, support for changing the issue, the influence of the issue on program implementation, and health outcomes.

Each participant was then asked to pick the top three clusters to focus on for strategy development. Individual preferences were collected anonymously and then tallied and weighted according to the ranked order of preference. After prioritizing, participants were asked to discuss and brainstorm additional details regarding strategies for addressing those prioritized issues. These strategies were then discussed using the CFIR constructs to provide a framework for referencing the specific domains and constructs involved. Because there was an overlap in the chosen priorities from the two separate community forums, the research team combined the strategies generated by both groups and worked to identify action plans accordingly. Suggested strategies were then reviewed by the research team and action steps for each strategy were taken to move forward on the prioritized areas.

Results

Characteristics of Participants

The majority (> 90%) of participants during the brainstorming phase were white and female. The duration of work at their respective organizations ranged from 6 to 23 years with an average duration of 14.5 years. The average time spent by these participants in their respective communities was 38 years with a range from 11 to 61 years. Of the 20 individuals who participated in the sorting and rating phase of the study, 25% were on the WKO Board of Directors. A subset of participants from prior phases, both internal staff and external partners, participated in the two prioritization meetings (n = 8 and n = 9, respectively).

Results from Concept Mapping

The final six-cluster solution, which represents groups of individual statements to promote HFA implementation and prevent child maltreatemt in the region, is depicted in Fig. 2 and described in Table 1, with clusters listed in order from higher to lower mean importance rating (range: 7.6–6.8). Cluster average bridging values (range: 0.15–0.44) are also reported, reflecting the likelihood that participants sorted statements similarly (lower value indicates greater similarity) (Streeter et al., 2011). A summary of each cluster with representative sample statements is described below:

Fig. 2
figure 2

Concept Map Representing Clusters of Factors Related to HFA Implementation, Six-Cluster Solution. Note This is a computer generated map (on a XY plane) based on the sorting and rating of multiple statements by individuals participating in the structuring phase of the concept mapping process. Each shape in the map represents a cluster of statements and each point represents individual statements. Overlapping of the shapes indicate the overlapping of ideas.

Full size image
Table 1 HFA implementation clusters (Six cluster solution), with importance ratings and bridging values
Full size table

Challenges Meeting Funder Expectations

This cluster included 7 statements related to the HFA program and standards. These standards ensure proper implementation of the HFA program to prevent child maltreatment. Some of the statements included in the cluster are: “one of the HFA standards is to have advisory and board meetings, but we (WKO) do not have enough people to be on all of those boards”, “WKO does not have the money to hire the OS needed to meet the HFA standards”, and “grant providers need to be realistic about the number of families that can be seen with the money they provide”.

Challenges Families Face Related to Home Visits

This cluster included 17 statements that represented families’ fears of having children taken away, not having tangible resources to support families, and preference for not having visitors in the home. This might create barriers for families to access support and services provided by home visitors to prevent child maltreatment. Statements in this cluster included “a lot of people just do not like intrusion and that is how they look at home visiting program”, “in some instances, there may be drugs in the home, and they fear we (OS) will call law enforcement or child services”, and “in some counties, WKO has a stigma of going into the home and having your kids taken away”.

Challenges Outreach Specialists Face Related to Home Visits

This cluster contained 17 statements depicting challenges that OS face when conducting home visits. These challenges might hinder the ability of OS to promote healthy parenting and prevent child maltreatment. This cluster includes statements like “OS caseload/ their daily quota of who they need to visit is too hard for the distance in which they travel to meet with families”, “the paperwork is taking (away from) our time to educate our moms”, and “I think everybody gives so freely of themselves that maybe sometimes we (OS) give so much, we don’t have anything left to pull for ourselves and our families”.

HFA Program, Assessment, and Curriculum-Related Issues

This cluster contained 13 statements that represented issues related to the HFA program itself and its curriculum that have been successfully tried and tested in other settings against child maltreatment (Healthy Families America, 2021). This cluster included statements like “HFA does not adapt well to our OS having to travel far to get their clients”, “HFA requirements for paperwork can get in the way of bonding with new families that are learning to trust us”, and “HFA (curriculum) materials are not written at the right literacy level for our families”.

Increasing WKO Exposure in the Community

There were 13 statements in this cluster, describing the need for WKO and its home visiting child welfare program to be more well known in the community. Representative statements included “community events like back-to-school fair need to be used more to get the words out about WKO and HFA”, “WKO should use social media more to advertise what they do”, and “I would think that if WKO had someone locally that people knew and trusted, that might help build future collaborations”.

WKO Working with Other Agencies

There were 17 statements clustered in this group centered on coordination between WKO and other local agencies to work collaboratively to prevent child maltreatment in the region and included statements like “some competitiveness with other programs”, “agencies need to know more about WKO and HFA to make referrals”, and “it is difficult for referring agencies to find the right opportunity and timing with clients to make referrals to WKO”.

Cluster Prioritization and Strategy Development

Table 2 describes prioritized clusters, aligned with the corresponding CFIR domains and constructs, with accompanying strategies and action steps to prevent child maltreatment in the study area. The first priority focused on the HFA program, assessment, and curriculum. Strategies centered on adaptations of aspects of HFA implementation and processes within the WKO organization. These included modifying data systems and protocols to streamline paperwork. In addition, suggestions that were made to center some of the data that home visitors are required to collect on resilience rather than trauma-focused questions prompted discussions with the HFA national program office. Through these discussions, several strategies were identified for working with HFA to adapt existing processes while maintaining fidelity, for example, postponing assessments of history of family trauma until after some level of relationship has been formed with the family.

Table 2 Prioritized clusters with strategies mapped to CFIR domains and constructs
Full size table

A second priority area was related to the challenges outreach specialists and others at WKO face related to home visits. The strategies here focused on identifying virtual structures to address the emotional content that comes up with the work and/or gets activated by a situation. This is seen as supplementing the support and reflective supervision available within WKO.

The last priority was related to increasing WKO exposure in the community. As these strategies were being developed, it was recognized that this overlapped with the cluster “WKO working with other agencies.” This was seen as particularly important in addressing challenges families face related to accessing resources to support child welfare. As a result, the strategies focused on not just helping people to become more aware of WKO by enhancing WKO’s virtual presence (e.g., Facebook and other social media) and creating a community liaison position, but also on identifying resources that could be brought to WKO and the families they serve (e.g., access to free telephone plans and broadband and basic needs such as food and diapers) from within as well as outside the WKO service area.

Discussion

Implementation research allows us to better understand how best to ensure that an intervention that works in one location will also have the intended impacts across multiple populations and contexts. While this kind of research is critical to optimize the impact and reach of programs to prevent child maltreatment, conducting this research requires working with internal organizational staff and external partners as well as understanding community culture and context. By using the CFIR as a framework and a structured concept-mapping process with nominal group technique, we were able to not only identify areas for improving implementation of HFA in rural settings, but also generate actionable and feasible strategies to enhance the program’s effectiveness.

The findings from the project highlight ways that the HFA implementation factors, beyond simply corresponding with respective CFIR domains and construct, impact each other. For example, HFA program components (intervention characteristics), though well aligned with WKO goals (inner setting), were also intertwined with perceived burden and stressors on WKO staff (inner setting), and this was in turn exacerbated by the lack of external resources in the area to provide additional supports for the OS, thus highlighting links between inner and outer settings. In addition, the outer setting was represented by the clusters depicting the need to increase the WKO exposure in the community and its collaboration with other organizations in the area (clusters 5 & 6). Discussions among participants recognized the interconnectedness between outer setting factors and the inner setting of the WKO and crafted strategy recommendations for WKO that were also appropriate for the external context. For example, issues of burden and stressors on staff were addressed by turning to virtual resources for reflective supervision. This acknowledged resource constraints both internal and external to WKO. In another example, our project enhanced the relationship between WKO (internal) and the HFA national office (external), which allowed for direct communication between home visitors and those responsible for making policy and practice decisions that affect home visitors’ day-to-day experiences with the families. Through a series of meetings, the group was able to address concerns around data collection expectations, particularly with regard to the timing of the collection of data for the families served and the focus on trauma rather than resilience. It was evident through communications with the national office that fidelity of implementation was influenced by their capacity to communicate expectations in ways that the local staff could integrate into practice.

The use of structured and collaborative concept mapping approach to identify barriers and develop strategies facilitated an understanding of key HFA implementation factors and the development of subsequent strategies to enhance its effectiveness and prevent child maltreatment in rural settings. By eliciting discussion and nominal group technique during the community forums, we applied criteria for cluster prioritization that went beyond importance ratings to include a broader set of considerations including feasibility, the potential to create change, support for changing the issue, the influence of the issue on either implementation factors or program implementation, and impact on health outcomes. This also captures the tension between the application of quantitative results (e.g., importance ratings) and priority setting for practice, with practice requiring more nuanced discussions that allow for consideration of the complex interplay of a wide variety of factors that cannot be easily quantified. Likewise, it highlights the benefit of iterative, consensus-building approaches to move from recognizing gaps to leveraging change (Rankin et al., 2016).

Our study supports the need to find the right balance between fidelity and adaptation when implementing evidence-based interventions developed in one context and implementing them in another, including different settings and cultures (Ingemann et al., 2023). Similarly, our findings align with previous literature that emphasizes the importance of studying factors related to HFA program success across multiple levels. Latimore et al. (2017) found that implementation factors varied across the level (families, home visitors, and implementation sites). Among the strongest predictors of implementation success for HFA (“high service dose”) were indicators of home visitor anxiety. This is consistent with our finding that reflective supervision may be an important strategy to address sources of stress and anxiety among home visitors. Furthermore, we found that programs may be more effective in preventing child maltreatment when parallel processes are created where the positive relationship developed between the program home visitor and parent, in turn, serves as a model for relationship building between parents and their children (i.e., via reflective supervision) (Tomlin et al., 2016; O’Rourke, 2011; Bernstein & Edwards, 2012).

Our study highlights the benefits of conducting research on implementation factors, assessing how best to disseminate programs in ways that allow adaptation to new settings and maintain key elements of fidelity. This may be particularly important in disseminating programs from high-resource to lower-resource communities, and in translating improvement strategies across settings both locally and internationally (Theobald et al., 2018). In doing so, it is important to use structured collaborative methods that promote stakeholder participation throughout the research process, ensuring that adaptations are appropriate to both organizational and community settings.

Limitations

There were several limitations to our work. For example, our findings were based on interviews and community forums with a relatively small number of individuals. While these were key partners in implementing the work, there may have been some concerns or issues that might have been raised by individuals who were less integrally involved in the program. It is also possible that the implementation challenges we observed and focused on might not be those that are most important in other settings or in other time periods. This highlights the importance of conducting these kinds of assessments more regularly to better create continuous improvement in the program. The engagement of multiple researchers and multiple points of partner validation (review of clusters, prioritization, and strategy development) was seen as critical to ensure that the actions taken reflected the partner’s desires. However, it is evident that not all the challenges identified were addressed through our work. Moreover, while examining how best to maximize the implementation of programs, future long-term collaborative research is needed to examine the effectiveness of the strategies implemented on preventing child maltreatment.

It should also be noted that our work was conducted prior to the most recent review of the CFIR (Damschroder et al., 2022). In spite of that, our work certainly fits within the updated CFIR domains and construct definitions. For example, our findings support the new guidance that suggests that local conditions as well as performance measurement pressure influence implementation. Similarly, our findings highlight the importance of considering information technology infrastructures and work infrastructures that influence implementation.

Conclusion

In order to optimize and foster the sustainability of interventions that aim to improve child well-being and reduce child maltreatment, researchers must evaluate not only the impact of interventions on health outcomes but also identify, seek to understand, and prioritize modifications that can be made while also maintaining fidelity. Using community-engaged processes, the CFIR framework was able guide this work and to identify key leverage points for improving implementation of HFA in the rural communities of Missouri including the importance of reflective supervision for home visitors/community health workers, modification of data systems and protocols to meet participant capacity and system and structural constraints, and building and/or enhancing collaborative relationships across community organizations.