Background

Overweight and obesity established in childhood has been shown to track into adulthood [1] increasing the future population health burden of the most prevalent causes of adult mortality and morbidity [2, 3]. Inadequate physical activity and poor nutrition are key risk factors related to the development of overweight and obesity [4, 5]. The early childhood years have been recognised as a critical time in the development of dietary intake and physical activity behaviours [6, 7].

The World Health Organization recently identified centre-based childcare services (including organised group pre-school early education programs, infant classes, reception classes, nurseries and long day care) [8] as an important setting for the delivery of population-based interventions in order to prevent childhood obesity [9]. They have the potential to reach a significant proportion of the population aged less than five [10] and existing infrastructure to facilitate, support, and promote child healthy eating and physical activity. In recognition, governments and private organisations both in the United States (US) and Australia have developed specific recommendations regarding the implementation of service policies and practices to improve diet and increase physical activity levels among children attending care [11,12,13,14]. Such policies and practices have included a written service nutrition and physical activity policy, the provision of nutrition learning experiences to children, provision of nutrition resources to parents, limiting the provision of sweetened drinks to children, structured physical activity opportunities (e.g. fundamental movement skills programs), and limiting opportunities for sedentary screen time (e.g. computer use).

Despite such recommendations, evidence suggests that healthy eating and physical activity-promoting policies and practices are not routinely implemented by childcare services [13, 15,16,17,18,19]. For example, in the US, studies have reported that half of services do not provide drinking water in classrooms [17], around a quarter serve juice [20], and less than one third have written nutrition policies regarding food consumed by staff [18]. A 2016 study conducted in Canada reported that only 37% of childcare services provide nutrition education/resources to parents (on a regular basis) [21]. Further, in the US, research suggests 47–50% of childcare services do not have a physical activity policy [13, 22] and more than a quarter (28%) do not provide teacher-led physical activity [23], while in Australia, half of services reported not having a physical activity policy (41–48%) and nearly a third (28–30%) allow 3- to 5-year-old children non-active small screen recreation daily [15].

To maximise the public health benefit of obesity prevention interventions delivered in childcare, widespread implementation of recommended policies and practices is required. However, evidence to inform interventions is currently limited [24, 25]. A recently published Cochrane review examined the effectiveness of childcare obesity prevention interventions seeking to improve the implementation of healthy eating and/or physical activity policy and practice change and identified just ten such controlled trials [25]. The review concluded that there was considerable heterogeneity in the implementation strategies tested which precluded isolation of the impact of individual strategies and found weak and inconsistent evidence regarding their effectiveness in improving implementation of obesity prevention policies and practices in this setting [25]. The review identified a need for further implementation research in childcare [25].

Audit and feedback (also known as performance feedback) has long been used to support organisational change, particularly in clinical settings [26]. Performance review and feedback interventions are defined as those where an individual’s professional practice or performance (usually objectively measured health professional practice or patient outcomes) is measured over a specified period of time and compared against professional standards or targets with the results fed back to the individual [27]. A Cochrane review including 140 randomised controlled trials (RCTs), primarily conducted in the outpatient setting to improve prescribing or testing, concluded that audit and feedback interventions resulted in improved professional practice and the quality of health care [27]. The findings indicated that following audit and feedback, desired practice behaviours improved by around 4% on baseline behaviour [27]. The magnitude of the effect of audit and feedback however appears greater when baseline adherence is low, feedback frequency is high, and multiple modes of feedback occur [27]. Additional attractive features of audit and feedback are its capacity to be routinely delivered using modalities, such as telephone that enable reach to large numbers of services at relatively low cost, and ability to address identified implementation barriers such as beliefs about capabilities and motivation [28]. Compared with more resource-intensive interventions delivered under tightly controlled research conditions, the impact of such interventions that are able to be delivered under ‘real world’ conditions are of particular interest to policymakers and practitioners as they are more likely to provide a better approximation of the effect of intervention when delivered in the absence of research support and expertise [28]. Audit and feedback therefore presents as a feasible whole of population approach to supporting childcare services to implement healthy eating and physical activity policies and practices. The primary aim of this study was to examine the effectiveness of a performance review and facilitated feedback intervention in increasing the implementation of healthy eating and physical activity-promoting policies and practices in childcare services.

Methods

Design

The study employed a parallel group randomised controlled trial (RCT) design [29]. The trial protocol has been described in detail previously [30]. A sample of eligible childcare services, within the Hunter New England Region of New South Wales (NSW), Australia, was approached to participate. This implementation trial was conducted as part of government-funded health promotion services routinely delivered to all childcare services by a regional population health unit (Hunter New England Population Health (HNEPH)) [31]. The unit is responsible for supporting the implementation of healthy eating and physical activity policies and practices among early childhood education and care services across the Hunter New England government health region [31]. The trial, conducted from August 2013 to December 2014, was registered with the Australian New Zealand Clinical Trials Registry (ACTRN12614000972628) and approved by the Hunter New England Human Research Ethics Committee (06/07/26/4.04) and the University of Newcastle Human Research Ethics Committee (H-2008-0343).

Participants and recruitment

The study recruited centre-based childcare services in the Hunter New England region. A list of all 366 childcare services in the region provided by the Early Childhood Education and Care Directorate (the Government Licensing Authority) served as the sampling frame. Study information and consent forms were mailed to services approximately 2 weeks before a telephone call from a research assistant assessing eligibility and requesting consent to participate in data collection. Services catering exclusively for children requiring specialist care, mobile preschools, and Department of Education and Communities preschools were excluded as they were outside of the ethics approval, as were services already involved in an alternative RCT currently underway in the Hunter New England region. Services already identified through local health service data as comprehensively implementing healthy eating and physical activity policies and practices were also excluded. Eligible, consenting services were randomly allocated to the intervention or control group.

Randomisation and blinding

Eligible childcare services were allocated to intervention or control in a 1:1 ratio using computerised random number function undertaken by an independent statistician. Randomisation occurred prior to service consent to participate and baseline data collection. Staff responsible for recruitment of childcare services and data collectors were blinded to group allocation. Childcare services were not made aware of which study group they were allocated to until after baseline data collection.

Implementation strategy

Policies and practices targeted for implementation

We sought to increase service implementation of healthy eating and physical activity policies and practices consistent with best practice Australian healthy eating and physical activity guidelines for the childcare setting [12], and evidence reviews of policies and practices shown to be associated with child healthy eating and physical activity [20, 32]. The policies and practices targeted for implementation included:

  1. 1.

    Service having written nutrition, physical activity, and small screen recreation policies.

  2. 2.

    Service providing information to families (healthy eating, physical activity, small screen time, and breast-feeding, where relevant)

  3. 3.

    Service providing structured and specific learning experiences about healthy eating at least two times per week

  4. 4.

    Service supplying age-appropriate drinks to children (only water and age-appropriate milk)

  5. 5.

    Service conducting fundamental movement skills activities for children aged 3–5 years every day to at least 90% of children

  6. 6.

    Service limiting the use of small screen recreation by children aged 3–5 years to only educational purposes and for learning experiences

The policies and practices were identified and targeted from a broader group of 15 practices for which Hunter New England Population Health was responsible for supporting childcare services to implement as part of a state-wide childhood obesity prevention initiative (Healthy Children Initiative). Policies and practices in this intervention were selected on the basis of existing health service monitoring data summarising policy and practice implementation by childcare services in the region [33]. The sub-groups of targeted practices were included as they were those identified to have low prevalence of implementation.

Strategies

The implementation intervention was primarily delivered to nominated supervisors of each childcare service via five cycles of performance review and facilitated feedback conducted over a period of 10 months. Formative work including consultations with local childcare services was conducted to identify relevant factors that may impede local implementation of the targeted policies and practices. Damschroder’s Consolidated Framework for Implementation Research (CFIR) was then used to help classify barriers and inform strategy content [34]. Damschroder’s Consolidated Framework for Implementation Research integrates 19 theoretical models and is composed of five major domains identified as influential in successful implementation: innovation characteristics, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation [34]. Table 1 provides a summary of the four targeted constructs and how these were applied.

Table 1 Application of the consolidated framework for implementation research
Full size table

The initial performance review was completed in-person by trained support officers (1.5-h duration). Detail regarding the content of the visit is described in the protocol paper [30]. The four subsequent reviews were completed via telephone (30-min duration). Support officers had qualifications in nutrition, exercise physiology, psychology, and health education, with at least 2 years’ experience working with childcare centres. Each performance review included a standard assessment of the implementation of the targeted policy and practice (completed via telephone by support officers). Facilitated feedback included a discussion of current implementation status, assessed on the basis of information reported by the nominated supervisor using standard criteria with support tailored to meet service needs based on the policies and practices not being implemented. During each contact, implementation support staff facilitated a discussion regarding how to implement policies and practices to best suit the service needs and advice provided on the use of tools and resources was tailored based on identified barriers in line with service identified priorities. Where services were already meeting a policy or practice, implementation support was directed towards policies and practices not yet achieved. A follow-up email was then sent after the contact summarising agreed actions and providing resources.

Based on reviews of empirical research evidence, we aimed to incorporate components shown to be associated with greater effect in audit and feedback interventions targeting professional practice change. A detailed outline of the evidence and description of how it was applied to the intervention in practice is reported in the protocol paper [30]. In brief, these included providing feedback from a reputable source via experienced government health service support officers known to the services [26]; multimodal feedback provided in person (at the initial visit), via telephone, and in written form [26]; feedback tailored to each individual service context and identified implementation barriers; and use of reinforcement, facilitated reflection, problem-solving, goal setting, and action planning [26, 35,36,37,38]. Support officers monitored and recorded intervention delivery in a database. In addition, nominated supervisors were expected by service staff to offer support for the implementation of the targeted practices and to communicate goals and action plans, as well as progress.

Intervention services were provided with resources to support the implementation of these policies (policy templates, DVD, manuals, posters, and parent lunchbox resources). The policy templates were word documents to enable the service to amend to suit the needs of their service and provided an example aim, rationale with sample strategies that supported the recommended practices. They also referred to relevant childcare standards and regulations and provided a prompt for a timeframe for policy review. The DVD described fundamental movement skills and demonstrated children practicing and educators engaging in intentional teaching of these skills. Posters encouraged healthy eating and physical activity in young children, for example, healthy drinks and active play (available from http://www.health.gov.au/internet/main/publishing.nsf/Content/phd-gug-posters). Lunchbox resources included a parent brochure providing a list of recommended foods to be packed including recipe ideas and label reading. Program resources are available from http://www.goodforkids.nsw.gov.au/early-childhood-services/. Services were also emailed four electronic newsletters which communicated key messages relating to the targeted healthy eating and physical activity policies and practices [39], provided web links for additional information, and included case studies from individual services reporting successful approaches and strategies to improve or sustain policy or practice implementation or describing new or novel problem-solving approaches to alleviate service implementation barriers.

Comparison

Control group services received the same four electronic newsletters (described above) during the intervention period, but did not receive any other resources. Staff could implement any of the recommended practices described in the newsletter, but did not receive support in doing so. At completion of the intervention period, control services were offered the complete intervention.

Outcomes

Computer-assisted telephone interviews (CATI) were used to collect data at baseline and 12-month follow-up. The interviews were conducted as part of government-funded service delivery with baseline and follow-up data collection aligned with 12-monthly routine practice assessment. The survey was undertaken with the nominated supervisor or lead educator at each service. In Australia, nominated supervisors are responsible for policy development, ensuring compliance with licensing and accreditation requirements and most also have teaching roles. The surveys assessed service characteristics and implementation of healthy eating and physical activity policies and practices [40] and took approximately 15–25 min to complete. CATI interviewers were blind to study group allocation (Additional file 1).

The primary outcome was the change in prevalence of services implementing all six targeted healthy eating and physical activity policies and practices at 12 months. Twelve items assessed the implementation of six policies and practices. The items have been validated and used in previous trials [15, 40,41,42,43] and included:

  1. 1.

    Services having written nutrition, physical activity, and small screen recreation policies (three items) (all three) (which may be combined with another policy) (yes/no to having all three).

  2. 2.

    Service providing information to families on healthy eating, physical activity, small screen time, and breast-feeding, where relevant (four items) (all topics). Information was required to be distributed at least once in the last 12 months and could include material handed directly to parents, mailed or emailed or placed in their child’s pigeon hole or bag, or information included in newsletters or at orientation (yes/no to all topics). Examples could include a list of recommended foods for lunchboxes and lunchbox ideas, physical activity and screen time recommendations for children, and breastfeeding guidelines.

  3. 3.

    Service providing structured and specific learning experiences about healthy eating at least two times per week (one item, yes/no). Examples included experiential activities about food, cooking skills, food growing (e.g. kitchen/vegetable gardens, planting seeds), tasting sessions, and discussion around “everyday” and “sometimes” foods.

  4. 4.

    Service only supplying age-appropriate drinks to children including water and reduced fat plain milk (one item) (yes/no). For children less than 2 years old, the service reported supplying plain full-fat milk.

  5. 5.

    Service conducting fundamental movement skills activities for children aged 3–5 years every day to at least 90% of children (two items). Where service reports the average number of days per week that educators lead structured activities to develop Fundamental Movement Skills (could be during transition activity, group or circle time, or during outdoor play) is equal to all service opening days, and where service estimate of the percent of children that usually participate is 90% or greater (or are encouraged to participate if special needs).

  6. 6.

    Service limiting the use of small screen recreation (one item) (i.e. TVs, videos, DVD, computers and other electronic games, iPads/tablets) by children aged 3–5 years to only educational purposes and for learning experiences such as to gain knowledge or share information about a specific learning area or child’s interest or to facilitate exploration of activity, dance, or movement (yes/no).

To provide greater description of changes occurring in the measures of practice implementation, we also report the proportion of services that implemented each of the policies and practices at baseline and 12 months, the mean number of practice services were compliant with, and the breakdown for each individual policy and for information home to families. These measures were not prospectively registered.

Service characteristics included service type, days open, ages catered for, hours of operation, number of children enrolled, and number of primary contact teaching staff. Intervention participants also completed a survey about their satisfaction with the intervention components including program resources, feedback reports, support calls, and newsletters at completion of the intervention. Eleven items were assessed using a 7-point Likert scale (1 = strongly disagree to 7 = strongly agree). Data regarding intervention fidelity was sourced from program records which were collected by research staff during implementation.

Sample size calculation

A sample size of 152 services (76 per group) can detect a difference of 20% in the prevalence of services implementing all targeted policies or practices with 80% power and alpha of 0.05, assuming 10% of control services implement all targeted policies and practices at follow-up. A difference of 20% was deemed by the Population Health Unit (HNEPH) responsible for improving policy and practice implementation as meaningful from a public health service delivery perspective and similar to policy and practice changes observed in previous studies conducted by the research team in this setting [42, 43].

Statistical analysis

All statistical analysis was performed using SAS v9.2 statistical software. Descriptive statistics were used to describe the demographic and service characteristics of the study sample. Sample characteristics were compared using Fisher’s exact test (percentages) and the t test (means). Intervention effectiveness was determined using logistic regression, controlling for baseline. The primary analysis was based on an intention-to-treat framework. Sensitivity analysis was undertaken using imputation of baseline values for missing follow-up data [44]. Additional sub-group analyses of the primary trial outcome were pre-specified and undertaken by childcare service geographic location (urban [regional cities and inner regional areas] or rural [outer regional and remote areas]) as determined by the Australian Standard Geographic Classification. Socioeconomic status was classified according to the Socioeconomic Indices for Areas, and services were grouped by postcode as being in the top 50% of New South Wales or the bottom 50% of New South Wales. All statistical tests were two-tailed with an α of 0.05.

Results

Response rates and sample characteristics

Figure 1 describes the participation of services in the trial. Of the 366 childcare services in the region, 128 were excluded given their involvement in an alternative RCT, a further 30 did not meet inclusion criteria, and an additional 77 were identified as comprehensively implementing healthy eating and physical activity policies and practices. A total of 131 services were randomised among which 68 were allocated to intervention and 63 to the control. Six services allocated to the intervention and 17 services allocated to the control group did not provide baseline data and were therefore excluded. There were 62 intervention services and 46 control services that provided baseline data, giving a consent rate of 82%. At 12-month follow-up, five intervention participants and three control participants were unavailable to complete follow-up.

Fig. 1
figure 1

Participant flow chart

Full size image

Service characteristics

Service characteristics by intervention and control group are shown in Table 2. Intervention groups were similar at baseline. The majority of services were long day care services (73.0%), and one third were preschool services (32.8%). Almost all services were open for 5 days per week (93.0%). Food was provided by the service for the children in 60% of services and by the family in 34% of services. Services were open for an average of 9 h per day. The majority of services in both groups were located in a metropolitan or inner regional city (61.4%) with 38.7% classified as outer regional or remote areas.

Table 2 Baseline demographic comparison of study groups
Full size table

Implementation of healthy eating and physical activity policies and practices

Primary trial outcome

There were no services that were implementing all of the six policies and practices at baseline, and few who were meeting all six practices at follow-up (9.7% [n = 6] intervention services and 17.4% [n = 8] control services) (mean difference 0.51; 95% CI 0.16 to 1.58; p = 0.24) (Table 3). Sensitivity analyses carrying baseline observation forward for missing data were also non-significant. There were no significant differences between groups on the primary trial outcome at follow-up by socioeconomic (mean difference 1.25; 95% CI 0.12 to 12.84; p = 0.85) or geographic subgroups location (mean difference 0.55; 95% CI 0.05 to 5.85; p = 0.62).

Table 3 Proportion of services implementing policies and practices at baseline and follow-up
Full size table

Table 2 shows the proportion of services that were compliant with each policy or practice at baseline and 12-month follow-up. At baseline, the proportion of services already implementing healthy eating learning experiences and limiting the use of small screen recreation was high. Between baseline and follow-up, there were increases in compliance for all policies and practices in both intervention and control services, with no significant differences between groups. The largest increases over time were observed for intervention services providing information to families (by 27.4%) and the service having written policies (by 17.8%).

Other outcomes

There were no differences between the groups in the number of practice services were compliant with (mean difference 0.1; 95% CI − 0.4 to 0.6; p = 0.71). At baseline, intervention services were compliant with a mean of 3.4 (95% CI 3.1 to 3.6) practices, which increased to 4.0 practices at follow-up (95% CI 3.8 to 4.3). Control services were compliant with 3.4 practices at baseline (95% CI 3.1 to 3.6) and increased to 3.9 practices at follow-up (95% CI 3.5 to 4.4).

Intervention fidelity and acceptability

Overall, more than half (56.5%; n = 35) of services participated in all five performance review and feedback cycles (contacts). Participation rate for each of the five contacts ranged between 83 and 93% (n = 52–58). The mean number of contacts was 3.56 (SD 1.58) and range 0 to 5. Most services reported that intervention components were helpful (Table 4) and that they were happy with the frequency of support calls and e-newsletters.

Table 4 Acceptability of the intervention
Full size table

Discussion

This study sought to test an audit and facilitated feedback intervention to improve the implementation of healthy eating and physical activity-promoting policies and practices in the childcare setting. The study found the intervention was not effective in improving, relative to control, the implementation of targeted policies and practices. Findings indicate that such an intervention in isolation may not be sufficient to improve childcare environments, suggesting policymakers and practitioners may require more intensive strategies to support practice change in this setting.

Study findings contrast with previous research in the field [25]. Three targeting the implementation of healthy eating and physical activity practices included in a recent Cochrane review trials tested multi-component implementation strategies (for example, staff workshops, policy support, resources, and follow-up contact/support) capable of addressing a number of barriers to change [45,46,47] and reported significant improvements in measure of policy or practice implementation. The lack of evidence of improvement in implementation found in the current study suggests future strategies may need to address a more comprehensive range of barriers to that achieved through audit and facilitated feedback (e.g. beliefs about capabilities and consequences) alone [48]. For example, views of parents and childcare service management are important in the implementation of obesity prevention practices, as is the accessibility of external resources to support implementation [49].

The potential impact of the implementation strategy may have been hindered by targeting nominated supervisors of childcare services. While nominated supervisors are ultimately responsible for service operations, actual implementation of obesity prevention practices requires the action of service staff. In this trial, we did not assess the extent to which nominated supervisors engaged with childcare service staff to support service improvement as a result of the intervention. Failure of the nominated supervisor to do so may have reduced the potential impact of the implementation strategy as previous research suggests that implementation is more likely to occur if appropriate individuals or groups are engaged in the change process [34]. The use of strategies to ensure greater penetration of performance review and feedback across childcare service staff may therefore be required [34]. Targeting multiple policies and practices may have also reduced the potential effect of the implementation strategy in this trial compared to targeting a single policy or practice. Interventions perceived as complex and placing greater demand on childcare services and their staff are less likely to be implemented [34]. In settings such as schools, strategies targeting single health policies have yielded large improvements in implementation [49]. Such an approach could be considered for future implementation initiatives in childcare.

Furthermore, the introduction of the National Quality Standards for the setting (including nutrition and physical activity elements) in the 12 months prior to the commencement of the trial and broad government support for implementation of policies and practices occurring state-wide may have also facilitated policy and practice implementation in the control group potentially reducing the likelihood of an intervention effect.

The trial used random assignment and blinding of outcome assessor to improve internal validity and used probability sampling, less stringent eligibility criteria, and was conducted in the context of a service delivery initiative, maximising external validity. However, there were a number of limitations. First was the use of phone survey for audits conducted remotely (off-site) and reliant on nominated supervisor self-report of implementation outcomes. While validated and used in previous trials, such measures are prone to reporting bias such as social desirability bias [50]. The use of on-site observation or measurement triangulation techniques should be considered for future intervention to improve the robustness of audit and research findings [51]. Secondly, the quantitative nature of the outcome measure limited our capacity to collect contextual information that may have provided greater insight into any systematic or structural barriers to implementation. Third, outcomes were assessed post-intervention at one time point only. Additional time points would allow for more sophisticated analyses and potential assessment of sustainability. Furthermore, there is possible bias in the sample given that potential participants had already declined to participate in another intervention. This may indicate the participating services may have been less likely to adopt the policies and practices targeted in the intervention. In addition, the study was under-powered. Finally, the measure used to assess the practices may not have been sensitive enough to detect changes in implementation.

Notwithstanding study limitations, the trial makes an important contribution to a limited evidence base on the effectiveness of performance review and feedback as a standalone implementation strategy. While self-assessment, audit, or feedback is a common element of most previously trialled strategies to improve implementation of nutrition and physical activity policies and practices [25], in isolation audit and facilitated feedback does not appear to be sufficient to achieve implementation improvements. For policymakers and practitioners, the findings underscore the need for consideration of additional implementation strategies in future trials to maximise obesity prevention initiatives in this setting. Strategies that address the reported barriers to implementation that include specific behaviour change techniques [48] (e.g. prompts/cues, material reward, habit formation) or accounting for policy context (e.g. changing licensure requirements) are two areas that may offer promise.

Conclusion

A performance review and facilitated feedback intervention was not effective in improving the implementation of healthy eating and physical activity-promoting policies and practices within childcare services. Future research targeting multiple health policies or practices in this setting should include additional implementation support. Gaps remain in available evidence to inform programs delivered in real-world conditions aiming to support the implementation of healthy eating and physical activity policies and practices as part of routine practice in childcare services [52].