Background

In 2011, the YMCA of the USA adopted the Healthy Eating and Physical Activity Standards (HEPA) for all their afterschool programs (ASPs) [1]. The standard for physical activity called upon ASPs to ensure each child accumulates a minimum of 30 min of moderate-to-vigorous physical activity (MVPA) each day during the ASP. The achievement of the MVPA standard holds considerable public health importance as it would ensure children accumulate at least half of their daily MVPA recommendation during the ASP. Although the MVPA standard was adopted in 2011, data collected during the spring of 2015 on children’s activity levels in YMCA-operated ASPs demonstrated that only 33% of boys and 17% of girls were achieving the standard [2]. This is consistent with other published evaluations of non-YMCA operated ASPs [3,4,5].

Given the adoption of the HEPA standards in 2011 was insufficient to improve routine practice within ASPs to achieve the 30 min MVPA standard, the YMCAs across a single southeastern state in the United States, South Carolina (SC), pledged to achieve the MVPA Standard using a coordinated statewide training framework to enhance the quality of their programming to promote greater amounts of physical activity. The initiative focused on providing professional development training for program staff, as well as modifying program structure to facilitate opportunities for children to engage in physical activity. The training built upon previous studies by focusing on enhancing the quality of existing physical activity opportunities and extending the amount of time allocated for physical activity [6,7,8,9,10,11]. Results from the first year (2015 baseline to 2016 first year) indicated no overall changes in the proportion of children accumulating 30 min MVPA [12]. However, this was greatly influenced by whether ASPs attended the trainings and what components, if any, the ASPs implemented. Based on the first year outcomes, the CEOs of each SC YMCA Association pledge to have all their ASP staff receive the professional development training for 2017. The objective of this study is to report the two year outcomes (2015 to 2017) from this statewide initiative.

Methods

The methodology and intervention for this study is described in detail in previous publications [2, 12] and will be overviewed in brief below. For this study, baseline data were collected during the spring of 2015. The delivery of the intervention occurred during the fall of 2015 and fall 2016, while outcomes following the trainings were measured during the spring 2016 and spring 2017. This study employed a single group quasi-experimental pre/post design, with all YMCA-operated ASPs in South Carolina eligible to participate in the professional development trainings. The study is reported in accordance with the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement [13]. Additionally, the study findings are reported using the RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance) [14, 15]. The maintenance component of the framework was not assessed given the absence of a post-treatment follow-up (e.g., > 6 months) [15]. Also, reach and adoption were considered analogous in this study given the intervention targeted change at the setting-level (i.e., ASPs) that was hypothesized to lead to changes in the MVPA levels of the children attending (i.e., individuals enrolled in the ASPs) [15, 16].

Sampling strategy for evaluation of ASPs

A detailed description of the sampling strategy for the evaluation ASPs is reported elsewhere [2]. In brief, the sampling strategy included a single program from each of the 20 Associations. For Associations that operated a single program (n = 5), that program was selected. For Associations that operated two or more programs (n = 15), the following sampling strategy was employed. For Associations where all programs enrolled fewer than 50 children (n = 3), the largest program was selected. For Associations that operated programs with more than 50 children enrolled (n = 12), a single program was randomly selected from these. All sites selected as an evaluation site, both randomly and non-randomly, agreed to participate as part of their Association’s statewide commitment to the initiative. All parents were informed of the study from their respective ASP location. Parents were provided an option to opt-out (passive consent) their child from participating. Children verbally assented on each day where data occurred to participate in the measures. All methods were approved by the Institution Review Board of the University of South Carolina.

Child and ASP characteristics

Child demographics were self-reported (e.g., age, grade, race/ethnicity), and standing height and weight were measured using standard protocols with children removing shoes and wearing light clothing [17]. The annual revenue for each YMCA Association was collected from their most recent publicly available annual reports from 2013. The percentage of households in poverty was used as a marker of socioeconomic status of the ASPs. This was based on US Census 2014 zip code information based on the operating location zip code for each ASP. Each ASP’s operating setting, either in a school or YMCA facility, was also recorded.

Intervention description

The intervention for physical activity followed the Strategies To Enhance Practice (STEPs) framework [18,19,20,21,22]. All intervention-related activities were coordinated and delivered by a single full-time employee with 15 years of experience operating YMCA youth programs. For the statewide initiative, the STEPs training was divided into two distinct, yet complimentary, trainings. The two different types of trainings were offered over the course of the fall academic year (August to October 2015 and 2016). The first training was for site leaders (i.e., the individuals directly responsible for a single ASP). The second was for site leaders and staff (i.e., those individuals responsible for interacting and overseeing children during the program, referred to as the all-staff training). For the first year, attendance at the trainings was voluntary, with ASPs opting to attend both, one, or neither of the trainings. For year two, the CEOs of the YMCA Associations pledged to have all their staff attend the trainings.

The site leader training lasted for two hours and provided an overview and history of the YMCA MVPA standard and strategies to increase children’s MVPA in ASPs. Strategies were based on the Theory of Expanded, Extended, and Enhanced Opportunities [22] which focused on extending the amount of allocated time for children to be physically active each day, creating schedules that clearly define the roles and responsibilities of staff during physical activity (PA) opportunities and other scheduled times, and enhancing the games commonly played in the ASPs by using the LET US Play principles (lines, elimination, team size, uninvolved staff/kids, and space, equipment and rules) for modifying games to maximize MVPA [18, 19, 23]. Programs were also asked that when time was scheduled for physical activity, that children did not have a choice of selecting a non-active alternative (e.g., play on playground or stay inside at computer lab). For allocated PA time, where ASPs were allocating 50% (i.e., 90mins) or more of their daily schedule for PA opportunities, they were asked to maintain this allotment. This was based on information collected from the baseline evaluation of ASPs in 2015, where ASPs scheduled a median of 59% (~115mins/d) of their daily program for PA opportunities, with 15 ASPs devoting 50% or more of their daily schedule for PA. For any ASP that scheduled less than 60 min for PA, they were asked to increase this to a minimum of 60 min per day (33% of a 3 h program). During the first year a total of three site leader trainings were provided, one per region in the state (i.e., Low Country, Midlands, and Up State). For the second year, no separate site leader only trainings were provided. This was based on the low attendance during the first year (only 16% attended). The content of the site leader trainings was, therefore, incorporated into the all-staff trainings and held for two hours prior to the start of the all-staff trainings. This eliminated the need for site leaders to schedule to attend a training on two separate days.

The second training, the all-staff training, lasted for two hours and provided an overview of the YMCA MVPA standard and strategies to increase MVPA in ASPs. Strategies included role modeling PA behaviors, scheduling time for children to be active, and following the LET US Play principles. The majority of the training focused on skill development for using the LET US Play principles. Attendees identified a game they commonly play, played the game using its traditional rules, and then played the game again with self-identified modifications based on the LET US Play principles. This procedure was repeated for up to five different games. Throughout the training, the LET US Play principles were continuously introduced, explained, demonstrated, and reinforced. For the first year, a total of 13 all-staff trainings were conducted. For the second year, a total of 16 all-staff trainings were conducted, with four Associations attending the same regional training. Additionally, monthly emails were distributed to all ASP site leaders. Emails included STEPs-related physical activity promotion content, such as examples of LET US Play principles, links to online two-minute videos depicting a LET US Play principle, and scheduling physical activity opportunities.

Reach/adoption, effectiveness, and implementation of STEPs

All measurements occurred during the spring (March through April) of 2015, 2016, 2017. Consistent with previously established protocols, each ASP was visited for data collection on four non-consecutive, unannounced days Monday through Thursday [5, 24,25,26].

Reach/adoption

For the purpose of this study, the measure of the reach/adoption of the intervention was determined by percentage of Associations, ASPs, site leaders and staff in attendance at the fall trainings for professional development at each of the intervention years. To document reach/adoption, at each training offered during the fall of 2016 and 2017 attendees provided their name, position, and their affiliated ASP and Association name.

Effectiveness

The effectiveness of the STEPs intervention was evaluated via accelerometry using a standardized protocol as the primary outcome [2, 12]. All children attending an ASP on measurement days were fitted with an ActiGraph GT3X+ accelerometer on the hip. Accelerometer data were distilled using 5-s epochs to account for the intermittent and sporadic nature of children’s PA [27] and to capture the transitory PA patterns of children [28, 29]. Upon arrival to the ASP, children were fitted with an accelerometer and the arrival time was recorded (monitor time on). Research staff continuously monitored the ASP for accelerometer wear compliance. At the time of a child’s departure, research staff removed the accelerometer and recorded the time (monitor time off). Children wore the monitors for the entire time in attendance at the ASPs. Cutpoint thresholds associated with moderate and vigorous activity were used to distill the PA intensity levels [30] and sedentary behavior [31]. Children were included in the study if they had one or more valid days of accelerometer data defined by a total wear time (time off minus time on) of ≥60 min [5, 25, 32]. The minutes all children spent in MVPA were dichotomized to represent those children who achieved (i.e., ≥30 min MVPA/day) and those that failed to achieve (i.e., < 30 min MVPA/day) the PA standard [33].

Implementation

Implementation of STEPs was measured via document review and on-site direct observation during all measurement occasions [34,35,36,37]. The amount of scheduled PA opportunities was determined via program schedules. Program schedules were collected on each of the four data collection days at each measurement occasion and the time allotted for PA opportunities was totaled for each day. We also measured the PA opportunities via direct observation using the System for Observing Staff Promotion of Activity and Nutrition (SOSPAN) [37]. The observational protocol consisted of continuous systematic scanning and rotating through pre-defined target areas across the entire duration of an ASP. Trained observers completed standardized reliability training and conducted all observations. Inter-rater agreement criteria were set at > 80% using interval-by-interval agreement for each category [37]. Consistent with published reliability protocols [37], reliability was collected prior to measurement and on at least 30% of data collection days. Inter-observer reliability for the LET US Play principles were estimated via interval-by-interval percent agreement and weighted kappa (κw). Percent agreement ranged from 90.5 to 99.2% and κw ranged from 0.21 to 0.96 (median 0.75). Reliability was checked weekly to identify disagreements. Operational definitions of variables with borderline or low reliability (< 90% agreement) were then discussed with observers to ensure reliability and prevent observer drift. A total of 22,626 scans were collected across 2015, 2016, and 2017 data collection periods.

Each scan was tagged with a “context” variable that identified whether the scan occurred during snack, academics, enrichment, physical activity, or some other scheduled ASP opportunity (e.g., water-breaks, transitions). The percentage of scans each day a context variable occurred was also used to assess time allocated for a given context. The percent of scans was compared to the written ASP schedule to determine consistency among the two. Where discrepancies occurred, the direct observation allocated time was used as an indicator of time allocated for a given context. Also, each scan was tagged with a location variable and this was used to calculate time spent outdoors. SOSPAN was used to evaluate the implementation of 12 LET US Play principles during PA opportunities. On each day of observation, the percentage of scans were computed for the following: children waiting in line for turn, children eliminated, small sided-games, staff actively engaged in activity, staff verbally encouraging activity, choice of two or more activity offered, girl’s provided with their own physical activities, staff giving instructions on how to play games, children waiting for activity to start (i.e., idle time), staff withholding PA as punishment, and staff disciplining children with PA. A twelfth variable was collected at the end of each day - all staff wear clothing to be physical active (yes/no). The distribution of each of the first eleven LET US Play principles were divided into tertiles based on their 33rd and 66th percentile for the principle at baseline [36]. Programs were assigned a zero (<33rd centile), 1 (33rd to 66th centile), or 2 (>66th centile). Staff wearing appropriate clothing was transformed into a 3-pt scale ranging from zero (none of the days), 1 (some of the days), or 2 (all days). Scores were summed to represent an overall LET US Play implementation score that could range from 0 to 24.

Overall implementation and changes in MVPA

First, ASPs were classified into one of three categories based upon their change in MVPA between adjacent years (2016 minus 2015 and 2017 minus 2016) for boys and girls, separately. Change in MVPA was benchmarked at ±3 min MVPA, which is consistent with the most recent meta-analysis on physical activity interventions using objective measures in youth [38]. Second, using the SOSPAN and document review, the following six implementation indicators were created and expressed as a percentage: LET US Play implementation score, amount of PA time observed, amount of PA time scheduled that was PA only time, the total amount of PA time scheduled as documented in daily schedules, and the amount of time the ASPs went outside during scheduled PA opportunities and across the entire duration of the program. For each evaluation ASP, changes in the implementation indicators were calculated between adjacent years. Programs were the classified as having either decreasing (<− 10%), increasing (> + 10%) or no change (> − 9.99 to <+ 9.99%) in level of implementation on each indicator separately. In absence of established benchmarks, change was defined as a ± 10% difference between adjacent years.

Statistical analyses

Analyses were conducted summer 2017. Descriptive statistics were computed by gender for the percentage of youth meeting the 30 min/day MVPA standard and in mins/d of MVPA. The overall test for the effectiveness of the intervention was examined using intent-to-treat (defined as all ASPs received the intervention) repeated-measures random effects logit models, with days measured nested within children, who were nested within ASPs. The models were estimated using the dichotomized MVPA variable (0 = < 30 min/d) as the dependent variable for boys and girls, separately, to compare changes in the percentage of children meeting the MVPA Standard over time. Race (African Americans), age (years), ASP operating in YMCA facility, and Census 2010 zip code percentage of households in poverty were included in the models as covariates. As a secondary outcome, the minutes spent in MVPA were estimated with random effects quantile regression models at the 50th quantile of the distribution and design-matrix bootstrapped standard errors, separately [39]. The same covariates were used in this model. All analyses were performed using STATA (v.14.0 College Station, TX).

Results

The descriptive characteristics of the evaluation ASPs and the children enrolled in the programs during spring 2015 (baseline), spring 2016 (end of first year of intervention), and spring 2017 (end of second year of intervention) are presented in Table 1. The reach/adoption of the trainings for the evaluation sites and all YMCA-operated ASPs across the state at the site leader and all staff training occurring fall 2016 and fall 2017 are presented in Table 2. Overall, the attendance at both the site leader and all-staff trainings during the second year were higher than those observed during the first year of the intervention, with 77% of site leaders attending the site leader trainings in year two (versus 16% in year one) and 67% of the staff attending the all-staff training during year two (versus only 45% during year one).

Table 1 Baseline (spring 2015), year 1 (spring 2016), and year 2 (spring 2017) characteristics of the afterschool programs and children in attendance
Full size table
Table 2 Attendance at the professional development trainings offered year 1 fall 2016 and year 2 fall 2017
Full size table

For effectiveness, the change in the proportion of boys and girls achieving the 30 min/d MVP Standard and the minutes/d of MVPA accumulated from across all three measurement occasions are presented in Table 3 and changes in mins/d MVPA for each ASP for boys and girls separately are presented in Fig. 1. Overall, from 2015 to 2017, there was a statistically significant increase in the odds of both boys and girls achieving the 30 min/d MVPA Standard by 1.31 (95CI 1.04–1.64) and 1.35 (95CI 1.01–1.80), respectively. This represented an increase of 5 and 3% for boys and girls achieving the MVPA Standard, respectively. Correspondingly, during the last year of the intervention, boys increased the number of mins/d of MVPA by 2.6 (95CI 1.0–4.1) and girls by 1.5 (95CI 0.1–2.9).

Table 3 Estimates and changes in accelerometer-derived moderate-to-vigorous physical activity (MVPA) from 2015, 2016, and 2017
Full size table
Fig. 1
figure 1

Changes in moderate-to-vigorous physical activity minutes per day for boys and girls by afterschool program from 2015, 2016, and 2017

Full size image

For implementation, changes across the implementation indicators by ASPs that either gained (+3mni/d), lost (− 3 min/d), or maintained MVPA across each adjacent measurement occasion (2015 vs 2016 and 2016 vs 2017) are presented in Tables 4 and 5. For boys, 7 of the possible 9 patterns of gain-lost-maintained classifications were observed, with only 2 ASPs classified as maintain-maintain and only a single ASP exhibiting gains across both years (i.e., gain-gain). Similar findings were found for girls, with 8 of the 9 patterns observed, with only 3 ASPs classified as maintain-maintain and 1 classified as gain-gain. Importantly, the gain-gain ASP for boys was not the same gain-gain ASP for girls. Across adjacent years, and for both boys and girls, ASPs that gained MVPA had a consistently higher supportive to non-supportive change ratio (range 1.0 to 2.0) in the implementation indictors compared to ASPs classified as maintain (range 0.6 to 0.8), followed by ASPs classified as lost (range 0.1 to 0.4).

Table 4 Levels of implementation by gain, maintain, or lost minutes of moderate-to-vigorous physical activity for boys
Full size table
Table 5 Levels of implementation by gain, maintain, or lost minutes of moderate-to-vigorous physical activity for girls
Full size table

Discussion

This study reports the outcomes from a statewide initiative to achieve the 30 min/d MVPA standard adopted by YMCA-operated afterschool programs after two years of implementation. Overall, across the two-year implementation/dissemination effort, small improvements were observed in the proportion of boys and girls achieving the 30 min/d of MVPA benchmark. This was in spite of an increase in the attendance of both site leaders and staff that received the training in 2017 compared to the percentage that received the training in 2016. Detailed process evaluation revealed there was substantial variability in the uptake and continuance of different intervention strategies within and between afterschool programs, across the years. This is an important finding as level of implementation was associated with considerable increases or decreases in the amount of MVPA children accumulated during the programs. This bidirectional pattern across years demonstrates the complexity of modifying this environment to promote higher levels of MVPA. Yet despite variability in implementation and the associated improvements or decreases in child activity levels, ASPs were routinely affording children meaningful amounts of MVPA (> 15 to 20 min/d MVPA), indicating that these programs, even without fully achieving the MVPA Standard, can have a positive impact on child activity levels.

Important findings from this study are those associated with the implementation of the strategies that targeted modifiable aspects of daily programmatic structure (e.g., increasing the amount of time allocated for physical activity opportunities, staff enhance commonly played games to promote more MVPA), that when changed, should lead to increased child MVPA during the program. Across both intervention years, ASPs elected to improve upon one or more of the targeted intervention strategies. Conversely, ASPs also elected to modify one or more of the targeted intervention strategies in a less supportive way. This led to a scenario where during each year of evaluation all programs were observed making both supportive and non-supportive changes to how their program operated. Further, no two programs made all the same changes to all the same implementation indicators. This complex pattern of changes suggests that programs can, and will, make alterations to their routine operation that fit best with their local conditions and resources, and that no one strategy is ultimately responsible for improvements (or losses) in child MVPA. For example, an ASP could adopt activity supportive practices such as scheduling more time for children to be physically active each day and schedule more of this time to occur outside, yet simultaneously adopt activity non-supportive practices such as decreasing the quality of the activities and the amount of the scheduled activity opportunities where children cannot elect to engage in a sedentary activity (e.g., play outside or choose to go to a computer lab). Conversely, another ASP may reduce allocated time for activity opportunities, yet make all the allotted time ‘PA only’ time (i.e., children cannot select to do a non-active activity).

The patterns observed from these ASPs are consistent with the theoretical tenets of equifinality [40, 41], whereby the same outcome (in this case increases/decreases in MVPA) can be achieved via varying/multiple pathways (in this case different implementation indicators). The results presented in Tables 4 and 5 illustrate this phenomenon, with programs across the three categories of gain, maintain, and lost MVPA showing both supportive and non-supportive changes in the implementation indicators. An important aspect is that ASPs that gained MVPA during any adjacent year comparison elected to make more supportive changes than non-supportive changes compared to those programs that either maintained or lost MVPA. Overwhelmingly, programs classified as lost MVPA consistently made more non-supportive than supportive changes to their daily program which led to an average decrease in MVPA ranging from 5.0 to 7.7 mins/d. Moreover, the majority of ASPs showed variable levels of MVPA across the three years of evaluation, with some of the changes (both supportive and non-supportive) being very large. This suggests that within any two or three year time period, major swings in MVPA may occur and that ASPs voluntarily make both supportive and non-supportive changes to their daily programmatic schedule that influences child levels of MVPA. Unfortunately, the only other evidence of this phenomenon can be found in the authors’ previous studies [20, 21] that report estimates of MVPA across multiple years for each ASP individually. What these data show is that during any two-year comparison, there appears to be natural “ups and downs” within and across these settings – even for ASPs operated by the same organization [20, 21]. Nevertheless, programs looking to increase MVPA should seek to make more overall supportive changes to their programmatic structure. Which changes they should target, however, is unknown, only that more supportive than non-supportive changes need to be made and ASPs need to do this consistently across years.

Statewide commitment by YMCA CEOs to ensure all staff receive training in August 2016 led to higher levels reach/adoption as indicated by the attendance by both site leaders and staff to the professional development trainings compared to the attendance during trainings in August 2015. The reason 100% attendance was not achieved during the second year was due to conflicting schedules of some employees with the dates of the trainings. Despite this, a much higher percentage of site leaders attended the site leader training during 2016 (77%) compared to only 16% across the state in 2015, whereas 67% of staff received training in 2016 compared to 45% in 2015. Unfortunately, this increase in the number of individuals trained did not translate into full achievement of the MVPA standard. Based on the activity levels presented in Table 2, even though no ASP fully met the 30 min/d MVPA standard, all programs afforded children the opportunity to engage in a large volume of MVPA across all measurement years. This amount of MVPA is consistent with other intervention studies [6, 8, 20, 21, 32] within the ASP setting that report children, prior to their ASP receiving a formal intervention, engage in anywhere from 15 to 30 min of MVPA/d. These MVPA levels suggests the ASP setting may not be a place in need of a physical activity-focused intervention and that efforts should be focused on increasing the number of children who can access, and thereby, benefit from attending an ASPs. National estimates indicate ~ 10 million children currently attend an ASP [42], yet an additional 19 million children would attend if one were available [42]. Providing access to ASPs for these children may result in greater public health gains for MVPA compared to the minimal improvements that can be achieved for the limited number of children who currently have access to ASPs.

The current study has several strengths. These include objective process measures that were linked to objective outcome measures, evaluation of a statewide effort to achieve a nationally endorsed policy, a large number of participating ASPs and children, and multiple years of evaluation. Conversely, the limitations include the lack of a control/comparison group, the findings only represent efforts from a single state, and no planned follow-up to monitor maintenance. It should also be noted that the intervention delivered may not be entirely effective and that alternative approaches may be necessary. However, previous studies that have employed pre-packaged curricula [6, 8,9,10, 43, 44] or dedicated greater amounts of time to professional development [6] have shown no more effectiveness than the current approach. Thus, questions still remain as to the “best” approach for achieving the 30 min/d MVPA standard and, as mentioned above, whether the time and resources dedicated to increase MVPA within this setting are warranted.

Conclusion

In conclusion, this two-year evaluation found that, overall, minimal gains in MVPA were made. Detailed process evaluation, however, indicated that all ASPs made both supportive and non-supportive changes to their daily programmatic structure which led to increases and decreases in children’s MVPA, with those ASPs electing to make more supportive changes exhibiting the largest levels of MVPA whilst those ASPs electing to make a greater number of non-supportive changes exhibiting the greatest declines in child MVPA. Future studies need to take account for the substantial within and between program variability across years. Furthermore, intervention scientists need to determine whether intervening within this setting, given baseline MVPA levels, is necessary or whether placing resources towards increasing access to these programs may provide a greater public health benefit.