Appendix
Full Results for Refining the Constructs Tested in the Measurement Model
In this sample, between .0 and .9% of the item-level data were missing. Due to the low level of missing data, we assumed all data were missing at random (MAR). The MAR mechanism renders the missingness functionally random (Little 2013) and is thus termed ignorable (Rubin 1976). To account for the participants nested in project sites, we used maximum likelihood estimation with robust standard errors (MLR) methods in these analyses—as all items were on a five-point Likert scale (see Rhemtulla et al. 2012)—and a sandwich estimator (TYPE = COMPLEX in Mplus).
Analyses were conducted using Mplus Version 8 (Muthén and Muthén 1998–2017). We used multiple goodness of fit indices as recommended by Brown (2006). Absolute fit was tested by checking for χ2 significance and the standardized root mean square residual (SRMR), with values closer to 0 indicating better fit (Brown 2006). Parsimony-corrected fit was assessed by evaluating the root mean square error of approximation (RMSEA) and its confidence interval, with values closer to 0 indicating better model fit (Brown 2006). The suggested upper bounds, or cut-off values, of acceptable fit for the SRMR and RMSEA are .08, and ideally less than .05 (Brown and Cudeck 1993; Hu and Bentler 1999; Steiger 1990). Comparative fit, the evaluation of the specified solution in comparison to a null model in which no items are correlated, was tested with the comparative fit index (CFI) and the Tucker-Lewis index (TLI), with values closer to 1 indicating better model fit (Brown 2006). The suggested lower bounds, or cut-off values, of acceptable fit for the CFI and TLI are .90, and ideally above .95 (Bentler 1990). We next present the results for each respective construct.
Exploring the Factor Structures by Construct
As noted in the main body of this article, we took several preliminary steps in order to enable testing of the final model for between-group measurement invariance in order to compare the latent means and latent correlations between CI-supported youth and non-CI-supported youth (see Little 1997, 2013).
Spirituality
To explore spirituality, we tested 11 items pertaining to a factor of Transcendence from the Measurement of Diverse Adolescent Spirituality (MDAS; King et al. 2016) using a CFA. The model tested on the full sample indicated poor fit to the data: χ2 (44) = 165.8460, p = .000; RMSEA = .056 (90% CI .047–.065); CFI = .871; TLI = .838; SRMR = .049. The poor fit persisted across the subgroups present within the data set (see Table 4 for model fit statistics for the full sample, the CI-supported group, and the non-CI-supported group, for each purported construct, respectively).
Table 4 Model fit statistics for the initial confirmatory factor analyses of the purported constructs of spirituality, hopeful future expectations, and the five Cs of positive youth development, respectively Accordingly, we conducted a series of exploratory factor analyses (EFAs) to determine the best model for the data. For the EFAs, Geomin rotation (Mplus default) was used for a parsimonious factor pattern matrix. A combination of criteria was used to determine the adequate number of factors to retain, including a scree plot (Cattell 1966), parallel analysis (Horn 1965), and the above-noted multiple goodness of fit indices as recommended by Brown (2006). Out of necessity for fitting the EFA models, we had to ignore nesting.
Tested on the full sample, the scree plot and parallel analysis suggested a two-factor solution. We fit an initial series of EFA models ranging between one and four factors, and the suggested two-factor solution provided good fit: χ2 (34) = 95.07, p = .000; RMSEA = .045 (90% CI .034–.056); CFI = .958; TLI = .931; SRMR = .028. MDAS Items 1 and 2 did not load strongly on either factor. The remaining nine items formed two factors related to transcendent experiences or awe of God (MDAS Items 3, 4, 5, and 6), and to adherence or fidelity to spiritual or religious beliefs (MDAS Items 7, 8, 9, 10, and 11). Accordingly, we refer to these factors as Transcendence and Fidelity.
To test the robustness of these findings, we then attempted to replicate the analyses across pertinent subgroups present within the sample, namely, CI-supported youth and non-CI-supported youth (see Duncan et al. 2014). For CI-supported youth, the scree plot and parallel analysis eigenvalues suggested a two-factor solution, which provided the best fit to the data: χ2 (34) = 77.54, p = .000; RMSEA = .054 (90% CI .038–.070); CFI = .945; TLI = .910; SRMR = .035. The factor loading patterns were consistent with the findings from the EFA tested on the full sample. Again, MDAS Items 1 and 2 did not load strongly on either factor.
For non-CI-supported youth, the screen plot and parallel analysis suggested a one-factor solution; however, the two-factor solution remained the better fit to the data: χ2 (34) = 74.57, p = .000; RMSEA = .052 (90% CI .036–.068); CFI = .941; TLI = .904; SRMR = .035. The factor loading patterns were not consistent with the patterns found in the CI-supported group. Whereas MDAS Items 1 and 2 did not load strongly on either factor, in the non-CI-supported sample, neither did MDAS Items 3, 8, 9, 10, or 11. One factor consisted of MDAS Items 4, 5, and 6; and the other factor only included MDAS Item 7.
After examining these initial results, we first removed two items (MDAS Items 1 and 2) that did not load strongly on any factor in any group. We then attempted to replicate the above analyses, first on the full sample, and then on the CI-supported and non-CI-supported groups. The EFA of the remaining nine items tested on the full sample suggested a two-factor solution based on the scree plot and parallel analysis eigenvalues. The two-factor solution provided excellent fit: χ2 (19) = 35.94, p = .011; RMSEA = .032 (90% CI .015–.047); CFI = .986; TLI = .974; SRMR = .018. The two factors remained the same as previous findings: four items related to Transcendence; and five items related to Fidelity.
For the CI-supported group, the EFA with the nine items also suggested a two-factor solution based on the scree plot and parallel analysis eigenvalues. The two-factor solution demonstrated good fit: χ2 (19) = 43.63, p = .001; RMSEA = .054 (90% CI .033–.075); CFI = .965; TLI = .934; SRMR = .029. For the non-CI-supported group, the scree plot suggested a two-factor solution; but the parallel analysis suggested one factor. The two-factor solution provided the best fit to the data: χ2 (19) = 38.03, p = .006; RMSEA = .047 (90% CI .025–.069); CFI = .966; TLI = .936; SRMR = .028. However, MDAS Item 9 did not load on the Transcendence factor as it did within the full sample and the CI-supported group; instead, it loaded significantly on the Fidelity factor. In addition, MDAS Item 11 loaded strongly on both factors.
Based on these findings, we then removed MDAS Item 9 from the solution, as it loaded discrepantly across groups, and tested a CFA of the two-factor solution. Tested on the full sample, the scree plot of the remaining eight items (MDAS Items 3, 4, 5, 6, 7, 8, 10, and 11) and the parallel analysis suggested a two-factor solution. The EFA two-factor solution indicated excellent fit to the data: χ2 (13) = 21.02, p = .073; RMSEA = .026 (90% CI .000–.046); CFI = .993; TLI = .984; SRMR = .015. Geomin rotated factor loadings ranged from .43 to .62 across the two factors. The latent factors were significantly correlated, r = .58, indicating the two factors were related but distinct constructs.
We then replicated the analyses across the CI-supported and non-CI-supported groups to test for robustness and found consistently good fit with the two-factor solution. Table 5 presents the model fit statistics for the full sample, the CI-supported group, and the non-CI-supported group, for each refined construct, respectively. For the CI-supported group: χ2 (13) = 23.39, p = .037; RMSEA = .023 (90% CI .010–.070); CFI = .983; TLI = .964; SRMR = .023. Geomin rotated factor loadings ranged from .44 to .75 across the two factors. The latent factors were significantly correlated, r = .50, indicating the two factors were related but distinct constructs. For the non-CI-supported group: χ2 (13) = 24.53, p = .027; RMSEA = .045 (90% CI .015–.071); CFI = .977; TLI = .950; SRMR = .025. Geomin rotated factor loadings ranged from .32 to .74 across the two factors. The latent factors were significantly correlated, r = .28, indicating the two factors were related but distinct constructs.
Table 5 Model fit statistics for the refined constructs of spirituality, hopeful future expectations, and the five Cs of positive youth development, respectively We thus decided to retain these eight items related to spirituality as a robust and parsimonious two-factor solution. One factor included MDAS Items 3, 4, 5, and 6: “I find meaning in life when I feel connected with God;” “I marvel in front of nature and God’s creation; “I feel God’s presence in my life;” and “I feel that there is someone bigger than me (God) that is concerned for me”. We thus termed this factor “Transcendence”. The second factor included MDAS Items 7, 8, 10, 11: “I try to incorporate my religion or spirituality in every aspect of my life;” “My spiritual beliefs define the way I see the world;” “I face the obstacles and problems in life when I think that my life is part of God’s plan;” and “Religion or spirituality is a big part of who I am”. We thus termed this factor “Fidelity”. In subsequent analyses we therefore specify Transcendence and Fidelity as two distinct aspects of spirituality in this sample (see also King et al. 2014).
Hopeful Future Expectations (HFE)
To explore the construct of HFE, we tested 12 items derived from the 4-H Study of Positive Youth Development (see Schmid et al. 2011) using a CFA. The model tested on the full sample indicated moderate to poor fit to the data: χ2 (54) = 158.03, p = .000; RMSEA = .047 (90% CI .038–.055); CFI = .926; TLI = .910; SRMR = .042. The poor fit persisted across the subgroups present within the data set (see Table 4 for model fit statistics for the full sample, the CI-supported group, and the non-CI-supported group, for each purported construct, respectively).
Accordingly, we then tested a series of EFA models on the 12 items related to HFE. The initial set of EFAs with 12 items tested on the full sample suggested a two-factor solution based on the scree plot and parallel analysis eigenvalues; however, a three-factor solution provided the best fit: χ2 (33) = 59.59, p = .003; RMSEA = .030 (90% CI .017–.042); CFI = .987; TLI = .974; SRMR = .018. The three factors were related to education (HFE Items 1 and 2), to financial success (HFE Items 3, 4, 6, and 7), and to happy life (HFE Items 5, 9, 10, 11, and 12). HFE Item 8 (“Do the things you would like to do”) did not load strongly on any factor.
For CI-supported youth, scree plot and parallel analysis suggested a possible two-factor solution; however, a three-factor solution appeared to provide the best fit to the data: χ2 (33) = 64.11, p = .001; RMSEA = .046 (90% CI .026–.063); CFI = .973; TLI = .947; SRMR = .026. The factor structure differed as compared to the full sample: whereas HFE Items 1 and 2 remained a factor related to education, the remaining items factored differently. HFE Items 3, 4, 5, 9, 10, 11, and 12 formed one factor, including items related to financial success and happy life; and HFE Items 6, 7, and 8 formed a factor related to financial or lifestyle independence.
For non-CI-supported youth, the scree plot suggested a two-factor solution; but the parallel analysis suggested one factor. The two-factor solution appeared to provide the best fit: χ2 (26) = 75.61, p = .002; RMSEA = .041 (90% CI .025–.056); CFI = .987; TLI = .974; SRMR = .018. The factors appeared to be related to education and financial success (HFE Items 1, 2, 3, 4, 7, and 8) and to happy life (HFE Items 5, 6, 9, 10, 11, and 12).
To refine the model for parsimony and robustness, we first removed three items: HFE Items 3, 4, and 8. HFE Item 8 did not load strongly on any factor when tested in the full sample and, as well, did not consistently load on any one factor in the by-group analyses. HFE Items 3 and 4 also did not load consistently on one factor across groups. Tested on the full sample, the scree plot and parallel analysis suggested a two-factor solution. The full-sample EFA of the nine items indicated a two-factor solution with good fit: χ2 (19) = 31.26, p = .038; RMSEA = .027 (90% CI .006–.043); CFI = .990; TLI = .981; SRMR = .018. The two factors were related to education (HFE Items 1 and 2) and happy life (HFE Items 5, 6, 7, 9, 10, 11, and 12).
When we tested for robustness across the CI-supported and non-CI-supported groups, the factor structure remained consistent, except for HFE Item 7 (“Buy the things you need”) which had a strong factor loading (> .30) on both factors in the non-CI group. We thus decided to remove HFE Item 7 from the final model. In addition, although the education items (HFE Items 1 and 2) consistently formed a strong factor in prior analyses, we decided to also remove those two items, in order to have a parsimonious and robust one-factor model to represent HFE.
An EFA of the remaining six items (HFE Items 5, 6, 9, 10, 11, and 12) suggested a one-factor solution based on the scree plot and parallel analysis eigenvalues. The one-factor solution in the full sample indicated excellent fit to the data: χ2 (9) = 4.80, p = .852; RMSEA = .000 (90% CI .000–.021); CFI = 1.000; TLI = 1.009; SRMR = .010. Geomin rotated factor loadings ranged from .47 to .65.
To test the robustness of these findings, we replicated the analyses across the CI-supported and non-CI-supported groups and found consistently good fit with the one-factor solution (see Table 5). For the CI group: χ2 (9) = 10.84, p = .287; RMSEA = .022 (90% CI .000–.060); CFI = .996; TLI = .993; SRMR = .019. Geomin rotated factor loadings ranged from .48 to .65. For the non-CI group: χ2 (9) = 9.75, p = .371; RMSEA = .014 (90% CI .000–.056); CFI = .998; TLI = .996; SRMR = .021. Geomin rotated factor loadings ranged from .45 to .64. We thus decided to retain these six items related to HFE as a robust and parsimonious one-factor solution.
Positive Youth Development (PYD)
We first conducted a CFA of the Five Cs (Competence, Confidence, Character, Caring, and Connection) of PYD. Within each factor, errors were allowed to correlate for related items (e.g., within the Connection factor, the two items that pertained to peers were allowed to correlate). A total of 14 pairs of same-facet items were allowed to correlate a priori. The model tested on the full sample indicated moderate to poor fit to the data: χ2 (503) = 951.27, p = .000; RMSEA = .032 (90% CI .029–.035); CFI = .909; TLI = .899; SRMR = .048. The poor fit persisted across the subgroups present within the data set (see Table 4 for model fit statistics for the full sample, the CI-supported group, and the non-CI-supported group, for each purported construct, respectively).
After examining initial results, six items related to three factors were removed—the two items related to physical competence within the Competence subscale; the two items related to physical appearance within the Confidence subscale; and the two items related to conduct behavior within the Character subscale. We removed these items based on modification indices (e.g., PYD Item 22 indicated cross-loadings with Chi square reduction values ranging from 78.62 to 106.67) and based on previous findings from Geldhof and colleagues (2014) in which physical competence items, physical appearance items, and conduct behavior items were found to be problematic in the PYD model.
A CFA of the remaining 28 items tested on the full sample demonstrated good fit: χ2 (329) = 505.38, p = .000; RMSEA = .025 (90% CI .020–.029); CFI = .958; TLI = .952; SRMR = .035. However, the latent variable covariance matrix (psi) was not positive definite, and the warning message indicated a program with the latent variable Connection. The warning may have been due to high correlations between the latent variables of Connection and Competence (r = .87) as well as between Competence and Confidence (r = .92).
To address the psi matrix problem and to test for robustness, we then attempted to replicate the CFA of the 28 items on the two groups (see Table 5). For the CI-supported group, the model provided good fit: χ2 (329) = 480.73, p = .000; RMSEA = .032 (90% CI .026–.038); CFI = .936; TLI = .926; SRMR = .048. However, the latent variable covariance matrix (psi) was not positive definite, and the warning message indicated a problem with the latent variable Confidence. There were high correlations between the latent variables of Confidence and Competence (r = 1.22) as well as between Connection and Competence (r = .93). For the non-CI-supported group, the model provided good fit and had no warning message regarding the psi matrix: χ2 (329) = 393.32, p = .009; RMSEA = .021 (90% CI .011–.028); CFI = .968; TLI = .964; SRMR = .041.
The by-group analyses revealed that psi matrix problems were present in the CI-supported group and not in the non-CI-supported group. We used the Wald test of parameter estimates (MODEL TEST in Mplus) to compare the correlation between Confidence and Competence across groups, which indicated that the relation did indeed significantly differ across groups (Wald test value = 5.58, p = .018). Therefore, to accommodate the apparent collinearity between Confidence and Competence in the CI-supported group (as demonstrated by the correlation greater than 1.00), we combined the factors in that group by fixing the latent correlation to 1.00 and equating the relevant elements in psi. The resulting multiple-group CFA provided good fit: χ2 (662) = 880.87, p = .000; RMSEA = .027 (90% CI .022–.032); CFI = .950; TLI = .943; SRMR = .045.