Background

It is estimated that there will be more than 1,700,000 new cases of cancer in the U.S. in 2018 (1). Many survivors will receive state-of-the-science biomedical treatment, but will not receive care that adequately assesses or addresses their psychosocial concerns (2). This is problematic in part because distress and unmet psychosocial needs can have adverse implications for health outcomes, health service use, treatment adherence, and health care costs (3, 4). Further, there is research to suggest that treating and addressing cancer patients’ distress can lead to improved overall health and reduced health care costs (5, 6).

Given the immediate and long-range implications of distress among persons with cancer, the Institute of Medicine (7) and the National Comprehensive Cancer Network (8) have recognized that screening, referral, and follow-up for psychosocial concerns are critical to ensuring quality cancer care for the whole patient. Further, the Commission on Cancer of the American College of Surgeons released accreditation standards requiring practices to screen cancer patients for psychosocial distress (9), and the American Society of Clinical Oncology recommends regular screening of adults with cancer for depression and anxiety (10). Providing patients and clinicians with a distress screening and referral program that is comprehensive, easily accessible (i.e., by Internet), and patient-centered in terms of resource/referral preferences can offer benefit to patients, and potentially, the broader system of care. A number of validated measures exist to assess distress (11,12,13) and identify areas of unmet need (14, 15). Among those tools that assess patient preference for help, few ask the patient to delineate the type of help they wish to receive or have the ability to integrate across electronic systems via a web-based platform. Even fewer were developed and validated for use in community-based care settings, despite the fact that a majority of cancer patients are treated in community cancer centers, hospitals, and private practices (16).

The Cancer Support Community (CSC) developed CancerSupportSource® (CSS), a web-based psychosocial distress screening, follow-up, and referral program to assist community-based cancer centers in meeting distress screening accreditation standards linked to patient-centered cancer care (17), and to facilitate access to and provision of psychosocial care to cancer patients in the community. The initial theoretical framework and item content of CSS are grounded in the critical domains of psychosocial, practical, and physical needs (7, 18). Further, the design of the screening program reflects CSC’s model of patient empowerment (19). In this model, patients are supported and encouraged to identify their needs and play an active role in partnering with their health care providers, make important lifestyle changes, and enhance their ability to access information and resources that can address their needs.

Preliminary development of CSS has been previously described (17). Scale content was informed by patient focus groups and cognitive interviews, as well as expert input; initial psychometric validation was conducted among CSC’s community-based affiliate sites with a convenience sample of 251 cancer patients. In completing CSS, respondents are asked to rate their level of concern for 25 items, and in its full implementation, respondents also indicate desired help (additional information or talk to a staff member) and are provided tailored, automated referrals to support resources and follow-up care. The preliminary scale demonstrated high internal consistency reliability (Cronbach’s α = .92) and moderate to strong concurrent validity via associations with the FACT-G (r = − .70, p < .001), National Comprehensive Cancer Network (NCCN) distress thermometer (r = .62, p < .001), and CES-D (r = .69, p < .001). Items most highly correlated with the CES-D included feeling sad or depressed (r = .72, p < .001) and feeling lonely or isolated (r = .72, p < .001); these associations informed early conceptualization of a depression risk screening subscale. In the initial validation, items most frequently endorsed for additional follow-up with staff included worrying about the future and what lies ahead, feeling sad or depressed, and eating and nutrition. Intraclass correlation coefficients (ICCs) were used to examine test–retest reliability with a time interval between administrations of approximately 2 hours; items with ICCs < .75 (n = 9) were revised to assist with clarity and interpretability, resulting in a revised 25-item version of CSS.

The aims of the current study were to (1) examine the psychometric properties and multi-dimensionality of a revised 25-item version of CSS and (2) examine the scale’s ability to identify individuals at risk for clinically significant levels of depression and anxiety.

Methods and results

From March 2013 to December 2017, validation of CSS was completed in two phases: (1) exploratory factor analysis and measure validation and (2) confirmatory factor analysis and test–retest reliability.

Phase I—Exploratory Factor Analysis and measure validation

Participants

Data were collected through the Cancer Support Community’s Cancer Experience Registry® (CER), an online, community-based research initiative examining the social and emotional impact of cancer. Recruitment for the registry occurred through CSC’s network of community-based affiliates/chapters, online communities, CSC’s toll-free Cancer Support Helpline®, other cancer advocacy organizations, and social media. Ethical and Independent Review Services (E&I, Independence, MO) served as the IRB of record. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the registry. Eligibility to participate in the CER includes being 18 years of age or older and having ever been diagnosed with cancer. Potential participants for phase I of the current study participated in the CER survey from March 2013 to December 2016 (n = 4668).

Procedures

Participants provided demographic information (age, gender, race, ethnicity, education, employment status, and household income). Self-reported cancer diagnosis, stage at diagnosis, and time since diagnosis and treatments received were also obtained. We excluded survey participants living outside the U.S. (n = 288) and those missing > 50% of CSS distress items (< 1% excluded); those included also completed at least one comparative measure scale, resulting in a total analytic sample size of 1436 (Table 1). Compared to the sample of registrants who provided basic background information but did not complete the full length of the survey (n = 2944), the analytic sample for this study (n = 1436) was more likely to be male (28% vs. 17%, p < .001) and older (mean age = 58 years vs. 55 years, p < .001).

Table 1 Descriptive characteristics of the sample
Full size table

Measures

CancerSupportSource

Cancer-related distress was assessed using the revised 25-item version of CancerSupportSource (CSS-25). Patients rated their level of concern (0 = not at all, 1 = slightly, 2 = moderately, 3 = seriously, 4 = very seriously) for each item plus one additional exploratory item assessing concerns about thinking clearly; request for follow-up services was not assessed for the current study. Completion of CSS takes approximately 5 to 10 min. A total distress score was calculated as the sum of item ratings. For those participants missing CSS responses, the mean of available items for each individual was used to impute an overall score. The square root of the total distress score was approximately normally distributed. A 2-item depression risk score was calculated by summing two items (feeling sad or depressed; feeling lonely or isolated); a 2-item anxiety risk score was calculated by summing two items (feeling nervous or afraid; worrying about the future and what lies ahead).

PROMIS-29

Participant self-reported symptoms and functioning were examined using the Patient-Reported Outcomes Measurement Information System—29 (PROMIS-29 v2.0), a collection of 4-item short forms plus one item assessing pain intensity (20). Five domains assess symptoms with higher scores corresponding to worse symptomatology (depression, anxiety, pain interference, fatigue, sleep disturbance) and two assess function with lower scores corresponding to worse functioning (physical function, ability to participate in social roles and activities). Participants rate each item with reference to the past 7 days; function scales have no timeframe specified. Scale scores are converted to standardized T scores (mean = 50, SD = 10); normative reference groups are the U.S. general population, except sleep disturbance, where the reference is based on the U.S. general population and a clinical sample which was generally more enriched for chronic illness. In this case, a score of 50 likely represents somewhat sicker people than the general population.

General health

Participants rated their perceived level of general health (1 = poor; 5 = excellent) (21).

Analysis

Data analysis was conducted using Stata 14.2 (22) and R 3.4.0 (23), with GPArotation (24) and psych (25) R packages. Descriptive statistics were calculated for socio-demographic, disease, distress, general health, and patient-reported outcome variables. One sample t tests of means were used to compare means for PROMIS scales to the national average of T = 50 for adults in the U.S. Exploratory factor analysis with direct oblique rotation and principal axis factoring (PAF) extraction was used to analyze patterns in the inter-item correlation matrix and to assess the dimensionality (number of factors) needed to represent the variability in the data. Both parallel analysis and Cattell’s scree test, as well as examination of fit indices, were performed in order to determine the number of optimal factors for the scale (26). We used factor loadings to indicate the relevance of the variables to each factor. Items were considered to cross-load if the item had a factor loading > 0.4 on more than one factor, and the difference between factor loadings was < 0.2. Items that cross-loaded were not included in the final factor structure. Internal consistency reliability was estimated using Cronbach’s alpha coefficient (27, 28). Nonparametric receiver operating characteristic (ROC) curve analyses were used to determine scoring thresholds for depression and anxiety risk scales, using PROMIS depression (T ≥ 60) and anxiety (T ≥ 62) scales as criterion scores (29, 30) as these correspond to conventional cutoffs for clinical risk significance using the PHQ-9 and GAD-7 legacy instruments (31,32,33). We assessed convergent validity with Pearson correlation coefficients (34). Discriminant validity was determined using the known-group validation method; two sample t tests were used to verify whether summary scores could differentiate according to demographic and clinical variables: gender, currently receiving treatment for cancer, and time since diagnosis. Cohen’s d was used to estimate effect size between groups using the square root of the distress score.

Results

Clinical description of the sample

Participant socio-demographics

Participants were predominantly female (72%), White (90%), an average of 58 years of age (range = 19–87), and 4.6 years from their first cancer diagnosis (range = < 1 to 52 years; Table 1). The sample comprised a wide range of cancer patients and survivors representing various cancer diagnoses.

Cancer-related distress

The items of greatest distress (rated moderately to very seriously concerned) were the following: eating and nutrition (57.8%), exercising (50.0%), worrying about the future (47.6%), feeling too tired (46.9%), and health insurance or money worries (42.1%; Table 2). Of note, 20.3% of participants reported serious to very serious concern about thinking clearly; this new CSS item was selected to replace an infrequently endorsed item (9.1% indicated serious or very serious concern) assessing interest in complementary and alternative treatments, and was also used in calculating CSS total distress score for the 25-item scale.

Table 2 Factor loadings and communalities based on exploratory factor analysis
Full size table

Patient-reported outcomes

Among the PROMIS-29 scales (mean ± SD, p-value; % scoring > 1SD than reference group) (35), participants reported poorer quality of life than the general U.S. population with respect to fatigue (53.6 ± 11.4, p < .001, n = 1398; 30.3%), anxiety (53.4 ± 10.5, p < .001, n = 1407; 28.7%), pain interference (52.2 ± 10.0, p < .001, n = 1158, 24.3%), physical function (45.7 ± 9.0, p < .001, n = 1417; 29.6%), and ability to participate in social roles (48.8 ± 10.2, p < .001, n = 1160; 18.8%). Participants also reported worse sleep disturbance (52.0 ± 8.3, p < .001, n = 1375; 13.2%) than a reference group that included both the general U.S. population and a clinical sample. Average depression scores were not significantly different from the general U.S. population (50.4 ± 9.6, p = .12, n = 1413, 19.2%).

General health

Mean (SD) rating for general health was 3.18 (1.00; n = 1138); 9.0% indicated their general health was excellent and 24.3% indicated fair or poor.

Psychometric analysis of CSS-25

Exploratory factor analysis

Examination of scree plot and parallel analysis, as well as fit indices for a range of models (three to seven factors), supported a five-factor solution that explained 52% of the total variance in the 25 distress items and produced the strongest model statistics (SRMR = 0.02; RMSEA = 0.06; see Table 2). The first factor explained 18% of the variance in distress and included eight items assessing emotional well-being. The second factor explained 13% of the variance and included eight items assessing symptom burden and impact. The remaining factors included items assessing body image and healthy lifestyle, health care team communication, and relationships and intimacy. Tobacco or substance use did not have a factor loading > 0.30 on any of the five factors.

Internal consistency reliability

Internal consistency reliability of the total scale score and factors were evaluated using Cronbach’s alpha (36) (Table 3). The factors demonstrated moderate to large inter-correlations, but were not redundant. Cronbach’s alpha for the full 25-item scale was .94.

Table 3 Scale and factor descriptive characteristics, inter-correlations, and internal consistency reliability
Full size table

Receiver operating characteristic curve analysis

Using a PROMIS depression score of ≥ 60 to indicate risk for clinical levels of depression, a score of ≥ 3 on the 2-item CSS depression risk scale yielded a sensitivity and specificity of 91.4% and 79.5%, respectively (AUC = .923; Table 4). Using a PROMIS anxiety score of ≥ 62 to indicate risk for clinical levels of anxiety, a score of ≥ 3 on the 2-item CSS anxiety scale yielded a sensitivity and specificity of 91.8% and 70.9%, respectively (AUC = .903). Based on a cutoff score of 3 for each CSS risk scale, 33.9% of participants were at risk for clinically significant levels of depression and 42.9% at risk for clinically significant levels of anxiety.

Table 4 Percentages for sensitivity and specificity for CancerSupportSource depression and anxiety risk scales
Full size table

Convergent validity

The mean (SD) total CSS distress score for 25 items was 27.9 (19.1; median = 25; range 0–99). Total distress was moderately to strongly correlated with PROMIS-29 scales (rs -.67 to .70, ps < .001). The CSS depression risk scale was strongly correlated with PROMIS depression (r = .79, p < .001), and the CSS anxiety risk scale with PROMIS anxiety (r = .74, p < .001; Table 5). All five CSS factors were associated with PROMIS scales, with stronger associations exhibited between similar domains. The first factor encompassing emotional well-being was most highly correlated with PROMIS depression and anxiety scales (rs = .77 and .76, ps < .001). The second factor encompassing concerns about symptom burden and impact was most highly correlated with PROMIS fatigue, pain interference, and physical and social function scales (rs − .74 to .72, ps < .001). The third factor, body image and healthy lifestyle, was most highly correlated with PROMIS fatigue and social function scales (rs = .49 and − .49, ps < .001). The fourth factor, health care team communication, was most strongly associated with the PROMIS anxiety scale (r = .45, p < .001), and the fifth, relationships and intimacy, with the PROMIS depression scale (r = .42, p < .001).

Table 5 Pearson correlations between CSS, PROMIS validation measures, and general health ratings
Full size table

Discriminant validity

Several group comparisons supported known-group validity. The CSS total distress score was significantly (t = 7.37, p < .001) higher among those in active cancer treatment (n = 770) than among those who were not (n = 653; Cohen’s d = 0.39) and among those who were within 5 years of their cancer diagnosis (n = 922) than among those who were beyond 5 years (n = 508; t = 5.79, p < .001; Cohen’s d = 0.32), consistent with a small to medium value of d. Female participants reported more distress than male participants (Cohen’s d = 0.21; t = − 3.60, p < .001). The square root of the total distress score was inversely associated with age (r = − .30, p < .001, n = 1309) and time since diagnosis (r = − .13, p < .001, n = 1430).

Phase II—Confirmatory Factor Analysis and test–retest reliability

Participants

Participant eligibility was the same as phase I of the study (see above). From January to December 2017, a separate sample of 1457 cancer patients and survivors participated in the survey.

Procedures and measures

Procedures and measures were identical to phase I of the study. We excluded survey participants living outside the US (n = 28) and limited the sample to those who answered at least 50% of the distress items and completed at least one quality of life scale, resulting in a total analytic sample size of 1167 (Table 1). Compared to the sample of registrants who provided basic background information but did not complete the full length of the survey (n = 290), the analytic sample for this phase of the study (n = 1167) was more likely to be White (84% vs. 65%, p < .01) and, on average, older (58.4 vs. 55.7 years p < .01). Additionally, a subset of participants (n = 249) was asked to complete the revised 25-item CancerSupportSource a second time at the end of the survey to examine test–retest reliability with a time interval between test administrations of approximately 30 to 90 min. CSS was administered before and after multiple questionnaires in order to reduce the effect of memory. The socio-demographic and clinical history variables for this subset of participants did not differ significantly from other phase II participants.

Analyses

Confirmatory factor analysis was conducted with maximum likelihood factor extraction and listwise deletion of missing data. The factor loadings for the first indicator in each factor were fixed to 1.0, and we allowed for correlations between factors, using a target matrix for our rotation method with the highest loading factors from the exploratory factor analysis (EFA) specified (37). Both absolute fit indices and relative fit indices were used to measure goodness of fit (38). Test–retest reliability was measured with intraclass correlation coefficients.

Results

The five-factor model explained 53% of the variance and again demonstrated good fit (RMSEA = .067; SRMR = .042; CFI = .922; χ2(242) = 1132.59, p < .001). All items demonstrated stability, with equivalent or higher loadings on their respective factors. Full scale test–retest reliability was .91, while all individual scales ICCs were ≥ .78, exceeding the threshold of .75 for excellent (39) test–retest reliability (emotional well-being = .91; symptom burden and impact = .91; relationships and intimacy = .85; health care team communication = .79; body image and healthy lifestyle = .78).

Discussion

CancerSupportSource (CSS) is a reliable, valid, multi-dimensional distress screening program with the capacity to identify those at risk for significant anxiety and depression, known indicators of poor health outcomes (40,41,42,43). The current study findings provide the psychometric foundation to support the use of CSS in community-based and hospital-based oncology settings, and CSS is currently implemented across a network of community-based cancer support facilities (Cancer Support Community, Gilda’s Club) as well as oncology practices and hospital cancer centers nationwide. The tool demonstrates strong internal consistency reliability and test–retest reliability, a factor structure that is replicable, and adequate convergent and divergent validity. Additionally, the depression and anxiety risk scales demonstrate high sensitivity coupled with adequate specificity. When implemented in oncology practice, this tool fulfills the American College of Surgeons Commission on Cancer patient-centered standards for distress screening (9), NCCN Guidelines for distress management (8), and the American Society of Clinical Oncology (ASCO) Quality Oncology Practice Initiative (QOPI) certification standards (44).The strengths of this study include that it is a community-based research program, currently validated across a broad sample of survivors representing diverse cancer care settings and geographic regions. Further, our results support that CSS is a relatively brief but multi-dimensional tool, which can assist providers in developing a personalized, integrated care plan. Additionally, our results provide psychometric support for the inclusion of depression and anxiety risk scales. These scales provide a brief screening of individuals at risk for clinically significant levels of depression and anxiety, without additional administration burden to patients or providers, an important consideration given that distress screening uptake continues to be a challenge across cancer care (45). Integrated distress and risk screening can be beneficial in community-based settings in particular by facilitating a first step in triaging patients to appropriate levels of care, as determined by each care system in which it is implemented.

Limitations of the current study include self-selected samples of participants who have Internet access, are predominantly female, White, fairly educated, and include a substantial proportion of individuals with breast and hematologic cancers. These limitations impact the study results’ generalizability to a more diverse socio-economic population. This sample is not representative of all cancer patients and survivors across the U.S.; however, it is a representative of those who seek social and emotional support in the community. Future work is required to understand the effectiveness of implementing the tool and to evaluate its applicability and cultural sensitivity among other diverse samples and settings. Additionally, the time interval for test–retest reliability was relatively short, although CSS was administered after multiple questionnaires in order to reduce the effect of memory. Internal consistency reliability for two factors was .60 < α < .70, which may in part reflect the limited number of items that loaded on these constructs; however, these factors were retained due to their relevance to needs assessment for cancer patients and survivors. The depression and anxiety risk cutoff scores were developed in comparison to the PROMIS-29; future evaluation and testing of these risk scales will consider integration of structured clinical interviews. The current study did not examine participants’ desired follow-up on items (e.g., information or talking to a staff member); these will be explored in future implementation studies.

Distress screening is the necessary first step in a comprehensive assessment of the patient experience, functioning, and unmet needs related to emotional well-being, symptom burden and impact, body image and healthy lifestyle, health care team communication, and relationships and intimacy. This study focuses on the validation and psychometric evaluation of CancerSupportSource and its content. Future work will focus on implementation of distress screening, uptake of referrals and follow-up, and the impact of distress screening on quality- and cost-related outcomes in diverse settings. Particularly in settings where high socio-economic need exists and health and support services are remote, efforts are needed to evaluate innovative approaches that leverage technology and community resources to deliver tailored interventions that meet patient and family needs. Such efforts will support the overarching goal of examining a stepped care approach to delivering psychosocial care tailored to the individualized needs of cancer survivors over time.

Acknowledgements

Support for the Cancer Experience Registry was provided by AbbVie, Inc.; Amgen Oncology; Astellas Pharma US, Inc.; AstraZeneca; Bayer HealthCare; Boehringer Ingelheim; Bristol-Myers Squibb; Celgene Corporation; Genentech, Inc.; Janssen Oncology; Jazz Pharmaceuticals; Lilly Oncology; Novartis; Pfizer Oncology; Pharmacyclics, Inc.; and Takeda Oncology.