Background

The World Health Organization estimates that 55 million people globally have dementia, with the number expected to rise to 78 million within the next decade [1]. They also note that dementia tends to be under-diagnosed and the diagnosis often comes relatively late in the person’s disease trajectory [2]. Early detection of changes to a person’s cognitive performance (i.e., changes to a person’s function that are indicators of and associated with cognitive impairment) that may suggest cognitive impairment (CI) is key for several reasons. Early detection of cognitive impairment enables clinicians to identify and treat some modifiable contributors to cognitive changes, such as delirium, increased pressure or bleeding in the brain, vitamin deficiencies, or depression [3]. CI is a known risk factor for several negative outcomes including caregiver burden or distress [4, 5], repeat visits to the emergency department [6], and admission to a long-term care (LTC) facility or nursing home [7,8,9,10,11]. It also is important for clinicians to identify and to track changes to cognitive performance over time, because of its association with CI, and since some evidence suggests that mild CI is an intermediate step in the development of dementia [12, 13]. Furthermore, despite the lack of disease altering treatments for dementia, it is important to identify its onset and to provide symptom altering treatment options as early as possible [14]. Early detection of people who may be at risk for cognitive decline is critical therefore to enable timely intervention, which could delay disease development or progression [15]. Information about risks for decline in cognitive performance could be used to identify non-pharmacological approaches to address critical risk factors for decline such as diet, physical inactivity, obesity, hearing loss, and social isolation [16], among those who are and are not diagnosed with a disease that causes CI.

A simple but robust method is needed to assist clinicians in identifying individuals who are at risk of a deterioration in their cognitive performance. Similar types of risk prediction tools, or nomograms, have been used widely in cancer care [17,18,19]. However, very few exist for predicting changes in cognitive performance [20]. Unlike multivariable regression models that typically focus on exposure-outcome effect estimates, nomograms focus on estimating an individual’s predicted outcome probability based on their specific profile of characteristics.

Understanding the risk of deterioration in cognitive performance is particularly important within the home care sector. Roughly two million Canadians receive publicly-funded home care annually, and around 40% of them are aged 65 + [21]. It is also recognized that home care clients are generally more impaired in their cognitive functioning as compared to other older adults not receiving this type of care. Studies in home care report rates of CI ranging from 30% in Europe [22, 23], 27%-38% in Australia [24, 25], and 40%-60%in Canada [26,27,28,29,30], as compared to roughly 3% among, community-dwelling older persons not receiving home care [31].

The interRAI Cognitive Performance Scale (CPS) [32] was designed to be a functional measure and to act as a brief screen for impaired cognitive performance. Among Canadian home care recipients, roughly 20% would be expected to experience a decline in their CPS score over a one-year period [26]. Although it is not a diagnostic tool, some data have shown that the proportion of individuals diagnosed with dementia increases with each 1-point increase (i.e., worsening) on the CPS score [33]. Furthermore, a one-point change on the lower values of the CPS (e.g., those scoring a zero or one on the CPS) corresponds to a roughly 2.4- to 3.0-point difference on the Mini-Mental State Examination (MMSE) [32, 34]. These values exceed the 1.4-point change on the MMSE, suggested by Howard et al. [35], to represent the minimal clinically important difference. In a cognitively healthy cohort, changes on the MMSE over three months were roughly 0.35 to 0.69 [36]. All of this supports the use of the CPS in detecting a clinically relevant change over time.

The CPS is a hierarchical scale that includes ratings of two domains found on traditional cognitive assessments (e.g., difficulties in short-term memory, daily decision making) and two items reflecting functional status (e.g., expressive communication, independence in eating). The scale ranges from zero to six (0 = no impairment in cognitive functioning, 1 = borderline intact, 2 = mild impairment, 3 = moderate impairment, 4 = moderately severe impairment, 5 = severe impairment, and 6 = very severe impairment in cognitive functioning). The CPS is embedded within multiple clinical assessment tools developed by interRAI, a non-profit consortium of researchers, clinicians, and policy makers from roughly 37 countries. The items within the CPS have excellent inter-rater reliability within the LTC population (average kappa = 0.85, which measures the extent to which assessors assign the same score) [32] and good reliability in home care (average kappa = 0.65) [37]. In multiple studies, the CPS has demonstrated at least moderate correlation (values of 0.45 and higher) with performance on two cognitive screening measures, namely the Mini-Mental State Examination [32, 34, 38,39,40,41,42] and the Montreal Cognitive Assessment [43].

In this paper, we aimed to develop and validate a new nomogram, the risk of CPS decline (RCD). Specifically, this tool was created to estimate the predicted 6-month risk of a decline on the CPS among individuals with a baseline CPS score of zero or one. The long-term goal of this work would be to have the predicted probability, for an individual client, included in the outputs available to clinicians, similar to the other scales and algorithms that can be generated with the interRAI Home Care assessment. As part of the validation process, another objective was to explore the characteristics of clients in the highest risk versus the lowest risk groups. During our preliminary analysis, it was clear that individuals with a baseline CPS score of two or higher had a different risk profile, and warranted a unique nomogram. Those results will be reported in a separate manuscript.

Methods

Data source

We conducted secondary analysis of data collected using the Resident Assessment Instrument for Home Care (RAI-HC) across five provinces (British Columbia, Alberta, Manitoba, Ontario, Newfoundland and Labrador) and one territory (Yukon Territory) in Canada. The development and validation of the prediction tool took place between Nov. 2020 and July 2022. The RAI-HC is a standardized assessment that is used routinely for all home care clients expected to receive at least 60 days of service [44]. The assessment has established reliability and validity and contains roughly 300 items which capture key domains, including cognitive functioning, sensory impairments and functional ability [37]. Assessments are completed by trained care coordinators (typically registered nurses) through discussion with the individual, their informal care providers and other health care professionals, as needed. Re-assessments are typically completed every 6–12 months or following a change in health status [45]. Missing data are rare as the electronic assessment does not allow assessors to close an assessment when fields are left blank.

At the time of the analysis, two different interRAI home care instruments were available for use, namely, the RAI-HC and also the newer version of this instrument, the interRAI Home Care Assessment (or interRAI HC). These two assessments are very similar, but there are roughly 100 items within the interRAI HC which are not found on the RAI-HC assessment. Our goal was to create a nomogram that would be compatible with either version of the instrument. As a result, we excluded from consideration any item that was not available on the interRAI HC instrument. We opted to analyze existing RAI-HC data across several jurisdictions in Canada since this yielded the largest database available. At the time, very limited interRAI HC data were available, and only for the province of Ontario. The Research and Ethics Board at Wilfrid Laurier University reviewed and approved the design of this study (#6504).

Sample

The sample included all home care clients who were 18 years of age or older and who had at least two RAI-HC assessments completed between January 2010 and December 2018. Only individuals whose first assessment was an intake assessment were retained, and each individual had to have at least one additional assessment completed within six months following their baseline (intake) assessment. For the vast majority of clients (76.9%), the reason for the re-assessment was either a regular follow-up assessment or a routine assessment at a fixed interval. Home care clients had to be receiving home care throughout the entire duration of the six-month period to be included. Since the main objective of the current study was to develop a nomogram predicting any decline on the CPS (vs. no decline) within six months for individuals with a baseline CPS score of zero or one, only those individuals were included in the sample (n = 39,292). The choice to define our main outcome as dichotomous (any decline vs. no decline on the CPS) was an a priori clinical decision, not a statistical one.

Covariates

All characteristics were measured once at baseline. They included: demographic characteristics (age at intake, sex [male vs. female], marital status, caregiver relationship to client, disease diagnoses [stroke, congestive heart failure, coronary artery disease, Alzheimer’s dementia, another type of dementia, hemiplegia/hemiparesis, multiple sclerosis, Parkinson’s disease, any psychiatric diagnosis, hip fracture, other types of fractures, pneumonia, urinary tract infection, cancer, diabetes, chronic obstructive pulmonary disease], sudden/new onset change in mental function [each coded as yes or no]), sensory and communication challenges (hearing impairment [HI], vision impairment [VI], dual sensory impairment [DSI; yes or no], the ability to understand others), health conditions and responsive behaviours [each yes or no] (wandering, verbally abusive, physically abusive, socially inappropriate, resists care, chest pain, no bowel movements, dizziness or light-headedness, edema, shortness of breath, delusions, hallucinations, smoked/chewed tobacco).

Additionally, items around physical functioning and health status (client believes they are capable of increased functional independence [yes or no], number of falls [0, 1 or 2 +], unsteady gait, bladder incontinence, client believes they have poor health, client has condition or diseases that make cognition, activities of daily living [ADL]), mood or behaviour patterns unstable, flare-up of recurrent or chronic condition, prognosis of less than six months to live, difficulty swallowing, ate one or fewer meals in the last three days, unintended weight loss [each yes or no]), service utilization (hospital admissions, emergency department visits [both 0, 1, or 2 +], made economic trade-offs during the last month [yes or no] and social functioning [client indicates that they feel lonely, change in social activities causing distress; code as yes or no]) were all explored in the model.

Finally, seven health index scales/algorithms embedded within the RAI-HC which are automatically generated upon completion of the assessment also were explored. Across all scales, a higher value indicates a great degree of impairment.

  1. 1.

    The Activities of Daily Living (ADL) Self-Performance Hierarchy Scale includes four items, namely, bathing, dressing, toilet use, locomotion, and eating. It is scored from zero (no difficulty) to six (major difficulty), where a cut-point of two or higher was used to indicate at least moderate difficulty completing ADLs independently [46].

  2. 2.

    The Instrumental Activities of Daily Living (IADL) Involvement Scale is a summative scale across seven IADLs (meal preparation, housework, managing finances, managing medications, phone use, shopping, and transportation), which ranges from 0 to 21, where a cut-point of 14 or higher was used to indicate moderate difficulty completing these tasks. Both the ADL and IADL scales are valid and reliable measures of functional ability [46].

  3. 3.

    The Depression Rating Scale (DRS) includes seven items related to mood and behaviour. The scale ranges from 0 to 14 where a score of three or higher is predictive of a clinical diagnosis of depression [47].

  4. 4.

    The Pain Scale uses two items, one related to pain frequency and one related to intensity. It is measured from zero (no pain/less than daily pain) to four (daily/severe pain) and a cut-point of two or higher was used to indicate pain that was daily or severe. The scale has been validated against the vertical version of the Visual Analog Scale [48].

  5. 5.

    The Changes in Health, End-Stage Disease Signs and Symptoms (CHESS) Scale uses nine items including shortness of breath, vomiting, dehydration, and prognosis. It can range from zero to five. For every one-point increase on the scale, there is a nearly two-fold increased risk of mortality [49].

  6. 6.

    The Pressure Ulcer Risk Scale (PURS) is scored from zero to eight and groups clients into low, moderate, high, and very high risk of experiencing a pressure ulcer. It includes seven items such as bowel incontinence, weight loss, history of a resolved pressure ulcer, and impaired bed mobility [50].

  7. 7.

    The Caregiver Risk Evaluation (CaRE) is a decision-support tool that generates the risk of caregiver burden among informal caregivers. It contains six individual items or scores on the health index scales (e.g., CPS scale and DRS scale) and assigns caregivers into one of four groups, ranging from low risk (score of zero) to very high risk (score of four) of experiencing burden [7].

Analysis

Developing the prediction model

We randomly selected 75% of eligible home care clients for model derivation and used the remaining 25% for validation. To check how well random sampling produced equivalent groups, we compared the distributions of baseline characteristics between the derivation and validation cohorts. Using the derivation cohort, we used a multivariable logistic regression model to predict a decline on the CPS score within 6 months of an individual’s baseline assessment. Numerous variable selection techniques were initially explored for deriving a parsimonious model (e.g., backward, forward, and stepwise procedures), and these techniques showed consistent preliminary results. Similar to prior work, our final model focused on the backward selection procedure for variable selection [18, 19]. We chose a more liberal two-tailed alpha value of 0.10, to ensure that important interaction terms would not be neglected. In addition to the p-value, we examined the AIC and log-likelihood values from our series of models during the iterative model building process. Our final model had the lowest AIC and highest log-likelihood, while retaining main effects and interactions that were clinically meaningful. Continuous variables such as age were explored using both linear and quadratic terms. Missing data were only an issue when using the CaRE algorithm to categorize the risk of caregiver burden, as this measure is only calculated when all items in the algorithm are not missing. If an individual did not have a primary caregiver, then the value would be set to missing. Because there was no obvious missing pattern, we created a missing category for these individuals rather than impute or remove them from the analysis. As decided a priori, all two-way interactions with age and sex were explored, along with other two-way interactions with each of the three types of sensory impairments. For example, we explored two-way interactions with each of HI, VI and DSI with all of the following covariates: Alzheimer’s dementia or another type of dementia, Parkinson’s disease, any psychiatric diagnosis, sudden/new onset change in mental function, the CaRE algorithm, number of falls and ability to understand others.

Validating the prediction model

Once the optimal regression model for predicting CPS decline was developed using the derivation cohort, the validation cohort then was used to assess the performance of this model. Specifically, for each individual in the validation cohort, the estimated predicted probability of decline on the CPS score within 6 months was calculated based on their specific baseline covariate values and the corresponding regression coefficient (beta) estimates from the regression model. Calibration was examined by grouping individuals into deciles (or groups) of lowest to highest risk and then plotting the observed proportion of CPS decline within a decile against the corresponding mean predicted risk within that decile. Points closer to the 45-degree line indicate better calibration [18]. We examined the calibration plot overall, as well as examined the plots within various sub-groups (e.g., sex, baseline CPS score, HI, presence of any type of dementia, number of falls [0 vs. 1 +], ability to understand others [any difficulty vs. none]) in order to assess whether model calibration was different within these groups. The model’s discriminative ability (i.e., ability to discriminate between those who declined from those who did not decline) was measured via the area under the curve (AUC) statistic, where a value of 1.0 implies perfect discrimination and a value of 0.5 implies the model classifies no better than chance [18, 19]. All analyses were performed using SAS software, version 9.4 [51]. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis reporting guidelines [52].

Results

A total of 39,292 individuals had a baseline CPS score of zero or one between 2010 and 2018. The median age of the overall sample was 79.0 years (interquartile range = 62–96 years), 62.2% were female, 38.8% were married and 38.6% lived alone. Distributions of characteristics between the derivation and validation cohorts were nearly identical at the intake assessment (Table 1). On average, 30.3% experienced a worsening on the CPS score within 6 months following their baseline assessment, which was nearly identical between males and females (males: 30.9%; females: 30.3%). Most clients experienced a one-point decline (70.2%; from CPS zero to one or from CPS of one to two), 25.9% experienced a two-point decline, and the rest, a three-point worsening on the CPS (3.9%).

Table 1 Comparison of the derivation and validation cohorts across all baseline variables under consideration from the RAI-HC
Full size table

After utilizing a backward variable selection approach, a total of 34 main effects and 12 two-way interactions were included in the final risk prediction model. Of these 12 interactions, seven involved sex, including a significant interaction between age and sex. It was seen that the effect of females versus males increased with age. For example, for an individual who was 69 years of age, there was virtually no difference in the odds ratio comparing males and females (OR = 0.99). However, for an individual who was 79, the OR decreased to 0.94, and then decreased again, to 0.90 for someone who was 86.

A number of factors were associated with an increased risk of experiencing a decline on the CPS score of more than 10%, including age (5-year increments), baseline CPS score, IADL impairment, a diagnosis of Alzheimer’s dementia, another type of dementia, Parkinson’s disease and any psychiatric diagnoses, vision impairment, dual sensory impairment, the ability to understand others, wandering, verbally abusive behaviour, hallucinations, the number of falls, and self-reported loneliness (Table 2). The AUC value was 0.6576 in the derivation cohort.

Table 2 Estimates from the regression model (main effects onlya) following backwards elimination for the derivation cohort from the RAI-HC
Full size table

When assessing performance of the risk prediction model using the validation cohort, we found that the discrimination was good, with an area under the receiver-operating characteristics curve of 0.6516 (Fig. 1) and calibration was excellent (Fig. 2). Calibration plots, among the various sub-groups, revealed only minor deviations from the overall results (data not shown), indicating that there was virtually no change in model performance based on any of the characteristics that were explored (e.g., among men and women, the c statistic was 0.65 in both cases).

Fig. 1
figure 1

Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) for the validation cohort

Full size image
Fig. 2
figure 2

Calibration plot in the validation cohort. Dots represent each bin’s observed 6-month probability of CPS decline plotted against the 6-month predicted probability of CPS decline (among individuals in that bin)

Full size image

Table 3 provides a comparison of baseline characteristics among individuals in the lowest and highest risk prediction deciles from the 10-bin calibration plot.

Table 3 Comparison individuals in the lowest and highest risk deciles from the 10-bin calibration plot under the validation cohort
Full size table

Compared to the lowest risk prediction bin, individuals in the highest bin were more likely to be 85 + years of age (55.9% vs. 0.3%), widowed (40.5% vs. 10.0%) and have a baseline CPS score of one (75.6% vs. 9.3%). Additionally, the highest bin was also more likely to experience moderate/major difficulty completing IADLs (51.8% vs. 13.7%) and have caregivers at high risk of experiencing caregiver burden based on the CaRE algorithm (46.9% vs. 32.4%). In terms of disease diagnoses, those in the highest risk group were more likely to have a diagnosis of Alzheimer’s dementia or another type of dementia, Parkinson’s disease, any psychiatric diagnoses, and stroke. Experiencing two or more falls (32.1% vs. 1.8%), having bladder incontinence (53.0% vs. 21.8%) and self-reported loneliness (24.7% vs. 8.4%) were all more likely in the highest bin.

Conversely, individuals in the highest bin were less likely to experience severe/daily pain (11.2% vs. 25.6%) and to have a cancer diagnosis (7.6% vs. 39.5%). Additionally, those in the highest bin were less likely to experience edema (19.4% vs. 28.3%), feel as though they were capable of increased independence (14.2% vs. 48.0%), have a prognosis of less than six months to live (0.3% vs. 6.7%) and be admitted to the hospital at least once in the last 90 days (24.3% vs. 47.9%; Table 3).

Discussion

In this study, we developed and validated a novel prediction tool to mark decline on the CPS score that can be used for home care clients. By using a very large sample of clinical assessment data of home care clients from across Canada, we achieved good discrimination and excellent calibration, supporting the use of these data for risk prediction. While knowledge of the factors that increase risk are of interest, the strength of the nomogram is that it provides a unique risk score for an individual as a function of their own characteristics including age, various disease diagnoses, sensory and communication status deficits, as well as self-reported loneliness, and the risk of caregiver burden. To our knowledge the interRAI data have never been used to explore this particular outcome, although interRAI data have been used to create other prediction tools for those with cancer [18, 19].

There are a few features of our nomogram that are unique. For example, multiple prediction tools, or risk models, have been developed to understand the progression from mild CI to a dementia diagnosis [53,54,55,56,57,58,59], but we found no other studies that used a nomogram to predict the risk of cognitive performance decline. Given the breadth of the RAI-HC assessment, we were able to tap into multiple domain areas known to be associated with the risk of cognitive impairment or dementia. For example, in two recent reports from the Lancet Commission, 12 potentially modifiable risk factors were shown to account for roughly 40% of the risk of cognitive impairment or dementia [16, 60]. In our modelling, we were able to consider nine of these 12 risk factors, but were unable to look at traumatic brain injury, obesity, and air pollution, since they are not included on the RAI-HC assessment.

In our final model, individuals at highest risk for a decline on the CPS score were older, more likely to be widowed, to have difficulty completing IADLs independently, and to have care providers at high risk of experiencing caregiver burden. They were also more likely to have several diagnoses related to neurological and psychiatric conditions such as dementia. While some of these factors are non-modifiable (e.g., age and sex), some clearly are able to be addressed by the home care team. For example, caregiver burden is associated with the care recipient’s cognitive status [4, 5], and is a known risk factor for placement in a LTC facility or nursing home [9, 28, 61, 62]. Some recent studies suggest that supporting caregivers can lead to reduced or delayed LTC admissions [63, 64].

While the inclusion of variables related to a dementia diagnosis may seem counter-intuitive, we chose to keep them for a couple of reasons. For example, less than 5% of the sample (4.3%) had a dementia diagnosis. In addition, previous research reported that 12% of those with a CPS score of zero had a neurological diagnosis. The authors felt that this was because the assessor believed that the disease was present, but the person was not exhibiting symptoms, resulting in the low score on the CPS [32].

Our nomogram can support home care clinicians in this process as they engage in shared decision-making with the person and their family. However, like other health index scales and algorithms embedded within the interRAI tools, clinicians should use the information from this new prediction tool to augment, but not replace, their clinical judgement. Home care clinicians would ideally use this information, in conjunction with conversations with the person and their family, to ensure that decisions are tailored to best meet their needs.

Our risk prediction model utilized data from five Canadian provinces and one territory. This large cohort represented adult (aged 18 +) home care clients with a variety of illnesses and symptoms and in various stages of their illness trajectory. As a result, it may be generalizable to other publicly-funded home care systems. Although we found no other prediction tools for a decline on the CPS, several studies have created prediction tools for predicting a diagnosis of dementia. For example, two studies reported the c statistics across data analyzed from several existing cohort studies, ranging from 0.64 to 0.78 [57, 58]. The lower values in these studies are in line with our level of discrimination (c = 0.65). However, our model was not as good as two other studies, both of which used existing administrative or survey data, and had values of 0.74 [55] and 0.82 [54]. This likely reflects the added difficulty and complexity of predicting the risk of deterioration on the CPS score versus predicting a dementia diagnosis.

One limitation to our work is that the RAI-HC assessment included no information about biologic markers such as imaging data and serologic findings, which can be important in determining the risk of cognitive impairment or dementia [56, 59]. Since we explored changes on the CPS, individuals with a single assessment were necessarily excluded from the cohort. However, it is unclear how this may have biased the sample since several scenarios would be operating simultaneously within the six-month timeframe. For example, some home care clients would have a single assessment because they died or were admitted to a LTC facility. Others, however, would only have one assessment since they improved and were subsequently discharged from the home care program. Furthermore, the CPS is a functional screening tool and cannot be used for diagnostic purposes, although it has been noted that individuals with higher scores (i.e., three or higher on the CPS) are more likely to have a diagnosis of Alzheimer’s dementia versus those scoring two or lower [40].

Conclusions

Our model demonstrated that the risk of a decline on the CPS score can be accurately predicted using existing data from the RAI-HC. Using data from multiple domain areas showed the prediction tool tap into risks related not only to neurologic conditions, such as Alzheimer’s or a related dementia, but also the risks associated with sensory impairments, caregiver burden, and communication status. As a result, the tool possesses strong potential to provide clinicians with unique risk information for a given person. It can be used by them to guide further assessment, referrals, and help to support their clinical decision-making as they work with clients and families to navigate the health care system. Since the interRAI tools are widely used around the world, there is strong potential for this new nomogram to be utilized to generate information for use by home care professionals in multiple countries.