Introduction

Cognitive difficulties are common manifestations during the acute phase of stroke and can persist after a seemingly successful neurological recovery affecting the daily lives of people who had a stroke1,2. Cognitive deficits are linked with poor outcomes3,4; thus, early identification of stroke-related cognitive impairments is important.

Pinpointing the superior assessment tool for evaluating cognitive function after stroke is difficult5. The National Institute of Health Stroke Scale (NIHSS) is a routinely used instrument for the assessment of stroke-related neurological deficits, but limited sensitivity for detecting cognitive deficits has been shown6,7. However, Cumming et al.8 suggested that the NIHSS subscale, the Cog-4, could be used to make a statement about cognition8. The Cog-4 is estimated based on four items of the NIHSS - orientation, executive ability, language skills and extinction and inattention. The score range is 0–9 points, where 0 points indicates no cognitive deficits.

Conflicting results have been reported regarding the capacity of Cog-4 to identify cognitive deficits after stroke8,9,10,11. Some studies have reported that the Cog-4 cannot be considered a useful cognitive scale9,10. Another study indicated that the Cog-4 is almost as good as other commonly used screening tools for cognition in patients with severe cognitive deficits8. In these studies, the Cog-4 was compared with the Mini-Mental State Examination (MMSE)8, the Montreal Cognitive Assessment (MoCA)10, and follow-up assessment with the Cog-411. Assessments were performed within 1–4 days10, 90 days9, and 18 months8 after stroke. It still remains unclear whether the Cog-4 can identify cognitive difficulties after stroke, especially very early after the onset of stroke.

Cognitive performance can fluctuate at an early stage of stroke; therefore, a full neuropsychological assessment that is time consuming is seldom prioritized at early stage of stroke. However, it is still important to identify cognitive difficulties for timely treatment and rehabilitation planning. The aim of this study was to investigate whether admission Cog-4 has the potential to detect cognitive deficits by comparing it with the MoCA—a commonly used screening tool for cognition in acute stroke12,13.

Materials and Method

The Standards for Reporting Diagnostic Accuracy (STARD) statement was used as a guideline for reporting the study14.

Study design and participants

This was a cross-sectional, retrospective study. The study sample consisted of people with stroke who were enrolled at the acute comprehensive stroke unit in Gothenburg, Sweden between May 2011 and April 201615. The participants were screened for cognitive function and activities of daily living within the first two days after onset of stroke and included in the research database2. In total 2727 people were screened and 2474 received a stroke diagnosis. The complete data on the MoCA were available on 550 people2.

The inclusion criteria for the current study were stroke diagnosis, age of the participants >18 years, MoCA scores within 36–48 h after onset of stroke, and complete data on the NIHSS items. Patients with subarachnoid haemorrhage were excluded.

The Declaration of Helsinki was followed. The Data Inspection Board in Sweden states that data that are handled in quality registries are considered an exception to the general rule of requiring written informed consent to promote improvements in care and treatment, which is of general interest. Therefore, the current study did not obtain consent from the participants. Nevertheless, the participants were informed that their data could be used for research when their data were reported to the quality registers, and they had the right to withdraw their data at any time. The participants were anonymized and cannot be identified. The Regional Ethical Review Board in Gothenburg approved the study (042-11, amendment 2019-02882).

Procedure

Participants were screened within 36–48 h of admission to an acute stroke unit by clinical occupational therapists (OT) who administered the MoCA. The MoCA is a common assessment instrument used at the stroke unit by the OTs. The OTs working at the stroke unit have attended workshops on a regular basis regarding cognitive functions after stroke. They have also participated in peer discussions about the MoCA assessment as well as scoring. Patient medical charts were used to extract information about stroke-related neurological deficits upon admittance to the hospital (NIHSS, administered by trained stroke physicians at the stroke unit), activity status prior to stroke, comorbidities, risk factors for stroke, reperfusion treatment, and type of stroke.

Assessment instruments

The index test, the Cog-48, was calculated based on the following four items from the admission NIHSS: orientation – person’s awareness of current month and own age; executive functions–ability to follow a 2-step command; language skills–evaluator’s judgement about impaired language skills; and extinction and inattention (Supplementary Table S1). The Cog-4 scores can range from 0 to 9 points (p), and “0” indicates no problems8.

The reference standard test, the MoCA12, was used for assessment of cognitive functions. The MoCA assesses 6 cognitive domains: short-term memory, visuospatial abilities, executive functions, attention and working memory, language, and orientation to time and space12 (Supplementary Table S1). The score range on the MoCA is 0 to 30; the cutoff value of ≥26 indicates normal cognitive functioning12.

Stroke-related neurological deficits were assessed with the NIHSS16. The total NIHSS scores can range from 0 to 42; a higher score indicates more severe neurological impairment.

Activities of daily living (ADL) were assessed with the Barthel Index (BI)17. The total score range on the BI is 0 to 100, and higher scores indicate higher levels of ADL independence.

Ischaemic stroke was classified according to the Oxfordshire Community Stroke Project Classification (OCSP)18.

Statistics

Demographic and stroke-related characteristics of the participants are presented with descriptive statistics. The data had a skewed distribution, variables had mostly nominal and ordinal levels, and thus non-parametric statistics were used. Fisher’s exact tests for categorical variables and the Mann–Whitney U tests for continuous variables were applied for group comparisons.

The first null hypothesis (H0) in this study was that the Cog-4 and the MoCA could identify the same proportion of people with cognitive deficits. To test the hypothesis, we studied the agreement between the thresholds of Cog-4 and the MoCA with cross tabs and Cohen’s kappa. The threshold for impaired cognition was set at ≥1 p on the Cog-48. Three MoCA thresholds at ≤19 p, ≤23 p, and ≤25 p were chosen for impaired cognitive functioning according to previous literature19. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), and negative likelihood ratio (NLR) were calculated, and a 95% confidence interval (95% CI) was presented for each analysis. The Cog-4 and the MoCA were further compared using a receiver operating characteristic (ROC) curve. For this analysis, the index instrument, the Cog–4 (range 0–9 points) was entered as a test variable, and the previously mentioned three thresholds of the reference standard instrument, the MoCA were entered as a state variable. The area under the curve (AUC) results were interpreted as follows: 0.7–0.9 as moderate accuracy and 0.5–0.7 as low accuracy20.

The second H0 was that correlation between the total scores of the Cog-4 and the MoCA, and the correlation between the cognitive items of the Cog-4 and the corresponding cognitive domains of the MoCA equals “−1”. For testing the hypothesis, Spearman’s rank correlation test (rs) was used. The rs “−1” was chosen as a value for the perfect correlation, since the Cog-4 and the MoCA have reversed values for normal cognitive functioning. The correlation values were interpreted as small (r < ±0.29), medium (r = ±0.30 to ±0.49) or high (r = ±0.50)21.

Statistical Package for the Social Sciences (IBM SPSS®, version 25) was used for these analyses. The α was set at 5% for all statistical tests.

Results

Study participants

In total, 550 participants were registered in the research database. Detailed information is presented elsewhere2. Of these, 19 participants were excluded: 17 participants were missing a total NIHSS score; for 2 participants, the Cog-4 could not be obtained because of missing values on NIHSS items (Fig. 1). There were no statistically significant differences between the included and excluded participants with respect to sex (p = 0.64), age (p = 0.14), and stroke classification according to OCSP (p = 0.77).

Figure 1
figure 1

Flowchart of the study participants.

Full size image

The current study included 531 participants with stroke, with a mean age of 69 years (range 19–97 years); 515 (97%) and 524 (99%) participants were independent in ADL and mobility prior to stroke, respectively. The median score of admission NIHSS was 1 p (range 0–22). The Cog-4 threshold of ≥1 p for impaired cognition identified 151 (28%) participants with cognitive difficulties. The MoCA threshold of ≤25 p for impaired cognition identified 316 (59%) participants with cognitive deficits (Table 1).

Table 1 Characteristics of the study participants (n = 531) and participants stratified according to the National Institute of Health Stroke Scale scores.
Full size table

Test accuracy and evaluation of the Cog-4

The MoCA was treated as the reference standard instrument and the Cog-4 was treated as an index test. The Cog-4 and the MoCA were dichotomized for these analyses. Three different cut-offs of the MoCA were used. H0 was rejected, and the Cog-4 failed to identify cognitive deficits in 65%, 58%, and 53% of patients when the MoCA thresholds for impaired cognition were set at ≤25 p, ≤23 p, and ≤19 p, respectively (Table 2a–c). Moreover, the agreement between the Cog-4 and different MoCA cut-offs was poor; Cohen’s kappa was between -0.210 and -0.109 depending on the MoCA cut-off (Table 2a–c). The Cog-4 showed low sensitivity for identifying cognitive deficits; however, specificity was somewhat better (Fig. 2, Supplementary Table S2).

Table 2 Test accuracy of the Cog-4 and the Montreal Cognitive Assessment (MoCA) with ≤25 p, ≤20 p and ≤19 p as the thresholds for cognitive impairment (n = 531).
Full size table
Figure 2
figure 2

Diagnostic evaluation of the Cog-4 under different cut-offs of the Montreal Cognitive Assessment (MoCA) (n = 531). Abbreviations: 95% CI, 95% confidence interval. The figure shows exact values and 95% CI for positive predictive value, negative predictive value, sensitivity, and specificity.

Full size image

The results of the AUC models showed that the overall accuracy of the Cog-4 (score range: 0–9 points) was low when it was tested against three thresholds of the MoCA. The AUCs (95% CI) were 0.59 (0.54–0.64), 0.62 (0.57–0.67) and 0.63 (0.56–0.69) for the MoCA thresholds for impaired cognition were ≤25 p, ≤23 p and ≤19 p, respectively (Fig. 3).

Figure 3
figure 3

Receiving operating characteristic (ROC) curves for different cut-off points of the Montreal Cognitive Assessment (MoCA) for impaired cognition and Cog-4 scores (scores range from 0–9 points). Abbreviations: AUC – area under the curve; 95% CI, 95% confidence interval.

Full size image

Correlation between the Cog-4 and MoCA

The H0 was rejected, and a significant but small correlation was found between the total scores on the Cog-4 and the MoCA (rs = −0.29, p < 0.001). A small, partly significant correlation was found between individual items of the Cog-4 and the corresponding cognitive domains on the MoCA (Fig. 4).

Figure 4
figure 4

Correlation between individual items of the Cog-4 and corresponding cognitive domains on the Montreal Cognitive Assessment (MoCA), *p < 0.05, **p < 0.01. Rs - Spearman’s rank correlation coefficient.

Full size image

Discussion

The results of this retrospective cross-sectional cohort study showed that admission Cog-4 has a limited ability to detect cognitive deficits compared to the MoCA assessed within the first two days after hospital admittance. The median score on the Cog-4 was 0 p, and on the MoCA 25 p, for the total study sample as well as for the participants with mild stroke (NIHSS ≤2 p). These clinically important findings, together with other studies9,10,22, indicate that the NIHSS cognitive subscale Cog-4 has limited accuracy in identifying cognitive impairments early after stroke. Hence, other standardized screening tools should be applied for more reliable results regarding the assessment of cognitive difficulties very early after stroke. The screening tools must be chosen based on their validity, reliability, normative data and good psychometric properties.

In the present study, the Cog-4 defined 28% of participants with cognitive deficits, while the reference standard instrument (the MoCA) defined 59% when the cut-offs for impaired cognition was set at ≥1p on the Cog-4 and ≤25p on the MoCA. Somewhat higher proportions were previously presented, but the proportional difference between the Cog-4 and the MoCA was similar10. Furthermore, when the Cog-4 was compared with the MMSE at a later stage of stroke, it was concluded that although the Cog-4 was able to detect severe cognitive deficits, it could not be used as a cognitive screening tool8. There are several reasons why the Cog-4 has poor discrimination for cognitive deficits. The Cog-4 is a subscale of the NIHSS, which was originally developed for clinical trials. Thus, as a bedside assessment, it can have elusive accuracy for covering cognitive deficits6,7,23. Furthermore, scores on the NIHSS and Cog-4 also depend on the lesion side; persons with left-side lesions score higher than those with right-side lesions22,23.

In the current study, many items of the Cog-4 showed small correlations with the MoCA’s cognitive domains. One probable explanation could be that the items of the Cog-4 and the domains of the MoCA measure different things and do not correspond well to each other. Further, the items of the Cog-4 are less specific and cover fewer aspects of cognition than the MoCA10. The MoCA has shown poor accuracy for identifying domain-specific cognitive impairments24, when compared to more comprehensive domain-specific neuropsychological tests. Since MoCA is feasible in the acute care setting, using the total score of the MoCA is more reliable than the Cog-4 for understanding cognitive impairment very early after stroke.

An optimal cut-off point for normal cognitive functioning has been previously discussed19. To address this problem, we studied the relationship between a dichotomized Cog-4 and three different cut-off points for the MoCA. The Cog-4 was unable to identify cognitive deficits in 53% to 65% of the study sample, depending on the MoCA cut-off. We have further tested the total score of the Cog-4 against three MoCA cut-offs: the AUC curves showed low accuracy but statistically significant results. The statistical significance of the results could be explained due to the large sample size. Our results are in line with other studies10,13, and strengthen the recommendation of not using the Cog-4 as a screening tool for cognitive functions. Accordingly, the MoCA is a more feasible cognitive screening tool during the acute phase of stroke13. The MoCA is feasible in 80% of acute stroke patients, but it is also lesion side and type biased13; thus, there is a potential risk to overlook patients with cognitive deficits. Comprehensive neuropsychological assessments are usually time consuming and with the short length of stay at the stroke units, often not feasible. The reliability of a full assessment this early can be questioned due to the unstable nature of cognitive performance very early after stroke. Therefore, cognitive screening is thought to be more relevant in acute stroke settings.

There are some strengths and limitations of the study. The research database comprises clinician-gathered data2,15; thus, it can be assumed that the results have ecological validity. However, there were different assessors, which may have affected results on both the NIHSS and the MoCA. The opt-out consent used for quality registers increases the possibilities of a representative sample. In the research database, many people did not have the MoCA registered and the missing data was unlikely to be missing at random. The analyses performed elsewhere showed that people with missing MoCA were older2. This means that there is a risk that a larger proportion of people with older age was missed. The number of patients missing the Cog-4 scores was small, thus a data imputation was not performed.

Severity of the neurological symptoms was predominantly mild, but 10% had moderate or more severe stroke. Recent data from the Swedish stroke registry (Riksstroke, brief summary of data for the full year 2018) shows that -63% of people with stroke in Sweden has mild stroke (NIHSS 0–5 points). Thus, we can assume that the results can be generalized to the population with mild to moderate stoke. The neurological assessment was performed at admittance to the hospital, and cognitive assessments were performed within the first 2 days after admittance. This time difference is one possible explanation when participants were identified as having cognitive deficits according to the Cog-4 but not according to the MoCA. Another explanation could be that more patients with higher NIHSS received reperfusion treatment, and two days after stroke, the cognitive function, assessed with MoCA was somewhat recovered. While the NIHSS is feasible for the majority of people with stroke, the MoCA can be performed mainly in people with mild to moderate stroke13. Since people with mild to moderate stroke have short hospital stays there is a risk of overlooking people with cognitive deficits at discharge. The importance of early cognitive screening has increased as a basis for further planning and rehabilitation interventions after discharge.

In conclusion, clinicians who work with stroke should be aware that the Cog-4 at admittance has a limited ability to detect cognitive deficits compared with the reference standard instrument, the MoCA. This limitation was valid, even after comparing Cog-4 with different cut-offs of the MoCA for normal cognition. Thus, we can say that the Cog-4 should not be used as a screening tool to assess cognitive functions early after stroke.