Background

Cognitive deficits are highly prevalent in psychotic disorders [1], including schizophrenia, bipolar disorder, and schizoaffective disorder [2–6]. Numerous studies suggest that patients with severe psychiatric disorders have impaired sustained attention [7] and memory [8–10]. A wide spectrum of executive deficits have also been described, including problems performing goal-oriented tasks, recognizing priority patterns, and planning [11, 12], along with diminished verbal fluency [13] and information processing speed [14, 15]. Increasing recognition that psychosocial prognosis is directly related to the severity of the cognitive impairments [16–19], has resulted in a paradigm shift that may expand the targets for treatment beyond the mere symptom suppression and necessitate an integration of cognitive assessment into routine psychiatric practice.

The importance on the field of this study is emphasized by a long-standing initiative of the National Institute of Mental Health, known as MATRICS [20, 21], which has now been subdivided into three different programs: CNTRICS [22], TURNS [23], and TENETS (Treatment and Evaluation Network for Trials in Schizophrenia). The aim of these initiatives is to unify and standardize the types of deficits to be measured and the tests to use, with the final objective of developing effective new treatments for the neurocognitive deficits that occur in these patients. Recently, the MATRICS initiative has proposed a consensus battery that takes between 60 and 90 minutes to administer and is composed of 10 paper-and-pencil tests specifically for cognitive assessment of patients with schizophrenia – the MATRICS Consensus Cognitive Battery [24, 25]. Given the difficulties of performing an assessment lasting more than one hour in standard clinical practice, in the past few decades, considerable effort has been made to create brief cognitive batteries that facilitate an overall understanding of the individual’s cognitive status, without overly sacrificing the sensitivity and specificity of these new instruments. Some examples are the Cognistat, before 1995 known as the Neurobehavioral Cognitive Status Examination [26], the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) [27], the Woodcock-Johnson III Test of Cognitive Abilities (WJ III COG) [28], and the Brief Assessment of Cognition in Schizophrenia (BACS) [29]. These instruments have decreased the time it takes to assess patients to about 40–50 minutes, but even so they have a high cost in terms of time and economics due to time constraints on practitioners in their daily clinical practice.

More recently, other types of studies have focused on the development of cognitive screening tools – scales that do not require additional materials in order to be administered, tools that have different interchangeable versions, tools that are simple and easy to administer, and have an administration time that is appropriate and manageable in clinical practice, i.e., approximately 15 minutes. Some examples are the Brief Cognitive Assessment (BCA) [30], the Screen for Cognitive Impairment in Psychiatry (SCIP) [31], and the Brief Cognitive Assessment Tool for Schizophrenia (B-CATS) [32]. All of these have good psychometric properties [30–35], but still no standardization data have been established for any of them.

A Spanish translation of the SCIP was recently introduced (SCIP-S) [36] which demonstrated appropriate psychometric properties both for patients with a diagnosis of schizophrenia [34] and those with bipolar I disorder [33], with regard to equivalence between parallel forms, internal consistency, temporal stability, dimensional structure, and convergent validity. Tentative cut-scores for identification of significant cognitive impairment irrespective of diagnosis are available [35], but the resulting binary classification is insufficient for description of the severity of identified impairment relative to a patient’s clinical cohort after adjustment for age, gender, and education. Guidelines for the interpretation of the SCIP-S would thus represent a step forward in the clinical utility of this instrument and add valuable information on its proper use.

Normative data represent performance on a measure or test by a standardization sample against which other performances on the measure can be compared [37]. Lack of normative data limits the interpretation of scores in individual cases as well as in treatment outcome research (as we cannot know if a score is typical, high or low for the population being studied) [38]. This has implications for our ability to assess the clinical significance of a score (or change in a score). Norm scores can assist clinicians in providing quantitative labels for the degree to which a raw score is to be considered average, elevated, or extreme and might be useful for diagnostic purposes, clinical decision making, or evaluation of treatment effects [37]. A traditional approach to deriving norm scores is to compare an individual’s raw score to a reference group with the same condition matched for background variables such as age and gender. In addition, clinicians using norms for comparison can more readily interpret a patient’s performance on a number of relevant self-report dimensions as well. This should assist in the determination of whether or not an individual’s responses are unusual for someone experiencing, in this case, cognitive deficits. In turn, this may suggest possible courses of action, such as further investigation or treatment (whose outcome can be evaluated against the normative dataset) [37, 38].

When evaluating cognitive function in routine practice, clinicians usually compare the patient score against the norms in the general healthy population to ascertain whether the patient cognitive function is preserved or impaired. In such case, comparison allows to determine the distance to what a particular patient separate from the mean score. Nevertheless, in many cases the practitioner refers the patient to a specialist for formal recognition when his/her performance is unusually low compared with patients with same condition. Particularly when additional etiologies (in addition or replacement of schizophrenia) responsible for the high cognitive impairment observed is suspected [39]. Concerning this point, patient score should be compared with the norms belonging to subjects with the same health condition to assess how the different is the patient scoring related to his/her population of reference. As stated by Irvison et al. [40], this information can improve the clinician´s understanding of patient´s cognitive strength and weakness, put a patient’s cognitive abilities into perspective for their diagnosis, and facilitate multidisciplinary treatment decisions.

In this context, to date there are no standardization data for the SCIP-S scale in psychiatric patients that allow the examiner to interpret the patient scores relative to the cognitive performance of their peers. Thus, the objective of this study is to provide the first clinical normative data for the SCIP-S in patients with functional psychosis, and specifically with schizophrenia-spectrum disorder or bipolar I disorder.

Methods

Participants

Patients diagnosed criteria with schizophrenia, schizoaffective disorder, schizophreniform disorder, or bipolar I disorder according to DSM-IV-TR [41] were enrolled in this study. To take part, patients had to be between 18 and 55 years of age and in a stable phase of the disease. In the case of patients with schizophrenia spectrum disorders, stability was defined by: a) no hospitalization in the past 3 months, and b) a total score of less than 70 on the Positive and Negative Syndrome Scale (PANSS) [42, 43]. In the case of patients with bipolar I disorder, stability was defined as: a) 6 or more months in remission, b) a score less than or equal to 7 on the 17-items Hamilton Depression Scale (HAM-D) [44], and c) a score less than or equal to 6 on the Young Mania Rating Scale (YMRS) [45]. We excluded individuals that were participating in a clinical trial, and those with a serious medical or neurological condition, another psychiatric disorder as a primary diagnosis or main reason for treatment, major depression, or difficulty reading and/or writing. The process of recruitment began with a consensus conference on the diagnostic criteria for the different schizophrenia spectrum disorders and the bipolar disorder I. This consensus was adopted by all participating psychiatrists. This conference dealt mainly with the standard psychiatric interview based on the DSM-IV diagnostic criteria (anamnesis and the exploration of the mental condition), the PANSS, HAM-D, and YMRS scales, and the different inclusion/exclusion criteria of our study.

Instrument

The SCIP [31] is a brief screening tool designed to assess cognitive impairment in psychiatric patients. It has five subtests for evaluating immediate (Verbal Learning Test-Immediate; VLT-I) and delayed (Verbal Learning Test-Delayed; VLT-D) verbal learning, working memory (Working Memory Test; WMT), verbal fluency (Verbal Fluency Test; VFT), and processing speed (Processing Speed Test; PST). It may be administered without the need for additional equipment, i.e., a pencil and a stopwatch, and requires nearly 15 min. Three alternative forms of the scale are available to facilitate repeated testing. Table 1 contains the description and the main characteristics of the SCIP subtests.

Table 1 Description of the SCIP subtests
Full size table

The psychometric properties of the SCIP were studied in a sample of patients with schizophrenia [34] and in a sample of bipolar I patients [33], and were shown to be adequate. Specifically, both studies demonstrated the equivalence among the three parallel forms, internal consistency (Cronbach’s alpha of 0.73 and 0.74, respectively), and test-retest reliability (intraclass correlation coefficient of 0.90 and 0.87, respectively, for the SCIP total score). Convergent validity was supported by the associations between SCIP subtests and conventional neuropsychological instruments applied in routine clinical practice. The scores also converged on a single cognitive factor accounting for around 50% of the total variance, suggesting a one-factor internal structure in both samples named cognitive performance. Finally, when comparing cognitively-impaired individuals and those with adequate functioning, the proposed cut-off point of the SCIP (< 70) was associated with a sensitivity of 87.9 and specificity of 80.6.

Procedure

This study was approved by the University of Barcelona Ethics Committee. The SCIP-S was administered to all patients, who were systematically tested once it was confirmed that they met the study inclusion criteria and gave their written informed consent to voluntarily participate in the study; data confidentiality was maintained at all times. The data were collected at 119 Spanish mental health centers, selected by probability sampling adjusted by population weights from the 17 Spanish Autonomous Communities, with the participation of 132 psychiatrists duly trained in administering the instrument with a video designed for that purpose. Before the start of the process, a neuropsychologist experienced in administration of neuropsychological tests and batteries trained a sub-set of forty-four psychiatrists in a 60-minute session to ensure consistency in SCIP administration and correction. The training phase was completed with a kappa index of agreement in scale correction and scoring of .99.

Data analysis

The analyses were done using the SPSS statistical package version 15 and the significance level was set at α = .01. The internal consistency of the SCIP was assessed by computing Cronbach’s alpha coefficient, treating each of the SCIP subtests as variables. We compared the SCIP scores of patients with schizophrenia and bipolar I disorder, as well as between males and females, using a t test for independent samples. In both cases, the statistical significance was supplemented by calculating Cohen’s d. Likewise, the differences between the specified groups were analyzed by age and educational level. The normal distribution of the data was tested using the Kolmogorov-Smirnov (KS) test for normality.

Patient performance on the SCIP was shown using various descriptive statistics (mean, standard deviation, median, asymmetry, kurtosis, and range of scores). As for the transformation of SCIP scores, a percentile, z-score, T-scores (T = 50 + 10 » z), and intelligence quotient (IQ = 100 + 15 » z) were calculated.

Results

Sample description

A total of 514 patients diagnosed according to DSM-IV-TR [41] criteria with schizophrenia (41.5%), schizoaffective disorder (6.4%), schizophreniform disorder (1.4%), or bipolar I disorder (50.7%) participated in this study. Within this group, 57.9% were males. Most patients with schizophrenia were being treated with a single antipsychotic (66.9%), although a large number were receiving a combination of two (28.0%) or three (3.1%) antipsychotics. At the time of assessment, 5 patients were not taking any antipsychotic. In addition to the antipsychotic medication, 52.6% of patients were receiving an additional psychoactive drug, primarily antidepressants and benzodiazepines. The mean age at onset of the illness was 24.25 (SD = 6.34), the mean number of months since the diagnosis was 156.78 (102.99), and the mean number of hospitalizations was 2.61 (3.05). Within the bipolar I disorder sample, 23.5% were taking lithium, while other patients were taking one (33.5%) or two (3.5%) antipsychotics in addition to lithium, and finally another group of patients were taking receiving antipsychotic medication in monotherapy (23.8%), or in a combination of two (5.0%), or three (0.4%) agents. Additionally, 75.4% of patients were receiving another type of psychoactive drug (i.e., antidepressants or benzodiazepines). The mean age at onset of the illness was 28.39 (8.34), the mean number of months since the diagnosis was 144.55 (95.78), the mean number of manic episodes they had experienced was 3.36 (1.86), and of depressive episodes was 1.22 (2.94), and finally the mean number of hospitalizations was 2.80 (3.67).

The participants were divided into two age groups (18–39 and 40–55) and two education level groups (less than secondary education and secondary education or higher). Based on these and other variables, Table 2 shows the main demographic information for each clinical sample used in the study.

Table 2 Sample descriptors
Full size table

Comparison between groups

By comparing the scores of patients with schizophrenia and bipolar disorder I, as well as between men and women, we obtained the means, standard deviations, t statistics, and effect sizes specified in Table 3.

Table 3 Mean SCIP scores and standard deviations by clinical diagnosis and sex of patients
Full size table

Both in the various subtests and in the total score, the mean performance of the patients with schizophrenia was poorer than that of the patients with bipolar I disorder, although in no case was the effect size measurement significant. With respect to sex, the mean scores were similar, with the exception of the VLT-D subtest, where women scored slightly better than men, although the corresponding effect size was small. As was to be expected, on all subtests, as well as on the total SCIP score, the patients with a higher education scored higher than those with a less than secondary education (all p values < .01), with effect sizes that varied between 0.50 for the VFT subtest and 0.70 for the PST subtest. The difference in total SCIP score, for the education variable, reached an effect size of 0.78. In the case of age, as the patients’ age increases, the mean scores decreased. The differences were statistically significant (p value < .01) for the total SCIP score and the various subtests, with the exception of the VLT-I and VFT. The effect sizes varied between 0.25 for the VLT-I subtest and 0.54 for the PST subtest, while the difference in total SCIP score was characterized by having an effect size of 0.53. For those reasons, the clinical normative benchmarks are presented jointly for male and female patients with schizophrenia and bipolar I disorder. On the other hand, given the existing differences, patient age and educational level were taken into account.

Standardization data

The internal consistency of the SCIP achieved a Cronbach’s alpha coefficient of 0.73, which is an acceptable value given the small number of variables. This alpha value did not increase when any of the component variables were eliminated. The item/scale correlations were between 0.44 for the VFT and 0.58 for the PST. The normal distribution of the data from the various subtests (and total SCIP-S score) was tested for each of the groups after combining the two age groups and the two participant educational level groups. In no case was the KS test statistically significant at a level of 0.01 (all p > .01) although in six cases there were p values below .05 (the WMT, VFT, VLT-D, and PST subtests in the 18–39 year old group and the VLT-D and PST subtests in the 40–55 year old group, in all cases with a secondary or higher education). Tables 4, 5, 6, 7 and 8 show the clinical normative data for each of the SCIP-S subtests in terms of percentiles, z-scores, T-scores, and IQ. Likewise, Table 9 shows this same information for the total SCIP-S score.

Table 4 Transformation of VLT-I subtest scores
Full size table
Table 5 Transformation of WMT subtest scores
Full size table
Table 6 Transformation of VFT subtest scores
Full size table
Table 7 Transformation of VLT-D subtest scores
Full size table
Table 8 Transformation of PST subtest scores
Full size table
Table 9 Transformation of total scale scores
Full size table

After administering the instrument, for norming the cognitive performance of a patient with schizophrenia or bipolar I disorder against the reference or comparator group, the examiner has only to locate the corresponding transformed score on the table by the patient’s age and educational level.

Discussion

The clinical value of a screening tool is directly related to either considering cognitive impairment a key aspect of schizophrenic psychopathology and, according to the proposed DSM-V revisions, recommending it as one key dimension to be measured in all patients with a psychotic disorder, or including cognitive deficit as one of the diagnostic criteria for psychoses as suggested by some authors [46, 47]. The practical utility of the administered tests should not be forgotten when conducting a neuropsychological assessment, and since there is a large number of psychiatric patients (accounting for around 2% of the general population) who require diagnosis, there is a growing need for cost-effective and highly efficient diagnostic tools. In this regard, the creation of the SCIP-S precisely had these two objectives. Previous studies [33, 34] have shown that the SCIP-S takes approximately 15 minutes to administer, compared to a mean of around 75 minutes for the administration of a full neuropsychological battery or between 60–90 minutes for the MCCB, and it has good validity and reliability. Furthermore, Rojo et al. [35] reported the good sensitivity and specificity of the test and its high diagnostic value for appropriately distinguishing cognitively preserved from cognitively impaired individuals.

This study goes a step farther and presents the diagnostic-specific norms and performance for the SCIP-S according to the age and educational level of subjects in a large sample of patients with schizophrenia and bipolar disorder. Some studies report differences in neuropsychological performance between subjects with different educational levels [48, 49], and such differences were also found in this study, which is why in the SCIP-S standardization data the sample has been divided based on educational level. It should be pointed out that, although there is not always a direct correspondence between educational level and years of education, we may consider that in the vast majority of cases a less than secondary education implies fewer than 12 years of education, while a secondary or higher educational level implies at least 12 years of education.

Another aspect known to influence cognitive performance is age, as over the years a series of cortical changes occurs resulting in a loss of brain volume [50, 51] associated with a decrease in cognitive performance in the general population [52, 53]. Our norms take this aspect into account by dividing the sample according to age and limiting patient age to 55 years in order not to introduce bias due to patients whose performance could be affected by early onset of a picture of dementia.

One item that bears emphasizing refers to that fact that a certain pattern was observed to repeat in the various subtests and total SCIP-S score. Specifically, the median score in the two age groups is similar (a maximum difference of 1 point) when the patients have a high educational level, while differences of up to 8 points are found in the groups with a primary or lower educational level. This may be explained by an interaction between age and educational level and the possibility that age at onset of illness plays an important role, since some studies show that both verbal intelligence and impairment of verbal memory and executive functioning could be affected in patients before they experience their first psychotic episode [54–56], such that the earlier the onset of illness, the greater the potential for limiting the patient’s ability to normally pursue an education. Therefore, the effects of age and educational level and their interaction were explored by adding age at onset of illness as a covariate. Such interaction was not statistically significant in any SCIP-S score (all p > .01).

One of the aspects highlighted by this study is that of all the tests mentioned above that have been developed for the purposes of cognitively assessing psychiatric patients, we find diagnosis-specific standardization data for patients with schizophrenia only for the RBANS [39, 40]. And as stated by Iverson et al. [40], being able to describe a patient’s cognitive performance in terms of expectation for their peer group is more useful to multidisciplinary treatment teams than just comparing them to a healthy population. So the present study provides the tools necessary to interpret the score obtained by a patient with functional psychosis relative to other patients after administration of the SCIP-S scale. As an example, let us apply the SCIP-S to an imaginary 38 year old college graduate diagnosed with schizophrenia who obtains a direct score of 13 on the WMT subtest. After determining their performance relative to healthy controls (healthy control norms are under elaboration), the clinician interest could move to answer the question: Where do we situate that individual with respect to other patients? Looking at Table 4, we see that, based on his age and educational level, this patient is in the 9th percentile, has a z-score of −1.45, a T-score of 36, and an IQ of 78. This tells us that only 9% of his comparison group has obtained a score below his and that his working memory score of 13 is situated approximately 1.5 standard deviations below the other patients.

Conclusions

The SCIP and the SCIP-S provides a quick and convenient cognitive diagnosis and, in that regard, its usefulness extends to areas such as detection, cognitive assessment of large samples, epidemiological and screening diagnostic studies more than to specific cognitive domains or type of impairment in patients with functional psychosis. In that way, it is a complementary test that is not intended to replace complete neuropsychological batteries. Future studies should explore how performance on the SCIP relates to other cognitive domains that it does not measure directly (e.g., problem solving, social cognition, etc.).

A study that could continue this one should be perform standardization data for patients over 55 years of age, since, at a cognitive level, that is a critical age where the SCIP-S could help us reach a differential diagnosis between onset of dementia versus cognitive dysfunction associated with functional psychosis. Future research with this scale should also incorporate the development of guidelines for interpreting the scoring according to results of treatment of patients. Additionally, last evidences in schizophrenic and bipolar patients from recent studies have suggested that the history of psychosis explain part of the neurocognitive performance [57], thus in future studies with cognitive screening tools would be interesting to take this variable into account.

In short, this study presents the first jointly developed diagnostic-specific norms for the SCIP for functional psychosis (schizophrenia and bipolar disorder), providing increased information about their cognitive abilities.