Background

Suicidal ideation occurs in more than 10% of the population during their lifetime [1]. Each year, 2% of the population contemplate suicide and 0.3% attempt suicide [2]. Across all age groups, the incidence and prevalence of suicidal behaviour have increased considerably over the past two decades. Suicide is now a more common cause of death than motor vehicle accidents [3–5]. Suicide does not happen without warning. People frequently make contact with health services in the months leading up to their suicide attempt. There is a recognised need to identify those at risk [6, 7] for targeted interventions to stop suicide before it happens [8].

Risk factors for suicide are well-documented, but the list is long. These include: age, male gender, chosen method of attempted suicide, and number of previous attempts; [9] psychiatric diagnoses including anxiety and depression, psychosis, and bipolar disorder; [10–12] social isolation; [13] and potential lethality of previous attempts [14]. Despite the effort to combine these risk factors into risk scores and algorithms to predict suicide risk, [15–17] the predictions are often too poor to be clinically useful [18, 19].

Alternative to a constant set of risk scores that we hope would work for each individual, a broader source of information is available: the comprehensive electronic medical record (EMR). The time-stamped administrative, clinical, and investigative data in the EMR includes information on known risk factors for suicide. Additionally, it contains information on health service encounters that might not be directly related to suicidal behaviour. It is known that patients who attempt suicide frequently attend the emergency department (ED) in the months before their suicide attempt [20, 21]. Many of these attendances are not directly related mental health problems or self-harm. Also, many suicidal patients attend primary care facilities in the months before their suicide attempt, again commonly for reasons not directly related to their psychological distress [22]. Up to 85% of suicidal patients attend hospital outpatients in the twelve months before their suicide attempt, [23] again often for reasons not directly related to psychological morbidity. There is a high prevalence of coexistent physical illnesses amongst patients who exhibit suicidal behaviour [24]. All these contextual factors may contribute to suicide risk, but for each individual patient, they are difficult to be assessed objectively and consistently without a proper tool.

We develop a statistical risk stratification model based on EMR data. The model results from examining suicide attempts and completed suicide in a large cohort of patients who underwent suicide assessment in a regional health service. We compare EMR-based predictions of high-risk suicidal behaviour with clinician predictions, which are based on an 18-point assessment instrument.

Methods

Study design and population

This was a retrospective study using electronic records of inpatient admissions and emergency department (ED) visits within Barwon Health, a regional health service in Australia. As the only tertiary hospital in Greater Geelong, and a catchment area with over 350,000 residents, the hospital’s patient database provides a single access point for information on hospitalisations, ED visits, and medications. Although different IT systems are used by hospitals, EMRs often share similar underlying logical structure. All the diagnoses are often coded using the data standard ICD-10. For generalisability of the model, we focus on EMR data that are either generally available in mental health services or coded using the ICD-10 disease classification.

Patients were included in the study if they were aged 10 years or over, and had received at least one suicide risk assessment between April 2009 and March 2012. A cohort of 7,399 patients (with 16,858 risk assessments) met the inclusion criteria. Risk assessments were carried out either in Barwon Health following presentation to ED or another department or in one of five community health centres. Patient outcomes were observed for a 180 day period following a risk assessment and details of repeat suicide attempts or completed suicides were recorded. A risk assessment was considered a clinically relevant time point for prediction, as care models are devised according to patient risk at that time. If a patient had more than one assessments, one of them would be randomly selected.

Patient records prior to the selected assessment date were used to construct the independent variables. The same data were also available to clinicians during risk assessments. Suicide attempts, of varying lethality, in a 180 day period following the assessment were treated as the dependent variable. The true outcome, for a given period, is the actual risk class determined from patient records for that period. As outlined before, direct modelling of suicide risk is difficult because of low base rate of suicides. Modelling risk of suicide attempts, a strong predictor of suicide, [25] is a pragmatic measure to circumvent this difficulty. Moreover, it allowed us to define a set of risk classes for the suicide attempts of varying lethality levels, consistent with previous published work [26].

Self-harm events are recorded in the hospital database using ICD-10 codes. For example, an ED visit due to by attempted hanging, would be assigned the ICD-10 code X70. We exclude those events which may result from accidents or assaults (instead of intentional self-harm). In Australia, such events are coded with the following ICD-codes: Transport Accidents (V01-V99, Y85); Falls (W00-W19); Accidental Poisoning (X40-X49); Assault (X85-Y09,Y87.1).

For each stipulated period following a risk assessment, the risk class was defined as follows:

  1. 1.

    The period has high-risk if, during the period, the patient had an ED visit or inpatient admission with a high-lethality diagnosis (Table  1).

Table 1 ICD-10 codes identified to correlate with moderate or high lethality events
Full size table
  1. 2.

    The period has moderate-risk if, during the period, the patient had an ED visit or inpatient admission with a moderate-lethality diagnosis (Table  1).

  2. 3.

    Otherwise, the period has a low-risk (patients with low lethality suicide attempts or no suicide attempts).

Predictive algorithms were constructed from the derivation cohort to predict risk in different post-assessment periods—30, 60, 90, and 180 days. The predictive model takes demographic and historical data prior to a risk assessment, and identifies the post risk-assessment class, for a desired period.

Ethics approval was obtained from the Hospital and Research Ethics Committee at Barwon Health, with whom Deakin University has reciprocal ethics authorisation (approval number 12/83).

Clinician risk assessment

The clinicians’ prediction of risk involves using a risk assessment checklist. Clinical protocols at Barwon Health require a suicide risk assessment to be completed on intake to care, every 91 days during an episode of care and on discharge. For the purpose of this study all available suicide risk assessment data was extracted from the data warehouse. The primary purpose of the assessment is to ensure that consideration of suicide risk is medico-legally documented. Under the Victorian Mental Health Act, mental health services are required to report deaths of patients who are either currently receiving care or have had contact with the service in the past six months. Patient deaths are reported through a centralised risk management system, which is also reconciled against death certificates and a registry of deaths. Deaths reported to the coroner are also tracked to the point where the coroner determines whether the death was due to suicide. All suicides and other unnatural deaths are subject to a comprehensive case review. Data on self-harm was identified by diagnostic codes for intentional overdoses and self-inflicted injuries from admission and emergency presentation coding. The suicide risk assessment instrument used was developed by Barwon Health in 1999, based on known risk factors in the literature. It has been in use for more than 13 years as no alternative risk scores have been shown to be more effective. One of the purposes of this study was to validate use of this risk assessment tool. The tool consists of 18 items, each graded on a 3-point scale (low/moderate/high) covering suicidal ideation, suicide plan, access to means, prior attempts, anger/hostility/impulsivity, depression (current level), anxiety, disorientation/disorganisation, hopelessness, identifiable stressors, substance abuse, psychosis, medical status, withdrawal from others, expressed communication, psychiatric service history, coping strategies and supportive others (connectedness).

Derived predictor variables

Patient specific information—such as interactions with health services, ICD-10 diagnostic codes (mental disorders and other comorbidities), diagnosis-related groups (DRGs) and procedures, demography, and medications—were used to derive predictors. Diagnostic codes were used to derive following subsets: injuries classified as moderate or high risk, mental diagnoses mapped into Australia’s Mental Health Diagnostic Groups, and relevant codes mapped into Elixhauser comorbidities [27]. To ensure generalisability, medications were mapped according to the World Health Organisation’s Anatomical Therapeutic Chemical (ATC) classification system. Different from most existing clinical predictive models, derived predictors are specific in the temporal dimension, encoding the knowledge that prior events may influence future risks at multiple time scales. More specifically, occurrences of historical data were aggregated and normalised over several time periods before the assessment point: 0–3 months, 3–6 months, 6–12 months, 12–24 months, and 24–48 months. Overall 5,322 variables were derived, but only those that appeared more than 4 times among the patients at risk in the derivation cohort were kept, resulting in 202 variables. For robustness, variable values outside the 1st and the 99th percentiles were considered outliers and rounded to the nearest percentiles.

Derivation of the risk stratification model

For predictive modelling, we adopted a recently established statistical technique L1-penalised continuation-ratio model for ordinal outcomes [28]. Different from traditional ordinal regression models that assume monotone effects of predictor variables [29], the continuation-ratio model uses separate coefficients for different outcome classes, at the same time automatically selects highly predictive variables (and their time scales) for each class. As prior studies suggest that predictors of the two classes may be different [30], it is desirable to model the moderate-risk and the high-risk classes using separate coefficients. The variable selection capability comes from the lasso-like shrinkage [31] that forces variables with weak association with the outcomes to have zero coefficients. The shrinkage strength was adjusted so that the predictive performances on the derivation and validation cohorts were similar, and thus ensuring no over-optimism in the estimated model [32].

A single prediction risk score was computed, representing the expected lethality level. The risk score takes a value from 0 to 2, with 0, 1, and 2 representing low, moderate, and high risk respectively. This score results in a simple decision rule: Given two thresholds in the range 0 – 2, the outcome class is low-risk if the score is less than the lower threshold, moderate-risk if the score is below the upper threshold and high-risk otherwise. The thresholds can be adjusted according to the clinical need.

The contribution of individual risk factors toward each risk class was assessed using bootstrap. That is, the models were estimated multiple times from bootstrap samples, and variable coefficients were then collected and statistics were derived. Among them are Wald statistics, confidence intervals, variable importance, and selection probability. Variable importance is defined as the product of absolute mean coefficient and variable standard deviation. Selection probability is the chance that a variable is being selected by applying the lasso shrinkage.

Model validation

To assess the performance of prediction methods, the 7,399 patients were randomly divided into a derivation cohort of 4,911 patients and a validation cohort of 2,488 patients. The risk score assigned by the predictive model and clinician-assigned overall risk were validated against true outcomes on the validation cohort. The risk score makes it simple for estimating performance measures for multiple binary decisions such as high-risk versus the rest, or moderate/high-risk versus low-risk. We reported here Area under the ROC Curves (AUC), sensitivity and specificity. The confidence intervals for these measures were estimated using bootstrap.

Results

Patients

Characteristics of patients included in the study are summarised in Table 2. In the follow up periods of 30 days, 90 days, and 180 days, there were 7, 9, and 13 suicides. The mean age was 41.2 years (11 – 101 years). Overall, 2,479 (33.5%) patients had changed postcode in the preceding 12 months. 3,953 (53.2%) attended ED in the 3 months before suicide assessment, and 4,436 (60.0%) had at least one hospital admission during the preceding 12 months. Of the 7,399 patients, 1,562 (21.1%) had a moderate lethality suicide attempt in the 12 months preceding their assessment and 646 (8.7%) patients had a high lethality attempt. A further breakdown of factors according to the outcomes within 90-day is presented in Table 3. Overall, risk factors were more prevalent in the more risky groups. The prevalence of males, pensioners, prior moderate-risk events, past 3-month admission and alcohol abuse were highest in the moderate-risk group. Other risk factors were most prevalent in the high-risk group.

Table 2 Demographic and clinical characteristics
Full size table
Table 3 Risk factors within outcome groups (low, moderate and high-risk events) in 90-days
Full size table

Validation of the 18-point risk score

Risk scores from the 18-point risk assessment instrument with respect to 90-day outcomes are shown in Table 4. Overall most identified factors were scored higher in the high risk group. Not surprisingly, the differential between low and high risk groups was highest for the factor ‘prior attempt’ (the average score for the high risk group was 1.70 compared to 1.26 for the low risk).

Table 4 Average risk scores assigned to 18-item checklist within true outcome groups (low, moderate and high-risk events) in 90-days
Full size table

Comparison of clinician and machine model predictions

To assess the performance of the predictive model using EMR data, we compared it with the clinical assessment (prompted by the 18-point check list). The discriminative performance of these two methods, in terms of area under the ROC curve (AUC), is presented in Table 5. Prediction was compared over 30, 60, 90, and 180 days. The clinician prediction had relatively low predictive ability with AUCs of 0.55 to 0.59 over the four time points when predicting high-risk events. AUCs for the EMR model were consistently better, ranging from 0.73 to 0.79. Similar differentials were also observed when predicting either moderate or risk events, where the AUCs were in the range 0.52 - 0.54 for clinicians and 0.71 - 0.79 for EMR models.

Table 5 Area under the ROC curve (AUC, 95% CIs) of clinicians versus EMR-based model on the validation cohort
Full size table

In identifying high risks within 90 days, clinicians achieved sensitivity of 0.08 (95% CIs: 0.02-0.2) at specificity of 0.97 (95% CIs: 0.96-0.98). The EMR model, on the other hand, reached sensitivity of 0.28 (95% CIs: 0.16-0.41) at the same specificity. Note that this low sensitivity was due to unusually high specificity of 0.97. At specificity of 0.72 (95% CIs: 0.70-0.73) the EMR model had sensitivity of 0.70 (95% CIs: 0.56-0.83). For either moderate or high-risks, clinicians scored 0.27 (95% CIs: 0.19-0.35) sensitivity and 0.82 (95% CIs: 0.81-0.84) specificity. The scores for the EMR model were higher at specificity of 0.58 (95% CIs: 0.50-0.66) at the same specificity.

Of two patients in the validation cohort who completed suicide during 30 days of follow-up, neither was identified as high-risk by clinicians while one was categorised as high-risk by the EMR model. Three patients completed suicide within 90 days. None were identified as high-risk by clinicians while the EMR model correctly classified two of the three patients.

Risk factors

Table 6 shows the relative importance of the top factors that are predictive of high-risk events in 90 days following suicide assessment. Prior high-risk events, injuries and poisoning were indicative of subsequent high-risk events. Mental health male patients who moved homes (as approximated by postcode change) were also at risk. These risk factors were related but different from those associated with subsequent moderate-risk events (Table 7). For example, prior emergency visits, addictive drugs treatment and moderate-risk events were predictive of the future moderate-risk but not the high-risk.

Table 6 High-risk predictive factors in the EMR model for 3-month prediction
Full size table
Table 7 Moderate-risk predictive factors in the EMR model for 3-month prediction
Full size table

Discussion

We have developed a robust predictive model that takes information rich administrative EMR data and stratifies mental health a patient into three ordinal categories: low, moderate and high-risks of suicidal behaviours in 30, 60, 90 and 180 days following a risk assessment. For comparison against the common practice of assessment by clinicians, we validated an 18-point checklist developed and used at Barwon Health. The EMR model showed much greater predictive ability than clinicians using the checklist in identifying high-risk patients.

We followed 7,399 consecutive patients judged to be at risk of suicide for 180 days following a clinical assessment. Although this time frame is considerably short for rare events such as suicides, this was chosen for practical purposes in treatment and resources planning. Our first aim was to document risk factors associated with high and moderate lethality behaviours. We have confirmed that high lethality suicidal behaviour is associated with social and demographic factors and prior high-risk events of injuries and poisoning. On the other hand, moderate lethality behaviour is related to prior interactions with health services, history of moderate-risk events, mental disorders and lifestyle problems. These results are in line with previous studies, in that the number and potential lethality of previous suicide attempts predict lethality of future attempts [9, 14]. Also, a range of mental health diagnoses were, as previously shown, predictive of future attempts during follow-up [10–12]. Social factors such as postcode change are strongly associated with suicide risk, concordant with recent findings [13, 33]. We document that 66.8% of patients undergoing suicide assessment attended ED and that 41.8% of patients had been admitted to hospital in the year before their assessment. The majority of these encounters were not for episodes of self-harm or even for mental health problems. This information added stimulus to exploring use of administrative EMR data in risk stratification.

Our second aim was to examine the contribution of mental and physical health diagnoses to suicide risk. It appears that mental health diagnoses are powerful predictors of moderate-risk behaviour. However, medical conditions also contribute to risk. Previous studies have confirmed that hospital admission for non-psychiatric conditions are associated with increased suicide risk [23, 24].

The next aim was to validate the 18-point risk checklist. Most of the factors in the checklist were more prevalent in patients with higher risk. To verify that if the effectiveness of checklist could be improved further, we tested same statistical framework in this paper on predictors derived from the 18 aspects and demographic data. The results were very encouraging – the algorithm outperformed the clinician’s overall risk assignment, and the difference was statistically significant. We conclude, therefore, that the risk checklist is a valid decision support tool. Although there has been some pessimism about the ability to predict suicide risk in the clinical setting, [18, 19] recent reports of predictive models suggest that clinically useful risk stratification is possible using recognised risk factors [15–17].

Our fourth aim was to document the predictive performance of clinicians prompted by the 18-point risk checklist. This was relatively poor, underlining the potential utility of a decision support tool. A recent study documented the potential for clinical decision rules to augment clinical performance in risk stratification of patients who might be at risk of suicide [34]. The models studied had high sensitivity but low specificity, unlike those reported in our study which had lower sensitivity but high specificity. Recent systemic studies of the factors associated with high risk in suicide-prone patients have added to knowledge on the potential contribution of known risk factors to prognostic stratification, and may help in the design of effective decision support algorithms [35, 36].

Our final aim was to use predictive models applied to data from the EMR to derive predictive models for high-risk suicide behaviour. Models for prediction of high-risk behaviour used administrative, social and demographic, and clinical data. Risk stratification using these models was more accurate than clinical stratification. The potential of EMR data to identify patients at high-risk of suicide from overdose has been documented in a recent study of paediatric patients [37].

The strengths of this study include the large size of the dataset, the fact that patients were consecutively recruited from a defined geographical region, and the comprehensive follow-up. Suicide assessments in both hospital and community settings were included. The models based on EMR could be updated in real-time, and make use of data that are routinely collected. The predictors derived from the EMR data were standardised, and thus the tools can be generalisable to sites with similar EMR systems. Limitations are that it is a retrospective study confined to a single location. The predictive models were validated in the setting in which they were derived but their performance may differ in other settings. The problem studied is inherently difficult to predict and there may be limits to the performance of predictive models due to the fact that individual factors are only weakly predictive.

Conclusions

In conclusion, demographic, social, mental and physical health, and administrative data related to health service encounters can all be used to identify patients at risk of self-harm behaviour. A history of high risk events are indicative of short-term future high-lethality suicide attempts. Clinicians’ assessment could be enhanced by using data from a simple checklist of known risk factors. Modern statistical techniques applied to complex data from a comprehensive EMR has considerable potential to improve risk stratification of patients that may engage in repeated self-harm attempts.