Introduction

In Denmark, 580,000 people have been diagnosed with a severe mental illness (SMI), defined as psychotic disorders and severe affective disorders (bipolar disorder, and severe unipolar depression) in 2023 [1, 2]. Globally, 970 million people were living with a mental illness in 2019, of which 2.5% had schizophrenia, 4.1% had bipolar disorder and 28.9% had a depressive disorder [3]. SMI has serious health implications and has been associated with a shorter life expectancy of up to 20 years compared to the general population [3]. The lower life expectancy is likely attributable to the higher prevalence of somatic diseases, such as cancer, cardiovascular disease, and type 2 diabetes mellitus [1, 4,5,6].

In most government-funded healthcare systems, the general practitioner (GP) functions as a gatekeeper to the secondary healthcare system. Moreover, the GP has the responsibility of identifying and treating somatic diseases and also referring some persons to specialized treatment in the secondary health care system [7,8,9]. Research in the general population has shown, that continuity of care with a regular GP has a beneficial impact on disease trajectories and mortality in the general population [9]. Therefore, regular contact with a GP might promote the prevention and timely treatment of somatic health problems in individuals with SMI.

In line with the higher somatic mortality, previous studies have found that persons with SMI tend to have a higher contact frequency with GP compared to the general population [10,11,12]. However, reaching people in greatest need of new initiatives and interventions in different healthcare settings has shown to be unequally distributed [13] and a study have shown that people with SMI in general have a lower socioeconomic position, and are less likely to be identified, approached, and participate in trials and that the recruitment strategy for this population should be considered carefully [14]. In addition to this another study discovered, that healthcare providers also encounter difficulties in identifying persons with SMI with unidentified or poor management of somatic disease [15]. Potential initiatives such as an outreach service by GP could promote patient engagement and improve treatment and prevention of somatic illness in this group. However, such initiatives would require efficient identification of persons in most need. To facilitate the identification of persons at risk of somatic disease, there is a growing interest in prediction models [16]. These models often use biological measures, however such models could benefit from including sociodemographic variables in their prediction [17]. Previous studies have explored which sociodemographic factors that characterize persons with SMI with and without contact with a GP. However, to our knowledge, no studies have aimed to create a prediction model for contact with GP for people with SMI based on sociodemographic factors. The objective of this study was therefore to fill the existing gap in the literature by developing a prediction model to identify persons with SMI without contact with GP by using data from the Danish national registers.

Method

Study design

We have designed a prediction model for no contact with GP based on a cohort of persons with SMI, where information on the study population was collected retrospectively. Data was collected from nationwide registers and merged by the Civil Registration Number (CPR number). The study population included those persons registered with contacts in secondary care due to SMI in the period 2011–2015. For these persons, the number of contacts with GP was recorded over the period 2016–2017 (Fig. 1).

Fig. 1
figure 1

Study design

Full size image

Registers

The study population was identified using the Danish Psychiatric Central Register (DNPR-Psych), a sub-register of the Danish National Patient Register (DNPR). These registers contain information on all public and private hospital sector contacts, and the DNPR is one of the oldest nationwide hospital registers globally [18, 19]. The DNPR was also used for information on comorbidities.

The Danish National Health Service Register for Primary Care (NHSR) [20] was used to gather data on GP contacts and Out-of-hours contacts. The register contains information about the activities of health professionals in the public healthcare system, such as general practitioners, psychologists, and physiotherapists. Each contact is registered with a CPR number [20].

The Civil Registration System (CPRS) is a nationwide register for all Danish citizens, where each person is assigned a CPR number. The CPR number allows for linking data across registers and IT systems [21]. Additionally, the register includes information on demographic variables such as sex, age, ethnicity, marital status, and region of living.

The Danish Education Register (DER), Income Statistics Register (ISR), Employment Classification Module (ECM), Danish Register of Causes of Death, and the Danish Prevention Register were also used to collect information on demographic variables, as well as information about the time of death and migration out of Denmark during the study period [22,23,24,25,26].

Study population

Our study population included adults above 18 years of age registered in the DNPR-Psych with an SMI from the International Classification of Diseases Tenth Revision (ICD-10) classification system [27] defined as psychotic illness (F20-F29), bipolar disorder (F31), or severe depression (F32.2, F32.3) as a main diagnosis in the inclusion period. If a person was diagnosed with multiple SMI diagnoses, a hierarchy was applied. Persons with multiple SMI diagnoses were allocated to the diagnostic group with the greatest disease severity. The hierarchy is illustrated in Fig. 2. Persons who did not have a valid birthdate or CPR number, died, or moved out of the country between 2011 and 2017 were excluded. The final study population consisted of 48,804 persons. A flow chart of the study population selection is depicted in Fig. 3.

Fig. 2
figure 2

Diagnosis hierarchy

Full size image
Fig. 3
figure 3

Flowchart of the study population

Full size image

Outcome

The outcome was the total number of contacts to GP in the follow-up period for each person in the study population. Data were derived from the NHSR and included all services in GP during the opening hours, including contact by telephone and email. We excluded out-of-hours contacts. “No contact” was defined as less than three contacts with GP in the two-year follow-up period. The threshold was set after considering the overall distribution of contact with GP in the study population.

Other variables

Demographic data (age, sex, marital status, ethnicity, region) were obtained from CPRS at baseline. Information on education, income, and occupation were obtained from DER, ISR, and ECM at baseline. Comorbidity was measured using a modified version of the Quality and Outcomes Framework (QOF) (Supplementary file 1). QOF is a simple measure that counts and summarizes the number of comorbidities for each individual based on a list of frequent chronic diseases that are particularly treated in general practice [28]. Data on comorbidity was obtained from DNPR in the inclusion period and divided into categories of 0,1 or 2 or more comorbidities.

In the follow-up period data on contact with additional health services was obtained from NHSR, DNPR, and DNPR-Psych and divided into three variables: Out-of-hours contacts with GP, contacts to secondary services (hospital), and contacts to psychiatric services.

See supplementary file 2 for a list of all the included variables.

Model performance

Receiver Operating Characteristic (ROC) curves are graphical plots illustrating the models’ performance conditional to different thresholds of the probability of the outcome [29]. The ROC curves therefore illustrate the models’ ability to predict and differentiate between the binary outcome at varying thresholds [30]. The area under the ROC curve (AUC) is an overall measure of the model’s discriminative ability. A perfect discriminative ability is indicated by AUC = 1 and a value of AUC = 0.5 indicates that the discriminative ability is not better than chance.

A concordance statistic (ROC curves) and goodness-of-fit statistic (Hosmer-Lemeshow (HL) test) were calculated to evaluate the discriminative performance and calibration of the models. The HL test is a goodness-of-fit test that assesses the conformity between the model’s predicted probability of the outcome, and the actual probability of the outcome in the data [29, 31]. The outcome of the test is a p-value. A significant p-value indicates that the model’s prediction differs significantly from the observed values, indicating that the model has an insufficient calibration.

Statistical analysis

The statistical analyses were conducted in SAS version 9.4. Chi-square tests (categorical variables) and general linear model (continuous variables) were used to assess differences in the baseline characteristics across diagnoses of the study population at baseline.

Multivariable logistic regression analysis with a backward step-by-step approach was conducted to develop the model structures of two prediction models for no contact with general practice. Hence, influential variables to be included in the predictive models were identified through significance tests in these models, where the only implicit assumption of consequence is of linearity in the logit for continuous variables [32]. A simple model included the demographic and contact with additional health services variables. An extended model included both demographic and contact with additional health services variables and two-way interactions between these. All 14 demographic and contact with additional health services variables were included in both models, and all possible two-way interactions was also included in the extended model as a starting point. Thereafter, incrementally, the variable with the highest p-value was removed, until all variables included in the simple model and all variables or interactions in the extended model had a p-value < 0.001. For the extended model, insignificant variables remained in the model if they were part of a significant two-way interaction.

The parameter estimates of the two models were found by subsequently carrying out the models on a random selection of 90% of the study population. The two models that were developed were a simple model that included only significant variables and an extended model that included significant variables and significant two-way interactions (p < 0.001).

Finally, the models were cross-validated on the remaining 10% of the study population according to the hold-out method. Hence, the performance test on the 10% holdout data of the models, found through the model search and the belonging parameters estimated in the 90% holdout data, was assessed through the AUC of the two ROC curves (discrimination) and the HL test (calibration).

The procedure for the cross-validation is displayed in Fig. 4.

Fig. 4
figure 4

Procedure for cross-validation

Full size image

Results

Descriptive results

Table 1 shows the distribution of the included variables in the study population and across the SMI (diagnoses) groups at baseline. The mean age was 43.5 years and 50.0% were women. The distribution of all variables differed significantly between the SMI diagnoses at baseline (p < 0.001). People with high odds of 0–2 contacts with GP were characterized by having a psychotic disorder, being male, living in region Zealand, being of other origin than Danish, being unmarried, having a low income, not having any comorbidities, and having high contact with psychiatric services.

8.9% of the study population had 2 or fewer contacts with a GP in the follow-up period. Persons with psychotic disorders had the highest proportion of persons with 0–2 contacts with GP, and persons with bipolar disorder had the lowest proportion with 0–2 contacts (p < 0.001). A sensitivity analysis using a cut-off point at < 7 contacts to GP showed only marginal changes in the associations. 79.6% of the study population had ≥ 7 contacts (Supplementary file 3).

Table 1 Characteristics of the study population
Full size table

Prediction models for 0–2 contacts with GP

The simple logistic model based on a random sample of 90% of the study population yielded a model including the variables SMI, sex, age, region, ethnicity, marital status, income, comorbidity, out-of-hours contacts with GP, contacts to secondary services (hospital) and contacts to psychiatric services, which all had a significant association with 0–2 contacts witch GP (see Supplementary file 4). The extended logistic model yielded a model, that included the same variables as the simple model as well as occupation, education, and ten two-way interactions (Fig. 5).

Fig. 5
figure 5

Model structure for the simple and extended model

Full size image

Receiver Operating Characteristic (ROC) curves were applied to assess the prediction models’ discriminative ability. The curves show the models’ ability to classify the outcome correctly conditional on consecutive thresholds [29], and are presented in Fig. 6. The overall discriminative ability of the models are measured by the AUC, where perfect discriminative ability is indicated by AUC = 1. The %-points on the curves represent the threshold for when the model classifies an observation as an outcome. The simple model had an AUC of 0.789 and the extended model had an AUC of 0.790.

Fig. 6
figure 6

ROC curves for the simple and extended model. a ROC curve for the simple model. b ROC curve for the extended model

Full size image

The goodness-of-fit test (HL test) was applied to assess the performance of the prediction models in terms of calibration that is, the conformity between the model’s predicted probability of the outcome, and the actual probability of the outcome observed in the data [29, 31]. The results of the tests are presented in Fig. 7. The calibration curves show the correspondence between the observed and expected probability for the outcome. In the figure, the dotted line indicates perfect calibration, where the expected probabilities for the outcome predicted by the model, correspond to the observed probabilities. The p-value of the HL test for the simple model was 0.349 and 0.676 for the extended model.

Fig. 7
figure 7

Calibration (HL test) for the Simple and extended model. a Calibration for the simple model. b Calibration for the extended model

Full size image

An overview of the models’ performance (discriminative ability, AUC, and calibration, HL test) can be seen in Table 2.

Table 2 Overview of the results of the models’ performance
Full size table

Discussion

Principal findings

The objective of the study was to develop a predictive model that could identify persons with SMI and no contact with a GP using sociodemographic and clinical data collected from national registers. We developed two models: a simple prediction model with 11 variables and an extended prediction model with 13 variables and 10 two-way interactions. While a simple model may have better practical applicability [29], the extended model showed a slightly better performance. This was expected since the simple model was included in the expanded model. AUC showed slightly better predictive ability of the extended model, compared to the simple model. Though the p-value from the HL test was higher for the extended model compared to the simple model this does not necessarily mean that we can conclude a better calibration of the extended model. When testing the calibration of a model, that has been developed in a large dataset, in a smaller dataset, a lack of statistical significance can simply be due to the smaller sample size, and thus, lack of statistical power [33]. To deal with this issue, it has been suggested to assess the calibration curves’ deviation from the 45° line of perfect fit [33]. The calibration curves shown in Fig. 7 indicate greater conformity between the predicted probabilities and the observed probabilities for no contact with the GP in the extended model.

As depicted in the ROC curves, the threshold for when the predicted probability for no contact with GP is classified as the outcome is related to the model’s sensitivity and specificity [29]. In light of the aim of the study, we would like to argue for selecting a lower threshold. A low threshold permits the classification of persons at risk of not having contact with GP even when their predicted probabilities for no contact are relatively low. With a lower threshold we prioritize sensitivity over specificity as we consider the consequences of false negatives to be more severe than those of false positives, i.e., wrongly identifying a person as having regular contact with a GP, more severe than false positives, i.e., mistakenly identifying a person as having contact with a GP, whilst this is not true.

The sociodemographic variables predicting no contact with GP in our simple model, are generally in line with the existing literature. Similar to our study, young men were found to have a lower contact frequency with their GP in Germany and Australia [34, 35]. Although evidence is limited, ethnic minorities also seemed to have less access to healthcare services in accordance with the results [36]. Furthermore, people with SMI who are married or in a relationship have been documented to have more contact with GP than people with a single status [34, 37] Persons with SMI living in rural areas have limited access to healthcare often due to lack of transportation [36, 37]. This observation is likewise relevant to our study population, as the number of GPs per capita is lower in low population-density areas in Denmark [38]. The finding that the absence of comorbidity in persons with SMI yielded a higher risk of non-attendance with the GP, aligns with another Danish study showing higher contact with GPs among persons with schizophrenia and comorbidity as well as in an Australian study with people with psychotic disorders, where frequent attenders were more likely to suffer from chronic somatic health problems [11, 35]. The inclusion of interactions in the extended model improved the identification of persons with SMI and no contact with GP. To our knowledge, no studies in the literature have examined interactions between sociodemographic variables and contact with GP, making it difficult to discuss and compare our findings related to the extended prediction model.

In our study, we used income, occupation, and education as measures of socioeconomic position (SEP). In the simple model low income was associated with no contact with GP, as opposed to findings from Australia [35] and Latvia [39], where low SEP was associated with a higher contact frequency. In Denmark, it has been shown that people with a low SEP experience more barriers to accessing GP in general [40, 41]. Occupation and education only had a significant effect in the extended model.

In Denmark, it is possible to receive services from different healthcare sectors free of charge, which might influence the frequency of contacting a GP. Previously, secondary services have been considered when investigating contact with a GP among persons with schizophrenia in Denmark [11]. In Norway, continuity in contact with a GP was also assessed in relation to the use of out-of-hour services and acute hospital admissions among the general population [9]. In addition to sociodemographic factors, this study, therefore, analysed the effects of contact with different healthcare services on contact with GP. The results showed that contact with other healthcare services, such as secondary or psychiatric services, indeed can affect the contact frequency with a GP. This highlights the importance of considering multiple healthcare services, since not having contact with GP, does not necessarily indicate not having contact with all health care services.

Strengths and limitations

A number of limitations in our study should be taken into account. We defined no contact as 0–2 contacts with GP in the follow-up period. This was because of the possibility of the GP reaching out to patients by telephone or email and registering this as a contact point in DPNR, even though no actual contact had been established. A cut-off for no contact at 0 contacts in the follow-up period, would entail a risk of misclassifying persons with SMI and no contact with GP as having contact. We considered it more important to capture all persons with SMI and possibly no contact with GP, than misclassifying some persons with few contacts (1–2 contacts in the follow-up period) as having no contact. There is a risk of misclassification of SMI due to its broad definition and complexity, although the DNPR-psych has almost complete coverage [18, 42], and the ICD-10 classification ensures standardization in diagnosis [43]. To differentiate between multiple diagnoses a hierarchy of SMI diagnoses based on their severity was created. However, misclassification within diagnoses might obscure or reduce the associations within each diagnostic group and contact with the GP.

Apart from misclassification, a risk of selection bias cannot be excluded. Recovery from diagnosis is not registered in the DNPR-Psych. As the inclusion period of the study consisted of five years, there is the potential of included persons with SMI recovering from their diagnosis during the inclusion period. Incorrect classification of SMI can impact the associations with contact with GP in the study population. Another risk of selection bias concerns persons with low functional capability due to SMI who might be too affected to seek health care and are therefore not registered with a diagnosis in the DNPR-Psych. If this is the case, the lack of inclusion of this group might reduce the associations with contact with GP further.

Comorbidity in the study population was estimated using a modified version of the Quality and Outcomes Framework (QOF). In the literature, the Charlson Comorbidity Index (CCI) has also been used [44]. QOF is a simpler measure than CCI, as it counts the number of comorbidities based on a list of frequent chronic diseases treated in GP and predicts variation in mortality in GP more effectively [28, 44]. However, we were only able to include diagnoses registered during hospital contacts, leaving out those comorbidities treated only in GP. The possibility of unmeasured comorbidity in the study population therefore exists. Moreover, there is a risk of confounding due to unmeasured variables. Loneliness and lifestyle factors in the literature were identified as influencing contact with GP among persons with mental illness [35, 45]. However, it was not possible to include these factors and the proxies for measuring these factors were also very limited. Therefore, these variables would not be possible to include in a register-based identification of persons with SMI and no contact with GP.

As our study population was identified using national registers the risk of selection bias in our study is minimal, thus making the study population representative for persons with SMI in Denmark. Representativity improves external validity, making the findings applicable to countries with comparable healthcare systems and similar populations [46]. The collection of data on contact to GPs in NHSR eliminated the risk of recall bias and GPs are financially incentivized to report all consultations, ultimately improving the completeness of the measurement of the variable [20]. Another strength of the study is the relatively long follow-up period. The 2-year follow-up period is comparable to other related studies [11, 34,35,36,37, 45, 47] and allows precise measurement of the outcome while maintaining the same exposure-outcome association over time [48].

Implications

This study found the extended model optimal for the identification of persons with SMI and no contact with GP. While the models employed are relatively simple, logistic regression models, their performance is unlikely to be much improved from the use of more modern machine learning methods; the latter considerably increasing complexity.

The choice of the classification threshold for outcomes plays a pivotal role in determining the balance between false negatives and false positives in the identification. Consequently, it’s imperative to tailor the intervention strategy, considering factors such as invasiveness and associated costs. Further development and application will need to incorporate the perspectives of general practitioners and persons with SMI to validate and study the future use of such a model [49]. An existing intervention in Denmark focused on identifying somatic disease among persons with SMI [50], in which patient medical records were used for the identification of the target group. This identification method has been shown to be time-consuming, administratively heavy, and imprecise [50]. On a central level, the sociodemographic variables identified in this study can be used for developing a tool that physicians can use, potentially elevating the chances of reaching those persons with SMI, who do not have regular adequate contact with their GP.

Conclusion

This study developed a prediction model to identify persons with SMI and no contact with GP using data obtained from Danish national registers. The model might be used in population programs focusing on prevention and timely treatment of somatic diseases, thereby possibly reducing premature mortality among persons with SMI. Developing a prediction tool based on sociodemographic variables can help physicians reach individuals who do not have regular contact with their GP. However, considerations of the optimal classification threshold for the model should be made, and future studies are needed to provide insight into the utility and effect of the prediction model in the identification of patients with SMI and no contact with a GP.