Background

Malaria remains a major public health concern globally and is considered as one of the most prevalent and lethal human infectious diseases among children in sub-Saharan Africa [1]. Despite the drastic reduction in the number of malaria cases and deaths in all ages globally, it still accounts for 10% of child deaths in sub-Saharan Africa [1], and mortality is mostly higher among children below the age of 5 years. Individuals in endemic regions increasingly develop immunity to malaria with age and this is conventionally thought to reflect a slow and gradual acquisition of protective antibodies [2]. Asymptomatic carriers may be a reservoir for malaria transmission [3]. It has recently been shown that interaction between naturally acquired antibodies to Plasmodium falciparum and polymorphisms in host FCGR3B gene, encoding the Fc Gamma Receptor IIIB (FcγRIIIB) plays a key role in immunity against malaria [4]. The FcγRIIIB is exclusively expressed on human neutrophils and crosslinking with immunoglobulin (Ig) G antibodies mediates neutrophil degranulation and generation of reactive oxygen species (ROS) [5], which kills intra-erythrocytic P. falciparum [6]. It is conceivable that other host genes may also modify the protective effect of malaria antibodies. This emphasizes the need for robust modelling approaches to effectively address such confounders in malaria vaccine studies. It is quite plausible that the long delay in attaining an effective malaria vaccine may partly be due to inadequacies of traditional statistical approaches used in malaria immuno-epidemiological studies to determine the performance of predictors in classifying or predicting malaria risk [7,8,9,10]. Traditionally, most studies use generalized linear models (GLM) depending on the measurement scale of clinical malaria. GLM provides an extensive class of tools for modelling the effect of predictors. Statistical prognostic modelling techniques have been applied primarily in the area of non-communicable diseases such as cardiovascular diseases and lung cancer. For instance, Gail et al. [11] developed a model of breast cancer risk prediction and implications for chemoprevention which was later validated by Rockhill et al. [12]. Several risk prediction models for other cancers and cardiovascular diseases [12,13,14,15,16,17,18,19,20,21] have also been developed. For clinical malaria, on the other hand, personalized risk estimation has not been extensively studied. As indicated by several authors [7,8,9,10, 22], markers such as polymorphisms and antigen-specific antibodies proposed for classifying or predicting risk in individual subjects must be held to a much higher standard than just assessing associations based on odds ratio estimates. Pepe et al. [22] showed that strong statistical associations (including odds ratio, relative risk) between disease and host-specific factors found in literature do not necessarily imply that those factors can discriminate between a subject who is likely or not have the disease in a specified time. A risk prediction model exploits the joint predictive power of several variables on the risk of an event or disease. A robust malaria risk prediction model based on epidemiological predictors may contribute to finding possible answers to the question of which parasite antigens and host factors should be the main research focus in the efforts to find optimal control strategies and vaccines. This is particularly important as the number of malaria-specific antibodies and host gene polymorphisms found to be associated with clinical malaria have increased significantly over the past few years [3] but with little impact on malaria control. Using a more rigorous prediction modelling approach, this study aims to evaluate the predictive performance of malaria antibodies and FCGR3B gene polymorphisms on P. falciparum infection outcome. To assess the performance of the models, this study used Brier scores and area under the receiver operating characteristic curve (AUROC) through appropriate bootstrap cross-validation design.

The identified model was obtained by comparing Brier score estimates and the AUROC curve of several models that integrate malaria antibodies and host gene polymorphisms.

Methods

Data source

Data used for modelling the risk of malaria was secondary data obtained from a prospective longitudinal malaria cohort study which was conducted from May 2008 to January 2009 among children under 13 years of age in five different communities in the Shai Osudoku (formerly Dangme West) district of Ghana (Asutsuare, Kewum, Mafikorpe, Osuwem, and Volivo) [20, 21]. These children were observed both actively and passively for malaria case detection. The primary outcome measure was clinical malaria defined as fever with any level of P. falciparum parasitaemia plus at least one clinical symptom of malaria, such as vomiting, joint pains, diarrhoea. In this study, the term “Susceptible” and “Protected” are used to represent clinical malaria and no malaria case detection respectively over the study period. The proportion of children that developed malaria in the 1 year follow up was 15.0% (incidence proportion) and there were approximately 1.7 malaria cases per 100 children per month (incidence rate) [23]. The predictors of clinical malaria included age in years, sex, sickle cell status, blood group, haemoglobin level, malaria antigen-specific immunoglobulin (Ig) G and subclasses (IgG1, IgG2, IgG3, and IgG4) and FCGR3B polymorphisms (c.108C>G-rs403016, c.114T>C-rs447536, c.194A>G-rs448740, c.233C>A-rs5030738, c.244A>G-rs428888 and c.316A>G-rs2290834). Malaria antigens studied were Merozoite surface proteins (MSP-1 and MSP-3), Glutamate Rich Protein (GLURP)-R0, R2, and the Apical Membrane Antigen (AMA-1). Antibody levels were measured by sandwich ELISA and optical density values converted to antibody units using a four parameter curve fitting program by means of a reference curve on each plate generated by serial dilution of malaria hyperimmune sera [3].

This study includes a summary of the climatic variables at the time of the study to serve as a guide for future studies that may wish to compare their findings to this study. Changes in relative humidity (RH), which is the ratio of the partial pressure of water vapor relative to saturated vapour at the specified temperature was assessed over the study period. The minimum and maximum relative humidity over the study period were 71.0% and 98.0% respectively. The total monthly rainfall ranged between 0 and 273.5 mm with the month of May 2008 recording the highest total rainfall. The total rainfall between June and December 2008 ranged from 19.7 to 125.4 mm. The maximum daily rainfall was recorded in the month of June 2008 (60 mm). There was no rainfall in the month of January 2009. The minimum and maximum air temperature ranged from 19.3 to 36.8 °C.

Malaria risk prediction model and performance measures

Let \( D_{n} = \left\{ {Y_{i} ,\varvec{X}_{i} } \right\}_{i = 1, \ldots ,n} \) be a malaria data set with n number of children aged less than 13 years. \( \varvec{X}_{i} = \left( {X_{ip} } \right)_{p = 1, \ldots ,P, i = 1, \ldots ,n} \) be an input matrix of P predictors of malaria (age, antibodies) and

$$ Y_{i} = \left\{ {\begin{array}{*{20}c} {1\;if\;the\;child\;is\;positive } \\ {0\;if\;the\; child\; is\; negative} \\ \end{array} } \right. $$

Let \( G \subseteq \left\{ {1, \ldots G} \right\} \) be a subset of the available predictors where \( p = p_{1} ,p_{2} , \ldots ,p_{g} \). Let \( \beta = \beta_{1} , \beta_{2} , \ldots ,\beta_{g} \) be the regression coefficient to be estimated. The predicted risk of malaria is modelled using the logistic regression model. Specifically, for the trained prediction model, \( \hat{\tau }_{n} \) which assigns to each child the probability of developing clinical malaria, the estimated malaria risk prediction model is given by:

$$ P\left( {Y_{i} = 1|\varvec{X}_{i} } \right) = \hat{\tau }_{n} \left( {\varvec{X}_{i} } \right) = \hat{\tau }_{n} = \left[ {1 + \exp \left\{ {\hat{\beta }_{0} - \mathop \sum \limits_{g \in G} \hat{\beta }_{g} \left( {X_{p} } \right)} \right\}} \right]^{ - 1} $$

where \( \hat{\beta } \) is a vector of estimated parameters of the model associated with predictor variables, \( \hat{\beta }_{0} \) is the estimated intercept of the logistic regression model considered as the baseline risk of malaria and \( \hat{\tau }_{n} \left( {\varvec{X}_{i} } \right) \) is the predicted risk of malaria for the ith child with baseline characteristics \( \varvec{X}_{i} \).

The modelling strategy was to first identify a baseline model out of several other competing models relating to the prevalence of malaria to baseline covariates. Comprehensive discrimination and calibration assessments of the fitted models were explored based on the AUROC curve, Brier score, root mean squared error and the total explained variations in predicted probabilities (R2) using bootstrap cross-validation from 200 bootstrap samples with replacement. In selecting predictor variables to be included in the final model, candidate predictor variables were screened using tools for discriminative and calibration abilities such as the area under the receiver operating characteristics curve, the Brier score and explained variation in predicted risk. The model’s ability to predict accurately was further assessed by means of the calibration plot. According to Gerds et al. [24], the values of the Brier score could be interpreted as the loss or regret which is incurred when the prediction model \( \hat{\tau }_{n} \) is applied to a child whose true malaria status is \( Y_{i} . \) Model performance was assessed via bootstrap cross-validation. Selection of predictor variables (age, sickle cell, blood group, haemoglobin, parasite density and, mosquito net use) for the baseline model was premised on subject matter knowledge and prior evidence of association with the risk of malaria from literature. Continuous predictors were fitted via restricted cubic splines. To quantify over-fitting and to recalibrate the model, the heuristic shrinkage estimator \( \hat{\gamma } = \frac{{model\chi^{2} - p}}{{model\chi^{2} }} \) was used where p is the number of predictors (regression parameters including both linear and non-linear and possible interaction terms), \( \chi^{2} \) is the likelihood ratio \( \chi^{2} \) test statistic computed using the full set of p parameters to determine whether any of the predictor(s) is/are associated with log-odds of developing clinical malaria. For the model to be calibrated well for future data, \( \hat{\gamma } \) was multiplied by \( X\hat{\beta } \) and that defines shrinkage. The penalty factor was determined by means of repeated cross-validation of the data. All antigen-specific antibodies were log-transformed to base e in subsequent analysis. All models were fitted with R programming software version 3.2.4 with the following specialized packages: Design [25], Penalized [26]. Data cleaning and all other forms of data preparations were done with Stata SE version 13. A p-value of < 0.05 was considered statistically significant.

Results

Description of study participants and clinical malaria distribution

The study recruited 799 children of which 393 (49.2%) were males and 406 (50.8%) were females. Complete information on candidate predictors was available for 395 children. The missing data were due to children for whom either no or insufficient plasma was available for all antibody measurement. Additionally, children for whom there was no DNA or poor quality DNA that resulted in poor genotyping data were excluded. The overall median age for this study sample was 5.0 years (Interquartile range = 3.0–8.0). The cumulative incidence of malaria was 13% (53 out of 395 children). The bed net use among the children was 40.5%. The analysis of sociodemographic characteristics and baseline biomarkers on the risk of clinical malaria indicated that the cumulative incidence of malaria did not differ significantly among the baseline predictors that were studied (p-value ≥ 0.05). Distribution of other predictors and clinical malaria status can be found in Table 1.

Table 1 Bivariate analysis of malaria predictors and clinical malaria
Full size table

Modelling results on baseline covariates and socio-demographic characteristics

Four different initial models were rigorously assessed and the best model was chosen to serve as the baseline model using the aforementioned indices. These models are presented in Table 2. ‘Model.standard’ is a linear additive model of age in years, haemoglobin level in (g/dl), sickle cell, blood group, bed net use and parasite density categorized into positive and negative for presence and absence of parasite in blood at enrollment, respectively. The ‘Model.spline’ is the additive model (Model.standard) but it was modelled the nonlinear effect of age and haemoglobin using 5 knots restricted cubic splines, and also included an interaction term of bed net use, parasite density. The ‘Model.slope.optimism’ (SCFM) is the ‘Model.spline’ but with regression coefficients shrunk by slope corrected optimism. Finally, a ‘PMLE.model’, which is the ‘Model.spline’ but has estimates of the β coefficient that were based on the Penalized Maximum Likelihood Estimation procedure [27]. Upon careful consideration based on the above model prediction performance measures, the ‘PMLE.model’ was chosen as the baseline model. The bootstrap cross-validated estimates of AUROC, Brier score, root mean squared error and explained variations in predicted probabilities of the ‘PMLE.model’ are 50.0%, 10.0%, 0.32 and 17.0%, respectively. Although these performance indices were generally poor, it was better as compared to the other models (Table 2). The probability that PMLE.model will assign a higher predicted risk to a randomly chosen child with malaria compared with a randomly chosen child with no malaria is 50.0% (discrimination ability = 50.0%; Table 2), which is not better than random prediction. The Brier score of 10.0% is the expected loss or regret incurred when predicted risk from PMLE.model is issued to a child whose true malaria status is either susceptible or protected. The variations in predicted risk of malaria explained by the chosen model is 17.0% (Table 2). Furthermore, the calibration plot in Fig. 1 shows that the PMLE model underestimates children at low risk of malaria and overestimate children at higher risk of malaria.

Table 2 Predictive effect of socio-demographic indices and baseline covariates
Full size table
Fig. 1
figure 1

Calibration plots comparing the four baseline models

Full size image

Modelling results on antigen-specific antibodies and FCGR3B polymorphisms

The predictive effect of each antibody (IgG and subclasses) and FCGR3B was evaluated by introducing them one after the other in the selected baseline penalized maximum likelihood model (PMLE.model) as shown in Table 3. All antibodies were modelled via 5 knots restricted cubic spline. Admittedly, none of the antibodies nor the FCGR3B could significantly improve the performance of the PMLE.model after introduction but it was observed that IgGAMA1, IgG1AMA1, and FCGR3Bc.233C>A were better than all the other predictors in relation to their bootstrap cross-validated estimates of relatively higher AUROC, lower Brier score, and and relatively higher R2.

Table 3 Assessing the predictive effect of each malaria antibody and FCGR3B polymorphisms on the risk of malaria
Full size table

Evaluating the joint effect of IgGAMA1, IgG1AMA1 and dominant gene c.233C>A

The selection of IgGAMA1, IgG1AMA1, and dominant gene c.233C>A in the subsequent model building were based on the fact that they had a higher AUROC and R2, and a smaller Brier Score as previously shown in Table 3. First, the analysis involved a model which was basically the baseline model (PMLE.model) together with the three predictors (IgGAMA1, IgG1AMA1 and dominant gene c.233C>A) but with no shrinkage adjustment to the regression coefficients. This model was denoted as the final model with no shrinkage (FMNS). The second model was a baseline model (PMLE.model) together with the three predictors (IgGAMA1, IgG1AMA1 and dominant gene c.233C>A), but with shrinkage of the regression coefficient using van Houwelingen-Le Cessie heuristic estimate. This was done to improve the calibration ability of the model. This was denoted as slope corrected optimism with the van Houwelingen-Le Cessie heuristic (SCOV) model. Finally, a model was fitted and evaluated with the only IgGAMA1, IgG1AMA1 and dominant gene c.233C>A. This was denoted as the no baseline line covariate (NBC) model. The predictive performance indices of these three final models were evaluated and the results showed that the NBC model with only IgGAMA1, IgG1AMA1, and dominant gene c.233C>A predicted malaria incidence better as this model had the highest bootstrap cross-validated AUROC and R2 with a smaller Brier score as shown in Table 4.

Table 4 Prediction performance measures of the final selected models
Full size table

Assessing the prediction performance of the three models using calibration plots

The prediction performance of these models was explored by examining calibration plots, that is the plots of sensitivity versus (1-specificity) (AUROC). There was no significant difference in their respective areas under the curve before internal validation, although the model with penalty corrected parameter estimates (SCOV model) had a higher AUROC (75.3%). After internally validating these models, it was observed that the AUROC of the model with only IgGAMA1, IgG1AMA1, and dominant gene c.233C>A (NBC model) performed better than the other two models (AUROC = 61.5%) (Fig. 2).

Fig. 2
figure 2

Discrimination and calibration ability of the three final models. FMNS = Final model with no shrinkage, SCOV Final model with slope corrected optimism using Van Howelligen estimator, NBV Model with no baseline variables (only IgGAMA1, IgG1AMA1, and dominant gene c.233C>A); BCV Bootstrap cross-validation

Full size image

The calibration plots in Fig. 3 shows that the same model was well calibrated as it appears to be closer to the 45° line compared to the two other models.

Fig. 3
figure 3

Calibration plots comparing the three final selected models

Full size image

In relation to how these models assign the low and higher predicted risk of malaria using the box-whisker plots, there was not much difference between the model with slope corrected optimism and the model with only IgGAMA1, IgG1AMA1, and dominant gene c.233C>A although both were slightly better than the PMLE model updated with 3 more predictors (Fig. 4).

Fig. 4
figure 4

Evaluating the discrimination ability of the three final selected models. Abbreviations: SCOV: slope corrected optimism model, NBC: no baseline covariate model, that is model with only IgGAMA1, IgG1AMA1, and c.233C>A genotype, FMNS: Parameter estimates via penalized maximum likelihood estimation

Full size image

Discussion

The study investigated the predictive performance of several malaria-specific antibodies (IgG and subclasses) and FCGR3B polymorphisms on the malaria risk. This was achieved by comparing a baseline malaria model with a prediction model that integrates the antibody and genetic data. Malaria prognosis of a child is an estimate of the child’s future malaria risk. The prognosis in this study is based on the child’s baseline socio-demographic characteristics, blood group, sickle cell status, the use of bed net, presence of malaria parasite in blood and malaria-specific antibodies and FCGR3B genotype. Most of the socio-demographic factors, malaria antigen-specific antibodies and the FCGR3B variant used in this study as predictors have been found to be associated with clinical malaria in various malaria seroepidemiological studies [23, 28,29,30,31,32,33,34]. The discrimination and calibration performance of the baseline and integrated models measured via AUROC and the Brier score were far less than the generally recommended 80% or more for AUROC and smaller Brier score (closer to zero). Although most of the antibodies did not improve the performance of the baseline model, it is worth noting that AMA1 specific antibodies and FCGR3B-c.233C>A under the additive and dominant models of inheritance, can discriminate children of low and higher risk of malaria. Admittedly, the improvement in AUROC and Brier score was not very substantial from the baseline model in this study but they showed signs of improving the performance of the baseline model. This finding was consistent with other studies [31, 34,35,36,37,38] which identified AMA1 as an important blood-stage malaria vaccine candidate.

Traditional statistical methods of estimating odds ratios, relative risk and hazard ratio’s in epidemiological studies to assess associations between antibodies, genetic polymorphisms, and clinical malaria may not adequately determine the performance of each biomarker for predicting the risk of malaria for a child [22]. An antibody or gene variant associated with protection against malaria does not necessarily imply that it can significantly discriminate between randomly chosen children with a low or high risk of malaria and consequently, will not improve model prediction performance. This may have contributed to the poor predictive performance of most of the antibodies and genotypes in this study. For classification of children into the high and low risk of malaria, statistical techniques should be used that directly address classification accuracy (e.g. Brier score, AUROC) rather than traditional regression models for assessing associations (reporting of the odds ratio, relative risk and hazard ratio’s for time to event outcomes). Studies that link antibody responses and gene polymorphisms to clinical malaria either control for age or restrict the analysis to individuals who have been exposed to P. falciparum [31]. In situations where they have adjusted for most of the predictors of malaria, the predictive performance of certain antibody responses was reduced and only a few remained statistically significant [31]. Other important parameters such as haemoglobin level and sickle cell status should all be controlled for in a prediction model and its predictive performance assessed [39]. The inconsistencies in malaria risk prediction model performance indices may also be due to misclassification of the outcome variable (clinical malaria). There are several different case definitions for clinical malaria based on different parasitaemia threshold values and what would have been recorded as a case (malaria) in a particular study may be recorded as control (no malaria) in a different study and vice versa. The number of events (malaria cases) studied is important for prognostic research [41] because there is the risk of overestimating the predictive performance of the model when the number of predictors is much larger than the number of outcome events (malaria). Besides, different prognostic studies have suggested that for each candidate predictor (antibodies, host genes, age, sickle cell, haemoglobin, parasite density) studied, at least 10 events (malaria cases) are required [40,41,42,43] but these numbers could be lower in certain circumstances [44]. If for instance the number of cases studied is 200 based on fever with a threshold of 2500 parasites per microlitre of blood, then there will be differences in the number of events in a different study where clinical malaria was defined as fever with any level of parasitaemia with the latter having a larger number of cases in general to improve the power of the test. So long as malaria case definitions are not clearly defined, there will be inconsistencies in results of which antibody, host genetic factor and or their interactions have higher predictive performance on the malaria risk prediction model. Thus, in practice, different modelling strategies may result in similar or very different results even if they are applied to data from the same study. This study did not quantify the predictive performance of climatic factors on the model performance because children enrolled in the study were clustered around the same endemic area with no significant difference in climatic variables. This study recommends that prospective studies may consider including climatic variables in the malaria predictive model in situations where there are climatic differences in the geographic locations of subjects being studied.

Other factors that contribute to the incidence of malaria but were not measured to assess their predictive effect may also be of interest. These include drug resistance in parasites, vector species, parasite strain and insecticide resistance in mosquitoes [32, 45,46,47,48]. The study’s inability to observe these factors may have contributed to poor model performance. The proportion of missing values on covariates was relatively high and can result in loss of statistical power, efficiency and precision of the predicted risk notwithstanding the fact that standard statistical procedures of handling missing data were adhered to. This is particularly so when advanced statistical techniques that handle missing completely at random, missing not at random and missing at random make critical but untestable assumptions about how the data went missing [39]. The proportion of missing observations was common among immunoglobulins and host genetic factors which coincidentally were key predictors of interest in the study. To accurately evaluate the effect of malaria antibodies and host genetic factors on the risk of malaria, steps should be taken to reduce to the barest minimum the proportion of missing values among candidate predictors of malaria risk as this may bias model performance metrics.

Conclusion

This study has found that the FCGR3B-c.233C>A genotype and IgG against AMA1 were relatively better compared to the other antibodies and FCGR3B genotypes in classifying or predicting malaria risk among children. The findings support Apical Membrane Antigen 1 as an important malaria vaccine antigen while FCGR3B-c.233C>A under the additive and dominant models of inheritance were also identified as an important modifier of the effect of malaria protective antibodies. Furthermore, the goal of malaria etiological risk factor studies may be quite different from studies where antibodies and host genes are used in classifying a child into high or low-risk groups. Hence the statistical methods between such studies differ to a large extent. If the latter is required, then it would be appropriate to e use model discrimination and calibration indices, such as Brier score, RMSE, and AUROC.