Abstract
Objectives
Multivariate metaanalysis allows the joint synthesis of multiple outcomes accounting for their correlation. This enables borrowing of strength (BoS) across outcomes, which may lead to greater efficiency and even different conclusions compared to separate univariate metaanalyses. However, multivariate metaanalysis is complex to apply, so guidance is needed to flag (in advance of analysis) when the approach is most useful.
Study design and setting
We use 43 Cochrane intervention reviews to empirically investigate the characteristics of metaanalysis datasets that are associated with a larger BoS statistic (from 0 to 100%) when applying a bivariate metaanalysis of binary outcomes.
Results
Four characteristics were identified as strongly associated with BoS: the total number of studies, the number of studies with the outcome of interest, the percentage of studies missing the outcome of interest, and the largest absolute withinstudy correlation. Using these characteristics, we then develop a model for predicting BoS in a new dataset, which is shown to have good performance (an adjusted R^{2} of 50%). Applied examples are used to illustrate the use of the BoS prediction model.
Conclusions
Cochrane reviewers mainly use univariate metaanalysis methods, but the identified characteristics associated with BoS and our subsequent prediction model for BoS help to flag when a multivariate metaanalysis may also be beneficial in Cochrane reviews with multiple binary outcomes. Extension to nonCochrane reviews and other outcome types is still required.
Similar content being viewed by others
Introduction
Conventional methods for metaanalysis produce a single summary result, for example about one particular outcome. In particular, in most intervention reviews an inversevariance weighting metaanalysis is typically applied to each outcome of interest separately, and so each metaanalysis utilises just one intervention effect estimate per study. This process can be coined a univariate metaanalysis, with the word ‘univariate’ indicating a single summary result is of interest. However, most metaanalysis projects aim to produce multiple summary results, especially because multiple outcomes are of interest, such as a hypertensive participant’s systolic (SBP) and diastolic (DBP) blood pressure, a migraine sufferer’s levels of pain and nausea, a cancer participant’s diseasefree and overall survival times, and pregnancy outcomes for both the mother and her baby [1]. This potentially motivates a multivariate metaanalysis, to produce multiple summary results (one for each outcome) jointly from the same metaanalysis model [2].
The key advantage of a multivariate metaanalysis of multiple outcomes is to account for their correlation [3]. At the participant level, multiple health outcomes are often correlated with each other, and this leads to correlation amongst multiple effect estimates from the same study. Such correlation of a pair of effect estimates is known as withinstudy correlation. For example, in a randomised trial of antihypertensive treatment, the estimated treatment effects for SBP and DBP are likely to have a positive withinstudy correlation, caused by a positive correlation at the participant level between SBP and DBP. When there is betweenstudy heterogeneity in effects there may also be betweenstudy correlation, which arises when the true effect for each outcome is correlated with the true effect for another outcome. For example, the true effect of antihypertensive treatment on SBP usually has a positive betweenstudy correlation with the true effect on DBP, caused by changes in study and participant characteristics (such as dose and mean blood pressure at baseline) which modify the true treatment effects on SBP and DBP in the same direction.
Accounting for outcome correlation in a multivariate metaanalysis allows more studies to contribute toward the metaanalysis results for each outcome (i.e. a study can be included in the analysis even though it has not measured all the outcomes of interest), which may improve efficiency and even decrease bias (e.g. due to selective outcome reporting [4]) compared to undertaking a separate univariate metaanalysis for each outcome. In particular, alongside any direct evidence, the multivariate metaanalysis allows the summary result for each outcome to depend on correlated results from other outcomes. The rationale is that by observing information from related outcomes we can learn something about the missing outcomes of interest, and thus gain some knowledge that is otherwise lost; a concept known statistically as borrowing of strength (BoS) [5, 6]. However, a downside of multivariate metaanalysis is that the approach is more complex than undertaking separate univariate metaanalyses (one for each outcome), as it requires the metaanalyst to obtain or derive withinstudy correlations between pairs of treatment effect estimates in the same study [7, 8]. Furthermore, the amount of borrowing of strength is often small. Trikalinos et al. examined 45 Cochrane reviews that contained two or three binary outcomes that could be analysed using either univariate or multivariate metaanalysis [9, 10]. They conclude that if the “focus is on the summary effects and the confidence intervals then the choice between the univariate and multivariate metaanalysis has limited practical importance” [9]. Yet, isolated examples within the Trikalinos review do exhibit important differences between univariate and multivariate metaanalysis, and other examples exist where multivariate metaanalysis has an important impact [3, 11,12,13,14].
Guidance is therefore needed to help researchers identify (as part of a prespecified analysis and project plan) whether a multivariate metaanalysis may be useful in their particular review. To address this, in this article we use the set of Cochrane reviews identified by Trikalinos et al. to investigate the characteristics of metaanalysis datasets that are associated with larger BoS when applying a multivariate metaanalysis. We then derive a multivariable prediction model for predicting the amount of BoS conditional on these characteristics, which might be useful as part of a prespecified analysis plan to flag when to use the multivariate approach in practice. Note that we focus on the benefits of multivariate metaanalysis for estimating the summary effect for each outcome, and do not consider functions of the outcomes (e.g. differences in summary effect for outcomes 1 and 2), as then the benefit of a multivariate metaanalysis to account for correlation is more obvious (i.e. to determine the correct variance of the function) [15].
Methods
We firstly introduce the Trikalinos dataset to be used through the article. Then we describe our methods to identify characteristics associated with BoS, and for developing a multivariable prediction model for BoS.
Data analysis using 43 reviews of binary outcomes identified by Trikalinos
There were 45 reviews included in the empirical review by Trikalinos et al. [9, 10]. Each review contained at least seven studies that reported both outcomes or at least half the studies with both outcomes if the total number of studies was greater than 14. Each of the studies satisfying the previous requirement (i.e. at least seven studies with both outcomes) had at least 10 patients and at least two events. There were two reviews that contained three outcomes, but these were excluded for our purposes since we decided to focus on bivariate models. The remaining 43 reviews were included, and these contained between 7 and 132 studies with two binary outcomes of interest each with a cross classification (two by two) table summarising the number of outcome events and nonevents for the treatment and control groups. The relationships between the pair of binary outcomes was either mutually exclusive or an issubsetof relationship. An issubsetof relationships refers to when one outcome is contained within the other. For example, the number of patients that have survived with a particular condition at, say, 6 months and a year. A mutually exclusive relationship is when the outcomes are independent of each other and therefore occur separately. An example is death from breast cancer and death from other causes, excluding breast cancer.
For each of the 43 metaanalysis datasets, we used the twobytwo tables for each outcome in each trial to derive treatment effect estimates (log odds ratio estimates) and corresponding error variances. A fixed 0.5 continuity correction was required if any denominator in the equation for the variance was equal to zero [16, 17]; that is, if a study had a zero cell in the twobytwo table, then 0.5 was added to all cells for that study. This is a similar approach to the normal approximation analyses in the original Trikalinos review [9, 10]. For a pair of treatment effect estimates in the same trial, we also derived their withinstudy correlation using the formula provided for an issubsetof relationship [7, 18], and by Trikalinos and Olkin for a mutually exclusive relationship [19]. In some studies, the withinstudy correlation was +1 or −1, which can cause issues of singular variance matrices during the multivariate model estimation. To avoid this issue, we replaced any ±1 values with ±0.99., although other approaches are possible [18].
To each of the 43 metaanalysis datasets, a univariate commoneffect metaanalysis was applied to each outcome separately, using maximum likelihood (ML) estimation. Then we also fitted a bivariate commoneffect metaanalysis using ML estimation, to jointly analyse both outcomes whilst accounting for any withinstudy correlations. The ordering of outcome 1 or 2 was irrelevant (i.e. same results obtained regardless), though for the issubsetof reviews outcome 2 was designated to be the subset of outcome 1.
A bivariate metaanalysis can be conducted in many statistical packages, for example in Stata and R there is the mvmeta package (which, in Stata, also provides an option to calculate the BoS), and formula for univariate and bivariate metaanalysis models is shown elsewhere [2, 3].
Following the bivariate analysis, the BoS was quantified for each outcome by calculating the BoS statistic proposed by Jackson et al. [6]:
The BoS statistic provides the percentage reduction in the variance of a particular summary result that is due to (borrowed from) data from other correlated outcomes. It is the percentage weight toward the summary result for, say, outcome 1 that is given to the study data for other correlated outcomes [6]. For example, in a bivariate metaanalysis, a BoS of 0% for outcome 1 indicates that the summary result for outcome 1 depends only on data for outcome 1, i.e., that outcome 2 does not add any information for estimating outcome 1. As BoS increases, outcome 2 provides more and more information toward the estimate of outcome 1, reflected by reducing its variance.
The distribution of BoS statistic values was summarised using descriptive statistics and graphically via histograms (see Supplementary material).
The process was repeated but rather using univariate and bivariate randomeffects models, which allow for betweenstudy heterogeneity. Similar conclusions were drawn and so we focus on the results from the commoneffect metaanalyses in this paper. Further, some of the bivariate randomeffects models suffered from problems estimating the betweenstudy correlation (often ‘converged’ at 1 or +1, for reasons explained elsewhere [20]), and so we deemed it more reliable to focus on BoS observed for the bivariate commoneffect model. Results from the randomeffects models are shown elsewhere [21].
Examining characteristics associated with BoS
The following seven metaanalysis level characteristics were selected for examination of their association with BoS statistic values from a bivariate commoneffect metaanalysis:

The percentage of studies with missing data for the outcome of interest

The percentage of studies with missing data across both outcomes

The number of studies in the metaanalysis

The number of studies with only the outcome of interest

The number of studies with both outcomes

The average absolute withinstudy correlation

The largest absolute withinstudy correlation
These characteristics were identified by the research team based on analytic reasoning (see Supplementary material), and our previous (applied and methodological) experience [3, 12, 15, 22]. The unadjusted effect of each characteristic on the magnitude of BoS was estimated by fitting a linear regression with BoS as the outcome and the characteristic as the only covariate. Two BoS values were available for each of the 43 reviews (one for each outcome), and so the dataset had 86 outcome values in total. A random intercept was used to account for clustering of BoS values from the same study. We also considered modelling BoS on the log scale, but this did not change the findings importantly, and therefore we present results on the BoS scale to aid interpretation.
Development and internal validation of a prediction model for BoS
A multivariable prediction model was developed for predicting BoS in a new bivariate metaanalysis dataset. The 7 characteristics previously listed were candidate predictors for inclusion. A summary of the characteristics of the 43 metaanalyses examined by Trikalinos et al. [9, 10] is provided in Table A in supplementary material. As there were 86 BoS values for the modelling, this corresponded to 12.3 values per candidate predictor. At the time of model development, this was considered appropriate as it was larger than ten subjects per predictor (often a rule of thumb for sample size), larger than a recent proposal of two values per predictor [23], and ensured a multiplicative margin of error less than 20% for the residual standard deviation (i.e. lower and upper bounds of 95% confidence for residual variance within 20% of the estimated value) [24, 25].
A multivariable linear regression model containing all the seven candidate predictors (forcing them all to be included, regardless of statistical significance) was fitted. The apparent model performance was quantified by the apparent R^{2} statistic. Internal validation was then undertaken to obtain optimismadjusted estimates of R^{2} and calibration slope, using bootstrap resampling with 1000 bootstrap samples, as described elsewhere [26,27,28]. The optimismadjusted calibration slope was then used as a uniform shrinkage factor; that is, we multiplied the predictor effects of the fitted model by the optimismadjusted calibration slope. Then, forcing the revised predictor effects to be held fixed, we reestimated the model intercept to ensure calibrationinthelarge. This produced our final model with all predictors.
In addition to fitting full models, a backwards selection procedure was undertaken to identify a simpler model, with p values less than 0.1 used to define predictor inclusion. Internal validation and optimism adjustment was again applied using bootstrapping, which accounted for the variable selection when estimating optimism.
Applications in new data
For illustration of their potential use, we applied the developed tools to predict BoS in two Cochrane reviews not included in the Trikalinos review, and to three nonCochrane reviews, with comparison to subsequent multivariate metaanalysis results and observed BoS values.
Results
BoS values and comparison of bivariate and univariate metaanalysis results
The 86 BoS statistic values from the 43 metaanalyses in the Trikalinos review are shown alongside the univariate and bivariate metaanalysis results within Figs. 1 and 2 for outcomes 1 and 2, respectively. (see Fig. A in the supplementary material for outcome 1 and outcome 2 ordered by metaanalysis ID). A large proportion of the BoS values were small; the median (mean) value was 9.0% (13.2%), and the interquartile range was 2.71% to 20.18%, with a minimum value of 0.05% (Fig. B in the supplementary material). However, 22 of the 86 BoS values were over 20%, and the largest value was 57.2%, indicating how a bivariate metaanalysis provides notably greater efficiency than separate univariate metaanalyses in some applications.
Important differences between bivariate and univariate metaanalysis results arise in some examples, and these tend to occur in situations with largest BoS values. For example, for outcome 1 in metaanalysis ID 23 (BoS = 35.1%), the summary log odds ratio was 0.06 (95% CI: −0.37 to 0.49, odds ratio = 1.06) from the univariate metaanalysis and −0.15 (95% CI: −0.50 to 0.19, odds ratio = 0.86) from the multivariate metaanalysis; hence the direction of the summary effect changed. Similarly, for outcome 1 in metaanalysis ID 9 (BoS = 36.3%), the summary log odds ratio was 0.65 (95% CI: 0.35 to 0.96) from the univariate and 0.32 (95% CI: 0.08 to 0.57) from the multivariate metaanalysis; a difference in the magnitude of the effect size. For outcome 2 in metaanalysis ID 26 (BoS = 57.2%), the summary log odds ratio was −1.22 (95% CI: −1.33 to −1.11) from the univariate metaanalysis and −0.81 (95% CI: −0.88 to −0.73) from the multivariate metaanalysis; hence the confidence interval was narrower and the treatment effect less strong after multivariate analysis. Also, for outcome 1 in metaanalysis ID 38 (BoS = 51.4%), the summary log odds ratio was −0.39 (95% CI: −0.81 to 0.03; p value = 0.068) from the univariate metaanalysis and −0.42 (95% CI: −0.71 to 0.12; p value = 0.005) from the multivariate metaanalysis; hence the statistical significance changed. Note that our focus on change in p values is merely for illustration, as in practice basing decisions solely on statistical significance is not recommended, and indeed most applications did not result in a change of statistical significance).
Characteristics associated with BoS
Linear regression analyses looking at each predictor individually found strong evidence that all our seven prespecified characteristics were positively associated with the magnitude of the BoS statistic value (Table 1). In particular, the amount of missing outcome data and the magnitude of withinstudy correlation appeared to be important. There was a 0.49% (95% CI: 0.24%, 0.74%) increase in BoS for every 1% increase in the percentage of studies missing an outcome and a 29.08% (95% CI: 16.30%, 41.85%) increase in BoS when the largest absolute value of a withinstudy correlation changed from 0 to 1. This is sensible, as the amount of BoS has been shown mathematically to depend on the size of correlation and the amount of missing data [3, 12, 15, 29]. Increasing the number of studies was associated with a small increase in BoS.
Multivariable prediction model for BoS
Multivariable modelling results are shown in Table 2. After fitting a full model including all seven characteristics, the apparent proportion of variability explained (R^{2}) was 0.58. Only four characteristics had strong evidence for an important adjusted association with BoS statistic values: the total number of studies, the number of studies with the outcome of interest, the percentage of studies without the outcome of interest, and the largest absolute withinstudy correlation value in the metaanalysis dataset. These all had a positive adjusted association with BoS values, except for the number of studies providing the outcome of interest which had a negative association. The latter makes sense as, after accounting for the total number of studies, a larger number of studies with the outcome of interest leads to less opportunity for borrowing strength from the correlated outcome. After using backwards selection, only these four characteristics were selected (Table 2), and the model performance was very similar to the full model (apparent R^{2} = 0.57). For parsimony, we therefore chose to use this reduced model going forward. Bootstrapping identified a small amount of overfitting in the final model after variable selection (optimismadjusted calibration slope = 0.96; optimismadjusted R^{2} = 0.50). After applying a global shrinkage factor of 0.96 to adjust for overfitting, the final prediction model was (Table 2):
A scatter plot of the observed BoS values against the predicted BoS values (before and after shrinkage) is shown in Fig. 3. The larger the predicted BoS then the greater the rationale for undertaking multivariate metaanalysis. We suggest a threshold of about 15 to 20% for flagging that a multivariate metaanalysis is worth considering. Note that the equation depends on knowing the largest withinstudy correlation. If this is not known, we suggest assuming a large value, say 0.8, for the largest absolute withinstudy correlation. This would then reveal the predicted BoS in a situation where the two outcomes are highly correlated.
Applied examples
We applied Eq. (1) to predict BoS in two Cochrane reviews published in 2017 by Buzzetti et al. [30] and Feinberg et al. [31], which both contained an isasubsetof relationship between two outcomes. Buzzetti et al. reviewed randomised trials comparing the use of glucocorticosteriods for alcoholic hepatitis compared with no intervention, and the two correlated outcomes were mortality at maximal followup and mortality at 30 days [30]. Feinberg et al. reviewed randomised trials that compared experimental nutrition support to a control for diseaserelated malnutrition in Intensive Care Unit participants, including trauma, characterised as “at nutritional risk”; their two correlated outcomes of interest were allcause mortality at the end of the intervention and at maximum followup [31].
The results are shown in Table 3. For the Buzzetti et al. review, using Eq. (1) the predicted BoS is only 10.8% for outcome 2, but larger at 20.1% for outcome 1 as calculated below.
Given the predicted BoS for outcome 2 is moderately large, it flags that a multivariate metaanalysis may be useful in this review, for outcome 2 to borrow strength from outcome 1. Subsequently, we applied a multivariate metaanalysis and the summary odds ratio for outcome 2 was 0.89 (95% CI: 0.69 to 1.15), which provided much weaker evidence of a beneficial treatment effect than the univariate metaanalysis results (summary odds ratio = 0.69, 95% CI: 0.48 to 0.98). Indeed, the actual observed BoS was 47.7% for outcome 2, and so our predicted BoS of 20.1% was an underestimate. Nevertheless, the predicted BoS still flagged that a multivariate metaanalysis was potentially important. For outcome 1, the observed BoS was 3.2%, and so multivariate and univariate metaanalysis results were similar, as anticipated by the lower predicted BoS value of 10.8%.
In the Feinberg et al. review, for outcome 1 the predicted BoS was 21%, again flagging that a multivariate approach may be worth the extra effort for outcome 1 in this review. When applying the multivariate model, the observed BoS was 32.8% for outcome 1, and this led to a narrower confidence interval for the summary treatment effect compared to univariate metaanalysis for outcome 1 (Table 3), although no change in statistical or clinical significance (unlike the Buzzetti et al. review).
Discussion
Key findings and recommendations
Methods for multivariate metaanalysis of multiple correlated outcomes are more complex for nonstatisticians and may not always be worth the extra effort for reviewers. However, on occasion, the multivariate approach provides important extra efficiency compared to a conventional univariate metaanalysis of each outcome separately and may even lead to different statistical or clinical conclusions. Our empirical examination using the datasets from the Trikalinos review shows that such situations occur mainly when the borrowing of strength amongst outcomes is large. Hence, it is potentially helpful to flag the importance of checking the predicted magnitude of BoS to (nonstatistical) researchers as part of a prespecified data analysis plan.
We identified four characteristics as strongly associated with BoS in Cochrane intervention reviews with multiple binary outcomes: the total number of studies, the number of studies with the outcome of interest, the percentage of studies missing the outcome of interest, and the largest absolute withinstudy correlation. Based on these we developed prediction Eq. (1), which can help predict BoS for outcomes in a new review. In particular, if this equation predicts BoS to be moderate or large (say greater than about 15 to 20%) then it may motivate reviewers to obtain additional statistical support to undertake the multivariate approach and to invest time and resources trying to extract or derive withinstudy correlations amongst outcomes or even to obtain individual participant data from studies to estimate them directly [32]. It should also be noted that, even if BoS is anticipated to be low, sometimes there are other reasons why a multivariate metaanalysis is needed; for example, if functions of summary results are required [9, 10, 12, 15], or if the actual estimate(s) of correlation are of interest.
Limitations
There are limitations of our work. Although the developed prediction equation explained 50% of the variation in BoS values, there can still be a reasonable discrepancy between predicted and observed Bos values (see Fig. 3 and Table 3). However, the model is not intended to perfectly predict BoS and is rather intended as a tool to provide additional insight for when the multivariate approach is worth considering. This was illustrated in our two applied examples, where outcomes with predicted BoS values of > 20% suggested multivariate metaanalysis may be useful, and subsequently applying multivariate metaanalysis improved precision and, in one example, even changed the statistical and clinical conclusions.
Another limitation is that the identified predictors of BoS and the developed model are based on Cochrane reviews of binary outcomes, and so further evaluation and extension are needed for other settings, especially continuous outcomes. It is likely that the same predictors will be important contributors of BoS to other settings, although their specific weight in a prediction model may vary.
Also, our prediction model equation requires the researcher to input the largest absolute withinstudy correlation; this might not be available and in the absence of other information we suggest assuming a large value such as 0.8, to predict BoS assuming correlations are high, to see if BoS is likely to be high even in this situation. However, it may be that although the withinstudy correlations are unknown, a more informed guess can be made based on past research or understanding the clinical outcomes and biology. For example, overall and diseasefree survival are likely highly correlated, as are stroke and CVD outcome events, which would justify assuming a large positive correlation. Conversely, benefit and harms might be assumed moderately or highly negatively correlated. Also, IPD could be obtained from a few studies, and the withinstudy correlations were obtained and used as input values.
The work presented is based on assuming common treatment effects. BoS results were also obtained in our examples after using randomeffects models to allow for heterogeneity, and similar conclusions found (results shown elsewhere [21]). However, predicting BoS in a random effects setting is more complex, due to the impact of betweenstudy variances and betweenstudy correlations, which are difficult to gauge in advance, although informative prior distributions could be considered [33, 34]. Furthermore, BoS is harder to define in random effects situations, as the univariate metaanalysis must be forced to have the same betweenstudy variance estimates as the bivariate metaanalysis, to make comparisons fair [6, 22]. Hence, we consider it simpler for researchers to focus on predicting BoS initially assuming no heterogeneity.
A further limitation is that the BoS statistic is focused on identifying changes in precision from univariate to multivariate metaanalysis; however, in some cases, multivariate metaanalysis may lead to important differences in terms of a change in effect size but not a change in precision, and so the prediction model may not necessarily identify these situations. Thus, choosing to conduct a multivariate metaanalysis over a univariate metaanalysis may depend on other considerations even when BoS is predicted to be small. This requires further research. The feasibility of multivariate metaanalysis should also consider whether withinstudy correlations and/or IPD can be collected and whether missing outcomes are likely to be missing at random [3, 4].
Summary
Though multivariate metaanalysis is a more complex evidence synthesis method, it may sometimes be important to consider in reviews of multiple outcomes. We have identified key characteristics and developed a prediction model to help to flag when a multivariate metaanalysis may be beneficial to reviewers with multiple binary outcomes, as part of a prespecified analysis plan. Extension to other settings, such as nonCochrane reviews and other continuous outcome types, is still required.
What this study adds

Multivariate metaanalysis jointly synthesises multiple correlated effect estimates from multiple studies which enables borrowing of strength across outcomes. This can sometimes lead to difference to results from a standard univariate metaanalysis.

An empirical examination of Cochrane intervention reviews with multiple binary outcomes shows that multivariate metaanalysis is most influential when the borrowing of strength (BoS) amongst outcomes is large.

Four characteristics were strongly associated with BoS: the total number of studies, the number of studies with the outcome of interest, the percentage of studies missing the outcome of interest, and the largest absolute withinstudy correlation.

We developed a prediction equation that included these characteristics, to help predict BoS for outcomes in a new review, so to flag when the multivariate approach may be worth considering instead of separate univariate analyses.

We suggest that if this equation predicts BoS to be moderate or large (say greater than about 15 to 20%) then a multivariate metaanalysis approach should be considered. However, further evaluation and extension to nonCochrane reviews and continuous outcomes are still needed.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
References
Riley RD, Gates SG, Neilson J, Alfirevic Z. Statistical methods can be improved within Cochrane pregnancy and childbirth reviews. J Clin Epidemiol. 2011;64(6):608–18.
Van Houwelingen HC, Arends LR, Stijnen T. Advanced methods in metaanalysis: multivariate approach and metaregression. Stat Med. 2002;21:589–624.
Jackson D, Riley RD, White IR. Multivariate metaanalysis: potential and promise. Stat Med. 2011;30:2481–98.
Kirkham JJ, Riley RD, Williamson PR. A multivariate metaanalysis approach for reducing the impact of outcome reporting bias in systematic reviews. Stat Med. 2012;31:2179–95.
Higgins JP, Whitehead A. Borrowing strength from external trials in a metaanalysis. Stat Med. 1996;15:2733–49.
Jackson D, White IR, Price M, Copas J, Riley RD. Borrowing of strength and study weights in multivariate and network metaanalysis. Stat Methods Med Res. 2017;26:2853–68.
Wei Y, Higgins JP. Estimating withinstudy covariances in multivariate metaanalysis with multiple outcomes. Stat Med. 2013;32:1191–205.
Riley RD, Thompson JR, Abrams KR. An alternative model for bivariate randomeffects metaanalysis when the withinstudy correlations are unknown. Biostatistics. 2008;9:172–86.
Trikalinos TA, Hoaglin DC, Schmid CH. An empirical comparison of univariate and multivariate metaanalyses for categorical outcomes. Stat Med. 2014;33:1441–59.
Trikalinos TA, Hoaglin DC, Schmid CH. Empirical and simulationbased comparison of univariate and multivariate metaanalysis for binary outcomes. Methods research report. Rockville: Agency for Healthcare Research and Quality (US); 2013. Publication no. 13EHC066EF.
Jones AP, Riley RD, Williamson PR, Whitehead A. Metaanalysis of individual patient data versus aggregate data from longitudinal clinical trials. Clin Trials. 2009;6:16–27.
Riley RD. Multivariate metaanalysis: the effect of ignoring withinstudy correlation. J R Stat Soc A Stat Soc. 2009;172(4):789–811.
Price MJ, Blake HA, Kenyon S, White IR, Jackson D, Kirkham JJ, et al. Empirical comparison of univariate and multivariate metaanalyses in Cochrane pregnancy and childbirth reviews with multiple binary outcomes. Res Synth Methods. 2019;10:440–51.
Moran JL. Multivariate metaanalysis of critical care metaanalyses: a metaepidemiological study. BMC Med Res Methodol. 2021;21:148.
Riley RD, Abrams KR, Lambert PC, Sutton AJ, Thompson JR. An evaluation of bivariate randomeffects metaanalysis for the joint synthesis of two correlated outcomes. Stat Med. 2007;26:78–97.
Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and avoidance of continuity corrections in metaanalysis of sparse data. Stat Med. 2004;23:1351–75.
Higgins JPT, Green SB. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]: The Cochrane Collaboration; 2011. Available from www.handbook.cochrane.org
Trikalinos TA, Olkin I. Metaanalysis of effect sizes reported at multiple time points: a multivariate approach. Clin Trials. 2012;9(5):610–20.
Trikalinos TA, Olkin I. A method for the metaanalysis of mutually exclusive binary outcomes. Stat Med. 2008;27:4279–300.
Riley RD, Abrams KR, Sutton AJ, Lambert PC, Thompson JR. Bivariate randomeffects metaanalysis and the estimation of betweenstudy correlation. BMC Med Res Methodol. 2007;7:3.
Hattle MSM. Development and application of multivariate metaanalysis in medical research: borrowing strength across multiple correlated outcomes. Doctoral thesis. Keele: Keele University; 2019.
Copas JB, Jackson D, White IR, Riley RD. The role of secondary outcomes in multivariate metaanalysis. J R Stat Soc Ser C Appl Stat. 2018;67:1177–205.
Austin PC, Steyerberg EW. The number of subjects per variable required in linear regression analyses. J Clin Epidemiol. 2015;68:627–36.
Harrell FE Jr. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. New York: Springer; 2015.
Riley RD, Ensor J, Snell KIE, Harrell FE Jr, Martin GP, Reitsma JB, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441.
Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009.
Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–73.
Riley RD, van der Windt D, Croft P, Moons KGM. Prognosis research in healthcare: concepts, methods and impact. Oxford: Oxford University Press; 2019.
Riley RD, Jackson D, Salanti G, Burke DL, Price M, Kirkham J, et al. Multivariate and network metaanalysis of multiple outcomes and multiple treatments: rationale, concepts, and examples. BMJ. 2017;358:j3932.
Buzzetti E, Kalafateli M, Thorburn D, Davidson BR, Thiele M, Gluud LL, et al. Pharmacological interventions for alcoholic liver disease (alcoholrelated liver disease): an attempted network metaanalysis. Cochrane Database Syst Rev. 2017;3:CD011646.
Feinberg J, Nielsen EE, Korang SK, Halberg Engell K, Nielsen MS, Zhang K, et al. Nutrition support in hospitalised adults at nutritional risk. Cochrane Database Syst Rev. 2017;5:CD011598.
Riley RD, Price MJ, Jackson D, Wardle M, Gueyffier F, Wang J, et al. Multivariate metaanalysis using individual participant data. Res Synth Method. 2015;6:157–74.
Burke DL, Bujkiewicz S, Riley RD. Bayesian bivariate metaanalysis of correlated effects: Impact of the prior distributions on the betweenstudy correlation, borrowing of strength, and joint inferences. Stat Methods Med Res. 2018;27:428–50.
Turner RM, Davey J, Clarke MJ, Thompson SG, Higgins JP. Predicting the extent of heterogeneity in metaanalysis, using empirical data from the Cochrane Database of Systematic Reviews. Int J Epidemiol. 2012;41:818–27.
Acknowledgements
Not applicable.
Author information
Authors and Affiliations
Contributions
RR, DB and MH planned the research question and study design. MH performed all the analyses using data originally extracted by TT and CS. All authors provided feedback on the methods and results. MH and RR drafted the paper and then revised following feedback and comments from all authors. The author(s) read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
Miriam Hattle was funded by a Keele University Acorn PhD studentship. This paper presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Reference Number RPPG061520002), with cofunding from Arthritis Research UK. Richard Riley is supported by funding from the Evidence Synthesis Working Group, which is funded by the NIHR School for Primary Care Research (NIHR SPCR) [ProjectNumber 390]. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. Yong Chen was supported in part by grant number 1R01LM012607 from the National Library of Medicine and 1R01AI130460 from the National Institute of Allergy and Infectious Diseases.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Hattle, M., Burke, D.L., Trikalinos, T. et al. Multivariate metaanalysis of multiple outcomes: characteristics and predictors of borrowing of strength from Cochrane reviews. Syst Rev 11, 149 (2022). https://doi.org/10.1186/s13643022019990
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13643022019990