Advertisement

Drug Safety

, Volume 35, Issue 12, pp 1127–1146 | Cite as

Individual NSAIDs and Upper Gastrointestinal Complications

A Systematic Review and Meta-Analysis of Observational Studies (the SOS Project)
  • Jordi Castellsague
  • Nuria Riera-Guardia
  • Brian Calingaert
  • Cristina Varas-Lorenzo
  • Annie Fourrier-Reglat
  • Federica Nicotra
  • Miriam Sturkenboom
  • Susana Perez-Gutthann
  • Safety of Non-Steroidal Anti-Inflammatory Drugs (SOS) Project
Open Access
Systematic Review

Abstract

Background: The risk of upper gastrointestinal (GI) complications associated with the use of NSAIDs is a serious public health concern. The risk varies between individual NSAIDs; however, there is little information on the risk associated with some NSAIDs and on the impact of risk factors. These data are necessary to evaluate the benefit-risk of individual NSAIDs for clinical and health policy decision making. Within the European Community’s Seventh Framework Programme, the Safety Of non-Steroidal anti-inflammatory drugs (NSAIDs) [SOS] project aims to develop decision models for regulatory and clinical use of individual NSAIDs according to their GI and cardiovascular safety.

Objective: The aim of this study was to conduct a systematic review and meta-analysis of observational studies to provide summary relative risks (RR) of upper GI complications (UGIC) associated with the use of individual NSAIDs, including selective cyclooxygenase-2 inhibitors.

Methods: We used the MEDLINE database to identify cohort and case-control studies published between 1 January 1980 and 31 May 2011, providing adjusted effect estimates for UGIC comparing individual NSAIDs with non-use of NSAIDs. We estimated pooled RR and 95% CIs of UGIC for individual NSAIDs overall and by dose using fixed- and random-effects methods. Subgroup analyses were conducted to evaluate methodological and clinical heterogeneity between studies.

Results: A total of 2984 articles were identified and 59 were selected for data abstraction. After review of the abstracted information, 28 studies met the meta-analysis inclusion criteria. Pooled RR ranged from 1.43 (95% CI 0.65, 3.15) for aceclofenac to 18.45 (95% CI 10.99, 30.97) for azapropazone. RR was less than 2 for aceclofenac, celecoxib (RR 1.45; 95% CI 1.17, 1.81) and ibuprofen (RR 1.84; 95% CI 1.54, 2.20); 2 to less than 4 for rofecoxib (RR 2.32; 95% CI 1.89, 2.86), sulindac (RR 2.89; 95% CI 1.90, 4.42), diclofenac (RR 3.34; 95% CI 2.79, 3.99), meloxicam (RR 3.47; 95% CI 2.19, 5.50), nimesulide (RR 3.83; 95% CI 3.20, 4.60) and ketoprofen (RR 3.92; 95% CI 2.70, 5.69); 4–5 for tenoxicam (RR 4.10; 95% CI 2.16, 7.79), naproxen (RR 4.10; 95% CI 3.22, 5.23), indometacin (RR 4.14; 95% CI 2.91, 5.90) and diflunisal (RR 4.37; 95% CI 1.07, 17.81); and greater than 5 for piroxicam (RR 7.43; 95% CI 5.19, 10.63), ketorolac (RR 11.50; 95% CI 5.56, 23.78) and azapropazone. RRs for the use of high daily doses of NSAIDs versus non-use were 2-3 times higher than those associated with low daily doses.

Conclusions: We confirmed variability in the risk of UGIC among individual NSAIDs as used in clinical practice. Factors influencing findings across studies (e.g. definition and validation of UGIC, exposure assessment, analysis of new vs prevalent users) and the scarce data on the effect of dose and duration of use of NSAIDs and on concurrent use of other medications need to be addressed in future studies, including SOS.

Keywords

Ibuprofen Celecoxib Naproxen Piroxicam Ketoprofen 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1. Background

NSAIDs are widely used for the symptomatic treatment of acute pain and chronic inflammatory and degenerative joint diseases. However, their use is restricted by the occurrence of upper gastrointestinal (GI) complications (UGIC) such as peptic ulcer perforations, obstructions and bleeding. The use of NSAIDs has been associated with a 3- to 5-fold increase in the risk of UGIC.[1,2] Clinical trials and observational studies have shown that the use of selective cyclooxygenase (COX)-2 inhibitors is associated with a lower risk of UGIC;[3, 4, 5] however, they have been also associated with an increased risk of serious cardiovascular (CV) events.[6] Further data are necessary to quantify the risk of UGIC associated with many individual NSAIDs, including selective COX-2 inhibitors, and to evaluate the benefit-risk balance of the NSAIDs most often used in regular clinical practice, taking into account dose, duration and effect of other risk factors. These data can help clinicians select treatments for individual patients and help health policy regulators assess the public health impact of therapy.

Within the European Community’s Seventh Framework Programme, the Safety Of non-Steroidal anti-inflammatory drugs (SOS) collaborative project started in 2008 with the goal of developing statistical and decision models to facilitate regulatory and treatment decisions based on the GI and CV safety of individual NSAIDs. One of the initial tasks of the SOS project was to summarize the data available on the risk of GI and CV events from observational studies. In this context, we conducted a systematic review and meta-analysis of published observational studies to provide pooled relative risks (RR) for UGIC associated with the use of individual NSAIDs versus non-use of NSAIDs. We followed the MOOSE guidelines for reporting meta-analyses of observational studies (http://www.equator-network.org/resource-centre/).

2. Materials and Methods

We performed a literature search in PubMed using medical subject headings (MeSH) and free-text terms for individual NSAIDs and selective COX-2 inhibitors, GI disease, case-control studies and cohort studies. The search was restricted to observational studies published in the English language between 1 January 1980 and 31 May 2011. Details of the search strategy are available in the supplemental digital content (SDC; http://links.adisonline.com/DSZ/A78). Studies had to be (i) cohort, case-control or nested case-control studies; (ii) provide odds ratios or RRs of UGIC comparing individual NSAIDs with non-use of NSAIDs; and (iii) provide effect estimates adjusted at least for age and sex. All titles and/or abstracts of the articles identified were reviewed to select those potentially meeting the inclusion criteria. Data from these articles were abstracted in a standardized database that included information on source population, inclusion and exclusion criteria, study design, case definition and validation, selection of controls, exposure definition, confounding factors and statistical analysis. The accuracy of the abstracted data was reviewed independently by two of the authors (NR-G, JC). References from relevant studies and prior meta-analyses were also reviewed. Study authors were contacted when additional information was needed.[7]

The methodological quality of each study was evaluated using the Newcastle-Ottawa Scale (NOS).[8] The NOS involves a score system in which the study design is evaluated on three broad categories: (i) selection of the study groups; (ii) comparability between the study groups; and (iii) exposure/outcome ascertainment. For each study, the NOS was evaluated independently by two of the authors (NR-G and JC), and any differences were resolved by consensus.

We estimated pooled RRs for those individual NSAIDs that had effect estimates reported in at least three different studies. Pooled RRs and 95% CIs were estimated using both the inverse-variance Lagrange fixed-effects method and the DerSimonian and Laird random-effects method.[9] We generated forest plots from the random-effects models. Heterogeneity between studies was assessed by graphical inspections of the forest plots and by Cochran’s Chi-squared (χ 2 ) test of homogeneity, and subgroup analyses evaluating methodological and clinical heterogeneity between studies. Subgroup analyses included stratification by study design, prior history of UGIC, bleeding complications, study period and dose of NSAIDs. Pooled estimates for dose were calculated according to the dose categorization used in each study. In the subgroup analyses, pooled RRs were also calculated for those NSAIDs with only two effect estimates available. The Higgins inconsistency I 2 statistic was used to describe the percentage of the variability in effect estimates that is due to heterogeneity rather than chance.[10] The meta-analysis was conducted using Review Manager (RevMan), Version 5.0.22 (The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen, Denmark, 2009).

3. Results

A total of 2984 articles on NSAIDs and GI complications were identified. Of these, 2974 articles were identified in the PubMed search and ten additional articles were identified through the references of relevant studies (figure 1). The review of titles and abstracts of these studies led to select 59 articles for full data abstraction. After review of the abstracted information, 28 studies on the use of individual NSAIDs and the risk of UGIC met the inclusion criteria and were included in the meta-analysis.[7,11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35] The remaining 31 articles were excluded for the following reasons: the reference group was other than non-use of NSAIDs in nine studies;[36, 37, 38, 39, 40, 41, 42, 43, 44] the outcome was overall upper and lower GI complications in three studies;[45, 46, 47] the outcome was uncomplicated upper GI events in two studies;[48,49] the study population was restricted to users of specific drugs or to patients with specific diseases in three studies;[50, 51, 52] the study population and the study period overlapped in four studies;[53, 54, 55, 56] and the study design did not meet the inclusion criteria in ten studies (i.e. different type of study or measures of association and exposure assessment).[57, 58, 59, 60, 61, 62, 63, 64, 65, 66]
Fig. 1

Flow diagram for identification of studies on upper gastrointestinal complications and individual NSAIDs. GI = gastrointestinal.

Selected characteristics of the 28 studies included in the meta-analysis are summarized in table I; 3 studies were cohort studies,[20,24,26] 10 were nested case-control studies,[7,11,17, 18, 19,23,25,33,35,67] and 15 were case-control studies.[12, 13, 14, 15, 16,21,22,27, 28, 29, 30, 31, 32,34,68] Twelve studies, all case-control studies, were field studies collecting individual information by standardized questionnaires. The 16 remaining studies used information recorded in healthcare databases.
Table I

Description of observational studies on the risk of upper gastrointestinal complications associated with the use of individual NSAIDs

Cases were defined as hospitalization or referral to a specialist for upper GI bleeding in 13 studies,[12,15,16,19, 20, 21, 22,27, 28, 29, 30,33,34] and for bleeding and/or perforation in 15 other studies; four studies also included cases of uncomplicated peptic ulcer.[14,17,33,68] The site of complication was defined as gastric and/or duodenal in all studies, and two included oesophageal complications.[12,32] Most studies, both field and database studies, required information from endoscopy or other diagnostic procedures to confirm UGIC. Six studies conducted in healthcare databases did not conduct any validation of the cases identified.[14,17, 18, 19, 20,68] Sixteen studies reported aggregate results for patients with and without a history of UGIC,[7,12, 13, 14,16,20, 21, 22, 23,28,31, 32, 33, 34,67,68] and 12 provided results for patients without a history of UGIC;[11,15,17, 18, 19,24, 25, 26, 27,29,30,35] the remaining study was a case-crossover study.[14] Most studies excluded subjects with a history of a known cause of UGIC, including the use of gastrotoxic medications and life-threatening diseases (table II). Four studies did not have any exclusion criteria.[20,22,28,34] Among the 14 case-control studies, seven included hospital controls;[12,13,16,21,29, 30, 31] five included both hospital and community controls;[22,27,28,32,34] one included community controls;[15] and two were case-crossover studies.[14,68] All case-control studies were matched on age, sex (except one study[16]), hospital or geographic area, and index date. Three of the case-control studies with hospital and community controls estimated separate results for each set of controls;[22,28,34] as results between the two sets were similar, we included in the meta-analysis results reported using hospital controls. In cohort studies, current use of NSAIDs was defined as the time covered by each prescription,[20,24] and one study extended the coverage by 15 days.[26] Most case-control studies defined current use of NSAIDs as any use ending at the index date or within 7 days before the index date. A few case-control studies considered current use as that ending up to 30 days[11,13,17,19,28,35] or 90 days[18] before the index date. Two cohort studies focused on new users of NSAIDs,[20,26] and one nested case-control study provided results for both new users and all users (incident and prevalent) of NSAIDs.[17] In addition to age and sex, the most frequent confounders considered were a history of peptic ulcer (21 studies),[7,11,15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,35,67] smoking (13 studies),[7,11,15,18,20,21,23,27, 28, 29, 30, 31, 32] alcohol use (9 studies),[7,15,19,21,27,28,30, 31, 32] use of proton-pump inhibitors and anti-ulcer medications (11 studies),[16, 17, 18, 19, 20, 21, 22, 23,30,67,68] and concurrent use of medications increasing the risk of UGIC (16 studies).[7,11,15,16,19, 20, 21,23,27,29, 30, 31,33,35,67,68] The quality of the studies measured with the NOS was, in general, very good: for the selection component, 12 studies had the maximum score of 4[7,11,15,22, 23, 24, 25,32,33,35,67,68] and 13 studies had the next highest score of 3;[13,14,16, 17, 18, 19,21,26,28,29,30,32,34] for comparability, 24 studies had the maximum score of 2;[7,11,14, 15, 16, 17, 18, 19,21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,35,67,68] and for the exposure/outcome component, 14 studies had the maximum score of 3[7,11,13,17, 18, 19,23, 24, 25, 26,33,35,67,68] and 10 studies had the next highest score of 2.[12,14,16,20,21,28, 29, 30,32,34]
Table II

Exclusions applied in observational studies on the risk of upper gastrointestinal complicationsa

The studies included in the meta-analysis allowed us to estimate pooled RRs for the current use of 16 different NSAIDs (table III and figure 2). Forest plots for each individual NSAID are available in the SDC. Using random-effects models, pooled RRs ranged from 1.43 (95% CI 0.65, 3.15) for aceclofenac to 18.45 (95% CI 10.99, 30.97) for azapropazone. Pooled RR was less than 2 for aceclofenac, celecoxib and ibuprofen; between 2 and less than 4 for rofecoxib, sulindac, diclofenac, meloxicam, nimesulide and ketoprofen; between 4 and less than 5 for tenoxicam, naproxen, indometacin and diflunisal; and greater than 5 for piroxicam, ketorolac and azapropazone. Pooled RRs from studies providing results for patients without a history of UGIC were similar to those from the overall analysis except for naproxen (more than 10% change), 3.10 (95% CI 2.45, 3.91) and diclofenac, 3.76 (95% CI 2.71, 5.21). Data in patients with a history of peptic ulcer disease were available only for celecoxib (two studies) and rofecoxib (one study).[67,68] The pooled RR for celecoxib in this population was 1.50 (95% CI 1.16, 1.94).
Table III

Individual and pooled relative risks (95% CIs) of upper gastrointestinal complications associated with the use of individual NSAIDs

Fig. 2

Pooled relative risks and 95% CIs of upper gastrointestinal complications associated with the use of individual NSAIDs. Vertical bars denote 95% CIs.

Pooled RRs from case-control studies were higher than those from cohort studies for all NSAIDs except ibuprofen, ketorolac and sulindac. In general, pooled RRs from fixed-effects models were slightly lower than those from random-effects models. In general, there was significant heterogeneity between studies, which decreased in the subsequent subgroup analysis exploring methodological and clinical diversity.

Pooled RRs for the effect of daily dose were estimated for eight different NSAIDs. Cut-off values used in each study to define the daily dose of each NSAID are presented in table IV. Variations in cut-off values were, in general, small except for those used for ibuprofen (200 mg) and naproxen (220 mg) in one study,[14] and for ibuprofen (≪2400 mg) in another study.[33] RRs for the use of high daily doses of NSAIDs were approximately 2- to 3-fold greater than RRs for low-medium doses (figure 3). The pooled RR for high daily dose of ibuprofen was similar to that for high daily dose of diclofenac. Exclusion of results from the studies with different cut-off values for ibuprofen[14,33] and naproxen[14] did not substantially change the pooled results for these individual NSAIDs. For ibuprofen, RRs were 2.15 (95% CI 1.66, 2.79) for low-medium dose and 4.22 (95% CI 1.76, 10.12) for high dose. For naproxen, the RR for low-medium daily dose was 3.62 (95% CI 2.62, 4.99) [the excluded study[14] did not provide data on high dose].
Table IV

Cut-off values (mg) used to define daily dosea of NSAIDs in observational studies on the risk of upper gastrointestinal complications

Fig. 3

Pooled relative risks and 95% CIs of upper gastrointestinal complications associated with the daily dose of individual NSAIDs. See table IV for cut-off values used in each study to define high dose and low-medium dose. Vertical bars denote 95% CIs. HD = high dose; LD = low-medium dose.

A total of 12 studies provided results specifically for upper GI bleeding for eight different NSAIDs; ten of these studies were case-control field studies. Pooled RRs were higher than those from all UGIC (bleeding, perforation and/or obstruction). Pooled RRs were 1.09 (95% CI 0.77, 1.53) for celecoxib; 1.43 (95% CI 0.65, 3.15) for aceclofenac; 1.88 (95% CI 1.00, 3.51) for ibuprofen; 2.25 (95% CI 1.56, 3.25) for rofecoxib; 4.20 (95% CI 3.03, 5.83) for diclofenac; 5.64 (95% CI 3.60, 8.83) for indometacin; 5.72 (95% CI 3.83, 8.53) for naproxen; and 13.36 (95% CI 9.62, 18.54) for piroxicam.

Pooled RRs from studies conducted from the year 2000 onward[7,15, 16, 17, 18,21,67,68] were slightly higher than those from studies conducted before the year 2000 for ibuprofen, 2.13 (95% CI 1.66, 2.73) versus 1.50 (95% CI 1.12, 2.01); ketoprofen, 4.28 (95% CI 2.36, 7.76) versus 3.70 (95% CI 2.27, 6.05); and nimesulide, 3.89 (95% CI 3.18, 4.74) versus 3.50 (95% CI 2.03, 6.03); but lower for diclofenac, 3.08 (95% CI 2.47, 3.84) versus 3.63 (95% CI 2.81, 4.70).

Only one study provided information on the effect modification of gastroprotective agents.[18] In that study, RRs for all individual NSAIDs were lower among patients receiving ulcer-healing drugs than among those not receiving them.

4. Discussion

The results from this meta-analysis confirmed the variability of RRs among individual NSAIDs. The lowest RRs were observed for the use of aceclofenac, celecoxib and ibuprofen, and the highest for the use of piroxicam, ketorolac and azapropazone. Intermediate RRs, between approximately 2 and 4, were observed for the rest of the NSAIDs for which at least three estimates were available: rofecoxib, sulindac, diclofenac, meloxicam, nimesulide, ketoprofen, tenoxicam, naproxen, indometacin and diflunisal.

The use of high daily doses of individual NSAIDs was associated with approximately a 2- to 3-fold increase of RRs for UGIC compared with the use of low-medium doses, except for celecoxib, for which we did not observe a dose-response relationship. For most NSAIDs, there were no major differences between studies restricted to patients without a history of peptic ulcer and those that included all patients, and between studies conducted before and after the year 2000. Data from the studies included in the meta-analysis were insufficient to estimate pooled RRs for the duration of use of individual NSAIDs and for the concurrent use of gastroprotective agents.

In general, there was significant heterogeneity among studies, although it improved or tended to disappear in the subgroup analysis conducted by study design, type of complication, history of peptic ulcer disease and dose of NSAIDs. However, the χ 2 test to detect heterogeneity is very sensitive to the number of studies (or sample size) included in the analysis. Meta-analysis of a small number of studies may be underpowered to detect heterogeneity. On the other hand, when the number of studies is large, the χ 2 test has a high power to detect a small amount of heterogeneity that may be clinically unimportant.

Most studies evaluated the role of confounding factors other than age and sex. One of the concerns in observational studies is the effect of confounding by indication. This is particularly relevant for selective COX-2 inhibitors as there is evidence that they have been preferentially prescribed to patients at high risk of UGIC.[20,46] Many studies attempted to minimize the effect of confounding by indication by adjusting for relevant comorbidity, including GI, CV and other chronic disease; concurrent use of medications; and prior use of healthcare services. Some studies also evaluated confounding by indication by conducting stratified analysis by markers of disease severity or restricted the study population to patients at high risk of UGIC.[7,19]

Pooled RRs from case-control studies were higher than those obtained from cohort studies, although CIs were wider. Most case-control studies were field studies restricted to upper GI bleeding. Thus, a more specific and strict definition of UGIC probably increased the internal validity of these studies. On the other hand, recall bias, leading to overestimation of effects, could be present in these case-control studies in which exposure to NSAIDs was assessed retrospectively by self-reported questionnaires.

Most studies included in this meta-analysis applied broad restrictions to the source population by excluding patients with major risk factors for peptic ulcer disease. Restriction results in more homogeneous study populations and increases the internal validity of studies.[69] On the other hand, absolute rates for UGIC obtained from studies conducted in restricted populations underestimated the rate of the disease.[24,26] This can impact public health policy, particularly when patients with risk factors for UGIC are excluded from studies estimating incidence rates.

Results for new users of NSAIDs were available for only three studies. The rest of the studies provided results for new and prevalent users overall. Inclusion of prevalent users of NSAIDs may result in an overrepresentation of the group of patients who tolerate the treatment (survivors). This is a selection process that may introduce bias if the risk varies with time since the beginning of treatment. In the case of UGIC and the use of NSAIDs, some studies have shown that the risk is higher at the beginning of treatment because susceptible patients are affected early in therapy.[13] Restricting the study to new users of NSAIDs prevents bias from overrepresentation of long-term users who tolerate the treatment.[70,71]

Most studies confirmed cases of UGIC by requiring a positive endoscopy or other diagnostic or therapeutic procedure. However, some studies conducted in databases, which identified potential cases of UGIC by discharge codes, did not conduct any validation of the cases identified, thus introducing misclassification of the outcome.[72] Studies conducted in Canada and Italy show that the positive predictive values (PPVs) for specific discharge diagnoses (gastric ulcer and duodenal ulcer) range from 92% to 100%, whereas PPVs for other specific diagnoses (peptic ulcer and gastrojejunal ulcer) range from 81% to 84%, and those for the non-specific diagnoses (GI haemorrhage) range from 54% to 68%.[67,73,74] Overall, the results from the studies included in this meta-analysis that did not confirm the cases identified may be biased towards the null, assuming non-differential outcome misclassification. Cohort studies estimating incidence rates of UGIC based on the cases identified through hospital discharge codes without performing any validation may overestimate absolute rates of UGIC by including non-cases or cases with uncertain clinical validity.[17,20,75]

The benefit-risk balance of individual NSAIDs is mainly driven by their GI and CV safety profile. Recent meta-analyses of observational studies on the CV risk associated with individual NSAIDs show that rofecoxib and diclofenac are associated with the highest RR and naproxen and ibuprofen with the lowest RR.[76,77] For cerebrovascular events, the increase in risk appears to be similar between naproxen, ibuprofen and diclofenac, and higher for rofecoxib.[78] In our study, ibuprofen was in the lowest range of pooled RRs for UGIC; rofecoxib and diclofenac were in the middle range; and naproxen was associated with a higher RR. However, estimates for individual NSAIDs varied by dose. The pooled RR of UGIC for high-dose ibuprofen was similar to the RR for high-dose rofecoxib and diclofenac, and the use of high-dose naproxen was associated with a higher RR.

5. Conclusions

We conclude that the results obtained in this meta-analysis are in line with those from other meta-analyses published in the last decade and confirm the variability of the risk of UGIC among individual NSAIDs as they are used in clinical practice.[1,2,5] Our meta-analysis provides pooled RRs for many individual NSAIDs, including, for the first time, nimesulide, tenoxicam and diflunisal. Aceclofenac, celecoxib and ibuprofen (analgesic and anti-inflammatory doses combined) were the NSAIDs with the lowest RR, whereas piroxicam, ketorolac and azapropazone were those with the highest RRs. Intermediate RRs were observed for the rest of the NSAIDs: rofecoxib, sulindac, diclofenac, meloxicam, nimesulide, ketoprofen, tenoxicam, naproxen, indometacin and diflunisal. The impact on the findings across studies related to varying study design approaches — including choices in the definition and validation of UGIC, exposure assessment and analysis of new or prevalent users — should be taken into account when designing new studies. For individual NSAIDs, data on the effect of dose, duration of use and concurrent use of other medications are still scarce. These gaps need to be addressed in future studies, including those to be conducted in the ongoing SOS project (http://www.sos-nsaids-project.org/).

Notes

Acknowledgements

We acknowledge Ceri Hirst, Miny Samuel and Lee Lanza for their contribution in the data abstraction; Lorenza Scotti for developing the data abstraction database; Carla Franzoni for other research support areas; and Adele Monroe for editing the manuscript.

The research leading to the results of this study has received funding from the European Community’s Seventh Framework Programme (FP-7) under grant agreement number 223495, the SOS project. The funding source had no role in study design, data collection, data analysis, data interpretation or writing of the report.

All authors contributed in conceiving and planning the study or played an important role in the acquisition, analysis, and interpretation of the data; drafted or made substantive suggestions for revision of the manuscript; and approved the final submitted version.

Jordi Castellsague, Nuria Riera-Guardia, Brian Calingaert, Cristina Varas-Lorenzo and Susana Perez-Gutthann are employees of RTI Health Solutions. Jordi Castellsague, Nuria Riera-Guardia and Susana Perez-Gutthann conducted research activities funded by Helsinn Healthcare S.A., a manufacturer of NSAIDs. Susana Perez-Gutthann and Cristina Varas-Lorenzo conducted research activities funded by NicOx S.A., a developer of NSAIDs. Susana Perez-Gutthann has served on advisory boards, provided consultation to pharmaceutical companies including NSAID manufacturers/developers such as Helsinn and NicOx S.A, and is a member of the scientific committee of a study on the safety of NSAIDs sponsored by Dundee University. Helsinn and NicOx were not involved in this study.

Miriam Sturkenboom is coordinating the FP-7-funded SOS project. She is heading a research group at the Erasmus University that occasionally does drug safety studies for pharmaceutical companies such as AstraZeneca, Pfizer, Eli Lilly and Novartis. Most of these companies produce NSAIDs but they were not involved in this study.

Annie Fourrier-Reglat and Federica Nicotra have no conflicts of interest that are directly relevant to the content of this study.

References

  1. 1.
    Henry D, McGettigan P. Epidemiology overview of gastrointestinal and renal toxicity of NSAIDs. Int J Clin Pract Suppl 2003 Apr; (135): 43–9PubMedGoogle Scholar
  2. 2.
    Hernández-Díaz S, Rodríguez LA. Association between nonsteroidal anti-inflammatory drugs and upper gastrointestinal tract bleeding/perforation: an overview of epidemiologic studies published in the 1990s. Arch Intern Med 2000 Jul 24; 160(14): 2093–9PubMedCrossRefGoogle Scholar
  3. 3.
    Silverstein FE, Faich G, Goldstein JL, et al. Gastrointestinal toxicity with celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and rheumatoid arthritis: the CLASS study — randomized controlled trial. Celecoxib Long-term Arthritis Safety Study. JAMA 2000 Sep 13; 284(10): 1247–55PubMedCrossRefGoogle Scholar
  4. 4.
    Bombardier C, Laine L, Reicin A, et al. Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in patients with rheumatoid arthritis. VIGOR Study Group. N Engl J Med 2000 Nov 23; 343(21): 1520–8PubMedCrossRefGoogle Scholar
  5. 5.
    Massó González EL, Patrignani P, Tacconelli S, et al. Variability among nonsteroidal antiinflammatory drugs in risk of upper gastrointestinal bleeding. Arthritis Rheum 2010 Jun; 62(6): 1592–601CrossRefGoogle Scholar
  6. 6.
    Cardiovascular and cerebrovascular events in the randomized, controlled Alzheimer’s Disease Anti-Inflammatory Prevention Trial (ADAPT). PLoS Clin Trials 2006; 1(7): e33Google Scholar
  7. 7.
    García Rodríguez LA, Barreales TL. Risk of upper gastrointestinal complications among users of traditional NSAIDs and COXIBs in the general population. Gastroenterology 2007; 132(2): 498–506PubMedCrossRefGoogle Scholar
  8. 8.
    Wells GA, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa (ON): Ottawa Hospital Research Institute, 2010 [online]. Available from URL: http://www.ohri.ca/programs/clinical_epidemiology/oxford,asp [Accessed 2011 Jun 20]Google Scholar
  9. 9.
    DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986 Sep; 7(3): 177–88PubMedCrossRefGoogle Scholar
  10. 10.
    Higgins JP, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ 2003 Sep 6; 327(7414): 557–60PubMedCentralPubMedCrossRefGoogle Scholar
  11. 11.
    García Rodríguez LA, Jick H. Risk of upper gastrointestinal bleeding and perforation associated with individual nonsteroidal anti-inflammatory drugs. Lancet 1994; 343(8900): 769–72PubMedCrossRefGoogle Scholar
  12. 12.
    Matikainen M, Kangas E. Is there a relationship between the use of analgesics and non-steroidal anti-inflammatory drugs and acute upper gastrointestinal bleeding? A Finnish case-control prospective study. Scand J Gastroenterol 1996 Sep; 31(9): 912–6PubMedCrossRefGoogle Scholar
  13. 13.
    Moride Y, Abenhaim L. Evidence of the depletion of susceptibles effect in non-experimental pharmacoepidemiologic research. J Clin Epidemiol 1994; 47(7): 731–7PubMedCrossRefGoogle Scholar
  14. 14.
    Biskupiak JE, Brixner DI, Howard K, et al. Gastrointestinal complications of over-the-counter nonsteroidal antiinflammatory drugs. J Pain Palliat Care Pharmacother 2006; 20(3): 7–14PubMedCrossRefGoogle Scholar
  15. 15.
    Sakamoto C, Sugano K, Ota S, et al. Case-control study on the association of upper gastrointestinal bleeding and nonsteroidal anti-inflammatory drugs in Japan. Eur J Clin Pharmacol 2006; 62(9): 765–72PubMedCrossRefGoogle Scholar
  16. 16.
    Lanas A, García-Rodríguez LA, Arroyo MT, et al. Risk of upper gastrointestinal ulcer bleeding associated with selective cyclo-oxygenase-2 inhibitors, traditional non-aspirin non-steroidal anti-inflammatory drugs, aspirin and combinations. Gut 2006; 55(12): 1731–8PubMedCentralPubMedCrossRefGoogle Scholar
  17. 17.
    Helin-Salmivaara A, Saarelainen S, Grönroos JM, et al. Risk of upper gastrointestinal events with the use of various NSAIDs: a case-control study in a general population. Scand J Gastroenterol 2007; 42(8): 923–32PubMedCrossRefGoogle Scholar
  18. 18.
    Hippisley-Cox J, Coupland C, Logan R. Risk of adverse gastrointestinal outcomes in patients taking cyclo-oxygenase-2 inhibitors or conventional non-steroidal anti-inflammatory drugs: population based nested case-control analysis. BMJ 2005; 331(7528): 1310–6PubMedCentralPubMedCrossRefGoogle Scholar
  19. 19.
    Nørgård B, Pedersen L, Johnsen SP, et al. COX-2-selective inhibitors and the risk of upper gastrointestinal bleeding in high-risk patients with previous gastrointestinal diseases: a population-based case-control study. Aliment Pharmacol Ther 2004; 19(7): 817–25PubMedCrossRefGoogle Scholar
  20. 20.
    Mamdani M, Rochon PA, Juurlink DN, et al. Observational study of upper gastrointestinal haemorrhage in elderly patients given selective cyclo-oxygenase-2 inhibitors or conventional non-steroidal anti-inflammatory drugs. BMJ 2002; 325(7365): 624PubMedCentralPubMedCrossRefGoogle Scholar
  21. 21.
    Laporte JR, Ibáñez L, Vidal X, et al. Upper gastrointestinal bleeding associated with the use of NSAIDs: newer versus older agents. Drug Saf 2004; 27(6): 411–20PubMedCrossRefGoogle Scholar
  22. 22.
    Lanas A, Serrano P, Bajador E, et al. Risk of upper gastrointestinal bleeding associated with non-aspirin cardiovascular drugs, analgesics and nonsteroidal anti-inflammatory drugs. Eur J Gastroenterol Hepatol 2003 Feb; 15(2): 173–8PubMedCrossRefGoogle Scholar
  23. 23.
    García Rodríguez LA, Hernández-Díaz S. Relative risk of upper gastrointestinal complications among users of acetaminophen and nonsteroidal anti-inflammatory drugs. Epidemiology 2001; 12(5): 570–6PubMedCrossRefGoogle Scholar
  24. 24.
    Menniti-Ippolito F, Maggini M, Raschetti R, et al. Ketorolac use in outpatients and gastrointestinal hospitalization: a comparison with other non-steroidal anti-inflammatory drugs in Italy. Eur J Clin Pharmacol 1998; 54(5): 393–7PubMedCrossRefGoogle Scholar
  25. 25.
    García Rodríguez LA, Cattaruzzi C, Troncon MG, et al. Risk of hospitalization for upper gastrointestinal tract bleeding associated with ketorolac, other nonsteroidal anti-inflammatory drugs, calcium antagonists, and other anti-hypertensive drugs. Arch Intern Med 1998; 158(1): 33–9PubMedCrossRefGoogle Scholar
  26. 26.
    McMahon AD, Evans JM, White G, et al. A cohort study (with re-sampled comparator groups) to measure the association between new NSAID prescribing and upper gastrointestinal hemorrhage and perforation. J Clin Epidemiol 1997; 50(3): 351–6PubMedCrossRefGoogle Scholar
  27. 27.
    Kaufman DW, Kelly JP, Sheehan JE, et al. Nonsteroidal anti-inflammatory drug use in relation to major upper gastrointestinal bleeding. Clin Pharmacol Ther 1993; 53(4): 485–94PubMedCrossRefGoogle Scholar
  28. 28.
    Langman MJ, Weil J, Wainwright P, et al. Risks of bleeding peptic ulcer associated with individual non-steroidal anti-inflammatory drugs. Lancet 1994; 343(8905): 1075–8PubMedCrossRefGoogle Scholar
  29. 29.
    Nobili A, Mosconi P, Franzoni MG, et al. Non-steroidal anti-inflammatory drugs and upper gastrointestinal bleeding, a post-marketing surveillance case-control study. Pharmacoepidemiol Drug Saf 1992; 1: 65–72CrossRefGoogle Scholar
  30. 30.
    Laporte JR, Carné X, Vidal X, et al. Upper gastrointestinal bleeding in relation to previous use of analgesics and non-steroidal anti-inflammatory drugs. Catalan Countries Study on Upper Gastrointestinal Bleeding. Lancet 1991; 337(8733): 85–9PubMedCrossRefGoogle Scholar
  31. 31.
    Savage RL, Moller PW, Ballantyne CL, et al. Variation in the risk of peptic ulcer complications with nonsteroidal antiinflammatory drug therapy. Arthritis Rheum 1993; 36(1): 84–90PubMedCrossRefGoogle Scholar
  32. 32.
    Henry D, Dobson A, Turner C. Variability in the risk of major gastrointestinal complications from nonaspirin nonsteroidal anti-inflammatory drugs. Gastroenterology 1993; 105(4): 1078–88PubMedGoogle Scholar
  33. 33.
    Griffin MR, Piper JM, Daugherty JR, et al. Nonsteroidal anti-inflammatory drug use and increased risk for peptic ulcer disease in elderly persons. Ann Intern Med 1991; 114(4): 257–63PubMedCrossRefGoogle Scholar
  34. 34.
    Somerville K, Faulkner G, Langman M. Non-steroidal anti-inflammatory drugs and bleeding peptic ulcer. Lancet 1986 Mar 1; 1(8479): 462–4PubMedCrossRefGoogle Scholar
  35. 35.
    Perez-Gutthann S, García-Rodríguez LA, Raiford DS. Individual nonsteroidal antiinflammatory drugs and other risk factors for upper gastrointestinal bleeding and perforation. Epidemiology 1997; 8(1): 18–24CrossRefGoogle Scholar
  36. 36.
    Carson JL, Strom BL, Morse ML, et al. The relative gastrointestinal toxicity of the nonsteroidal anti-inflammatory drugs. Arch Intern Med 1987; 147(6): 1054–9PubMedCrossRefGoogle Scholar
  37. 37.
    Strom BL, Schinnar R, Bilker WB, et al. Gastrointestinal tract bleeding associated with naproxen sodium vs ibuprofen. ArchInternMed 1997; 157(22): 2626–31Google Scholar
  38. 38.
    MacDonald TM, Morant SV, Robinson GC, et al. Association of upper gastrointestinal toxicity of non-steroidal anti-inflammatory drugs with continued exposure: cohort study. BMJ 1997; 315(7119): 1333–7PubMedCentralPubMedCrossRefGoogle Scholar
  39. 39.
    Perez-Gutthann S, García-Rodríguez LA, Duque-Oliart A, et al. Low-dose diclofenac, naproxen, and ibuprofen cohort study. Pharmacotherapy 1999; 19(7): 854–9PubMedCrossRefGoogle Scholar
  40. 40.
    Ashworth NL, Peloso PM, Muhajarine N, et al. Risk of hospitalization with peptic ulcer disease or gastrointestinal hemorrhage associated with nabumetone, arthrotec, diclofenac, and naproxen in a population based cohort study. J Rheumatol 2005; 32(11): 2212–7PubMedGoogle Scholar
  41. 41.
    Patterson MK, Castellsague J, Walker AM. Hospitalization for peptic ulcer and bleeding in users of selective COX-2 inhibitors and nonselective NSAIDs with special reference to celecoxib. Pharmacoepidemiol Drug Saf 2008 Oct; 17(10): 982–8PubMedCrossRefGoogle Scholar
  42. 42.
    Ray WA, Chung CP, Stein CM, et al. Risk of peptic ulcer hospitalizations in users of NSAIDs with gastroprotective cotherapy versus coxibs. Gastroenterology 2007; 133(3): 790–8PubMedCrossRefGoogle Scholar
  43. 43.
    Rahme E, Bardou M, Dasgupta K, et al. Hospitalization for gastrointestinal bleeding associated with non-steroidal anti-inflammatory drugs among elderly patients using low-dose aspirin: a retrospective cohort study. Rheumatology (Oxford) 2007; 46(2): 265–72CrossRefGoogle Scholar
  44. 44.
    Rahme E, Nedjar H, Bizzi A, et al. Hospitalization for gastrointestinal adverse events attributable to the use of low-dose aspirin among patients 50 years or older also using non-steroidal anti-inflammatory drugs: a retrospective cohort study. Aliment Pharmacol Ther 2007; 26(10): 1387–98PubMedCrossRefGoogle Scholar
  45. 45.
    van der Linden MW, van der Bij S, Welsing P, et al. The balance between severe cardiovascular and gastrointestinal events among users of selective and non-selective non-steroidal anti-inflammatory drugs. Ann Rheum Dis 2009 May; 68(5): 668–73PubMedCrossRefGoogle Scholar
  46. 46.
    MacDonald TM, Morant SV, Goldstein JL, et al. Channelling bias and the incidence of gastrointestinal haemorrhage in users of meloxicam, coxibs, and older, non-specific non-steroidal anti-inflammatory drugs. Gut 2003; 52(9): 1265–70PubMedCentralPubMedCrossRefGoogle Scholar
  47. 47.
    Blot WJ, McLaughlin JK. Over the counter non-steroidal anti-inflammatory drugs and risk of gastrointestinal bleeding. J Epidemiol Biostat 2000; 5(2): 137–42PubMedGoogle Scholar
  48. 48.
    García Rodríguez LA, Hernández-Díaz S. Risk of uncomplicated peptic ulcer among users of aspirin and non-aspirin nonsteroidal antiinflammatory drugs. Am J Epidemiol 2004 Jan 1; 159(1): 23–31PubMedCrossRefGoogle Scholar
  49. 49.
    Traversa G, Walker AM, Ippolito FM, et al. Gastroduodenal toxicity of different nonsteroidal antiinflammatory drugs. Epidemiology 1995; 6(1): 49–54PubMedCrossRefGoogle Scholar
  50. 50.
    Battistella M, Mamdami MM, Juurlink DN, et al. Risk of upper gastrointestinal hemorrhage in warfarin users treated with nonselective NSAIDs or COX-2 inhibitors. Arch Intern Med 2005; 165(2): 189–92PubMedCrossRefGoogle Scholar
  51. 51.
    Kaplan RC, Heckbert SR, Psaty BM. Risk factors for hospitalized upper or lower gastrointestinal tract bleeding in treated hypertensives. Prev Med 2002; 34(4): 455–62PubMedCrossRefGoogle Scholar
  52. 52.
    Kaufman DW, Kelly JP, Wiholm BE, et al. The risk of acute major upper gastrointestinal bleeding among users of aspirin and ibuprofen at various levels of alcohol consumption. Am J Gastroenterol 1999 Nov; 94(11): 3189–96PubMedCrossRefGoogle Scholar
  53. 53.
    García Rodríguez LA, Walker AM, Pérez Gutthann S. Nonsteroidal antiinflammatory drugs and gastrointestinal hospitalizations in Saskatchewan: a cohort study. Epidemiology 1992; 1992(3): 337–42CrossRefGoogle Scholar
  54. 54.
    Jick SS. The risk of gastrointestinal bleed, myocardial infarction, and newly diagnosed hypertension in users of meloxicam, diclofenac, naproxen, and piroxicam. Pharmacotherapy 2000; 20(7): 741–4PubMedCrossRefGoogle Scholar
  55. 55.
    Morant SV, Pettitt D, MacDonald TM, et al. Application of a propensity score to adjust for channelling bias with NSAIDs. Pharmacoepidemiol Drug Saf 2004; 13(6): 345–53PubMedCrossRefGoogle Scholar
  56. 56.
    Rahme E, Nedjar H. Risks and benefits of COX-2 inhibitors vs non-selective NSAIDs: does their cardiovascular risk exceed their gastrointestinal benefit? A retrospective cohort study. Rheumatology (Oxford) 2007; 46(3): 435–8CrossRefGoogle Scholar
  57. 57.
    Beard K, Walker AM, Perera DR, et al. Nonsteroidal anti-inflammatory drugs and hospitalization for gastroesophageal bleeding in the elderly. Arch Intern Med 1987; 147(9): 1621–3PubMedCrossRefGoogle Scholar
  58. 58.
    Fries JF, Bruce B. Rates of serious gastrointestinal events from low dose use of acetylsalicylic acid, acetaminophen, and ibuprofen in patients with osteoarthritis and rheumatoid arthritis. J Rheumatol 2003; 30(10): 2226–33PubMedGoogle Scholar
  59. 59.
    Jick H, Feld AD, Perera DR. Certain nonsteroidal antiinflammatory drugs and hospitalization for upper gastrointestinal bleeding. Pharmacotherapy 1985; 5(5): 280–4PubMedGoogle Scholar
  60. 60.
    Lanza LL, Walker AM, Bortnichak EA, et al. Peptic ulcer and gastrointestinal hemorrhage associated with non-steroidal anti-inflammatory drug use in patients younger than 65 years: a large health maintenance organization cohort study. Arch Intern Med 1995; 155(13): 1371–7PubMedCrossRefGoogle Scholar
  61. 61.
    Mellemkjaer L, Blot WJ, Sorensen HT, et al. Upper gastrointestinal bleeding among users of NSAIDs: a population-based cohort study in Denmark. Br J Clin Pharmacol 2002; 53(2): 173–81PubMedCentralPubMedCrossRefGoogle Scholar
  62. 62.
    Schneeweiss S, Solomon DH, Wang PS, et al. Simultaneous assessment of short-term gastrointestinal benefits and cardiovascular risks of selective cyclooxygenase 2 inhibitors and nonselective nonsteroidal antiinflammatory drugs: an instrumental variable analysis. Arthritis Rheum 2006; 54(11): 3390–8PubMedCrossRefGoogle Scholar
  63. 63.
    van Soest EM, Sturkenboom MC, Dieleman JP, et al. Adherence to gastroprotection and the risk of NSAID-related upper gastrointestinal ulcers and haemorrhage. Aliment Pharmacol Ther 2007 Jul 15; 26(2): 265–75PubMedCrossRefGoogle Scholar
  64. 64.
    McIntosh JH, Byth K, Piper DW. Environmental factors in aetiology of chronic gastric ulcer: a case control study of exposure variables before the first symptoms. Gut 1985; 26(8): 789–98PubMedCentralPubMedCrossRefGoogle Scholar
  65. 65.
    McIntosh JH, Fung CS, Berry G, et al. Smoking, non-steroidal anti-inflammatory drugs, and acetaminophen in gastric ulcer: a study of associations and of the effects of previous diagnosis on exposure patterns. Am J Epidemiol 1988; 128(4): 761–70PubMedGoogle Scholar
  66. 66.
    Pilotto A, Franceschi M, Leandro G, et al. The risk of upper gastrointestinal bleeding in elderly users of aspirin and other non-steroidal anti-inflammatory drugs: the role of gastroprotective drugs. Aging Clin Exp Res 2003 Dec; 15(6): 494–9PubMedCrossRefGoogle Scholar
  67. 67.
    Castellsague J, Holick CN, Hoffman CC, et al. Risk of upper gastrointestinal complications associated with cyclooxygenase-2 selective and nonselective nonsteroidal antiinflammatory drugs. Pharmacotherapy 2009 Dec; 29(12): 1397–407PubMedCrossRefGoogle Scholar
  68. 68.
    Chang CH, Chen HC, Lin JW, et al. Risk of hospitalization for upper gastrointestinal adverse events associated with nonsteroidal anti-inflammatory drugs: a nationwide case-crossover study in Taiwan. Pharmacoepidemiol Drug Saf 2011 Jul; 20(7): 763–71PubMedCrossRefGoogle Scholar
  69. 69.
    Rothman KJ, Greenland S, Lash TL. Design strategies to improve study accuracy. In: Rothman KJ, Greenland S, Lash TL, editors. Modern epidemiology. 3rd ed. Philadelphia (PA): Lippincott Williams & Wilkins, 2008: 168–82Google Scholar
  70. 70.
    Ray WA. Evaluating medication effects outside of clinical trials: new-user designs. Am J Epidemiol 2003 Nov 1; 158(9): 915–20PubMedCrossRefGoogle Scholar
  71. 71.
    Suissa S, Garbe E. Primer: administrative health databases in observational studies of drug effects — advantages and disadvantages. Nat Clin Pract Rheumatol 2007 Dec; 3(12): 725–32PubMedCrossRefGoogle Scholar
  72. 72.
    García Rodríguez LA, Ruigómez A. Case validation in research using large databases. Br J Gen Pract 2010 Mar; 60(572): 160–1PubMedCentralPubMedCrossRefGoogle Scholar
  73. 73.
    Raiford DS, Pérez Gutthann S, García Rodríguez LA. Positive predictive value of ICD-9 codes in the identification of cases of complicated peptic ulcer disease in the Saskatchewan hospital automated database. Epidemiology 1996 Jan; 7(1): 101–4PubMedCrossRefGoogle Scholar
  74. 74.
    Cattaruzzi C, Troncon MG, Agostinis L, et al. Positive predictive value of ICD-9th codes for upper gastrointestinal bleeding and perforation in the Sistema Informativo Sanitario Regionale database. J Clin Epidemiol 1999; 52(6): 499–502PubMedCrossRefGoogle Scholar
  75. 75.
    Lin KJ, García Rodríguez LA, Hernández-Díaz S. Systematic review of peptic ulcer disease incidence rates: do studies without validation provide reliable estimates? Pharmacoepidemiol Drug Saf 2011 Jul; 20(7): 718–28PubMedCrossRefGoogle Scholar
  76. 76.
    McGettigan P, Henry D. Cardiovascular risk with non-steroidal anti-inflammatory drugs: systematic review of population-based controlled observational studies. PLoS Med 2011 Sep; 8(9): e1001098PubMedCentralPubMedCrossRefGoogle Scholar
  77. 77.
    Varas-Lorenzo C, Riera-Guardia N, Calingaert B, et al. The SOS Project. Risk of acute myocardial infarction and use of NSAIDs: a meta-analysis of epidemiological studies. Pharmacoepidemiol Drug Saf 2010; 19(S1): S142–S3Google Scholar
  78. 78.
    Varas-Lorenzo C, Riera-Guardia N, Calingaert B, et al. Stroke risk and NSAIDs: a systematic review of observational studies. Pharmacoepidemiol Drug Saf 2011 Dec; 20(12): 1225–36PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2012

Authors and Affiliations

  • Jordi Castellsague
    • 1
  • Nuria Riera-Guardia
    • 1
  • Brian Calingaert
    • 2
  • Cristina Varas-Lorenzo
    • 1
  • Annie Fourrier-Reglat
    • 3
  • Federica Nicotra
    • 4
  • Miriam Sturkenboom
    • 5
  • Susana Perez-Gutthann
    • 6
  • Safety of Non-Steroidal Anti-Inflammatory Drugs (SOS) Project
  1. 1.RTI Health SolutionsBarcelonaSpain
  2. 2.RTI Health SolutionsUSA
  3. 3.Universite V. SegalenBordeauxFrance
  4. 4.University Milan-BicoccaMilanItaly
  5. 5.Erasmus University Medical CenterRotterdamthe Netherlands
  6. 6.RTI Health SolutionsBarcelonaSpain

Personalised recommendations