Selection of trials
We used a sample of randomised controlled trials that had been published in seven general medical journals: The New England Journal of Medicine, The Journal of the American Medical Association, The Lancet, The British Medical Journal, Annals of Internal Medicine, Archives of Internal Medicine, and The American Journal of Medicine. We selected these journals because they are highly ranked in the Science Citation Index and provide a relatively broad coverage of general medical topics. The Cochrane Controlled Trials Register was used to identify all the randomised controlled trials from these journals in the year 1997 [8]. As our primary objective was to evaluate the reporting of drug ADRs, we reviewed only those trials in which at least one treatment arm involved a pharmacological intervention. We excluded papers in which the main bulk of the data had been published elsewhere, e.g. post hoc or subgroup analyses of trials, or papers that stated that adverse effects data were or would be available in a separate publication. A complete list of the trials evaluated is available from the authors.
Two reviewers assessed all trials independently, using a checklist, and any discrepancies were resolved through discussion.
Evaluation
The review checklist was based largely on the recommendations of SORT; our main criteria were that:
-
1.
ADRs should be clearly and appropriately defined;
-
2.
there should be sufficient detail on how the reported ADRs were elicited and recorded (including definitions of severity) so that the reader can fully interpret the result or reproduce the study if necessary;
-
3.
frequencies of adverse effects should be reported in a simple, coherent manner.
We also measured the amount of space (area in cm2) in the Results and Discussion sections that was devoted to ADR reports.
Details are given below on the specific criteria used in evaluating ADR reports.
Clear and appropriate use of ADR definitions
Adverse drug reactions have been defined as those "where the causal relation between the product and an adverse event is at least a reasonable possibility", while the less specific (but widely used) term "adverse events" should be reserved for describing untoward occurrences that are not necessarily causally related to the treatment [9, 10]. The terms should not be used interchangeably.
The use of terminology was considered appropriate if trials reported drug safety with terms such as "adverse drug reactions", "adverse or side effects" or "drug toxicity", or if they explicitly stated which adverse events were possibly related to drug treatment. Reports of adverse outcomes without clarification of their relation to drug treatment were considered to be inadequate.
Sufficient detail on methods used in monitoring of ADRs for the reader to interpret or reproduce the result
When ADRs were reported, we asked whether it was possible for readers to tell which specific methods had been used in the detection or recording of ADRs. We evaluated the following areas of ADR monitoring – symptoms reported by patients, events diagnosed clinically by physicians, and abnormalities detected using investigative tests.
Patients' symptoms
We asked whether the trial report explained how the rates of symptomatic ADRs had been determined during the course of the study; for example, by spontaneous reporting, patient diary or checklist questionnaire.
ADRs presenting as clinical events diagnosed by physicians
We asked whether the reported ADRs had been detected with specific, targeted clinical assessments or purely from spontaneous reports as they arose. In particular, we looked for detail of systematic surveillance measures e.g. investigators were asked to take blood pressure readings at 15 minute intervals to look for hypotension; rates of gastrointestinal haemorrhages were monitored through routine checking of hospital records.
ADRs detected through the use of diagnostic tests
We asked whether the trial specified which diagnostic techniques had been used in detecting the reported ADRs; for example, was deep venous thrombosis diagnosed by ultrasonography or by contrast venography? In cases in which laboratory tests were used, we asked whether the laboratory values or criteria for diagnosing an abnormality were stated; for example, was hypokalaemia diagnosed at serum potassium concentrations below 3.0 mmol/l?
Severity
We looked for details on the severity of the reported ADRs and, if present, whether the measures of severity were made according to recognized scales (such as the National Cancer Institute Toxicity Grades[11]), or otherwise defined e.g. "major bleeds were those that were fatal or required transfusion".
Nature of ADR reporting
Numerical data
When ADRs were reported, we asked whether simple summary (unadjusted) data were always given for each treatment arm; were there instances in which data were given as a lumped total (e.g. headache noted in 5% of all patients in the trial), or for only one treatment arm?
We also looked at the prevalence of statements that could not be fully interpreted owing to lack of detail on the frequency and/or type of adverse effect; e.g. "The pattern and profile of adverse effects were similar" or "The treatments were well tolerated". Such generic statements, in the absence of any supporting information, are not helpful in evaluating the frequency of ADRs [2].
Space for reporting adverse events
We measured the area (cm2) used for reporting ADRs (text, figures and tables) in the Results and Discussion sections of the published paper, and expressed it as a percentage of the total area of the Results and Discussion sections.