Introduction

Hip fractures are a major cause of disability and functional impairment among the elderly [12]. Most cases result from fall-related direct impact on the greater trochanter of the proximal femur [20, 27]. Fall prevention programmes have been extensively studied. Multifaceted interventions may prevent falls. There is no evidence that fall prevention programmes reduce hip fractures [5]. National guidelines recommend offering hip protectors to all nursing home residents [4]. Hip protectors have been suggested to be the only non-pharmacological intervention to effectively prevent hip fractures in high-risk populations [10, 14, 16]. However, recently published trials did not demonstrate hip fracture reduction through hip protector intervention [18, 24]. An updated Cochrane review concluded that hip protectors are ineffective for community-dwelling elderly and their effectiveness remains uncertain for institutionalised elderly [19]. However, the effectiveness of a hip protector intervention depends on the baseline risk of the population and the adherence to wear the device. Hip protectors do not work unless they are worn at the time of the fall. Suboptimal implementation of hip protectors might lead to low adherence and, therefore, to ineffectiveness of the intervention. We aim to illustrate that quality assessment even within high-quality systematic reviews does not reflect if the intervention programmes were well planned and competently administered.

Materials and methods

We searched PubMed (to April 2006) to find systematic reviews on hip protectors. Reference lists of the retrieved two meta-analyses were reviewed to identify primary studies. We present a brief overview of the external evidence on hip protectors summarised in the Cochrane review [19] and briefly discuss disagreement with the review by Sawka et al. [23]. We analysed shortcomings in implementation techniques, which cannot be identified by the usual approach of critically appraising randomised controlled trials. Therefore, we exemplarily focused on two primary studies that the Cochrane reviewers rated as having very good methodological quality. Both studies [18, 24] received the largest weight in the meta-analysis of institutional setting studies and had a major impact on the interpretation of the effectiveness of hip protector interventions.

How hip protectors work

Hip protectors are devices designed to protect the hipbones during a fall. Different protectors have been studied in several randomised controlled trials in various countries [19]. Most types consist of plastic shields or foam pads that are held in place at the hips with specially designed underwear. Energy shunting (hard-shelled) hip protectors are made of durable plastic, designed to divert a direct impact away from the greater trochanter onto the surrounding soft tissue. Soft hip pads primarily absorb energy. Biomechanical studies suggest that the force-attenuating capacity of the hard-shelled hip protectors is superior to soft, energy-absorbing ones [11, 22, 25]. Only few hip fractures have been reported with documented use of the hip protector [16, 18, 24]. However, verification if the hip protector was correctly used at the time of fall is difficult [16].

The acceptance of hip protectors remains a problem. Structured education and involvement of caregivers can substantially improve adherence to hip protectors [16, 26].

External evidence on the effectiveness of hip protectors

The first randomised controlled trial investigating the effectiveness of hip protectors in nursing home residents was published 13 years ago [14], followed by several trials and reviews on hip protector intervention programmes.

Recently, the sixth update of the Cochrane review was published [19]. Figures 1 and 2 show the results of this meta-analysis. The authors have pooled five trials that used individual randomisation together with six cluster randomised trials. The studies were carried out in nursing homes or residential homes. The results show a significant reduction in hip fracture incidence [relative risk (RR) 0.77, 95% confidence interval (CI) 0.62 to 0.97]. However, the upper limit of the confidence interval indicates marginal statistical significance. Sensitivity analyses were carried out because of statistical heterogeneity within the group of cluster randomised trials. Removal of the trial by Kannus et al. [10] results in loss of significant effectiveness of hip protector interventions, whereas the significant effect remains while removing the trial by O’Halloran [18]. Data pooling of three individually randomised trials of hip protector interventions in community-dwelling elderly resulted in no reduction of hip fracture incidence (RR 1.16, 95% CI 0.85 to 1.59).

Fig. 1
figure 1

Incidence of hip fractures in elderly participants provided with hip protectors or not in an institutional setting

Full size image
Fig. 2
figure 2

Incidence of hip fractures in elderly participants provided with hip protectors or not in the community

Full size image

Discordance among systematic reviews on hip protectors

Sawka et al. [23] suggested a more pronounced relative risk reduction in nursing home residents allocated to hip protector intervention (RR 0.50, 95% CI 0.28–0.91). Assessment of study quality was a source of discordance among the two meta-analyses. Inclusion criteria were stricter in the review by Sawka et al. Five trials considered in the Cochrane review were excluded due to violation of randomisation [10, 14], lack of complete random allocation [2, 8] and unit of analysis bias [3, 10, 14]. Although the trial by O’Halloran et al. [18] has been published after finishing literature search for the review by Sawka et al., the authors stated that the trial would not have met their strict inclusion criteria due to major methodological shortcomings.

Sawka et al. classified the population investigated by van Schoor et al. [24] as community dwelling, whereas the Cochrane review defined it as institutionalised. As half of the study population consisted of elderly living in apartment houses or homes for the elderly and half of nursing home residents, both classifications have shortcomings. This problem is typical for meta-analyses that use aggregate, instead of individual, patient data. The latter one is judged to be the more valid method to summarise the results of multiple studies [15]. It has been suggested that meta-analyses based on aggregate patient data should be viewed as only exploratory, as they do not provide reliable information [21].

Diversity of hip protector interventions

The Cochrane review [19] as well as the review by Sawka et al. [23] quantitatively summarised studies regardless of different co-interventions or different control group interventions. Most studies investigated the effects of a single provision of protectors; other intervention programmes comprised both education and information approaches and provision of hip protectors [19]. The latter were more or less complex interventions, which consist of components that depend on each other and, therefore, cannot be separated [1]. All trials except one used no intervention as comparator. This one trial [16] offered optimised usual care to the control group through information to nurses about hip protectors and provision of protectors for demonstration purposes. In both systematic reviews, trials were synthesised through meta-analysis without exploring and considering clinical heterogeneity. Unlike statistical heterogeneity, clinical heterogeneity due to different inclusion criteria, manner of administration of interventions or measurement of outcome parameters can be identified before data analysis. Suggestions have been made on how to consider clinical heterogeneity in systematic reviews [6].

Clinical flaws despite high-quality rating of methodology

The Cochrane review [19] applied rigorous quality criteria well-known to influence the validity of randomised controlled trials [9]: allocation concealment, clear definition of inclusion and exclusion criteria, loss to follow-up, comparability of study groups, equal treatment of study groups, except of the intervention, and intention-to-treat analysis. Further criteria specially addressing hip protector trial methodology were applied: active or scheduled follow-up ascertainment and monitoring of adherence. However, appraising trials using formal checklists may overlook important clinical flaws in the original trials [7]. This can be illustrated exemplarily by critical appraisal of two included trials [18, 24]. The trials by van Schoor et al. [24] and O’Halloran et al. [18] were chosen, as they received the largest weight in the data pooling of institutional setting studies (Fig. 1).

Van Schoor et al. [24] investigated the effectiveness of a hip protector intervention programme in a heterogeneous population of elderly living in apartment houses, residential homes and nursing homes. The intervention did not result in a reduction of hip fractures, and the authors claimed the ineffectiveness of the hip protector.

However, the study has several important methodological shortcomings. The power calculation was based on a one-sided p value and a reduction of hip fractures of 75%. Such a pronounced effect has never been shown before. Sample-size calculation indicated the need of 700 participants. However, only 561 participants were enrolled. During the trial, follow-up was extended to 69.6 weeks to increase the number of events.

The education approach used in the intervention group was scarcely described. The underlying education theory was not mentioned. Precise details of the interventions intended for each group and how these were actually administered were not given. The implementation of the intervention may be less than optimal for the group of cognitively impaired persons because there was no structured involvement of caregivers.

Van Schoor et al. emphasized single randomisation as an important methodological strength of their study. However, the avoidance of cluster randomisation seems to be problematic in this setting. Contamination could have compromised the study, as the same caregivers looked after participants of both study groups.

Outcome assessment relied on participant-kept calendar. As baseline data indicated cognitive impairment in approximately 75% of participants, the reliability of the data is of major concern. None of these obvious shortcomings has been reflected in the Cochrane review, which classified the trial of high methodological quality with a score of 10 out of a possible 10.

The study by O’Halloran [18] is by far the largest trial investigating a hip protector intervention in institutionalised elderly. Thus, it received a weight of 46% in the meta-analysis of the Cochrane review. The sample consisted of 127 nursing and residential homes with 4,117 beds in Northern Ireland. The combined intervention of provision of hip protectors, education and information did not result in a reduction of hip fractures during 72 weeks of follow-up. However, this study also has important methodological shortcomings, which might have heavily affected the results. A mixed population was investigated, including nursing home residents and non-nursing home residents. Baseline data of participants are not reported. It remains unclear whether the two study groups were comparable at the beginning of the study. A flow of participants through the trial, which discloses the number of subjects with early study termination and the proportion of subjects that have been included after randomisation, is not given. The absolute number of participating residents remains unknown, as data are reported as events per 100 occupied beds or per 100 residents per year.

The multifaceted intervention consisted of an education session for home staff, distribution of manufacturer’s information material, provision of a videotape on hip protectors, information for residents and relatives on request, counselling by a nurse facilitator and provision of hip protectors for every resident agreeing to wear them. No details are reported on how the education and information programme was structured, whether it was based on an education theory and if so, whether it had been sufficiently explored before application. The reader cannot assess if an appropriate intervention was administered. Access to the education programme is not possible neither by further references nor authors’ offer to contact them.

It remains unclear whether newly admitted residents also received the information or whether residents and relatives requested information sessions. It is of special concern that the authors did not mention how and when the investigators determined agreement to wear the hip protector. Frequency of hip protector use in the control group was reported at baseline only. However, during the time course of the trial a successful hip protector trial has been published [10], and hip protector use became a more common prevention strategy. It is possible that an increasing number of control group residents also used hip protectors.

Outcome measurement was based on routine documentation of homes and chart review by the external nurse facilitator. Numbers of falls, fallers and total fractures were not reported and probably not ascertained. It remains unclear if participants of the intervention and control centres had a similar risk of suffering a hip fracture. The reported significant difference in pelvic fractures suggests that the use of the hip protector pads was not the only important difference among the study groups.

Furthermore, the trial was limited, as seven out of 40 homes were not compliant with the introduction of the intervention and outcome ascertainment compared to six out of 87 homes not cooperating in the control group. Management of missing data has not been discussed.

Although the clustering design effect was considered in the sample size calculation, the results were not adjusted for the cluster effect.

In conclusion, critical appraisal of the study by O’Halloran revealed several major methodological flaws. It seems that hip protectors were disseminated to a heterogeneous group of elderly using an unstructured, not theory-based education and information approach. These biases affecting the internal validity of the study have not been considered by the Cochrane review. On the contrary, quality assessment by reviewers’ checklist indicated a score of 9 out of 10 [19].

Conclusion

It has been suggested previously that the assessment using a methodological criteria checklist might end up with a high score despite major mistakes [13]. Although formal methodological criteria assessed within the Cochrane review were fulfilled, the two studies used suboptimal implementation practice and outcome collection. The inferior quality of these trials means that their findings must be questioned. The conclusion about ineffectiveness of hip protectors drawn by the Cochrane review should, therefore, be interpreted with caution.

What are the implications of our findings? They must not be misinterpreted as a pleading not to use methodological quality appraisal. Methodology of systematic reviews has developed considerably during the past years. Guidelines have been developed to improve quality of meta-analysis [17]. However, it is not enough to rely on the author’s self-reported formal quality criteria. Reviewers should go one step further and explicitly assess the quality of the intervention not only the methodological quality. A standardised set of criteria on the quality of implementation can only inadequately assess individual, study-specific methodological flaws. We suggest a narrative expert review on quality of implementation techniques and feedback to the authors of the primary study to clarify ambiguous issues and to confirm reviewers’ judgment. Our recommendation has been forwarded to the Cochrane Collaboration.