FormalPara Key Points for Decision Makers

VibraTip™, intended for the detection of diabetic peripheral neuropathy (DPN), has advantages in being readily portable, easy to use and provides a more consistent stimulus during examination than the 128 Hz tuning fork.

However, the available published evidence is insufficient to determine diagnostic superiority or equivalence of VibraTip™ confidently compared with the 10 g monofilament or the 128 Hz tuning fork. Additionally, the device is unlikely to reduce foot examination costs.

More research is therefore required to establish the place of VibraTip™ in the diagnostic management of DPN.

1 Introduction

The National Institute for Health and Care Excellence (NICE) provides evidence-based guidance for the National Health Service (NHS) in England and Wales with the aim of improving clinical outcomes for patients as well as delivering optimal use of finite NHS resources. The NICE Medical Technologies Evaluation Programme (MTEP) was established in 2009. The aim of MTEP is to evaluate and, where appropriate, encourage the adoption of, new and innovative medical devices into the NHS [1].

The Medical Technologies Advisory Committee (MTAC) is an independent body that works with MTEP and is responsible for the selection of medical technologies entering the programme, and the development of guidance from inception to final recommendations. To be selected, the technology must hold a current CE (Conformité Européenne) mark or be expected to gain one within 12 months, and must be considered by MTEP to have “plausible promise” [2]. This means that the technology must have the potential to have equivalent benefit to patients at lower cost to the NHS, or greater benefit with equivalent costs. Once selected by MTEP and assessment is commenced, medical technologies usually undergo a relatively rapid guidance development process of 38 weeks during which there is input from the sponsor (usually the manufacturer of the technology, who are responsible for submitting clinical and economic evidence), an External Assessment Centre (EAC, who evaluate the sponsor’s claims) and MTAC (who make recommendations) [3].

VibraTip™ (McCallan Medical) is a device resembling a small key-ring fob that provides a near-silent vibration, with specified amplitude and frequency similar to that of a 128 Hz calibrated tuning fork. It is indicated for use for the detection and assessment of diabetic peripheral neuropathy (DPN) in people with type 1 or type 2 diabetes mellitus. Adequate management of patients with DPN is believed to lead to a reduction in the risk of foot ulceration and associated complications [4], although good evidence for this, for instance the effectiveness of patient education interventions, is lacking [5]. Following MTAC selection of the topic in September 2013, assessment of VibraTip™ by the NICE EAC began in February 2014, with final recommendations published in December 2014 (NICE Medical Technology Guidance MTG22). This article provides an overview of the sponsor’s submission of evidence, the EAC’s critique of the evidence, and the formulation of final guidance. Full documentation of the process, supporting evidence and the final guidance can be found on the NICE website [6]. It is one of a series of NICE Medical Technology Guidance summaries being published in Applied Health Economics and Health Policy.

2 Background to the Condition and Device

Diabetes (type 1 and type 2) is a chronic disabling condition that is a major cause of morbidity and mortality, and is thought to affect around 3.38 million people within the UK in 2014 [7]. Left untreated, both forms of the disease have the potential to cause serious complications, including heart disease, stroke, blindness and nerve damage [8]. Diabetes is also a major burden to the UK economy, directly costing the UK £9.8 billion in 2010/2011, and this is forecast to rise to an estimated £16.9 billion by 2035/2036 [9].

Nerve damage caused by diabetes typically manifests itself as DPN, which is characterised by damage to, or degeneration of, peripheral nerves of the extremities including the sensory, motor and autonomic nerves. The prevalence of DPN in people with diabetes in the UK was estimated to be 28.5 % in a cross-sectional study (n = 6487), with higher prevalence associated with type 2 diabetes and greater length of time since onset of diabetes [10]. Patients with DPN typically have one or more of the following symptoms: numbness, tingling, pain or weakness. The symptoms typically begin in the feet and spread proximally, with deterioration of sensory symptoms more prominent than loss of motor function [11]. If left untreated, DPN can cause further serious complications. The main risk is numbness, as minor injuries of the foot may go unnoticed, leading to ulceration and secondary infection. In the worst cases, lower-extremity amputation may be required. Approximately 5 % of people with diabetes may develop a foot ulcer in any year, and amputation rates in diabetic people are around 0.5 % per year [4].

Although the later stages of DPN are poorly reversible or irreversible [12], there is evidence that improved glycaemic control can prevent the appearance and worsening of polyneuropathy in patients with type 1 diabetes [13], and although trial evidence is scant, the use of modified footwear and increased vigilance of ulcer formation is universally recommended for the prevention of foot ulcers in patients who have DPN [14]. However, to be effective, it is crucial that preventative measures are undertaken at an early stage in the development of DPN.

Current NICE clinical guidelines recommend that people with diabetes should undergo a thorough foot examination during their annual review [4, 15]. The foot examination should include testing of foot sensation (for DPN); palpation of foot pulses; inspection for any foot deformity; and inspection of footwear. A combination of these factors determines whether the person is deemed to be at low, increased or high risk of foot ulceration, which in turn informs referral pathways for intensive preventative management [4]. For people with type 2 diabetes, screening for the DPN element of the examination should be done using the 10 g monofilament or vibration (method unspecified) [4]. For people with type 1 diabetes, a non-traumatic pin prick is preferred over a vibratory test [15]. VibraTip™ is a new technology that is intended to be a replacement source of vibratory stimulus.

3 Decision Problem (Scope)

3.1 Population

The population described in the scope was “People (adults and children) with type 1 or 2 diabetes undergoing routine foot-care checks by health care workers in primary and secondary care settings” [16]. It was noted that diabetes affects a heterogeneous population, with the incidence and prevalence of diabetes increasing with age [17], and therefore caution should be used when generalising results from specific studies. Additionally, it was clear from the scope that the use of VibraTip™ by patients on themselves was out of scope.

3.2 Intervention (VibraTip™)

The intervention was the VibraTip™ device [16], which is illustrated in Fig. 1. VibraTip™ produces a vibratory stimulus similar to that of a 128 Hz tuning fork, and is intended to be used during examination of the foot for the detection of DPN. In the scope set by NICE [16], the sponsor claimed that the benefits of VibraTip™ for patients would be earlier diagnosis of neuropathy, leading to improved footcare and subsequent prevention of ulcers and amputation. For the healthcare system, the claimed benefits included improved consistency of testing for DPN, little need for training, greater portability and ease of cleaning.

Fig. 1
figure 1

Photograph of VibraTip™, the 10 g monofilament (Bailey’s Duraban Retractable shown) and the 128 Hz tuning fork (reusable Gardiner Brown)

Full size image

Although the description of the intervention as a physical entity was adequately described, the mode in which VibraTip™ should be used was not. This is because although the sponsor stated that VibraTip™ should act as a direct replacement for the 10 g monofilament or tuning fork (discussed below), there is currently no universally accepted guidance or consensus on how these devices should be used, for instance concerning the number and location of sites of the foot or feet the devices should be used on, or how many positive or negative tests should constitute a provisional diagnosis of DPN. This uncertainty is likely to impact on the generalisability of diagnostic accuracy studies to real-life clinical practice.

3.3 Comparators (Current Practice)

Three comparators were listed in the scope. These were the 128 Hz tuning fork, the 10 g monofilament and the biothesiometer [16]. Examples of the 128 Hz tuning fork and the 10 g monofilament are shown in Fig. 1. These devices are both widely used in NHS primary care, consistent with NICE guidelines for type 2 diabetes [4], and the Quality and Outcomes Framework (QoF) [18], which incentivises general practitioners to undertake DPN screening. The biothesiometer, which is functionally equivalent to, and has largely been superseded by, the neurothesiometer, is used predominantly by specialists, and is frequently the reference (or ‘gold’) standard used in clinical studies due to its diagnostic accuracy [19].

3.4 Outcomes

A mixture of diagnostic and management outcomes were described in the scope. These were sensitivity and specificity in assessment of vibration perception and/or light touch; sensitivity and specificity in assessment of grade of neuropathy; inter-rater agreement of assessment of grade of neuropathy; accuracy of risk assessment in ulcer formation; ulcer formation and amputation; time taken for sensory testing; quality of life and device-related adverse events [16]. However, only the first of these outcomes was explored in the clinical studies included in the submission.

4 Review of Clinical Effectiveness Evidence

4.1 Sponsor’s Review of Clinical Effectiveness Evidence

The sponsor performed a literature search and identified and presented a total of nine studies that they considered were relevant to the decision problem. The search identified seven published papers and two unpublished papers (in the form of conference posters and abstracts), and the sponsor excluded one study for not technically being in scope, leaving a total of eight studies. The literature search terms were not provided and study selection inclusion and exclusion criteria were not stated. As it was not replicable, the EAC performed its own literature search to identify papers published since 2007, following information that the device was first conceived of in late 2006 [20]. A flow chart illustrating the EAC’s literature search and sifting of published papers is provided in Fig. 2 according to PRISMA methodology [21], and full details are available on request.

Fig. 2
figure 2

PRISMA flow diagram for the External Assessment Centre (EAC) literature search

Full size image

Of the nine studies presented by the sponsor, the EAC identified eight papers from the literature search, and was supplied with an additional three unpublished papers from the sponsor (not identified by the EAC search), making 11 papers in total. Of these papers, the EAC excluded three papers on the basis one was a review and did not report additional primary data [22], one was a conference abstract of a study fully reported elsewhere [23], and one was an unpublished randomised controlled trial (that has since been published) but was out of scope on the basis of the population and intervention studied (lifestyle modification in people with diagnosed DPN) [24]. Of the eight remaining papers, two papers were technical studies and therefore considered out of scope [25, 26]; however, one of these studies did provide useful technical analysis which was used in the economic evaluation [26].

The six remaining papers were all cross-sectional diagnostic accuracy studies and are described in Table 1. Four were published studies [27–30], one was a conference abstract [31] and one was a conference poster [32]. These studies informed the clinical evidence submission for VibraTip™. The sponsor adequately reported the characteristics of the relevant studies, but did not critically appraise the studies to address methodological quality and potential sources of bias. Although the sponsor reported the primary results from the studies, they did not attempt to place the results in the context of the decision problem.

Table 1 Characteristics of six diagnostic accuracy studies that addressed the scope of the decision problem
Full size table

4.2 Critique of Clinical Effectiveness Evidence

The EAC considered that all six diagnostic accuracy studies were relevant to the decision problem, and undertook critical appraisal of these papers to establish their internal and external validity in this context (Fig. 3). The EAC used the QUADAS-2 tool (revised Quality Assessment of Diagnostic Accuracy Studies), which assesses the risk of study bias in four domains (patient selection, index test, reference standard, flow and timing), and the applicability of the study to the decision problem in three domains (patient selection, index test and reference standard) [33], and is recommended by NICE [34].

Fig. 3
figure 3

Pictogram summary of critical appraisal of diagnostic accuracy studies using QUADAS-2 (revised Quality Assessment of Diagnostic Accuracy Studies) [bias and applicability domains]

Full size image

A significant limitation of three of the studies was the lack of a reference standard [29, 31, 32], without which it is impossible to calculate diagnostic accuracy. Another study was performed in a population who had had DPN pre-diagnosed, and reported on the target condition of ‘at risk’ feet [27]. This study was considered to be of limited value because VibraTip™ or its comparators are not routinely used to detect DPN in this very high-risk population, and it is not clear how detection of DPN relates to the outcome reported. Another study used a modified reference standard (a neurothesiometer set to 20 V rather than the usual threshold of 25 V) [30]. The difference in vibration perception threshold over this range is likely to be clinically significant [35] and, for this reason, this study was judged not to be generalisable to the decision problem.

This left one study, that of Bracewell et al., that was judged to be most relevant to the decision problem [28]. This study recruited from the most appropriate population (diabetic patients), used a suitable reference standard (neurothesiometer set at 25 V) and compared the diagnostic performance of VibraTip™ with the 10 g monofilament and the 128 Hz tuning fork (both comparators specified in the scope). However, a potential weakness of the study was that it used Receiver Operator Characteristics (ROC) analysis to optimise the protocol for each device (number of insensate sites required for DPN detection), and used these retrospectively to set diagnostic thresholds. Therefore, the results may not be generalisable to usual clinical practice. Additionally, although this was the second largest study identified on VibraTip™ (n = 141), it was likely to have been underpowered and prone to type 2 error [36].

The results from Bracewell et al. indicated that (relative to the neurothesiometer), VibraTip™ had a sensitivity of 0.79 (95 % CI 0.69–0.90) and specificity of 0.82 (95 % CI 0.74–0.90); 10 g monofilament had a sensitivity of 0.84 (95 % CI 0.75–0.94) and specificity of 0.83 (95 % CI 0.75–0.91); and the 128 Hz tuning fork had a sensitivity of 0.69 (95 % CI 0.57–0.81) and specificity of 0.90 (95 % CI 0.84–0.97). Thus, the 10 g monofilament was slightly superior to VibraTip™ in terms of both sensitivity and specificity, and the 128 Hz tuning fork had poorer sensitivity but superior specificity. Although it was not possible to perform hypothesis testing because of the way the data were reported, it is unlikely that there was a statistically significant difference between the three devices used in this study. The results of this study are illustrated in Fig. 4.

Fig. 4
figure 4

Sensitivity and false positive rate (1—specificity) of VibraTip™, 10 g monofilament and the 128 Hz tuning fork (all relative to a neurothesiometer reference test)

Full size image

5 Economic Evidence

5.1 Sponsor’s Economic Submission

There was no indication that the sponsor performed a literature search for existing economic papers that might inform the decision problem, and no economic studies were supplied. However, the EAC did not identify any relevant economic studies from a subsequent literature search. As is standard practice for sponsor’s submissions for MTEP [37], the sponsor provided a de novo economic model to support claims that VibraTip™ was potentially cost saving to the NHS.

The model was written and executed in Microsoft Excel®. The basis of the model was that of a decision tree covering a 3-year time horizon with a starting population of people with diabetes in the UK (estimated at 2.9 million). The model had two arms, which consisted of a current practice arm (10 g monofilament or 128 Hz tuning fork) and an intervention arm, where patients were tested with VibraTip™ or retained practice (10 g monofilament or 128 Hz tuning fork). The decision tree simulated patients on a pathway where they had chances of becoming high risk for ulceration, developing ulceration, having continued ulceration or requiring amputation. Each stage in the model was associated with a transition probability, the number of patients in that state and the cost of being in that state (device costs and management costs). The sponsor stated costs were calculated at present day (2014) values and a discount rate of 3.5 % was applied.

In the base-case analysis, the sponsor assumed the VibraTip™ would be used in 40 % of foot examinations. In addition, the sponsors performed deterministic sensitivity analysis where the usage of VibraTip™ was assumed to be 20 or 100 %, and this was combined with a second sensitivity analysis, whereby an assumption was made that the introduction of VibraTip™ was associated with a 1 % reduction in ulcer formation (which was applied to the transitional probabilities used in the decision tree).

The sponsors reported that in the base case, the introduction of VibraTip™ would lead to overall cost savings to the NHS of approximately £50,000 compared with the 10 g monofilament and £40,000 compared with the 128 Hz tuning fork. From sensitivity analysis, the sponsor reported that increased adoption of VibraTip™ would lead to proportionate savings, as would be expected. It was calculated that if the use of VibraTip™ led to a 1 % reduction in ulcer formation, then VibraTip™ would lead to savings of £6,430,000 compared with the 10 g monofilament and £6,350,000 compared with the 128 Hz tuning fork, at the level of 40 % adoption.

5.2 Critique of Economic Evidence

The EAC critiqued the sponsor’s economic model, the accompanying narrative and conclusions. The EAC found several weaknesses in the model, which was not fully executable. This, as well as other inconsistencies in the model’s structure and populated parameters, made it difficult to replicate the sponsor’s results, as reported in their narrative, and the veracity of the results could not be confirmed.

The EAC found that a fundamental weakness of the model was that there was no diagnostic input into it, because VibraTip™ and the comparators were assumed to have diagnostic equivalence. Thus, the clinical evidence submission did not inform the economic model, and in the base case the model might be considered a de facto cost minimisation study, whereby diagnostic equivalence of devices was assumed with only the costs associated with device use per examination contributing to the overall costs. As a consequence of this, the overall cost savings reported by the sponsor for the base rate were very low compared with the overall burden of the condition.

The EAC considered that the first sensitivity analysis, whereby the sponsor adjusted the adoption uptake of VibraTip™, which caused a corresponding change in cost savings in a linear manner, was uninformative. The EAC considered that the assumption made in the second sensitivity analysis, that use of VibraTip™ would lead to a 1 % reduction in ulcer formation, had no evidence to support it, and thus did not inform the decision problem.

5.3 Additional Analysis

The EAC further considered the per examination costs used by the sponsor for each device, which were the sole drivers of costs reported in the base-case analysis. For its calculation of the per examination costs for the 10 g monofilament and the 128 Hz tuning fork, the sponsor used estimates based on simulated clinic use and expected useful life (assumed to be 1 year), and had estimated these costs were 1.0 and 0.8 pence, respectively. However, for VibraTip™, the sponsor calculated the per examination cost according to its battery life, irrespective of simulated clinical use. The sponsor estimated the per examination cost would be 0.2 pence, based on 5000 clinical examinations before battery discharge made the device unreliable. This was based on a technical paper by Horsfield and Levy [26]. The EAC considered that this was likely to be an overestimate and therefore the per examination costs would likely be higher. This was for two reasons: firstly, because the technical paper reported that the amplitude of VibraTip™ decays significantly after the first 1000 activations, leading the authors to conclude that “VibraTip would provide a very consistent source of vibration to test at least 100 patients, if not considerably more”; and, secondly, because the sponsor assumed that only one site on one foot would be tested. Whilst there is considerable uncertainty concerning the optimal clinical practice when testing for DPN, it is anticipated that as a minimum both feet are tested, but also often multiple sites on each foot.

It is known that the useful life of the 10 g monofilament is limited, and this issue has been addressed in the literature. A technical study performed by Lavery et al. on several brands of monofilaments showed considerable variation in their durability after repeat testing [38]. However, the Bailey’s 10 g monofilament (used widely in the NHS [39]) fared comparatively well, with the results indicating the device produced a buckling force within the limits of acceptability (between 9 and 11 g) after 1800–2400 tests. If the device is used five times on both feet (as is suggested in the product literature), this suggests the Bailey’s monofilament would have a useful life of approximately 200 patients before requiring replacement.

The EAC considered that, for the 128 Hz tuning fork, it was not possible to provide a meaningful per examination cost in this way, because it has an unlimited useful life. The per examination costs of VibraTip™ and the 10 g monofilament (Bailey’s) in various scenarios are reported in Table 2. As can be seen, in several of these scenarios, the cost of VibraTip™ (range 0.20–9.95 pence) exceeds that of the 10 g monofilament (range 3.04–19.00 pence), with the EAC’s most plausible estimated cost per examination for VibraTip™ (9.95 pence corresponding to the published literature) being more than the 10 g monofilament (7.60 pence).

Table 2 Per examination cost estimates for VibraTip™ and the 10 g monofilament. Cost estimates for the 128 Hz tuning fork are not possible using this methodology
Full size table

6 NICE Guidance

6.1 Provisional Recommendations and Consultation

In June 2014, MTAC met to make provisional recommendations on VibraTip™, aided by the EAC and guidance from three expert advisors. During the discussion, the Committee noted that VibraTip™ had some advantages over its comparators regarding its portability, ease of use and ability to produce a consistent stimulus (particularly compared with the tuning fork, which varies depending on how hard it is struck). However, it acknowledged that there was considerable uncertainty regarding the diagnostic accuracy of the device and the economic case had not been made. The Committee decided to make a research recommendation that would encourage the development of further evidence to support the case for adoption. MTEP plans to help facilitate this research so the guidance can be updated when further evidence is available.

Following the meeting, draft guidance was produced, which was released for public consultation between 9 July and 8 August 2014. In all, 27 comments were made from stakeholders, which were addressed at the MTAC meeting held in October 2014. However, none of the comments received offered new evidence and, as a result, only minor word changes were made to the final guidance.

6.2 Final Guidance

In December 2014, NICE made the following recommendations concerning the use of VibraTip™ in patients with diabetes: [6]

  1. 1.1

    VibraTip™ shows potential to improve the detection of DPN and to provide cost savings to the NHS. Although VibraTip™ appears to be easy to use, portable and reliable in its functionality, more evidence is needed on its clinical benefits and economic advantages to support the case for its routine adoption in the NHS.

  2. 1.2

    Research is recommended to address uncertainties in the potential benefits to patients and the NHS of using VibraTip™. Research is needed into the diagnostic accuracy of VibraTip™ compared with the 10 g monofilament and calibrated tuning fork in the diagnosis of peripheral neuropathy in people with diabetes. This research should also address the assessment of vibration perception compared with touch sensation in this clinical context. The research should gather information on the health system and economic impact of introducing VibraTip™ for detection of DPN. This should include longer-term outcomes so that an accurate and comprehensive cost consequences analysis can be carried out. NICE will review this guidance when substantive new evidence becomes available.

7 Key Challenges and Learning Points

There were several challenges associated with the development of this Medical Technology Guidance (MTG22). As is common with medical devices, particularly technologies produced by small manufacturers with limited access to funding [40], there was a lack of high-quality clinical studies available, for instance those demonstrating the diagnostic accuracy of VibraTip™ in comparison with the 128 Hz tuning fork or 10 g monofilament. Only one study was identified that matched the scope [28], and this was underpowered to clearly show non-inferiority of any of the technologies over another. It is unclear how generalisable these results are. It is noted that even small differences in diagnostic accuracy could have major implications for the subsequent patient pathways, especially when considered at the population rather than individual level. An additional area of uncertainty is that currently there appears to be a large degree of variation in the diagnostic management of patients at risk of DPN, especially concerning the specific sites of the foot that should be tested, as well as the number of positive tests (i.e. lack of sensation) that should facilitate a diagnosis. This level of detail is not described in existing NICE guidelines [4, 15], and there appears to be no national or international consensus regarding the optimal clinical testing protocol.

This lack of unequivocal clinical evidence meant that the sponsor was compelled to adopt a de facto cost-minimisation model for their economic submission. Because the modelled patient pathways were the same for each diagnostic technology, only variance in costs between the technologies themselves contributed to overall cost differences, but these were trivial in comparison to the overall burden posed by DPN. The uncertainty in diagnostic clinical practice also contributed to uncertainty regarding the per examination costs of the economics (Table 2).

The uncertainties surrounding the clinical efficacy of both VibraTip™ and its comparators, and the impact of this uncertainty on the economic potential of VibraTip™, meant that MTAC were unable to give a clear recommendation about the technology. However, MTAC considered the technology had promise and so decided to make a research recommendation that would encourage the development of further evidence.

8 Conclusion

Following assessment through the MTEP process, the NICE Medical Technology Guidance 22 describes VibraTip™ as having the potential to improve the detection of DPN and to provide cost savings for the NHS. However, there is a lack of available clinical evidence to prove this beyond reasonable doubt. The research recommendation in the guidance means that MTEP will try to facilitate the research so that the guidance can be updated when more relevant evidence becomes available.