The sponsor provided an evidence submission to NICE that reviewed the clinical and cost evidence for Vision ALD and presented a de novo cost-consequences model. Cedar a collaboration between Cardiff and Vale University Health Board and Cardiff University, was the EAC commissioned by NICE to produce the assessment report for this technology. The role of the EAC is to review and critique the sponsor’s submission: to ensure completeness and relevance of the evidence presented, to appraise its quality and to check the interpretation of this evidence with respect to the scope. The aim is to evaluate whether Vision ALD carries a diagnostic advantage and/or a reduced cost in comparison to current NHS standard care. The EAC was able to call upon clinical experts (research midwives and obstetricians) where clarification or advice on current UK practice was required.
Sponsor’s Submission of Clinical Effectiveness Evidence
The sponsor presented four studies, one of which was a laboratory study that assessed the ability of Vision ALD to detect small amounts of amniotic fluid . The other three studies were clinical trials of Vision ALD in which diagnostic accuracy was assessed against a reference test [21–23]. The EAC’s literature search and grey literature search did not identify any studies not included in the sponsor’s submission. The three clinical studies are summarised in Table 1. All three studies recruited women who presented at a labour or antenatal unit in secondary care. Comparative results were assessed in women presenting with unexplained vaginal wetness or a history suggestive of membrane rupture. In Bornstein et al. , two additional groups were recruited: a positive diagnosis group of women with overt spontaneous or artificial rupture of the membranes (n = 42) and a negative diagnosis group of women presenting for routine antenatal checks with no complaint of vaginal wetness (n = 27).
Only one of the studies (Mulhair et al.) used speculum examination alone as the comparator, as specified in the NICE scope. In the other two studies [21, 22] a positive comparator test was defined as a positive result from either speculum examination alone, or from both nitrazine pH and ferning tests.Footnote 1 In Bornstein et al. , the primary outcome was the comparison between the patient’s reading of the Vision ALD result and the initial clinical diagnosis. In this study, if the Vision ALD result was positive and the comparator was negative then the woman was tested for vaginal infection and underwent a second comparator test within 48 h.
The results as submitted by the sponsor indicate that Vision ALD has a high sensitivity ranging from 95.7 to 100.0 % and a reasonably high specificity of 75.0–84.5 %. (A lower specificity of 65.0 % was reported by Mulhair et al.  but was excluded by the sponsor.) The sponsor did not report any other outcomes but two studies also reported positive and negative predictive values: 67.4–87.0 % and 94.7–98.1 %, respectively [22, 23]. Mulhair et al.  estimated a 38 % reduction in referrals to their unit if the device were used in the community and also carried out a subgroup analysis on preterm subjects: sensitivity of 93 %, specificity of 76 % (n = 61).
Critique of Sponsor’s Submission of Clinical Evidence
The EAC excluded the laboratory study from the analysis as outside the scope of the evaluation . Each of the three clinical studies was consistent with the patient population in the NICE scope. Only Mulhair et al.  was also consistent with the scope intervention and comparator. The interventions varied slightly in the other two studies as both the patient and one  or two  investigators independently read the Vision ALD result. The Bornstein et al. studies [21, 22] used ‘clinical diagnosis’ as the comparator so that different reference tests were used for different patients (differential verification bias). Other outcomes identified in the scope were not reported: incidence of speculum examinations, speculum cross-infection, bed utilisation and staff time.
All three of the selected studies classed positive Vision ALD results due to vaginal infections as false positives, whereas the scope includes identification of vaginal infections as an outcome measure. In Bornstein et al. , six out of 34 women in the study group had a false positive Vision ALD result. Of these, four were diagnosed with bacterial vaginosis and the other two developed clinically identifiable PROM within hours of the initial examination. In the negative diagnosis group, two out of 27 women with no suspicion of ruptured membranes had a positive Vision ALD result but were subsequently identified as having ruptured membranes. In Bornstein et al. , 23 out of 309 women initially had a false positive Vision ALD result. Of these, four were rediagnosed as positive using another comparator test within 48 h.
The sponsor used an inappropriate appraisal tool, intended for randomised controlled trials, for reporting the quality of the included studies. The EAC used the QUADAS tool (QUality Assessment tool for Diagnostic Accuracy Studies)  to appraise the three diagnostic accuracy studies and rated them as good quality. Overall, the clinical evidence is good in terms of study design, but low in volume and only includes comparative diagnostic accuracy studies. It therefore provides little information about patient outcomes where clinical decision-making is based on Vision ALD readings. It is also unable to demonstrate whether Vision ALD is superior to the comparator.
Additional Work Carried Out by the External Assessment Centre (EAC)
The authors of the two of the Bornstein et al. studies [21, 22] provided the EAC with their study data upon request. With this the EAC was able to recalculate the diagnostic accuracy of Vision ALD using the intervention (Vision ALD assessed by a healthcare professional) and comparator (speculum alone) specified in the scope and to account for the re-classification of vaginal infections as true positives.
The diagnostic accuracy of Vision ALD was determined by the EAC using three different models:
Primary analysis: Intervention—clinician reading of Vision ALD. Comparator—initial clinical diagnosis using only speculum examination.
Post-hoc analysis: As for primary analysis, but women with initially false positive Vision ALD results who were later diagnosed with ROM or who gave birth within 72 h of the study are re-designated as true positives.
Infection as positive analysis: As for the post-hoc analysis, but false positives due to infections are re-designated as true positives. This is in accordance with the NICE scope in which identification of infections is an outcome measure.
These results are shown in Tables 2 and 3.
Sensitivity, specificity, positive predictive value and negative predictive value were calculated for each study, according to the model being used. The comparator was speculum examination only. Two patients from Bornstein et al.  were excluded from this analysis due to protocol violations. No patients were excluded from the other studies. Weighted means for sensitivity, specificity, positive predictive value and negative predictive values were used to take into account the sample sizes in the three studies.
The weighted mean false negative rate (1-sensitivity) for Vision ALD for the three models varied from 3.2–3.7 %, which is comparable to that for a speculum examination of 3.9 % [18, 19]. The number of speculum examinations avoided by using Vision ALD is the number who would be sent home following the Vision ALD result. This is derived from the negative Vision ALD test rate: Bornstein et al.  18/34, Bornstein et al.  131/307 and Mulhair et al.  53/139. The number of patients sent home erroneously following a Vision ALD test is determined from the number of patients with a false negative Vision ALD result as a proportion of all negative Vision ALD results: Bornstein et al.  0/34, Bornstein et al.  7/307 and Mulhair et al.  1/139. (This is not the same as the false negative rate, which is the number of patients with a false negative Vision ALD result as a proportion of all patients with a positive speculum result.)
The EAC recalculations show that Vision ALD has high sensitivity and reasonably high specificity and therefore shows promise in terms of non-inferiority to the reference standard. The high sensitivity (97 %) and high negative predictive value (96 %) means it can reliably rule out negative cases of PROM and PPROM in the populations included in these studies. If all women presenting with unidentified vaginal wetness were initially diagnosed with Vision ALD and only those with a positive result were then given a speculum examination, this indicates that around 42 % could avoid this procedure.
The sponsor identified Mulhair et al.  as a published economic study in their economic submission. In this the authors use their diagnostic accuracy results to speculate that the use of Vision ALD in the community to rule out PROM/PPROM could avoid 38 % of such referrals to the antenatal day unit (ADU) at a cost of £147 each. This is extrapolated to an approximate annual saving of £33,000 for that unit. However, the EAC considered this was not a valid economic study of Vision ALD.
The sponsor also provided a de novo economic model with an NHS perspective. The model structure was shown in a flow diagram comparing two diagnosis options for pregnant women with unexpected vaginal wetness who present at an ADU. The comparator is standard speculum diagnosis, in which the sponsor included a cardiotocograph trace (CTG, for detecting foetal heart rate and uterine contractions). In the intervention, Vision ALD is used as the initial diagnostic test. Patients with a positive Vision ALD result then go on to have a speculum examination to confirm membrane rupture (or infection), and a CTG. Those with a negative result are sent home. The timeframe for this model was the diagnostic tests only. No treatment or labour costs or costs associated with infections were included. The model was presented as a simple Excel spreadsheet.
The base-case inputs into this model and their sources are listed in Table 4. For the intervention the model resulted in a cost of £33.93 per patient. For the comparator, the model resulted in a cost of £43.94 per patient. Therefore the potential saving of using Vision ALD from the sponsor’s model is £10.01 per patient.
The sponsor conducted a one-way sensitivity and threshold analysis. The cost of the Vision ALD pads and disposable speculums were fixed, the range for bed day costs was taken from a published source (not identified) and other parameters were varied by ±50 %. For the threshold analysis the sponsor combined the midwife’s time to conduct a CTG trace and a speculum examination. Vision ALD remained cost saving over the one-way sensitivity ranges. Threshold analysis indicated that Vision ALD would no longer be cost saving if the cost of a pad exceeded £11.60, the time for a midwife to perform a speculum examination and CTG trace fell below 7.6 min, the time to administer Vision ALD increased above 12.3 min or the proportion of women with a negative Vision result fell below 19 %. Multiway analysis indicated the ‘best case’ (combination of inputs most favourable to Vision ALD) and ‘worst case’ (least favourable combination) scenarios for the maximum cost saved and minimum saved/maximum incurred, respectively, by using Vision ALD. The best-case scenario produced a cost saving of £54.60 per patient and the worst-case scenario incurred an additional cost of £3.52 per patient.
Critique and Interpretation of Cost Evidence
The sponsor’s de novo cost model was conservative; however, there were limitations and assumptions that weaken the model. Table 4 includes the EAC’s comments on the appropriateness of each input. The EAC conducted a more detailed sensitivity, threshold and impact analysis and determined that the cost savings were most sensitive to the midwife time for a CTG trace, followed by the time to administer a Vision ALD test and the proportion of patients with a negative Vision ALD test result. The higher cost of standard care was primarily due to the cost of midwife time to undertake the CTG trace. A clinical adviser confirmed that a CTG trace does not require 20 min of dedicated midwife time as modelled by the sponsor. The midwife would only stay to assess the trace if anything adverse is initially observed. The use of a CTG trace was not included in the scope, or in the sponsor’s clinical evidence or description of the clinical pathway. Additionally it could also be used in the intervention option, during the Vision ALD wear time. The EAC also considered the cost of a bed day in an ADU (£364 for 24 h) to be very high. The source of this value was unclear but is based on inpatient bed costs rather than outpatient or day unit; however, the impact of this is mitigated by the short stay-time (30–60 min). The sponsor obtained clinical advice regarding inputs from a single clinician and did not have their model checked by clinical advisers. The sponsor’s model does not account for false negative Vision ALD test results nor for any costs relating to the identification or treatment of infections as included in the scope.
The NICE scope states that Vision ALD can be administered in the community (primary healthcare) setting, which was not explored in the sponsor’s model. In light of this, NICE requested that the EAC model additional scenarios and inputs, including community use.
EAC Revisions to the Sponsor’s Economic Model
The EAC’s revisions to the economic model are separated into PROM and PPROM models, as the two conditions vary considerably in treatment, infection rates and costs. Subgroup analysis in Mulhair et al.  indicates that Vision ALD diagnostic accuracy is not different between these populations. In standard diagnosis, the speculum examination can be performed at an ADU, by a general practitioner (GP) or a midwife at the GP practice. In the intervention arm, Vision ALD can be administered in these same settings, or additionally by a midwife visiting the patient’s home. A positive Vision ALD result is followed up with a speculum examination, as in the sponsor’s model. The EAC’s model includes infection, with and without ROM, as a diagnostic outcome, and the timeframe includes infection treatment but not labour. The decision tree for PROM is shown in Fig. 1.
Base-case inputs attached to the model are given in Table 5. All inputs and the model structure were checked by clinical advisers. The positive speculum and Vision ALD result rates were determined as weighted means from the three clinical studies in the clinical evidence. We used the EAC post-hoc analysis to determine the rates for the speculum examination because this model provides the most complete estimate of the underlying prevalence rate. We used published infection rates rather than those from the clinical studies as infection testing was not universally conducted. We used the post-hoc plus infection analysis for the PPV for Vision ALD (86 %) as infection was included as an outcome. False positive results for the speculum examination are not accounted for as the PPV for this is very high (97 or 100 % ). The EAC model excluded Vision ALD false negative results because: (1) it has a very high negative predictive value, and (2) the false negative rates for Vision ALD and for standard diagnosis (speculum examination) are very similar. Thus any cost differences for these patients between the two arms of the model would be negligible.
There is no evidence or claim that Vision ALD increases the rate of diagnosis of ROM. The proportion of patients at each payoff should therefore remain constant, irrespective of whether Vision ALD or standard diagnosis is used. In practice there were small (<0.8 %) differences in the patient distributions that were considered to be due to rounding errors and to the assumption that the post-hoc analysis represents the true prevalence rate for ROM. Although we modelled treatment costs, these were not influential in the cost-consequence analysis due to these very small differences between outcome prevalences and are not shown here for simplicity. Infection rates were similarly unimportant in the cost differences between diagnosis options. Savings from the introduction of Vision ALD would therefore be realised if the cost of administering Vision ALD to all patients is offset by the savings from the number of speculum examinations avoided.
Results and Sensitivity Analysis from the External Assessment Centre (EAC)’s Revisions to the Economic Model
Base-case results are presented in Table 6 as per patient cost differences: Vision ALD minus standard care. Vision ALD is cost-incurring when administered in the same healthcare setting that a patient would attend for a speculum examination. Additional costs range from £1.28 to £38.28 depending on the clinician administering the tests and whether the condition is PROM or PPROM. Much higher treatment costs for PPROM patients meant that the small differences in outcome prevalences between the standard and Vision ALD arms of the model were more significant in this model. This leads to a consistent additional £5.75 cost for the Vision ALD arm as seen in the more positive values for PPROM in Table 6. Vision ALD was determined to be cost-saving only when it is administered at a lower clinician cost (for example, by a midwife at the GP practice or the patient’s home) when the patient would otherwise have gone to a high-cost scenario for a speculum examination (GP or ADU). One-way sensitivity analysis shows that clinician time is the most significant cost in the model in both PROM and PPROM, which supports the conclusions of the base-case analysis.