In this study, system accuracy of four SMBG systems in the hands of lay-users, i.e., diabetes patients, was evaluated with one test strip lot on the basis of testing procedures and accuracy criteria of ISO 15197:2013, clause 8. Additionally, accuracy in the hands of study personnel trained in operating the SMBG systems was evaluated with one test strip lot for each of the four systems. According to ISO 15197, user performance evaluation shall demonstrate that intended users are able to obtain accurate measurement results when operating the SMBG system, given only the instructions for use and training materials routinely provided with the systems.
Three out of four systems fulfilled accuracy criteria in the hands of lay-users with the evaluated test strip lot showing 96% (system A), 100% (system B), and 98% (system C) of results within ± 15 mg/dl for BG concentrations < 100 mg/dl and ± 15% for BG concentrations ≥ 100 mg/dl. All systems fulfilled ISO 15197:2013 accuracy criteria in the hands of trained study personnel. One system (B) showed 100% of results in the hand of lay-users and study personnel, and similar accuracy results for this system were obtained in a previous user performance evaluation .
The other three systems showed slightly better accuracy results for study personnel measurements compared to lay-user measurements. Different studies showed that accuracy achieved by trained study personnel is often better than accuracy achieved by lay-persons [11, 12, 26, 27]; however, only few user performance evaluations were performed on the basis of procedures described in ISO 15197:2013 [25, 26, 28,29,30].
For all four systems, all measurements were within zones of the CEG and SEG that are associated with no or minimal risk, irrespective of the user group. ISO 15197:2013, clause 6.3, requires a CEG analysis for system accuracy evaluation. The CEG is developed for type 1 diabetes patients and is based on data collected in 1994 . The CEG is divided into eight zones indicating the estimated risk to the patient. The SEG also assesses the clinical risk but is based on much more recent survey data . The SEG is designed as an assessment tool of SMBG systems’ postmarket risk. SEG risk zones are color-coded and data points are assigned to continuous individual risk scores.
For both comparison measurement methods used in our study, compliance with traceability requirements was confirmed by the manufacturers. In this study, the comparison measurement method/system applied had no impact as to whether the data obtained comply with ISO 15197:2013 accuracy criteria or not. However, differences in accuracy data depending on the applied comparison method/system were found which is consistent with results of two previous ISO 15197-based studies performed at our institute [9, 32]. This influence might be reduced if manufacturers would adjust calibration of their reference methods .
This study design has several limitations. According to ISO 15197:2013, the user performance evaluation shall also include the evaluation of the instructions for use and the messages displayed on the meter by a questionnaire in order to assess whether the instructions and messages are adequate and easy to understand. Since not all instructions for use were available in German, subjects received a summarized instruction for use (in German) for each system with relevant information for the preparation and performance of BG measurements. Therefore, instructions for use and the messages displayed on the respective meters were not evaluated. In this study, only one lot was evaluated on the basis of ISO 15197:2013, clause 8. According to ISO 15197:2013, clause 6.3, system accuracy evaluation in the hands of study personnel shall be performed with three test strip lots, since lot-to-lot variations between multiple test strip lots used for the same system can markedly affect a system’s accuracy results [11, 34, 35].
Regarding the subjects’ measurement technique, not checking the expiry date of test strips before the measurement was the most frequent error. It might be that subjects assumed that they were only provided with non-expired test strips. Because test strips were used in this study before their expiry date, this error had no impact on measurement results. However, different studies showed that the use of deteriorated test strips can affect SMBG measurement results [36, 37].
For all four systems, an incorrect blood application, e.g., applying an insufficient amount of blood, pressing the fingertip onto the test strip, or applying the blood not onto the test field, was the most frequent error during the measurement procedure. Since incorrect blood application can potentially affect SMBG measurement results, adequate SMBG training and education should be an integral part of diabetes therapy [36, 38].
Accurate SMBG measurement results are crucial for adequate insulin dosing decisions by patients on insulin therapy. Modelling analyses showed that inaccurate BG measurements can lead to insulin dosing errors [1, 3, 4, 39, 40] which can adversely affect glycemic control and increase the risk of long-term complications in patients on intensive insulin therapy. In this study, negative median insulin dosing errors (too small insulin doses) were calculated for three systems (A, B, D) and positive median insulin dosing errors (too high insulin doses) were calculated for one system (C). Except for one system (D), median calculated insulin dosing errors were larger when SMBG measurements were performed by lay-persons. In addition, the four systems showed differences in the width of the error range (99% ranges between the 0.5th percentile and the 99.5th percentile).