Accuracy of freely available online GFR calculators using the CKD-EPI equation

Purpose Estimated glomerular filtration rate (eGFR) as calculated by the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation is used for detection of chronic kidney disease and drug dose adjustment. The purpose of the present study was to investigate the accuracy of freely available eGFR online calculators. Methods All identified CKD-EPI online calculators were run with five reference cases differing in age, sex, serum creatinine, and ethnicity. Conversion from eGFRindexed (unit ml/min per 1.73 m2) to eGFRnon-indexed (unit ml/min) and creatinine unit from milligramme/decilitre to micromole/litre was checked, if available. Results Only 36 of 47 calculators (76.6%) produced accurate eGFR results for all reference cases. Eight of 47 (17.0%) calculators were considered as faulty because of errors relating to ethnicity (4 calculators), to conversion of the eGFR unit (2 calculators), to erroneous eGFR values without obvious explanation (2 calculators), to conversion of the creatinine unit (1 calculator), and to an error in the eGFR unit displayed (1 calculator). Overall, 28 errors were found (range 59 to 147% of the correct eGFR value), the majority concerning calculation of eGFRindexed and the conversion to eGFRnon-indexed. Only 7 of 47 (14.9%) calculators offered conversion of the eGFR unit. Conclusions Erroneous calculations that might lead to inappropriate clinical decision-making were found in 8 of 47 calculators. Thus, online calculators should be evaluated more thoroughly after implementation. Conversion of eGFR units that might be needed for drug dose adjustments should be implemented more often. Electronic supplementary material The online version of this article (10.1007/s00228-020-02932-x) contains supplementary material, which is available to authorized users.


Introduction
In clinical routine, glomerular filtration rate (GFR) is most commonly estimated with the creatinine-based Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation, which is used for detection and staging of chronic kidney disease (CKD) [1,2]. Drug dose adjustments in patients with renal impairment are traditionally based on creatinine clearance, using either the endogenous creatinine clearance or, more easily, Cockcroft and Gault's equation [3,4]. In this regard, creatinine clearance estimates should not be generally replaced by GFR estimates [5]. In elderly patients, for example, higher estimates by GFR equations could lead to inappropriately high doses [6][7][8] and an increased risk of adverse events, e.g. bleeding risk in case of anticoagulants [9,10]. Nevertheless, the GFR is already used for dose adjustments (e.g. carboplatin) and for contraindications (e.g. metformin) of some drugs and GFR equations will presumably be used increasingly also for drug dose adjustment in clinical routine and for approval of new drugs [11].
The creatinine-based CKD-EPI equation requires age, sex, ethnicity, and a standardised measurement of serum creatinine [2]. Standardised creatinine is determined by using a creatinine assay with calibration traceable to an isotope dilution Electronic supplementary material The online version of this article (https://doi.org/10.1007/s00228-020-02932-x) contains supplementary material, which is available to authorized users. mass spectrometry (IDMS) reference measurement procedure [12], which was developed to enable more accurate renal function estimates and to reduce differences between laboratories. Development of the CKD-EPI equations was based largely on patients with Caucasian or African-American ethnicity and has been evaluated in numerous studies in different patient populations [11]. Of note, the CKD-EPI equation provides a GFR estimate with the unit millilitre/minute per 1.73 m 2 (eGFR indexed ), i.e. an estimate normalised to a body surface area (BSA) of 1.73 m 2 , which appears appropriate for detection and staging of CKD [1]. In case of drug dose adjustment, a non-normalised individual estimate with units of millilitre/minute is desired, especially if a patient's BSA differs significantly from 1.73 m 2 . Thus, in such cases, the estimate should be individualised according to a patient's BSA (eGFR non-indexed = eGFR indexed / 1.73 · BSA) [1,11,[13][14][15][16][17]. Interestingly, in a recent analysis, eGFR non-indexed calculated with the CKD-EPI equation performed best at predicting dose requirements for carboplatin, closely followed by eGFR nonindexed calculated with the MDRD equation [18]. In obese patients, eGFR non-indexed calculated with CKD-EPI or MDRD equation was more accurate than eGFR indexed compared with measured 51 Cr-EDTA or 99m Tc-DTPA GFR [19,20].
The creatinine-based CKD-EPI equation is often implemented in laboratory systems but is also available in online calculators via the Internet. Such freely available online calculators might be used by medical or pharmaceutical professionals when a calculated eGFR is not immediately available from the laboratory or a calculation of eGFR non-indexed is required. The accuracy of such online calculators is unknown.
The aim of the present study was to systematically evaluate the accuracy of freely available online calculators using the creatinine-based CKD-EPI equation.

Materials and methods
Online calculators using the CKD-EPI equation were identified via Google search using the keywords 'CKD-EPI' and 'CKD-EPI calculator'. Five reference cases were developed, using parameter values that should cover different aspects of the CKD-EPI equation, because actual calculation can be implemented in online calculators as several different equations (where the applicable equation is selected based on sex, ethnicity, and creatinine category) or as a single equation [2]. These hypothetical cases included two Caucasian males with creatinine values below or above 0.9 mg/dl, two Caucasian females with creatinine values below or above 0.7 mg/dl, and one African-American male (Supplementary Table S1). A conversion factor of 88.4 was used to convert creatinine values with a unit of milligramme/decilitre to micromole/litre. BSA was calculated using Mosteller's equation [21,22] and Du Bois' equation [23]. In order to derive the correct eGFR values for the reference cases, the equations (Supplementary  Table S2) were entered into two separate Microsoft Excel sheets, which was done independently by two of the authors (SS, DC), and the results compared.
All reference cases were run with each identified online calculator. Deviations from the reference values where an obvious explanation could be identified (e.g. wrong application of BSA, leading to an inverse change in eGFR non-indexed ) were counted as an error. Deviations without obvious explanation were counted as an error only when the difference to the reference value was ≥ 1 ml/min per 1.73 m 2 . Otherwise, rounding inaccuracies were assumed. If calculators allowed different creatinine units (milligramme/decilitre or micromole/litre), one case was run with both units (in case of an error, the other cases were run with both units also), counting every deviation as an error. Cases where inadequate clinical application would not be expected (e.g. if calculation was not possible because of an upper age limit) were not counted as an error. Analyses were done by a clinical pharmacist (SS) and a clinical pharmacologist and nephrologist (DC).
We checked whether creatinine measurement by a standardised assay and information on the ethnic background of a patient was required and, if not, if the user is informed on these limitations. In addition, information on the online calculators was obtained from the respective site (country of origin, language, type of supplier, and provision of references).
Only 36 of 47 (76.6%) calculators provided accurate estimates for all reference cases (i.e. no errors were detected). After exclusion of two calculators where an arbitrary 70year age limit prohibited calculation for two cases and one calculator, which did not perform calculations for African-American ethnicity, but explicitly stated that this was the case, 8 (17.0%) calculators were considered as faulty.
Errors concerning ethnicity were found with four calculators. In two cases, no calculation for African-Americans was provided and no information on this fact was given. In one case, the same value was calculated independently from the selected ethnicity. In another case, the calculator provided a wrong eGFR value while an obvious reason could not be identified. For these errors, the average absolute difference from the correct value was 10.9 (range 4.3 to 13.6) ml/min per 1.73 m 2 with the reference cases. The average eGFR was 91% (range 86 to 104) of the correct value.
Errors concerning individualisation from millilitre/minute per 1.73 m 2 to millilitre/minute were found with two calculators. Obviously, calculations were done erroneously as eGFR indexed / BSA · 1.73, thus leading to a change in the opposite direction. As only 7 of 47 (14.9%) calculators offered this conversion, 2 of 7 (28.6%) were found to be faulty. For these errors, the average absolute difference was 18.0 (range 0.2 to 49.5) ml/min with the reference cases but could be much larger for patients with very low or very high BSA. The average eGFR was 101% (range 59 to 147) of the correct value.
One calculator produced an eGFR value for one reference case that was consistent with using the CKD-EPI equation for creatinine values > 0.9 mg/dl where application of the equation for values ≤ 0.9 mg/dl would have been correct. The absolute difference was 9.7 ml/min per 1.73 m 2 , and the eGFR was 110%.
An error in conversion of creatinine units between milligramme/decilitre and micromole/litre was found with one calculator. For these errors, the average absolute difference was 2.6 (range 1.4 to 5.2) ml/min per 1.73 m 2 , and the eGFR was 104% (range 101 to 105).
An error in the displayed unit of eGFR was found with one calculator, where millilitre/minute was shown for eGFR indexed and for eGFR non-indexed . For these errors, the average absolute difference was 12.0 (range 0.2 to 34.5) ml/min per 1.73 m 2 , and the eGFR was 98% (range 71 and 121) with the reference cases (assuming that eGFR indexed is used where eGFR nonindexed is actually required).
Two calculators produced erroneous eGFR values in one or more reference cases, where no obvious explanation could be identified. For these errors, the average absolute difference was 1.8 (range 1.8 to 2.0) ml/min per 1.73 m 2 , and the eGFR was 102% (range 98 and 102).
In total, there were 28 errors using the 47 calculators and 5 reference cases (1-11 errors per faulty calculator): 8 errors (7 calculators) concerning the calculation of eGFR indexed of which 4 did not have adaption for African-Americans and 4 where eGFR indexed -estimates differed more than 1 ml/min per 1.73 m 2 , 10 errors (2 calculators) concerning the conversion to eGFR non-indexed , 5 errors (1 calculator) concerning the Data are quoted as n (%) a Includes medical associations and patient associations conversion of creatinine units, and 5 errors (1 calculator) where the units of eGFR indexed were misstated. The extent of these errors ranged between 59 and 147% of the correct value (Fig. 1). When the limit for counting a deviation from the correct value as an error was set at ≥ 2, 5, or 10 ml/min per 1.73 m 2 (or ml/min), 19, 16, or 11 errors were found, and 7, 6, or 4 calculators were considered as faulty.
Furthermore, 45 (95.7%) of the Internet sites questioned or explicitly referred to ethnicity of a patient, 29 (61.7%) provided references on CKD-EPI, 26 (55.3%) allowed more than one unit of creatinine, 9 (19.1%) explicitly required standardised creatinine values or stated that such values were required, and 5 (10.6%) allowed for calculation of BSA based on height and weight.

Discussion
Estimation of GFR with the CKD-EPI equation is commonly used for detection and staging of CKD and is recommended for drug dose adjustment of some drugs (after individualisation to eGFR non-indexed , where required). After discovering an online calculator with erroneous conversion of units from millilitre/ minute per 1.73 m 2 to millilitre/minute (the index case), we performed a systematic analysis of freely available online calculators using the creatinine-based CKD-EPI equation. From online CKD-EPI calculators, we expect accurate calculations and would like to see a feature to estimate BSA and to calculate the individual eGFR (eGFR non-indexed ). Since accurate calculations require at least age, sex, ethnicity, and a standardised measurement of serum creatinine, the online calculators should ask for these variables.
Surprisingly, we found that only 36 of 47 (77%) freely accessible online calculators were able to calculate accurate values for our test cases. Eight of 47 (17%) calculators were considered as faulty, because of one or more errors that might lead to inappropriate clinical decision-making, as e.g. in one of the reference cases with erroneous conversion of eGFR indexed of 42.7 ml/min/1.73 m 2 to eGFR non-indexed of 51.6 ml/min instead of 35.2 ml/min, which could lead to quite different clinical decisions in therapy with metformin. In case of serious adverse events, potential legal implications are easily envisioned, when accurate calculations contraindicate for the use of a drug whereas the erroneous calculation does not. Frequency and consequences of eGFR online calculator use by patients are largely unknown. However, education and self-monitoring of CKD patients are increasingly advocated [24,25] and information on eGFR is part of patient education material [26]. In a retrospective study, 87% of patients using an electronic health record patient portal reviewed their laboratory results [27]. In a prospective study, providing a specific website to predialysis CKD patients, the GFR calculator was most commonly visited [28]. Thus, increasing eGFR online calculator use by patients can be expected, potentially confusing patients, when erroneous calculators are used. Identified errors included wrong calculations of eGFR for patients with African-American ethnicity, wrong calculation in a case with relatively low creatinine, individualisation of the GFR unit from millilitre/minute per 1.73 m 2 to millilitre/minute, conversion of the creatinine unit from milligramme/decilitre to micromole/litre, and further errors without obvious explanations.
The extent of the errors ranged from 59 to 147%, which could lead to erroneous staging of CKD and inadequate drug dosing. Even small discrepancies could affect clinical decision-making, if the value is close to the lower limit of a given range (e.g. in case of a drug that is contraindicated in patients with an eGFR below 30 ml/min). Obviously, the extent of the errors identified in the present study depends on the reference cases and could be larger, especially in patients with low or high BMI.
Only 7 of 47 (14.9%) calculators offered the possibility to calculate eGFR non-indexed , and in 2 of these, this calculation was incorrect. Calculation eGFR non-indexed may be required for drug dose adjustments for drugs with a narrow therapeutic range (e.g. carboplatin) and in obese patients, where application of eGFR indexed can lead to an erroneously low estimate of renal function and unjustified drug dose adjustment [11,16,17,19,20]. Two online calculators did not allow calculations for two cases because of age restrictions (> 70 years). Although CKD-EPI might be less accurate in older patients, there appears to be no general recommendation on an upper age limit. Only 9 of 47 (19.1%) of the calculators explicitly asked for creatinine values as measured by a standardised assay. The CKD-EPI equation requires standardised creatinine values, which were used in developing the equation. Non-standardised creatinine values can lead to erroneously low eGFR values [29,30]. Standardised creatinine can be determined by an enzymatic or alkaline picrate (Jaffé) method with calibration traceable to an IDMS reference [12,31]. Newer studies recommend the enzymatic method because of an even better performance than the IDMS-traceable Jaffé  [32,33]. Thus, it might not only be relevant if a standardised creatinine value was used, the method could also be important.
Of note, most providers of online calculators state that their calculators are not to be used for clinical application, are to be used for educational purposes only, or similar. However, such calculators may be used by medical or pharmaceutical professionals and by patients assuming that online calculators are properly evaluated after implementation. The Internet should always be used with caution as a source on information in medicine. However, we expect it will increasingly be used, especially by clinicians with fewer years of experience [34,35]. To our best knowledge, there are no published reference cases for evaluation of CKD-EPI calculators. Two online calculators from Germany showed the same error in conversion of eGFR units from millilitre/minute per 1.73 m 2 to millilitre/ minute. Thus, it is tempting to speculate that the algorithm from one of these calculators might have been used when programming the other. Thus, the use of reference cases should always be part of software validation.
Our study has some limitations. First, we considered only online calculators that are freely available in the Internet. Thus, our results cannot be generalised to smartphone apps or to calculations implemented in laboratory information systems. Second, background information on a calculator was obtained from the Internet site. In some cases, information, especially on the actual type of provider of the online calculator (e.g. private person or small company), were not entirely clear. However, we refrained from requesting further information from the providers and used what would be available to the typical user.
In conclusion, we found errors in one out of six (17%) freely available CKD-EPI online calculators that might lead to inappropriate clinical decisions if used without critically questioning the results by experienced medical professionals. We strongly recommend that online eGFR calculators (and possibly other calculators also) should be evaluated more thoroughly after implementation and provide reference cases that could be used for creatinine-based CKD-EPI equations. Conversion of normalised eGFR (eGFR indexed ) to an individual eGFR (eGFR non-indexed ), which may be needed for drug therapy, should be more often provided.