FormalPara Key Summary Points

Several barriers to physical attendance for visual acuity examination, such as increased age, difficulty in transportation, mobility issues, or pandemic crisis, could be overcome with the development of accurate vision-testing applications.

We designed and examined the validity of a new smartphone-based visual acuity test based on the standard ETDRS principles: the K-VA test.

The K-VA application performed well when compared with the standard ETDRS chart and provides repeatable and reliable measurements of VA across a wide range of VA.

The results of the K-VA test are automatically recorded and can be transmitted to the attending physician for further evaluation.

Introduction

Visual acuity (VA) is the most frequently performed ophthalmic examination of the visual function. The accurate evaluation of VA provides quantification of changes in central vision caused by various ocular diseases. Hence, VA examination is considered the most common primary outcome measure of ophthalmic clinical trials [1].

Vision impairment (VI) has a great impact on the quality of life of affected individuals [2] and is considered a major public health issue [3]. According to the World Health Organization (WHO), 285 million people worldwide suffer from some grade of VI, the majority of whom have poor access to eye care services [4]. The definition of VI is usually in terms of reduced VA, highlighting the central role of this test in the evaluation of ophthalmological patients.

There are several VA charts in use with the Snellen chart since it was developed in the 1860s, being the most widely available and subsequently used vision chart worldwide [5]. However, several characteristics of this chart, including the nongeometric progression in the size of the displayed letters and the variable number of letters per line, limit its use for research purposes and the evaluation of patients with low vision (LV) [6, 7]. Recently, the gold standard for the accurate evaluation of VA has been considered to be the Early Treatment Diabetic Retinopathy Study (ETDRS) acuity chart that is widely used in ophthalmic research [8]. This type of chart is recommended since it improves the precision and reproducibility of VA measurement at the penalty of significantly longer test duration [8]. The examination with these charts requires the physical presence of the examinee, which is not always possible for elderly, mobility-impaired, or rural patients, or patients in quarantine due to coronavirus disease 2019 (COVID-19) pandemic [9].

In ophthalmology, a promising approach includes the use of smart devices such as smartphones and tablet computers to perform portable assessments, screening, and evaluation of the visual function to address accessibility issues [9]. Currently, there are hundreds of vision-testing applications available; however, only a few have been validated and compared against the standard ETDRS chart [10,11,12,13]. Most of the existing validated applications for VA testing, such as the Eye Chart pro, the Eye Hand Book, and the Odysight, lack a significant index of validity, that is, the test–retest analysis [14,15,16,17]. The Peek Acuity test is a successfully implemented application for VA screening in school children and low-income areas; however, the testing procedure requires a tester to translate the gestures of the examinee [18]. The Easee eye test is an interactive, web-based VA test with proved accuracy in various study groups; though, the tool has not been evaluated in older patients, who are less familiar with digital devices [19, 20]. Furthermore, the inability of all patients to navigate such technology themselves highlights the need for appropriate instructions and remote consultation by their attending physician [10,11,12,13].

Within this context, a valid, repeatable, and interactive smartphone-based VA test could serve as a beneficial tool for both patients and their physicians. This study aimed to describe the development and validation of an interactive, smartphone-based VA test for self-performance, following the design principles of the standard ETDRS chart.

Methods

Study Participants

A total of 171 Greek-speaking participants (n = 250 eyes) with VI due to various ocular diseases were recruited from our outpatient unit at Aristotle University of Thessaloniki, School of Medicine in Greece. Participants were randomly selected among individuals who were referred to our unit and attended the LIFE4LV research program for patients with VI. The trial is registered at clinicaltrials.gov (NCT05184036). Patients were previously diagnosed and referred with their medical record. Most of them had undergone cataract surgery or were free of significant lenticular changes, based on their medical record. Participants with Best Corrected Visual Acuity (BCVA) levels ranging from logMAR 0.0 to 1.0 were included. Informed written consent was obtained from all subjects after an explanation of the nature of the study. Exclusion criteria included severe cognitive impairment as assessed by an expert psychologist with the Mini-Mental State Examination [21] and severe systemic comorbidities (Parkinson’s disease, etc.). Illiterate individuals, patients faced linguistic issues, or those who had a non-Greek primary language were also excluded from the present study. However, the selected letters could also be potentially used in English (already available by the application) or other languages with the appropriate modification. All study procedures were approved by the Committee for Bioethics and Ethics, Medical Department, Aristotle University of Thessaloniki (code#1.60/21.11.2018) and adhered to the principles embodied in the Declaration of Helsinki Code of Ethics of the World Medical Association.

Digital Visual Acuity Test

The K-VA test was developed as a smartphone-based application and is available for devices running Android OS 5.0 or later. The application has been created with Android Studio V4.1 and the software provided various options on the smartphone screen. The application has been demonstrated with a Samsung A30S smartphone (display: Super AMOLED, size 6.4 inches, resolution 720 × 1560 pixels, ratio 19.5:9, density ~ 268 ppi; GPU: Mali-G71 MP2) running Android OS 9.0 with maximal brightness of 489 cd/m2 (personal communication with Samsung technical support), and the VA testing was conducted using full brightness (100%). The device was switched on at least 5 min before each experimental session to allow its output to stabilize.

The K-VA test follows the standard ETDRS chart design [7, 22]. Monocular VA can be measured, with the appropriate correction, at 1 m and 40 cm, recorded, and saved following the General Data Protection Regulation (GDPR), and the physician has access to the results for further evaluation (Fig. 1).

Fig. 1
figure 1

Overview of the K-VA testing protocol, demonstrating the successive steps of testing

Full size image

The device should have a stable internet connection, and participants were instructed to read aloud letter by letter and were encouraged to guess the letter if not sure, as per the ETDRS protocol [1]. Each line contains five letters and the letter that should be spelled is pointed to guide the patient (Fig. 1). The letter contrast of the neighboring letters to the letter that should be pronounced was slightly lower to guide the user and to avoid losing their focus (Fig. 1). According to Chung and Mansfield, altering the contrast polarity of neighboring letters does not reduce the crowding effect since local uncrowding of individual letters seems to be counteracted by the grouping of letters [23,24,25]. Once the letter is read, either wrong or right, patients feel a vibration (40 cm examination) of the phone to better understand that they should read the next letter in the row. A speech recognizer records if the letter was recognized correctly or not. The crowding effect of the standard ETDRS chart was achieved by using a crowding bar at the right and left sides of each row of letters (Fig. 1).

As mentioned before, appropriate instructions written in a font corresponding to VA of 1.0 logMAR are shown on the landing page of the application, informing patients on how to perform the test correctly. Participants held the smartphone at 40 cm distance or supported the device with the back of a book at 1 m distance at a perpendicular angle, at the line of sight, to avoid reflection of light. In the case of an interruption, the application asks the users if they wish to continue or to repeat the examination. There is also an opportunity for font selection in cases of patients with visual acuity below 1.0 logMAR for the examination at 40 cm.

A clear statement is also shown before testing to inform users that this application is a nonclinical setting and should not replace the VA examination by an eye care professional. Instead, this application could be used as a useful tool by physicians to perform an immediate and accessible, remote examination.

For this test, a single-letter scoring methodology [26] was utilized and the score was converted to the logarithm of the minimal angle of resolution (logMAR) visual acuity. The application contains a set of letters that are commonly used among most European alphabets and can also be recognized by patients who are not familiar with some letters of the Roman alphabet [19, 27,28,29,30]. These are: A, E, H, Z, N, O, P, T, Y, X. The new letters were constructed on Snellen’s 5 × 5 grid and are compatible with the specification of the Sloan letters [31]. The Sloan letters originate from the Roman alphabet and cannot be used with individuals who are unfamiliar with this alphabet [19, 27, 28]. The new set of letters used in the K-VA application was previously evaluated for similarity with the standard ETDRS chart in 55 individuals; 15 controls and 40 with visual impairment and low vision. No obvious bias was detected between the proposed set of letters and the standard ETDRS chart [32].

A staircase algorithm was used to enhance testing efficiency as follows: if three out of five letters of each sentence are correctly identified, the next sentence representing the next size of letters is automatically presented. This sequence continues until the smallest optotype line is recognized or the logMAR 0.0 is reached. If less than three letters are correctly identified, then the test ends and the result is calculated according to the ETDRS algorithm [7, 22]. Results are displayed in logMAR form after testing and can be saved and shared via a secured backend system with the physician. The K-VA test was designed to be used for VA measurements and the monitoring of these measurements by the attending physician remotely. Within this frame, a single user would repeat the examination with the K-VA test in predetermined intervals, according to the physician’s instructions. Thus, letters are randomly presented during each time of examination to minimize the risk of learning and memory effects.

Testing Protocol

All study participants underwent near and distance VA testing twice using the ETDRS charts and the K-VA test for both distances on the same day, using a standardized protocol in a controlled examination environment. The K-VA test was performed by the participants themselves. Participants were supervised by a trained healthcare provider during the K-VA testing to keep the testing distance and to occlude correctly the eye with their palm, avoiding the application of pressure on the eye. An expert ophthalmologist examined participants with their best correction in the ETDRS charts, without knowing the score of the K-VA, which was automatically calculated by the application. Both eyes of the participants were examined.

For conventional VA testing, an externally illuminated 4-m ETDRS VA chart (CAT. No. 2111, LOT. 113529, Precision Vision Inc.) was used for the examination of distance VA at 4 m, and an ETDRS near VA card (CAT. No. 2106B, LOT110326, Precision Vision Inc.) was used at 40 cm distance for the evaluation of near VA. The testing distance for all tests was precisely measured with a measuring string before each examination and a trained healthcare provider observed that participants remained at the appropriate distance throughout the examination.

VA was examined using the K-VA application at 1 m and 40 cm. The participants were asked to follow the instructions presented on the smartphone’s screen and perform the test themselves, monocularly, occluding the other eye with the palm of their hand in the following sequence: right eye first and then left eye (Fig. 2). It takes approximately 2.5 min to perform the test for each eye (time also depends on the visual status of the patient). The K-VA test was performed in office lighting conditions and it is recommended to avoid excessive ambient lighting (sunlight) during the performance of the test [18]. The results were automatically recorded by the application.

Fig. 2
figure 2

Participant testing near visual acuity with the K-VA application with the supervision of a trained healthcare provider

Full size image

For ETDRS testing, participants were asked to read aloud the letters until the smallest row of letters in which they could identify less than three out of five letters, and results were recorded by the ophthalmologist. All participants were retested on the same day.

In total, participants conducted eight VA examinations: four with the K-VA test, at 1 m and 40 cm (test and retest for each distance), and four with the standard ETDRS at 4 m and 40 cm (test and retest for each distance). Whether ETDRS or the K-VA was tested first was decided at random, using an online randomization program (www.randomizer.org).

Questionnaire Feedback

A total of 97 participants, all Android users, were asked to answer a short questionnaire (Supplementary Material) regarding the ease of use of the test and their familiarity with such technologies (smartphones, tablet computers, etc.). Participants were asked to report the kind of mobile phone they used before answering the feedback questionnaire; those who were non-Android users were excluded from the feedback section to avoid possible effects of the unfamiliarity with the Android operating system in the feedback [33, 34]. The non-responders were users of different devices; in specific, 64 of them were iOS users and 10 were feature phone users.

Statistical Analysis

All VA measurements were converted to logMAR units [26]. Bland–Altman plots were employed to assess the pairwise agreement between the ETDRS charts and the K-VA test at 1 m and 40 cm examination distance. For most individuals, measurements for both eyes were taken and used in the analysis. The SimplyAgree [35] package in R, version 4.1.2 (R Foundation of Statistical Computing, Vienna, Austria) was used for estimating reliability and agreement between the different measurements while accounting for dependencies within individuals having both eyes included in the analysis.

Regarding the analysis of the feedback questionnaire, the Shapiro–Wilk test was used for the normality assessment of age. Continuous variables were described using median (IQR) while categorical variables were described using frequencies and relative frequencies. The chi-squared test was used to investigate for any dependencies between categorical variables. The Kruskal–Wallis H test was used to explore the differences between the groups of continuous variables. Bonferroni correction was used to adjust for pairwise comparisons. p-Values less than 0.05 were considered statistically significant. SPSS version 27.0 (IBM Corp., Armonk, NY) was used for the statistical analysis of the feedback questionnaire.

Results

In total, 171 patients (n = 250 eyes) were enrolled in the study. The mean age was 67.57 years [standard deviation (SD) 15.51 years, range 18–89 years] and 52% (n = 89) of the participants were female. A total of 53 subjects (31%) had Age-related Macular Degeneration (AMD), 40 (23.4%) had Diabetic Retinopathy (DR), 18 (10.5%) had glaucoma, 10 (5.8%) had Retinitis Pigmentosa (RP), 10 (5.8%) had treated retinal detachment, 9 (5.2%) had retinal vein occlusion, 6 (3.5%) had Stargardt disease, 6 (3.5%) had optic nerve atrophy, 5 (2.9%) had macular hole, 4 (2.3%) had epiretinal membrane, 3 (1.7%) had myopic maculopathy, 2 (1.2%) had cystoid macular edema, 2 (1.2%) had amblyopia, 1 (0.58%) had choroideremia, 1 (0.58%) had Fuchs’ dystrophy, and 1 (0.58%) had Birdshot Chorioretinopathy. The demographic characteristics of participants are presented in Table 1.

Table 1 Demographic data of participants
Full size table

The mean differences (95% confidence intervals) for the pairwise comparisons between ETDRS charts and K-VA test at both distances are shown in Fig. 3. The median VA for all measurements (ETDRS chart and K-VA test at both distances) are presented in Supplementary Table S1. The mean values for both eyes were also calculated and are presented in the same table.

Fig. 3
figure 3

Mean differences (95% confidence intervals) of K-VA and ETDRS chart at 40 cm denoted as “o” and K-VA test at 1 m and ETDRS at 4 m denoted as “Δ”

Full size image

The pairwise comparisons between ETDRS charts and K-VA application in measuring distance and near VA across participants are presented in Table 2. Bland–Altman summary statistics of mean bias and 95% Limits of Agreement (LoA) were calculated and are presented in the same table. Results suggest no significant bias between the methods studied. The mean difference between the K-VA test and the ETDRS charts was 0.006 logMAR (95% LoA −0.129 to 0.117) and −0.007 (95% LoA −0.105 to 0.090) for measurement at 1 m and 40 cm, respectively. Bland–Altman scatter plots were constructed to represent the graphical spread of each of the pairs of measurement results. No evidence of systematic association was found between the level of agreement and the underlying acuity (Fig. 4).

Table 2 Mean difference and 95% Limits of Agreement for intertest and test–retest agreement in logMAR units
Full size table
Fig. 4
figure 4

Bland–Altman plots of VA measurements by the standard ETDRS charts and K-VA application at 1 m (diagram A) and 40 cm (diagram B) distance. The grey line represents the mean difference and the orange line represents the upper and lower 95% limits of agreement

Full size image

The K-VA test–retest reliability was found to be high (Table 2). Bland–Altman summary statistics of mean bias and 95% Limits of Agreement were calculated and are shown in Supplementary Figure S1.

Feedback Questionnaire

Overall, 97 participants, all Android users, answered the questionnaire; 52 were female (53.6%) with a mean age of 67.42 ± 12.97 years. In total, 71 (73.2%) stated that the test was easy to very easy to use, and 40 (41.3%) stated having high to very high familiarity with new technologies (smartphones, tablet computers, etc.). There was a significant association between ease of use and familiarity with smartphones (p < 0.001). Age was found to significantly correlate with ease of use (p = 0.007). These correlations are illustrated in Fig. 5.

Fig. 5
figure 5

Participants’ feedback (Android users, n = 97) on the K-VA application using a short Likert scale questionnaire. The diagrams indicate the answer distribution for the ease of use of the app (diagram A) and their familiarity with the use of smartphones (diagram B), respectively

Full size image

Discussion

This study aimed to validate the performance of a new, smartphone-based VA test, the K-VA application, which was designed for the remote monitoring of patients by their physician. Overall, the K-VA test performed well and was proven to be repeatable and well-accepted. The mean difference between the K-VA test and the ETDRS charts was 0.006 logMAR (95% LoA −0.129 to 0.117) and −0.007 (95% LoA −0.105 to 0.090) for measurement at 1 m and 40 cm, respectively, and were within the accepted deviation of ± 0.15 logMAR [15, 16, 36]. The distribution of differences, expressed as 95% LoA, were also within the previously reported limits and indicate acceptable variability of the VA measurements with the K-VA test [16]. When comparing the repeatability of the K-VA test with that of the ETDRS charts, the LoA for the K-VA test–retest reliability were very close to the previously published estimates of test–retest variability (TRV) for the standard ETDRS charts, which ranges between 0.07 and 0.18 logMAR [28, 37,38,39].

The use of smartphones and other wireless technologies is rapidly increasing in clinical practice [40]. The role of mobile applications in delivering and measuring health-related outcomes was especially illustrated due to the coronavirus disease 2019 (COVID-19) pandemic [11, 41, 42]. Currently, there are many vision-testing applications available; however, only a minority of them have undergone rigorous validation, especially against the standard ETDRS chart [43, 44]. Even fewer have undergone test–retest analysis, a fundamental index of validity [15]. Unfortunately, healthcare applications with questionable validity could be misleading both for patients and healthcare providers [45].

In a recent systematic review evaluating the performance of other previously validated VA applications to reference standards, the mean differences ranged from −0.08 to 0.10 logMAR while the distribution of the differences, as expressed by the 95% LoA, ranged from ± 0.08 logMAR (lowest variability) to ± 0.47 logMAR (highest variability) from the mean difference [16]. Furthermore, test–retest variability of logMAR charts ranged from ± 0.07 to ± 0.18 logMAR from the mean difference [16]. Similarly, the K-VA test performed within these ranges, compared with the standard ETDRS charts.

Two important parameters must be taken into account when interpreting results of studies comparing VA assessments. Firstly, even when VA is stable, there is always variation of the repeated VA measurements of the same individual. This type of variation also depends on the visual status of the examinee; use of appropriate correction and better VA lead to more accurate results. For instance, the accuracy of the Easee VA test differed for various VA ranges and study populations (corrected VA versus uncorrected VA) [19]. Reports on test–retest variabilities of commonly used charts show a wide range of 95% LoA, with logMAR charts providing more reliable results than Snellen charts (95% LoA ranging from ± 0.18 to ± 0.33 logMAR) [46]. The Snellen chart illustrates only one or two letters in the poor vision lines and a line assignment method is used for scoring; this means that losing a letter in these lines makes a great impact in the score. However, time and convenience seem to outweigh the accuracy in clinical settings.

Secondly, another source of variation in VA measurements is the type of charts used for comparison [18, 47]. This phenomenon contributes partly to the obtained differences in outcomes between VA applications and the reference VA chart. This was obvious in the case of the Peek Acuity test when different reference charts were used (Snellen versus ETDRS) [18]. Siderov et al. proposed that differences of less than 0.15 logMAR do not indicate an actual change in VA, since they reflect the variability of VA measurements among different settings (different charts, different examiners, etc.) [36]. The high accuracy of the K-VA when compared with the standard ETDRS chart could be attributed to the design of the K-VA test that was based on the standard ETDRS chart’s principles. Specifically, the optotype presented by the K-VA application is very similar to the conventional ETDRS chart. Thus, a conversion effect was not detected in our study.

The use of digital tools for VA assessments may be limited due to the lack of appropriate instructions and remote consultation by the attending physician. The K-VA offers some advantages for self-performance: clear instructions are present prior to examination, the test interacts with the examinee, providing vocal recognition of the letter and a feeling of vibration (40 cm examination) to inform the patients to move to the next letter. The results are automatically recorded in logMAR (Fig. 1) units, minimizing possible technician-based errors. The data are saved and can be securely sent to the attending physician for further consultation. Importantly, since data from every examination are saved, they could be compared with those of a following examination by the physician. Using the K-VA test, the physician may detect changes remotely, indicative of the need for further evaluation of the patient. This could be of high clinical significance for patients who require frequent monitoring of their VA and face accessibility issues.

It is important to note that discrepancies when comparing examination in clinical setting with that in an unsupervised environment may be detected, especially if the test distance has not been accurately measured. In the case of the K-VA test, patients are strongly instructed to measure the testing distance and to maintain the same distance when performing the test (placing a distinctive mark). Clear instructions are provided prior to the test to ensure the consistency of testing conditions. In addition, a large print manual with more detailed instructions for the correct performance of the test is provided with the application (the manual can be sent by e-mail or downloaded with the application). The test distance is fundamental to the accuracy of the results, along with the refractive correction. The fact that the results can be evaluated directly by the attending physician and appropriate consultation can be provided accordingly aims to reduce the likelihood of inaccurate results. The accuracy of another well-established VA test, the Easee VA test, was recently evaluated in unsupervised home environments. The test provided reliable measures of VA in the majority of patients and the study highlighted important aspects on how to ensure the accuracy of the results in home settings [48]. Importantly, in suspicion of invalid outcomes, patients should be instructed to retest under optimal testing conditions following the instructions of the application and the attending physician.

Our study has certain limitations that warrant consideration. By convention, we chose to measure VA at a 1 m distance, a method used before for distance VA examination using smart devices [25]. In some patients, the 4 m and 1 m scores are not directly interchangeable; cataract, amblyopia, and inappropriate refractive correction in patients with high myopia could cause a disagreement between 4 m and 1 m VA scores [49,50,51]. However, a high degree of correlation between these VA measurements has been previously observed [52, 53]. VA less than 1.0 logMAR can only be measured at 40 cm and the examination is limited to 1.3 logMAR at this distance. Moreover, the accuracy of the VA assessment is often dependent on the competency level of the examiner, leading to inter-observer variability [1, 54]. In our study, a single researcher conducted all the measurements with the traditional ETDRS charts [54]. Furthermore, the study was conducted using only one type of smartphone and the application is available for devices running Android OS 5.0 or later. Thus, in the feedback section we included only patients who were Android users to avoid possible effects of unfamiliarity with the Android operating system in the feedback, since users of different operating systems have been previously found to differ in the usage of mobile health applications [55]. We are currently developing the application for the iOS operating system. According to recent data, Android was the leading mobile operating system worldwide with close to 71.8% of users, while iOS accounted for around 27.6% of the mobile operating system market [56, 57]. In our study, these percentages were found to be different; this may reflect differences in the sociodemographic composition of our sample [33, 34].

We should also acknowledge that the testing distance should be kept constant for accurate results. Moreover, previous studies suggest the susceptibility of smartphone-based tests to glare [58]. Keeping the device at a perpendicular angle at the line of sight to avoid reflection of light decreases the effect. Furthermore, the K-VA test was validated in a Greek-speaking population, which limits the generalizability of the results; however, the test utilizes a set of letters easily adaptable to other European languages. The K-VA application was assessed for self-performance at clinical settings. Thus, self-performance at home settings remains to be evaluated.

Conclusions

The study demonstrated that the smartphone-based K-VA test provides reliable and repeatable measurements of VA. This test seems to be a promising tool for quick and easily accessible measurements of VA. Patients who have difficulty accessing specialized healthcare services may benefit from applications such as the K-VA test, and the attending ophthalmologist has an additional tool for the follow-up of patients with accessibility issues. Future research should investigate the accuracy of the tool when performed independently by patients at home.