Participants
Study participation was offered to people who were visiting the hospital for out-patient visits, as well as to their accompanying friends and family members. Each participant was noted to have one or multiple eye diseases, or none.
Study Design
In order to assess the validity of the medical modules on OdySight, we conducted an open-label, single-arm, prospective, single-site study at the Quinze-Vingts National Ophthalmology Hospital, in Paris, France (ClinicalTrials.gov Identifier: NCT03457441). The research followed the tenets of the Declaration of Helsinki of 1964, as revised in 2013, and approval of the experimental protocol was obtained from an ethics committee (Comité de Protection des Personnes Ile-de-France VII, France, n° 18-005). Informed consent was obtained from each participant prior to any examination.
Participants in the study met the following key inclusion criteria: age ≥ 18 years; best corrected visual acuity (BCVA) between 0.0 and 1.0 logMAR; ability to recognize letters of the alphabet and to read French; ability to correctly distinguish body laterality; and affiliated with or beneficiary of the French healthcare system. Participants were excluded if: they presented with a pathology that was considered by the investigator to be capable of affecting the quality of the main evaluation criteria (e.g., mental illness, Parkinson disease); they were pregnant or breastfeeding; or they were not considered by the investigator or designee to correctly use the OdySight modules after the training session. To ensure a wide range of visual acuity, each participant’s eye was assigned to one of the three cohorts according to the level of near visual acuity, measured at a distance of 40 cm from the Sloan ETDRS letter chart (40-cm ETDRS): cohort 1, with near visual acuity of between + 1.0 and + 0.7 logMAR; cohort 2, with near visual acuity of between + 0.6 and + 0.3 logMAR; or cohort 3, with near visual acuity of between + 0.2 and 0.00 logMAR.
For the purpose of the trial, a scaled-down version of the OdySight app (in which the puzzle and the technical unit checking ambient light and distance to screen were removed) had been installed on two iPhone 6 smartphones. The OdySight medical tests and their algorithms had not been modified. Again for the purpose of the trial, the examiner was responsible for the navigation within the app, and the participants were asked only to perform the tests.
Sample Size
The sample size needed, driven by the number of eyes for the primary endpoint (to assess the agreement between a smartphone-based evaluation of near visual acuity with OdySight and a standardized method [Sloan ETDRS near vision letter chart]) was defined by performing a careful review of the relevant literature. The limits of agreement (LoA) technique was retained to analyze the results and to calibrate the sample size of this study (see Conduct of the Study, below).
The lack of agreement was estimated by the mean difference of the two measurements (d) and the standard deviation of the differences (s). Providing differences within d ± 1.96 s would not have been clinically important. Taking into account that these LoA are only estimates of the values which apply to the whole population, we viewed the sample size estimation as an attempt to obtain sufficient precision on these LoA. It can be shown that the standard error of d is root(s2/n), where n is the sample size and the standard error of d − 1.96 s and d + 1.96 s is about root(3 s2/n) [8, 9]. Bland and Altman recommended a minimum of 100 statistical units for a good sample size because this number leads to a 95% confidence interval (CI) of approximately ± 0.34 s which appears to be a reasonable and accurate estimate. To ensure that 100 studied eyes were available in the statistical reporting of agreement between the methods of measurements, and to take into account a possible lack of measurements for any reason, the study was planned to include between 60 and 120 patients as a whole.
Conduct of the study
During screening, inclusion criteria were checked, optical coherence tomography (OCT) was performed, and the participant’s BCVA was determined so that each eye was assigned to the corresponding cohort. The three medical modules were then presented to the participant. Because it is known that a learning phenomenon can occur, participants were asked to perform practice tests following instructions given by the examiner. At the end of this training session, the examiner estimated whether or not the participant was sufficiently capable of using the app. Eyes from participants who fulfilled all of the criteria were included, and a participant could be included for one or both of his/her eyes.
During the evaluation, participants performed tests to assess visual acuity, contrast sensitivity, and detection of scotoma and metamorphopsia with both standard tools and digitized tests from OdySight. Participants did not receive dilation treatment and, if relevant, were prompted to remove their contact lenses at least 1 h before any exam. All tests were performed at a distance of 40 cm, with the head of the participant leaning against an ophthalmological chin piece and the smartphone or chart fixed on a desk at an adequate height. Before each evaluation, distance was checked with a tape measure and luminosity was checked with a lux meter. All tests were monocular, the eye tested was equipped with the adequate correction, and the eye not being tested was covered with an occluder. The order of the tests (standard exams first vs. OdySight modules first) as well as the duration of resting breaks varied for each participant based on the discretion and assessment of the examiner.
All statistical outputs were generated using SAS® version 9.4 software (SAS Institute, Cary, NC, USA).
Visual Acuity
Visual acuity was assessed with two standard methods: with the ETDRS chart at a test distance of 4 m (4-m ETDRS)—and at 1 m if needed with + 0.75 D added for correction—and with the Sloan ETDRS chart at 40 cm (40-cm ETDRS). All tests were performed in a dedicated testing room with adequate illumination (≤ 161.5 lx). Visual acuity was determined by asking the participant to read each letter starting from the first line until the smallest letter readable on the chart.
During the same visit, visual acuity was also evaluated with the visual acuity module of OdySight.
Results of the visual acuity tests were recorded in terms of number of letters read.
Contrast Sensitivity
Contrast sensitivity was assessed with the Pelli–Robson chart at 1 m, in a dedicated testing room with adequate illumination (≤ 161.5 lx). Participants removed their own glasses and instead wore a trial frame equipped with the adapted correction for distant vision increased by + 0.75 D. The contrast sensitivity level was determined by asking the participant to indicate each letter, starting from the first letter until the last readable letter. The examiner recorded each letter identified correctly by the participant, and the last triplet of letters was validated if the participant was able to read at least two of them. Results were recorded in LogCS.
During the same visit, contrast sensitivity was also evaluated with the contrast sensitivity module of OdySight.
Results of contrast sensitivity were record in logCS.
Amsler Grid
The presence of metamorphopsia and/or scotoma was assessed using an empty Amsler grid printed on a white sheet of paper (10 × 10-cm grid) and a felt tip pen. The participant was asked to draw lines (or wavy lines) where distortion was seen, or to fill an area where any type of spot was seen (e.g., black spot, white spot, area where the grid disappears) while fixing the black dot.
During the same visit, the presence of metamorphopsia and/or scotoma was also evaluated using the digital Amsler Grid module on OdySight.
After the participants performed each test, the clinical team evaluated the presence or absence of metamorphopsia and/or scotoma for each type of grid. Results were recorded as the presence or absence of metamorphopsia and/or scotoma.
Statistical Analysis
For visual acuity and contrast sensitivity, statistical analysis was performed using the LoA technique provided by Bland and Altman (B&A). This method evaluates the agreement between two quantitative measurements by constructing LoA and deducing the resulting bias, which is the mean of differences between the two methods. The closer the bias is to 0, the more it indicates good agreement between the methods. LoA are an estimate of the true agreement limits for the entire population, whereby narrower limits reflect better agreement between two techniques. However, this method only defines the interval of agreements, it does not state whether those limits are acceptable or not. Acceptable limits must be defined a priori, based on clinical necessity, biological considerations, or other goals. Results are displayed on the B&A plot with the difference between the two measurements on the Y-axis, and the average of the two measurements on the X-axis [8, 9].
Acceptable Limits
Visual Acuity
The degree of improvement in logMAR or ETDRS letters that is considered to be clinically important is uncertain and has varied among studies. Statistically significant changes in visual acuity may not necessarily be clinically relevant. The minimal clinically important difference reported in the literature is five to ten ETDRS letters (one to two ETDRS lines) [10]. In fact, even in the absence of any clinical change (e.g., no change in room condition, no change in physician performing the exam), consecutive visual acuity measurements for a given person using the same visual chart are subject to a degree of variability. Rosser et al. [11] as well as a number of other authors found a test and retest variability for the ETDRS acuities of ± 0.18 logMAR (nine letters) [12,13,14].
Contrast Sensitivity
There is no strict consensus in the literature for an unacceptable limit. The limit reported in the literature varies from ± 0.14 (2.8 letters) to ± 0.48 logCS (9.6 letters). The average of these limits is ± 0.23 logCS and the median is ± 0.18 logCS [13, 14]. In current practice, some ophthalmologists may consider that there is a significant difference between two measurements if the difference is greater than ± 0.15 logCS (three letters).
Amsler Grid
For evaluation of the two versions (standard and OdySight module) of the Amsler grid, the presence or absence of metamorphopsia and/or scotoma was noted for each type of grid. A McNemar test was used to assess comparability between the results.