# Brain-inspired algorithms for retinal image analysis

- First Online:

- Received:
- Revised:
- Accepted:

- 2.1k Downloads

## Abstract

Retinal image analysis is a challenging problem due to the precise quantification required and the huge numbers of images produced in screening programs. This paper describes a series of innovative brain-inspired algorithms for automated retinal image analysis, recently developed for the RetinaCheck project, a large-scale screening program for diabetic retinopathy and other retinal diseases in Northeast China. The paper discusses the theory of orientation scores, inspired by cortical multi-orientation pinwheel structures, and presents applications for automated quality assessment, optic nerve head detection, crossing-preserving enhancement and segmentation of retinal vasculature, arterio-venous ratio, fractal dimension, and vessel tortuosity and bifurcations. Many of these algorithms outperform state-of-the-art techniques. The methods are currently validated in collaborating hospitals, with a rich accompanying base of metadata, to phenotype and validate the quantitative algorithms for optimal classification power.

### Keywords

Retina Diabetic retinopathy Multi-orientation Orientation scores Vessel analysis Tortuosity SE(2) Screening## 1 Introduction

Diabetes is reaching epidemic proportions worldwide, especially in Asia due to fast lifestyle changes and genetic factors. Today 11.6 % of the Chinese population is estimated to have diabetes-2. Diabetic retinopathy is also the main cause of newly formed blindness in the working population, leading to high societal costs. Early detection is the key to prevention and successful treatment of these forms of blindness. However, many cases still go unnoticed and are not treated in time, especially in rural areas.

To set up a screening program for the detection of early signs of diabetic retinopathy (DR), glaucoma and age-related macula degeneration (AMD), a Sino-Dutch consortium was formed and the project *RetinaCheck* was defined with the following four phases: (1) development of innovative algorithms for automated and quantitative detection of relevant bio-markers, (2) set up a significant validation study correlating the imaging data with relevant clinical metadata, and (3) roll out a screening infrastructure in the province of Liaoning, Northeast China, and (4) make a sustainable commercial infrastructure. The partners include the departments of Biomedical Engineering at Eindhoven University of Technology (TU/e, the Netherlands) and Northeastern University (NEU, Shenyang, China), which develop the computer-aided diagnosis (CAD) software, the clinical partners He University Eye Hospital (HUEH, Shenyang, China) and China Medical University (Shengjing Hospital, Shenyang, China) (Fig. 1), and the fundus camera manufacturer i-Optics (the Hague, the Netherlands).

This paper reports on phase 1 only, focusing in detail on algorithm design. The very important validation phase is ongoing, and will be reported in a forthcoming paper.

Several other early DR signs can be measured, such as nerve damage in the cornea with confocal laser microscopy, or changes in retina neural tissue layer thickness with optical coherence tomography (OCT), but these methods are more costly and more labor intensive, especially given the projected huge-scale screening and the much lower availability of OCT in China.

This paper gives an overview of the current brain-inspired applications in the RetinaCheck project. The paper mainly focuses on vessel analysis, while a forthcoming paper will focus on the automated detection of micro-aneurysms and background diagnostic features, as drusen and exudates.

The paper is organized as follows: first a short introduction is given to the physiological evidence of multi-scale and multi-orientation processing in the visual system in Sect. 2, and a theoretical model for brain-inspired multi-orientation computing in Sect. 3. Then a series of recently developed computer-aided diagnosis (CAD) algorithms is discussed in Sect. 4, mostly based on the mechanism of invertible multi-orientation scores. The paper ends with a description of the current validation study in Sect. 5 and a conclusion in Sect. 6.

## 2 Brain-inspired computer vision

Modern brain imaging techniques have revolutionized brain research: optical imaging methods such as voltage sensitive dye imaging, calcium intrinsic imaging and optogenetics, as well as structural and functional MRI techniques have revealed intricate details of brain function and structure, especially of the visual system, the best studied brain area today. Many models of the functional mechanisms in early vision have been proposed. This paper focuses on the geometric approach, starting from the multi-scale and multi-orientation structure found in the early stages of vision.

### 2.1 Multi-scale analysis

Multi-scale analysis is now well established [15, 44, 58]. The center-surround receptive fields in the retina are the first multi-scale sampling step. Our model for V1’s ‘simple cells’ (so called by Hubel and Wiesel) is that these have the function of multi-scale, regularized spatial differential operators, i.e., Gaussian derivatives, possibly up to fourth order [44, 58, 59, 60]. The simultaneous sampling of the multi-scale differential structure leads to ‘deep structure’ analysis [43], with applications as edge focusing, hierarchical top points and SIFT and SURF keypoints. Nonlinear adaptive multi-scale analysis is developed in the rich field of geometry-driven diffusion [32, 62], which introduced nonlinear PDEs and energy minimizing variational methods into this field of evolutionary geometric computing.

### 2.2 Multi-orientation analysis

Hubel and Wiesel [37] discovered that the receptive fields in cat’s striate cortex have a strong orientation-selective property (see Fig. 2). A so-called cortical hyper-column, with the characteristic pinwheel structure of equi-orientation lines radiating from a central singularity, can be interpreted as a visual pixel computer, neatly decomposed into a complete set of orientations.

This multi-orientation framework [14, 15, 21, 31] also allows us to generically deal with crossings, as we will show with the application to vessel tracking [6, 8, 9, 52] and segmentation [1, 33, 65]. Moreover, due to the neat organization of image data on the Lie-group SE(2), we are able to design effective detection algorithms [6, 7], geometric feature analysis techniques such as bifurcation detection/analysis [55] and local vessel curvature analysis [11], and enhancement methods [31, 66].

## 3 Theory

Motivated by this organization, so-called *orientation scores* are constructed by lifting all elongated structures (in 2D images) along an extra orientation dimension, see Fig. 2. Similar to the perceptual organization of orientation in the visual cortex, a 2D orientation score is an object that maps each 2D position and orientation \((x, y, \theta )\) to a complex scalar. So the original 2D image domain can be extended to the score domain. A great advantage is that it can deal with multiple orientations per position, and the extra dimension enables new techniques for, e.g., contextual geometric reasoning and crossing-preserving enhancement [31].

### 3.1 The Euclidean motion group SE(2)

### 3.2 Invertible orientation scores on SE(2)

*f*with a specially designed, anisotropic wavelet \(\psi \):

*k*th order B-spline, \(N_\theta \) is the number of samples in the orientation direction and \(s_\theta =2\pi /N_\theta \) is the angular step size. The function \(B(\rho )\) is a Gaussian multiplied with the Taylor series of its inverse up to order

*N*to enforce faster decay. The parameter

*t*is given by \(t^2=2 \hat{\rho }/(1+2N)\) with the inflection point \(\hat{\rho }\) that determines the bending point of \(B(\rho )\).

### 3.3 Left-invariant Gaussian derivatives in SE(2)

*d*—dimensional Gaussian given by

Detailed numerical approaches for linear left-invariant diffusions on SE(2) have been developed in [66].

## 4 Computer-aided diagnosis algorithms

- 1.
Quality assessment

- 2.
Masking and normalization

- 3.
Optic nerve head detection

- 4.Vessel enhancement and segmentation
- (a)
Left-invariant Gaussian derivatives in SE(2)

- (b)
Multi-scale and multi-orientation for SLO

- (c)
Crossing-preserving multi-scale vesselness

- (a)
- 5.
Multi-orientation vessel tracking

- 6.
Arterio-venous ratio

- 7.
Fractal dimension

- 8.Geometric vessel features
- (a)
Bifurcation/crossing detection

- (b)
Curvature/tortuosity

- (a)

### 4.1 Automatic quality assessment

Sometimes retinal images have a low quality, e.g., due to cataract or other pathology. To exclude non-diagnostic images automatically in the high-volume screening process, a method for image quality verification is developed [26], based on [50]. The supervised method is based on the assumption that sufficient image structure according to a pre-defined distribution must be available. Geometric differential invariants (expressed in 2D gauge coordinates [58]) up to second order and at 4 different scales are used to develop response vectors. Different combinations of features are trained by different classifiers on 100 normal and 100 low-quality images. The ground truth for normal or low quality images was specified by two expert ophthalmologists. Combining the image structure clusters with RGB color histogram features, the Random Forest classifier proved to be the best classifier, with a performance of 0.984 area under the curve (AUC) of the receiver operator characteristic (ROC), with 0.91 accuracy rate.

### 4.2 Masking and normalization

All pixels outside the camera field-of-view are masked, according to the camera type. Retinal images often suffer from non-uniform illumination and varying contrast, which may affect the later detection process. We exploit the luminosity and contrast normalization procedures proposed by Foracchia et al. [28].

### 4.3 Optic nerve head detection

The optic nerve head (ONH) or optical disk is a key landmark in retinal images, and many methods for detection have been proposed [4, 18, 45, 47, 54, 64]. For an extensive and recent overview see [51]. Correct segmentation of the ONH and its rim is an important biomarker for glaucoma identification [38]), and for establishing a metric for regions-of-interest on the retina [36].

Conventional fundus images show the ONH as a bright disk-like feature. However, scanning laser ophthalmoscopy (SLO) cameras generally show dark regions with considerably less contrast. In the presence of large pathologies, classical approaches typically show decreased performance, or fail. Better performance is obtained by including contextual information, e.g., by incorporating the characteristic pattern of large blood vessel arches in the upper and lower retina emerging from the ONH [45, 46, 64], but this comes at higher computational costs.

Exploiting the extra dimensions of orientation scores, we have developed a template matching application via cross-correlation including local orientations [7, 10] (see also [16]). We can now match patterns of orientation distributions, rather than pixel intensities.

*t*, we minimize

*P*(\(P=100\)) normalized positive patches with ONH (with \(y_i = 1\)), or negative patches without ONH (with \(y_i = 0\)) with typically \(\lambda = 10^{-1.5}\). The regularization term enforces smoothness by punishing the squared gradient magnitude \(||\nabla t ||^2\), and prevents over-fitting. Examples of both positive and negative training patches are given in Fig. 4.

*T*, a similar functional is minimized:

The orientation score SE(2) template method outperforms state-of-the-art methods on publicly available benchmark databases, as it correctly identifies the ONH in 99.7 % of 1737 images of the well-known MESSIDOR, STARE (with a wide variety of pathological images) and DRIVE databases. For more details, see [7, 10].

### 4.4 Vessel enhancement and segmentation

#### 4.4.1 Vessel enhancement filter

Vessels can be enhanced by oriented filters [5, 29], e.g., constructed from second-order Gaussian derivative operators at appropriate scales and orientations. To solve segmentation problems at complex structures like crossings, we have designed new filters in the orientation score matching the vessel profile as second-order left-invariant Gaussian differential operators perpendicular to the corresponding orientation [65], as described in Sect. 3.

*r*represents the radius of the vessel caliber [44]. Typically vessel calibers of the DRIVE and STARE databases range from 2 to 14 pixels, so we sample the spatial scales \(\sigma _s\) as \({\mathcal {S}}=\{0.7, 1.0, 1.5, 2.0, 2.5, 3.5, 4.5\}\) and angular scale \(\sigma _o=\pi /5\). We exploit \(N_o=36\) orientations between 0 and \(\pi \). Figure 5 shows our segmentation results on the DRIVE and STARE databases. From Fig. 6 we can see that the proposed orientation score based multi-scale filters show much better structure preservation ability on these special cases, as illustrated in Fig. 6.

#### 4.4.2 Multi-scale and multi-orientation segmentation (BIMSO) for SLO images

For retinal images taken with a scanning laser ophthalmoscope (SLO) camera, new segmentation techniques are required. Such cameras typically exploit two wavelengths, green and infrared and typically exhibit better contrast for hemoglobin (vessels and bleedings). However, they also exhibit speckle noise, due to which conventional analysis methods for RGB images often fail. Very few studies are dedicated to SLO image analysis (e.g., [63]).

The proposed method [1], termed BIMSO (Brain-Inspired Multi-Scale and multi-Orientation), has four main steps: preprocessing, feature extraction, classification and post-processing. The method is developed for the green channel (RGB and SLO).

Noise reduction is effectuated by a nonlinear gamma transform (\(\check{U}_f=\alpha ~{|U_f|}^\gamma \)). The orientation score is raised by a power factor \(\gamma >1\). The absolute value of the orientation score \(|U_f|\) is taken because of the quadratic property of the cake wavelets, and \(\alpha \) is determined by the sign of the real part of the orientation score (\({\hbox {Re}}(U_f))\). See Fig. 7.

#### 4.4.3 Crossing-preserving multi-scale vesselness

Frangi’s vesselness filter [29], which is based on geometric relations between the principal curvatures (eigenvalues of the Hessian second-order matrix) to extract cylindrical shapes, has been extended to the orientation score domain, which makes it crossing-preserving [33].

*s*and \(\mathbf {g}\in {\hbox {SE}}(2)\), normalized w.r.t. the \(\beta \)-metric. For the first we get:

### 4.5 Multi-orientation vessel tracking

Vessel tracking has the advantage over pixel classification that it guarantees connectedness of vessel segments. We have developed tracking via orientation scores [6], so classical difficulties as crossings, bifurcations, closely parallel vessels and vessels of varying width and high curvature can be dealt with naturally.

To handle bifurcations properly, one-sided kernels are exploited, constructed by decomposition of orientations scores in two opposite directions, weighted in the radial direction with an error function (cumulative Gaussian).

Two tracking methods are developed: Edge Tracking based on Orientation Scores (ETOS) and Centerline Tracking based on multi-scale Orientation Scores (CTOS). ETOS tracks both edges of the vessel simultaneously. The edge positions are detected in the orientation score, as a local minimum (left edge) and maximum (right edge) from the anti-symmetric imaginary part of the tangent plane. CTOS exploits the fact that the disturbing vessel central light reflex is filtered out by Gabor kernels with proper scales. The invertible orientation score has some distinct advantages over Gabor filtering: much lower computational costs as the orientation score filters for all scales simultaneously, whereas the Gabor filter only calculates a single spatial frequency at a time, and the orientation score filtering is more accurate.

Overall, ETOS outperforms CTOS. ETOS gives best results when applied on invertible orientation scores, can deal with many complex geometries, and gives reliable width measurements.

The algorithm performs well: ETOS detected 76 % (290/381) of the bifurcations and 96 % (109/114) of the crossings correctly. Most mistakes in bifurcations were actually crossings, only 5 % was misclassified. For more details, see [6].

#### 4.5.1 Sub-Riemannian geodesics in SE(2)

The algorithms for sub-Riemannian geodesic extraction are implemented via an anisotropic fast marching scheme [49, 52]. As a result, the curve extraction procedure is both fast and robust. In summary, the sub-Riemannian geodesic extraction in SE(2) allows for the robust extraction of curves because: (1) crossing structures are disentangled in the orientation scores, and (2) high curvature (e.g., discrete jumps to the left or right, or sudden change of direction) is punished due to the restricted Sub-Riemannian geometry. See examples in Fig. 9.

### 4.6 Arterio-venous ratio (AVR)

The separation of vessels in arteries and veins is crucial, as their different physiology, flow and mechanical properties respond differently to disease development, e.g., arteriolar narrowing, a decrease of the artery calibers relatively to the vein calibers is an important early biomarker for diabetic retinopathy. Also tortuosity measures are expected to be different for arteries and veins [39]. An automated artery–vein classification is required for high-volume screening.

The ratio of the arteriolar and venular diameters is called the arteriovenous ratio (AVR) and is classically computed from the six widest arteries and veins in a restricted zone around the optic disk [42].

We developed a novel method for artery/vein classification [25] based on local and contextual feature analysis of retinal vessels. Features are (a) the color, as arteries appear brighter and veins darker due to the oxygen content of the blood, (b) the transverse intensity profile, as arteries have a more pronounced central reflex to the camera flash, and (c) graph path properties of crossings and bifurcations of vessels, as these provide contextual information, because arteries never cross arteries and veins always cross arteries [17].

### 4.7 Fractal dimension

The concept of fractal dimension was initially defined and developed in mathematics. It measures the complexity of self-similar objects that have the same patterns across different scales, e.g., trees and snowflakes. The vascular tree on the human retina also has self-similar branching patterns over different scales. Therefore, there is a growing interest in retinal fractal analysis, exploiting fractal dimension as a biomarker for discriminating healthy from diabetic retinopathy.

Fractal dimensions have been widely investigated. However, conflicting findings are found [3, 12]. This motivated us to especially investigate the stability and reproducibility of fractal dimension measurements.

We examined the stability of the fractal dimension measurements with respect to a range of variable factors: (1) different vessel annotations obtained from human observers, (2) different automatic segmentation methods, (3) different regions-of-interest, (4) different accuracy of vessel segmentation methods, and (5) different imaging modalities. Our results demonstrate that the relative errors for the measurement of fractal dimensions are significant and vary considerably according to the image quality, modality and the technique used for measuring it. So automated and semi-automated methods for the measurement of fractal dimension are not stable enough, which makes fractal dimension not a proper biomarker in quantitative clinical applications [35].

### 4.8 Geometric vessel features

#### 4.8.1 Bifurcations/crossings detection

### 4.9 Curvature/tortuosity

Vessel curvature is an important biomarker [13, 34, 39, 53]. Most methods require pre-segmentation and center line extraction, but our brain-inspired orientation score method [11] works directly on the image data by fitting so-called exponential curves in the orientation score. An exponential curve is a curve whose spatial projection has a constant curvature (Fig. 13).

A confidence measure is acquired by calculating the Gaussian Laplacian in the plane orthogonal to the tangent direction of the vascular geodesic. From all pixelwise curvature measures, several global measures are derived, as the mean and standard deviation. The method was validated on synthetic and retinal images.

Tortuosity measures from the MESSIDOR database (1200 images) showed significant increase of tortuosity with the increasing severity of diabetic retinopathy (R0, R1, R2 and R3), see Fig. 14. For more details, see [11].

## 5 Next steps

A dedicated workstation has been designed, to program any pipeline of processes, in batch mode, and with automated report generation (see Fig. 15, see also [19]). Data are acquired with different fundus cameras (EasyScan, DRS, Topcon). The workstation is also used for annotations: at least three experienced ophthalmology experts diagnose and annotate the bio-markers on the fundus image datasets as ground truth or reference standard. We have established a detailed workflow protocol to minimize variations.

The crucial validation phase, not reported in this, paper, is ongoing. The algorithms are currently evaluated in the collaborating hospitals: image and extensive metadata have been acquired of 3000 diabetes patients in Shengjing Hospital, and over 20.000 normal images with first expert reading are acquired at He Vision shops in Shenyang, China. Studies are ongoing to correlate the rich set of geometric biomarkers described in this paper with the extensive Chinese metadata obtained in the clinical setting, and find the most predictive and effective (combination of) geometric biomarkers extracted from the retinal fundus images for DR, glaucoma, and AMD.

## 6 Conclusion

The brain-inspired multi-orientation approach turned out to be highly successful in dealing with many complex vessel geometries, such as crossings, bifurcations, curvature analysis without segmentation, enhancement and segmentation. The theory for orientation scores in SE(2) is now well developed. The generalized approach of a Lie-group analysis by the visual front-end is currently a broad topic of research. The paper gives a review of many current geometric developments for retinal vessel analysis with the goal of establishing biomarkers for screening for early signs of a range of systemic, neurodegenerative and cardiovascular diseases.

In the parallel deep machine learning field major developments take place. The RetinaCheck team ended at 17th position in the recent Kaggle challenge on Diabetic Retinopathy classification (https://www.kaggle.com/c/diabetic-retinopathy-detection), of 661 participating teams. It can be foreseen that the merge of geometric analysis and reasoning with deep learning, based on the large volumes of data acquired in this project, can be highly successful. This is ongoing development, and will be reported in forthcoming papers.

## Acknowledgments

The work is part of the Hé Programme of Innovation Cooperation, which is (partly) financed by the Netherlands Organisation for Scientific Research (NWO), dossier nr: 629.001.003. The work is also financed by a Lilly Collaborative Research Grant from the European Foundation for the Study of Diabetes EFSD together with the Chinese Diabetes Society, and by the EU Marie Curie program ‘Metric Analysis For Emergent Technologies’ (MAnET), Grant Agreement No.: 607643.

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.