A continuous-variable quantum-inspired algorithm for classical image segmentation

Quantum image processing has been an active area of research recently. The usual tasks of image processing are performed utilizing the theory of quantum mechanics. This includes image representation (Yan et al. 2016), image matching (Jiang et al. 2016), similarity analysis (Zhou et al. 2018b), interpolation (Zhou et al. 2018a), denoising (Mastriani 2015a), coding (Chapeau-Blondeau and Belin 2016), watermarking (Li et al. 2016), and segmentation (Caraiman and Manta 2015). These attempts are based on representing the pixels of an image as qubits operated on via suitable quantum circuits. However, there are major challenges facing this approach that need further investigation as in Mastriani (2015b). On the other hand, another approach exists in which classical images are processed on classical computers using quantum-inspired models. This includes for example the classical image segmentation algorithm in Youssry et al. (2015), and its application in segmenting biomedical retinal images (Youssry et al. 2016), quantum K-means (Casper et al. 2012), quantum Gaussian mixture models (Tanaka and Tsuda 2008), and quantum pattern recognition (Sergioli et al. 2016). Some of these methods can be ported to work on a quantum computer such as Youssry et al. (2015, 2016) which opens the path for many future applications.

Conventionally, discrete degrees of freedom of particles (such as the spin of an electron or the polarization of a photon) are used for encoding information in the form of qubits. In general, this approach faces some technological difficulties when it comes to the implementation. For example, we might have to control the polarization of a single photon system or the spin of a single electron which is not easily realizable. As a result, many of the developed ideas and algorithms have not been fully realized yet. Continuous-variable quantum information processing is another approach that depends on using the continuous degrees of freedom of the particle (such as position and momentum) for manipulating the information. This approach provides easier technological implementations, but can be challenging in porting algorithms and protocols from the discrete domain to the continuous domain. Examples of this category of information processing include quantum computation (Adcock et al. 2016), machine learning (Lau et al. 2016), quantum key distribution (Borelli et al. 2016), and identity authentication (Ma et al. 2016).

Image segmentation is an area of image processing that has many applications. It deals with delineating the significant objects in the image and isolating them from the background. Many methods exists to segment images including thresholding, edge detection, supervised and unsupervised machine learning, morphological methods, and deformable models. There exist also quantum-based methods for image segmentation. A short review on these techniques is given in Youssry et al. (2015). Some of the authors previously proposed a general framework that uses the theory of two-state quantum mechanics systems to process images (Youssry et al. 2015). Based on this framework, a general single-object image segmentation was developed and applied to generic images as well as in determining the vessel tree in retinal images. This algorithm showed high efficiency in segmenting images. Although the framework provides the theory for the extension to multi-object segmentation by utilizing discrete multi-state quantum systems, this extension is complex and computationally expensive. The continuous-variable quantum theory provides a solution to this challenge.

In this paper, we propose a new algorithm for image segmentation based on the continuous-variable coherent quantum states that occur in the theory of quantum harmonic oscillators. The work is built upon the framework presented in Youssry et al. (2015), but extends to the case of multi-object segmentation. The paper starts in Section 2 with a brief theoretical overview essential for introducing the new methodology. Next, the proposed algorithm is presented in Section 3. After that, the materials used for testing as well as the obtained results are shown in Sections 4 and 5. In Section 6, the analysis and the significance of the results are discussed. Finally, the conclusion and the future perspectives are given in Section 7. The Appendices A and B include some additional proofs given for the sake of completeness.

This section starts with a brief overview on the theory of quantum harmonic oscillators, needed for developing the proposed methodology. The details can be found in any standard reference on quantum mechanics or quantum optics such as Griffiths (2005) or Gerry and Knight (2005). Afterwards, a short review on the quantum fidelity measure is given. Finally, the performance measures used for evaluating the segmentation algorithm are discussed.

In this section, the proposed methodology is presented. First, an overview is given on the quantum-based framework upon which the proposed image segmentation algorithm is built. This framework has been proposed recently in Youssry et al. (2015). The challenges of extending this framework to the multi-object case are discussed, as well as how the novel algorithm overcomes these challenges. After that, the detailed steps of the developed algorithm for multi-object image segmentation are elaborated.

3.2 Proposed algorithm

Based on the previous discussion, the algorithm shown in Algorithm 1 is proposed and it consists of five main steps discussed as below.

Open image in new window

3.2.1 Reference state preparation

The first step is to associate each object in the classical image to a predefined coherent state, that will be referred to as the reference state. The choice of reference states is arbitrary. In this work, it is assumed that there are N − 1 objects to extract as well as the background. Thus, the reference states are chosen to take the form

$$ |\beta_{k}\rangle=\frac{e^{i\frac{2\pi}{N}k}}{2\sin\left( \frac{\pi}{N}\right)},k=0,1,...N-1 $$

(21)

Thus, on the complex plane formed of the real and imaginary components of the coherent state (phase space), the reference states are distributed evenly on a circle with radius \({\left (2\sin \limits \left (\frac {\pi }{N}\right )\right )}^{-1}\) centered at the origin. Once the reference states are selected, they do not change anymore. The magnitude of these states is scaled to provide enough separation between them. This is related to the uncertainty principle, as experimentally the real and imaginary parts of the state cannot be measured simultaneously. The actual representation of a coherent state in the phase space is a small circle to reflect this uncertainty. In order to prevent the overlap of the reference states, the amplitudes are scaled. However, the scaling factor does not change the result of the algorithm as shown in Appendix B; therefore, it can be chosen arbitrary. It may also be selected to have sufficiently large value such that the classical behavior dominates. Consequently, normal optical components can be used in this case for experimental realization. It is worth noting that the choice of the reference states affects the training of the Hamiltonian as the goal of this step is to find a set of optimal parameters of the Hamiltonian that results in evolving the initial states to the corresponding reference states as final states.

3.2.2 Pixel state initialization

In order to classify a particular pixel to one of the objects in the image, it is associated to a QHO system. The initial state of the pixel is taken to be the vacuum state.

$$ |\psi(0)\rangle=|0\rangle. $$

(22)

3.2.3 Feature extraction

In this step, the feature vector is extracted from the pixel and is denoted by x. Next, the features are combined together into a single feature, denoted by T = T(x). This function can be chosen arbitrarily. In this paper, it is chosen to take the following form

$$ T(\mathbf{x})=p(\mathbf{x})u\left( p(\mathbf{x})\right)+0.01. $$

(23)

p(∙) is a polynomial function chosen to be of 6th order, (in some cases it was selected to have a Gaussian form depending on the problem) while u(∙) is the Heaviside unit step function which returns 1 if its argument is greater than 0 and returns 0 otherwise. This form assures that T is positive-valued, and thus can represent a time variable. It also guaranties that T^− 1 is not singular at any point. These two requirements are needed as will be shown in the next step of the algorithm. The coefficients of the polynomial are evaluated during the learning phase of the algorithm and are kept fixed afterwards through out the testing.

3.2.4 State evolution

The QHO system of the pixel is then allowed to evolve to its final state governed by the Hamiltonian

$$ H=\hbar\omega\left( \hat{a}^{\dagger}\hat{a}+\frac{1}{2}\right) + \hbar f_{0}\left( \hat{a}e^{i\omega t}+\hat{a}^{\dagger} \hat{a}e^{-i\omega t}\right). $$

(24)

The final state under this form of the Hamiltonian is guaranteed to be the coherent state

$$ |\psi(t)\rangle=|\alpha\rangle, \alpha= -i e^{-i \omega t} f_{0} t. $$

(25)

This equation shows that the final state depends on three factors: ω, f₀, and t. But actually, there are only two degrees of freedom since the state is defined by a complex number that has only real and imaginary parts. The angular frequency is chosen arbitrary in this paper to be unity (ω = 1). The time instant at which we observe the state is chosen to be t = T(x). The force amplitude is chosen to be \(f_{0}=\frac {1}{2\sin \limits \left (\frac {\pi }{N}\right )T(\mathbf {x})}\). Therefore, the final state of the QHO is

$$ |\psi(T)\rangle=|\alpha(T)\rangle, \alpha(T)=\frac{-i e^{-i T}}{2\sin\left( \frac{\pi}{N}\right)}. $$

(26)

It is clear now that final state is a function in T which is itself a function of the image-based feature vector x. In other words, the image-derived features at the pixel control the final state of the QHO.

3.2.5 Measurement

In order to obtain the final outcome for the pixel under consideration, the final state is observed. If it is one of the reference states defined in the first step of the algorithm, then the corresponding class is the final outcome. However, practically this may not happen. In this case, we have to choose one of the reference states and consequently choose the corresponding class for this pixel. An intuitive solution is to choose the reference state that is nearest to the pixel’s final state. The fidelity measure in Eq. 19 can be used to obtain the distance between the final state and each of the reference states. Thus, the pixel is classified to belong to object j if it satisfies that

$$ j=\underset{i}{\arg\max} F_{i}=\underset{i}{\arg\max} |\langle{\alpha|\beta_{i}}\rangle|. $$

(27)

Following the work in Youssry et al. (2015), the proposed algorithm is tested on two datasets. The first dataset consists of synthetic images of geometric shapes with different types and different number of objects. The ground truth of these images is generated manually. Moreover, noise is applied to some of these images in order to test the performance of the algorithm in the presence of noise. The second dataset consists of natural images and is chosen from the publicly available image segmentation database of Alpert et al. (2007). This database provides images with single and double objects as well as the human segmentation for all images to test the accuracy of segmentation methods. The number of synthetic images are 19 images of which 11 of them are noisy images. Five natural images are included which adds up to a total of 24 images.

The presented algorithm is tested on the datasets described in the previous section. For each image in the dataset, features are selected to better represent the objects in the image and used to estimate the Hamiltonian parameters. Many features are used depending on the nature of the image which are mostly simple features such as the gray-level, mean, and median. Nevertheless, in few cases, more complex features like the Morlet transform–based features are utilized. The features used are based on the nature of the object and background in the image. For most of the simple noiseless images (examples are first and third images in Fig. 1 and first image in Fig. 2), the pixels’ gray values were sufficient to segment the object. However, in some cases there were no clear class separation between the object and background in the grayscale domain. Thus, other features were incorporated. For example, the second image in Fig. 2 is a textured image, so Morlet wavelet–based features were found to perform better than grayscale features. Moreover, in the presence of noise as in the images in Fig. 3, features from median filters with window size depending on the amount of noise were utilized and provided highly accurate segmentation. The feature selection is a design issue and should depend on the analysis of the images in order to estimate the Hamiltonian’s parameters that will derive the evolution and in turn the segmentation.

Open image in new window

Open image in new window

Open image in new window

The sensitivity and the specificity measures for each class in each image are calculated. First, the algorithm is applied to the images with single object that are used to validate the original framework (Youssry et al. 2015). This system is considered an enhancement to the previously introduced framework. Thus, the purpose of this step is to verify that the system can produce comparable results in case of single object before proceeding to multi-object images. Average sensitivity and specificity of 98.23% and 99.53% were obtained, respectively. This shows that the coherent state–based algorithm performs very efficiently in segmenting images with single object. Samples of segmented objects are shown in Fig. 1. In addition, the results are very similar to the former framework (sensitivity = 98.5% and specificity = 99.7%) which were shown to exceed other existing segmentation methods like active contours and graph cuts. In this paper, four methods are compared against the proposed algorithm. These segmentation algorithms are K-means clustering (Lloyd 1982), Otsu’s multithreshold (Otsu 1979), lazy snapping graph cuts (Li et al. 2004), and random forests (Sommer et al. 2011). The results of applying the five algorithms to the entire dataset are summarized in Table 1. Regarding the single-object images, the sensitivities of both the quantum (98.23%) and random forests (98.55%) methods were close and significantly higher than those of the other three methods. The quantum, Otsu, and K-means techniques produced similar specificities (around 99.72%) which were 1–2% better than the other two algorithms.

For the multi-objects’ dataset including synthetic, noisy, and natural images, the quantum method gave the highest sensitivity over all techniques in comparison with a sensitivity of 97.52%. The sensitivities of the other methods were approximately 4.5–9.2% lower than those of the proposed technique. In terms of specificity, the quantum had a value of 99.61% which is only 0.03% lower than K-means and higher than the rest by up to 7.25%. In order to assess the noise performance, Gaussian, salt and pepper, and compression noise are added to some of the synthetic images to create eleven noisy images. Salt and pepper noise was added to the three-objects image in Fig. 3 while other images were modified by the inclusion of Gaussian noise such as the first two images in Fig. 3. The compression noise was added to the single-object image in Fig. 4 at low and medium quality levels, as well as the two-object image in Fig. 4 at low- and high-quality levels. Compression noise was added by compressing then decompressing the image using lossy JPEG scheme at the corresponding quality level. Furthermore, the first image in Fig. 1 was blurred using a multiplicative noise. The test with the noisy images illustrates that the introduced method performed very well in the presence of noise with sensitivity and specificity of 98.30% and 99.30%, respectively. The best noise performing algorithm was the random forests which is slightly higher than the quantum approach by 0.5% and 1.2% for sensitivity and specificity, respectively. Nevertheless, the sensitivities of the quantum method were 3.3–7.2% better than those of the remaining three methods in comparison. Examples of segmented objects in images with multi-objects (Fig. 2) in addition to noisy images (Figs. 3 and 4) are demonstrated for qualitative assessment of the algorithm.

Open image in new window

The reported overall average performance measures indicate that the specificities from all methods, except the graph cuts (93.92%), are in close proximity to each other (99.00 to 99.69%) with quantum and K-means at the top of the range. However, the superiority of the proposed method over all the other methods under comparison in capturing the target objects for different types of images is evident from the sensitivity results. The quantum technique’s average sensitivity is 97.86% compared with the closest value of 94.58% from random forests and the least value of 90.03% from the graph cuts method.

The theory of quantum harmonic oscillators and coherent states provide the bases for the proposed quantum-based image segmentation algorithm. This method relies on treating each pixel in a classical image as QHO initially at the vacuum state. By allowing the system to evolve controlled by features extracted from the image’s pixels, the oscillator can reach any of the continuous eigenstates. Principally, this allows for the segmentation of an infinite number of objects. The Hamiltonian parameters are estimated by supervised learning from the image features to lead the evolution to the desired class. The results of applying the system to segment different images indicate that the algorithm can accurately segment multi-objects in many types of images including noisy ones.

The presented method inherits the design flexibility from the original framework (Youssry et al. 2015). So, many design aspects can be adjusted to suit different types of applications such as the form of the Hamiltonian. In this work, the Hamiltonian was selected to lead to a closed-form solution. However for other problems, a more complicated form might be needed which may necessitate obtaining a numerical solution of Schrödinger’s equation. The construction of the Hamiltonian was performed using supervised learning which can also be changed to possibly unsupervised learning approach. Additionally, the fidelity metric was adopted in this work as it can be optically implemented as will be discussed later in this section. However, other metrics can be used.

The analysis of the performance summarized in Table 1 shows that the quantum-based method outperforms two of the classical image segmentation techniques in addition to graph cuts and random forests in terms of overall accuracy. All objects in all images were correctly identified by the proposed algorithm. However, K-means, graph cuts, and Otsu’s method failed to identify objects in some of the images. For example, Otsu’s thresholding algorithm missed the second object in the natural image in the third row of Fig. 2 while all three techniques failed to identify the second object in the noisy image in the second row of Fig. 3. Also, it can be seen from Figs. 5 and 6 that K-means and Otsu’s methods tend to undersegment the objects in the natural image of the bird as well as in the gradient noisy image while graph cuts produced greater segmentation inaccuracies. Moreover, Otsu’s segmentation of the image in Fig. 7 tends to identify part of object 2 as object 1. The random forests method did not suffer from the aforementioned issues but performed poorly when using complex features in textured images as shown in Fig. 8. The proposed method succeeded in segmenting all objects in the different types of images without missing or undersegmenting any objects. Furthermore, the performance was robust to noise with minimal effect on accuracy. As can be observed from the average sensitivity and specificity for all four types of inserted noise which were only reduced by 1.70% and 0.31% in comparison with the noiseless case. K-means is initialized randomly and there is no guarantee that it will produce the correct segmentation. Also, the repeated runs of the algorithm do not yield the same segmentation results. For fair comparison, the tests were done numerously in order to achieve the best segmentation. Also, the features used in the quantum method were used with the K-means. The quantum method did not suffer from all the mentioned problems. The results do not change by repeated runs of the algorithm. Otsu’s method is not suitable for incorporating multiple features (i.e., it works using only one feature). A different approach for image segmentation is the active contours method (Chan and Vese 2001). Although it is very powerful in case of single-object images, it is not suitable for multi-object images. The only way to solve this problem is to customize the location and shape of the initial contour so that it only captures one object and then repeat for other objects. But this requires too much human intervention which is a drawback. In addition to its higher performance, the proposed algorithm does not face these challenges and can automatically do the task with minimal human intervention.

Open image in new window

Open image in new window

Open image in new window

In Sergioli et al. (2016), a framework for pattern recognition has been proposed in which features are mapped to quantum density matrices on the Bloch sphere via a stereographic mapping. This is suitable for a 2D feature vector. For larger number of features, the model can be generalized geometrically by using Bloch spheres of higher dimensions. In this case, higher-dimensional matrices are used. The classification is done by a nearest mean classifier rule based on trace distance. In a broader sense, the work in Sergioli et al. (2016) shares a quantum-based classification approach as the proposed method. Nevertheless, the two methods have many differences. First, the presented work is based on the theory of quantum harmonic oscillators which uses continuous states of infinite-dimensions and is independent of the number of classes as previously discussed. Second, the features are encoded through the Hamiltonian governing the evolution of states. Third, the identification of the final state is performed by first evolving the system then by using the fidelity as a distance metric. Fourth, the learning is done in a least-square sense for evaluating the Hamiltonian parameters. Finally, despite the ability of the suggested algorithm to be used as a classifier, the focus is on developing a complete for image segmentation technique.

Although the algorithm is designed to work on classical computer, the usage of coherent states opens the path for practical implementation of the algorithm using optical components. Laser beams are described using coherent states and the quantum effects dominate as the number of photons decreases (corresponding to reduction in the beam power). Using a local oscillator (LO) beam, other beams can be generated by splitting the LO beam and then phase-modulating each sub-beam alone. These beams correspond to the fixed reference states {|β_i〉}. For each pixel in the image, another beam from the LO is generated and modulated according to the value of the extracted feature at the pixel to produce the final state |α〉. The beams from the pixel and from the reference states can be combined in a Mach-Zehnder interferometer to estimate the fidelity using the method in Ekert et al. (2002). Then, the state with the highest fidelity is selected to produce the classification of the pixel.

Open image in new window

This paper proposes an algorithm for segmenting classical images that is formulated from the foundations of quantum mechanics. It can be considered as enhanced extension of the work done in Youssry et al. (2015). In addition to the ability to deploy beneficial aspects from quantum mechanics in image segmentation as the original framework, this algorithm has a major advantage as it can handle images with multi-objects at no additional computational complexity. This is accomplished by utilizing the theory of quantum harmonic oscillators rather than two-state quantum system. Since the number of coherent states is a continuum, their eigenstates represents a continuous-variable and thus can model any number of objects. The performance of the proposed method demonstrates its high performance in terms of accuracy even when noise is present, while being superior to the original work in Youssry et al. (2015) in terms of complexity. Despite being developed as a classical algorithm, we provided a suggestion on the quantum implementation of the system using the aforementioned optical hardware. The following points should be considered in the future to enhance the system. First, the interaction between neighboring pixels is considered only indirectly in the feature extraction. In order to provide a direct way to incorporate this information and replace the use of complicated image features, the mathematical model of coupled quantum harmonic oscillators could be exploited. Second, the algorithm was presented generally and tested on generic images to show its functionality. It remains to get advantage of its flexibility to efficiently apply it to a particular application. Finally, the system could be physically realized, as described previously, to validate its practically.