Image augmentation with conformal mappings for a convolutional neural network

For augmentation of the square-shaped image data of a convolutional neural network (CNN), we introduce a new method, in which the original images are mapped onto a disk with a conformal mapping, rotated around the center of this disk and mapped under such a Möbius transformation that preserves the disk, and then mapped back onto their original square shape. This process does not result the loss of information caused by removing areas from near the edges of the original images unlike the typical transformations used in the data augmentation for a CNN. We offer here the formulas of all the mappings needed together with detailed instructions how to write a code for transforming the images. The new method is also tested with simulated data and, according the results, using this method to augment the training data of 10 images into 40 images decreases the amount of the error in the predictions by a CNN for a test set of 160 images in a statistically significant way (p=0.0360\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p=0.0360$$\end{document}).


Introduction
A convolutional neural network (CNN) is a type of deep learning technique that is well-suited for processing images.It receives image data in a matrix format so that each element of the matrix corresponds to the value of one pixel in the image and then transforms this input through several layers by taking into account the spatial relationships between the data points.The CNNs have been noted to be very useful in different areas of research but training even a single CNN often requires a large number of labelled images, which can sometimes be difficult to obtain.
One possible solution to this problem is using data augmentation, which means that the amount of existing data is multiplied by using simple transformations to create new, slightly different versions of images.Typical transformations used for this purpose are rotations, reflections, and translations but they do not suit for all types of data.Namely, if the image is square-shaped, creating a new version of it with a translation always crops out some areas from the image close to its edges and there are only seven new images that can be created with such a rotation or a reflection that fully preserves the original square.Clearly, the issue would not be encountered if the images given to the CNN would be disk-shaped but this is rarely the case.
However, according to the Riemann mapping theorem from classical function theory, any simply connected proper subdomain of the complex plane can be mapped onto the unit disk with a conformal mapping and, in particular, a conformal mapping called a Schwarz-Christoffel mapping can be used to map a two-dimensional disk onto the interior of a simple polygon [8].Conformal mappings are much studied in complex analysis because they have many desirable properties such as preserving the magnitudes and directions of angles between curves even though they can turn straight line segments into circular arcs and vice versa.Furthermore, another subtype of conformal mappings are Möbius transformations which can be defined so that they map the interior of a disk onto itself but still significantly transform its contents.While conformal mappings could potentially be utilized in image data augmentation, their formulas are generally quite complicated and require integration of complex valued functions that cannot be computed directly with the existing functions in common programming languages.
In this article, we study if image augmentation can be performed by first mapping each square-shaped image onto a disk with a conformal mapping, then applying such rotations and Möbius transformations that preserve this disk, and finally mapping the resulting images back onto their original square shape.First, in Section 2, we present the usual formulas of conformal mappings and other mathematical theory needed.In Section 3, we show how these mappings can be written in Python or other programming languages and how images can be mapped with them.In Section 4, we test this method with one simple example about a CNN predicting a simulated data set.All the codes written in Python and MATLAB are also publicly available so the readers can access to these codes.

Preliminaries
Let G be the interior of the square with the vertices 1 Consider the complete elliptic integrals of the first kind K(r) and K ′ (r) defined for r ∈ (0, 1) by [3,16] (2.1) In many references, the complete elliptic integrals of the first kind K(r) are defined as in (2.1), see e.g.[4,7,12].However, in this paper, we will use the notations used in [1].Thus, the complete elliptic integrals of the first kind are defined by [1, p. 590] (2.2) where m = r 2 ∈ (0, 1) and m 1 = 1 − m.The incomplete elliptic integral of the first kind is defined by [1] (2.3) and then K(r) = F (π/2, m).See [1,3,4,7,12] for more information.
The Jacobian elliptic functions can be defined with the help of the incomplete elliptic integral (2.3), see [1,Ch. 16].If The exact conformal mapping from the square region G and its inverse can be written in terms of the elliptic functions and the incomplete elliptic integral.It follows from [11, p. 182] and [12, p. 242] that the conformal mapping f from the domain G onto the unit disk B 2 , f : G → B 2 , is given by [3, (3.19), p. 51].Then, it follows from (2.4) and (2.5) that the inverse of this mapping, f −1 : B 2 → G, is given which is a conformal mapping from the unit disk B 2 onto the square domain G.
Denote the extended complex space by C = C ∪ {∞} and choose some α ∈ B 2 .Define the Möbius transformation g : C → C, where α is the complex conjugate of α.This mapping g fulfills g(B 2 ) = B 2 , g(α) = 0, and g(0) = −α, which means that it preserves the unit disk B 2 but is not an identity mapping as long as α ̸ = 0.
Finally, we define formally the rotation about the origin for an angle k ∈ [0, 2π) as Figure 1 represents visually how the functions f , g, and υ and the inverse function f −1 affect the shape of an image, when g and υ are defined with constants α = 0.3 + 0.3i and k = π/3, respectively.

Code
Computing the mapping function f by (2.5) and its inverse function f −1 by (2.7) requires computing the Jacobian elliptic functions sn(ϕ, m) and dn(ϕ, m) for complex ϕ as well as computing the incomplete elliptic integral F (φ, m) for complex φ.Thus, the function f and its inverse function f −1 cannot usually be computed directly from their definitions (2.5) and (2.7) in the code because the functions used to evaluate the elliptic integrals are not defined for complex numbers in many programming languages.For instance, the complete elliptic integral K(m), the incomplete elliptic integral F (φ, m), as well as the Jacobian elliptic functions sn(ϕ, m), cn(ϕ, m), and dn(ϕ, m) can be computed for real arguments, , with the functions ellipk, ellipkinc, and ellipj of the subpackage special of the package Scipy [20] in Python.So, in this paper, we will use the properties of the Jacobian elliptic functions and the incomplete elliptic integral to compute their values without the need to compute elliptic integrals of complex variables.For example, the functions sn(z, m) and dn(z, m), for a complex variable z = x + iy where x and y are real variables, can be computed using the following formulas [1, §16.21.1]where and Similarly, for a complex variable z = x + iy, the incomplete elliptic integral F (z, m) can be computed by [1, §17.4.11] where x 1 and y 1 are real variables such that X = cot 2 x 1 is the positive root of the equation and tan 2 y 1 is given by Denote below the floor and the ceiling functions with floor() and ceil().Then, the values of x 1 and y 1 can be computed from (3.6) and (3.5), respectively, through (see [14]) At the beginning of the code, fix the constants r, m, and L as in (2.6).The conformal mapping f from the square G onto the unit disk can now be written with the following pseudo-code.
(3.9) define f (z) : Finding the inverse mapping f −1 requires solving the quadratic equation (3.4).The more unusual trigonometric functions such cot or csc and their inverses are in the package sumpy in Python.Note also that the parameter ϵ can be chosen to be any small positive number.
if X 1 tan 2 (x) < 1 : y 1 = 0 else : #Change of variables taking into account the periodicity ceil to the right After writing the functions f and f −1 with the pseudo-codes (3.9) and (3.10), they can be tested by choosing a random number z from the square G, computing w = f (z) and z ′ = f −1 (w), and printing the difference z − z ′ , which should be very close to zero.
An image can be mapped conformally onto a disk with the function squareToDisk presented in (3.11).First, suppose then that there is a square matrix called img which can be read as a square-shaped grayscale image by choosing the colour of each pixel according the corresponding element in img.Let h be the number of rows or columns in the matrix img, imgH a vector containing h evenly spaced points in the interval [−1, 1], and u the distance between two adjacent points in imgH.Then we initialize the new image img by creating a zero matrix of the same size as img.After that, we create a loop that goes through each element of img and expresses it as a point z ∈ G with the help of the vector imgH.If the point z belongs to the unit disk, we use the inverse mapping f −1 to find the point w in the square G that becomes z when the domain G is mapped conformally to the unit disk with the mapping f .For this point w, we find such points j 0 , k 0 ∈ (0, 1, ..., h − 1) and j 1 , k 1 ∈ [0, u) that w = j 0 + j 1 + (k 0 + k 1 )i, which gives us also the closest four points to w that correspond to the pixel locations of the original image matrix img and the distances between w and these locations.Finally, we compute the values of the pixel at the point w by using the weighted means of the four surrounding pixels and, since z = f (w), we have the value of the pixel at the point z in the unit disk.In the pseudo-code below, v[i] refers to the (i + 1)th element of the vector v as the indexing starts from 0.
for j in (0, 1, ..., h − 1) : for k in (0, 1, ..., h − 1) : The function in (3.11) returns such an image matrix that has the original image mapped onto the largest possible disk fitting inside the square-shaped image matrix, and the other values outside this disk are zeroes.Note that this function can be extended for also colour images of RGB or another similar format by just computing the weighted means for the values of each color channel at the end of the loop.Similarly, by replacing the inverse function f −1 with either the Möbius transformation g of (2.8) or the rotation υ of (2.9), we can map the input image so that the part inside the disk is transformed.To create the function that conformally maps the interior of this disk in the square-shaped image matrix onto the whole matrix, we just need to remove the condition |z| < 1 and replace w = f −1 (z) by w = f (z) in the code (3.11).Alternatively, we can use for instance the composed mapping w

Experiment
Here, we build a CNN for predicting how many small black disks an otherwise white image contains.We use Python (version: 3.9.9)[19] with packages TensorFlow (version: 2.7.0) [2], Keras (version: 2.7.0) [6], and SciPy [20] (version: 1.7.3).4.1.Data.The data set consisted of a collection of images from one to ten black disks on white background and an explaining variable containing the correct number of disks for each image.For a number n = 1, ..., 10, a single image was created by first initializing a 300×300 null matrix corresponding to the domain G, choosing n centers c j ∈ [−0.7, 0.7]× [−0.7, 0.7] such that min j,k∈{1,...,10} {|c j − c k |} > 0.4 and n radii R j ∈ [0.1, 0.17], and changing each element of the matrix from 0 to 1 if and only if the distance c j was less than R j for some j.Out of 170 different images, 10 were included into the training set and the rest 160 into the test set.Augmented version of the training set with 40 images was then created by adding three versions of each image in the original training set by mapping them with the composed mapping f −1 •υ •g •f , where f, g, υ are as in (2.5)-(2.9)for α = 0.1 + 0.1i and k = π/3, α = 0.1 + 0.3i and k = π, and α = 0.3 + 0.3i and k = 3π/2, as shown in Figure 2. We also created another augmented training set of 40 images by using rotations of k degrees, k ∈ (−15, 15).The final images were scaled to the size of 128 × 128 pixels.

Convolutional neural network.
The CNN used here is the same as in [10].It is based on the U-Net architecture introduced in [18], which is commonly used in segmentation of medical images.While a typical U-Net contains first an expanding path and then a contracting path to perform the segmentation, our CNN only has the contracting path after which it returns a single numeric value.The contracting path consist of four sequences, each of which contains two convolutions and one pooling operation, and, after that, there are four dense layers.The activation function of the last layer is a linear function and, for all the other layers, the ReLu function is used.The CNN was trained on 130 epochs for the non-augmented data set and two augmented data sets by using Adam as the optimizer, the mean squared error (MSE) as the loss function with learning rate of 0.001, and validation split of 30%.

4.3.
Methods.The CNN is first initialized, trained with the non-augmented data set, and used to predict the values of the test set.Then we compute the squared error between the predicted number of disks and the real number of disks for each image of the test set.After that, the CNN is re-initialized to its initial state and the experiment is re-run by using the augmented training data sets instead.The three methods are compared by computing the MSE as the mean value of the squared errors and, to see if the differences in these means are statistically significant or not, the Student's t-test is performed for the distributions of the squared errors.

4.4.
Results.The MSE of the predictions of the test set was 1.742 when the CNN was trained for the data augmented by using conformal mappings.The corresponding MSE was 2.381 for non-augmented data and 2.095 for the data augmented with just rotations.According the Student's t-test between the squared errors of the predictions from the non-augmented and the augmented model, the difference between the MSEs was statistically significant with a p-value 0.0360.However, the difference between two different augmentation types was not statistically significant (p-value=0.196).Pearson's correlation coefficient was 0.884 between the predictions on the test set by the non-augmented CNN and the real numbers of disks, 0.883 for the rotation augmentation, and 0.924 for the conformal mapping augmentation (Fig. 2).

Discussion
Firstly, it must be noted that the experiment above is very simplified example and better results could be obtained by building a suitable algorithm instead of using a CNN.However, there are numerous practical reasons why a CNN that recognizes certain unusual disk-shaped areas are useful.In fact, this experiment was inspired by the CNNs detecting tumors from tomography images of cancer patients in [10,13,17].
It also should be taken into account that this method is quite complicated and designed for situations where the typical transformations cannot be used.For instance, if some of the disks would be so close to edges of the images in our experiment that they would be cropped completely or partially out when the square-shaped images are directly rotated and re-calculating their number is not possible, using this new method rather than the usual rotation is justified.The rotation used as a comparison augmentation method in our experiment did not crop the images but rather included more background.Given how commonly CNNs are used nowadays, there are likely many practical examples of cases where information could be lost if the typical rotation is used.Note that these mappings can be also extended to map any rectangle-shaped image into a disk or vice versa because the rectangle can be very easily stretched into a square.
Another reason for using this method is wanting to utilize the properties of the Möbius transformation in particular.Because of this mapping, the images change in more complicated ways under this augmentation method than they do in simple rotations.We can see that the right-most disk in Figure 2(D) is larger than any of the disks in the original image in Figure 2(A).By increasing the absolute value of the point α used to define the Möbius transformation, the differences between the images before and after the composed mapping are also increased.Still, all these disks stay circular because Möbius transformations can only map circles into lines or circles and the disks stay inside the edges of the image.The use of Möbius transformations in image augmentation has been also studied by Zhou et al. [21] but they did not use other conformal mappings and their augmented images therefore contain much empty background outside the original photographs.
One alternative method is using general adversarial network (GAN) augmentation first introduced in 2014 by Goodfellow et al. [9].GANs are a class of neural networks that generate synthetic samples resembling the real images of the original data set.However, it might be difficult to predict what sort of augmented data GANs produce while the use of conformal mappings only distort the images.This means that the augmentation based on the conformal mappings preserves the number of disks in the images of the data set used here, while GANs might change the number of disks.Furthermore, GANs also require some amount of data so that they can be trained for their work, while the amount of the data does not affect how the images change under the conformal mappings.Our method of augmentation could be also compared with such procedures that use prior information about the data distribution in augmentation [5,15].

Conclusion
We used conformal mappings to create a new way to augment the image data of a CNN and, according to our result, this method both works and produces better predictions than a CNN trained with non-augmented data set.
Data and code availability statement.Available at https://github.com/rklen/Conf_map_augmentation Conflict of interest statement.On the behalf of all authors, the corresponding author states that there is no conflict of interest.Funding.The first author was financially supported by the Finnish Culture Foundation and Magnus Ehrnrooth Foundation.

Figure 1 .
Figure 1.An image of parallel line segments before and after different mappings, including the conformal mapping f from the original square onto the unit disk, the Möbius transformation g defined with α = 0.3 + 0.3i, a rotation υ with k = π/3, and the inverse mapping f −1 .

2 Figure 2 .
Figure 2.An image with n = 7 disks and three augmented versions of this image that have been created with the composed mapping f −1 •υ •g •f , where the mappings f, g, υ are as (2.5)-(2.9)for the specified choices of α and k.

Figure 3 .
Figure 3.The predictions of the augmented CNN against the real values of the test set with the least squares regression line.The slope of the line is 0.793 and the intercept is 1.846.The correlation between the predictions and the real values is 0.924.