An overview of mixing augmentation methods and augmentation strategies

Deep Convolutional Neural Networks have made an incredible progress in many Computer Vision tasks. This progress, however, often relies on the availability of large amounts of the training data, required to prevent over-fitting, which in many domains entails significant cost of manual data labeling. An alternative approach is application of data augmentation (DA) techniques that aim at model regularization by creating additional observations from the available ones. This survey focuses on two DA research streams: image mixing and automated selection of augmentation strategies. First, the presented methods are briefly described, and then qualitatively compared with respect to their key characteristics. Various quantitative comparisons are also included based on the results reported in recent DA literature. This review mainly covers the methods published in the materials of top-tier conferences and in leading journals in the years 2017-2021.


Introduction
In recent years Deep Learning (DL) research has made an incredible progress in solving complex image-related problems. This progress and the accompanying raising interest in this area were greatly motivated by the first widely-known successful application of Convolutional Neural Networks (CNNs) to the problem of image classification in 2012, when AlexNet [39] clearly outperformed shallow methods. There were three main reasons for this success: advancement in deep network architectures, the availability of huge computation power, and access to large amounts of training data. An acclaimed example of a large data set that fostered further development in this area is ImageNet [9] around which a competition called ImageNet Large Scale Visual Recognition Challenge (ILSVRC) was organized. This competition enabled tracking the performance of various CNN models over time and spurred the development of several renowned architectures like VGG [64], GoogleNet [66] or ResNet [22]. Throughout the years one could observe how the classification top-5 error plummets from more than 25% in 2010-2011, when shallow methods were used, to less than 5% in 2015 with the use of DL.
CNNs are successfully used in various computer vision tasks like image classification, object localization, object detection, image segmentation, action recognition in videos or image captioning (in the last task usually in combination with Recurrent Neural Networks). The main focus of this survey is on image classification problem, since the vast majority of mixing augmentation methods were designed specifically with this task in mind.
Generally speaking, when creating any DL algorithm for image categorization the goal is to train the model to correctly predict classes for new (previously unseen images) based on available training data. This ability is called generalization as the model is able to generalize the knowledge extracted from training images to correctly classify images not encounter during training. A failure to generalize is referred to as model over-fitting and is an unwanted property. The goal of DL practitioner is to find a model that is complex enough to learn based on all training samples (training examples) but simple enough so as not to memorize these samples. In many practical cases a viable approach is to create a model that is overcomplex for a given training set but, at the same time, introducing mechanisms that prevent the algorithm from over-fitting. There are various ways to increase generalization ability of DL models (and avoid over-fitting), for example by means of regularization mechanisms [41] such as dropout, weight decay or batch normalization, as well as an application of specific training strategies like transfer learning or zero-shot learning.
Although many tasks can be solved using the data sets, model architectures and computational power available right now, in quite many domains the lack of sufficient amounts of training data remains one of the key obstacles in DL application. This problem is especially severe in the areas where gathering of labeled training samples is costly or requires special skills (e.g. medical images), or the scale of the business case is not large enough to justify an investment into labeled training data gathering.
One of possible remedies, leading to partial alleviation of this problem, is data augmentation (DA), which is one of the regularization methods [41] designed to create additional observations based on available ones and thus increase the size of the training set. The focus of this survey is on DA methods that apply certain transformations to the source (original) images in order to create new training examples. Both, data warping (i.e. applying any label preserving transformation to the image) and synthetic over-sampling (i.e. creation of artificial samples) approaches are considered. Additionally, we discuss strategies for selection of the augmentation technique(s) best suited for a given task.
This review paper complements recent DA survey [62] that covered a full spectrum of various data augmentation methods, ranging from traditional augmentation methods to Generative Adversarial Network (GAN) based augmentations. The paper significantly extends the above-cited work in the area of mixing augmentation methods and augmentation strategies, which are only briefly covered in [62].
The main source of papers for this review are recent top-tier conferences related to Artificial Intelligence, Computer Vision, and Machine Learning, from the period 2017-2021. Certain exceptions include the cases of relevant and highly influential papers published in other venues.

Taxonomy of presented data augmentation methods
Presented methods are divided into two high level categories: data augmentation methods and strategies for data augmentation method(s) selection. The first group is further divided into approaches that consist in erasing part of the image and those relying on image mixing. Technically, the former methods can also be thought of as mixing ones, in which a given image is mixed with an empty image. However, due to their somewhat different specificity, and following the taxonomy generally agreed within the DA community, these two groups of methods are considered separately.
Strategies for data augmentation selection are mostly applied in the literature to traditional DA techniques, i.e. label preserving geometric transformations or operations in the color space, and only occasionally to more advanced augmentation methods.
While these two areas (mixing DA methods and DA selection strategies) are compared in the literature they are rather not combined within one DA system. In this review both groups of methods are intentionally considered together, since we hypothesize that future development of DA research will largely include application of DA selection strategies to advanced mixing or GAN-based augmentation methods.

Notation and general remarks
Let us start with an introduction of the base notation. First of all, mixing DA techniques are divided into two main classes: those that mix images using pixelwise weighted average (referred to as pixel-wise mixing) and those that mix images spatially by means of extracting patches from different images and joining them together (referred to as patch-wise mixing methods). Examples from both classes are presented in Figure 1. Furthermore, whenever a result of applying any given DA technique is presented, the baseline result or simply baseline would refer to the respective result obtained using the same architecture, trained on the same data and using the same training hyper-parameters, but with no application of the analyzed DA method. For the sake of consistency all DA methods will be written in italic font, e.g. Mixup, CutMix or Attentive CutMix.
The remainder of the survey is organized as follows: section 2 presents DA methods that rely on erasing part of the image and section 3 introduces the image mixing methods. The mixing methods are further qualitatively compared and analyzed in the context of their specific aspects (e.g. in which part of the learning model the DA is applied, how many images are involved in a single DA sample preparation, whether DA method is pixel-wise or patch-wise, whether or not the DA technique mixes labels, and other) in section 4. A thorough quantitative experimental comparison of the methods is presented in section 5. The vast majority of results refer to CIFAR-10 [37] and CIFAR-100 [38] data sets which are most frequently used in DA methods assessment. Section 6 presents and compares strategies for optimal DA method selection. The last section concludes the paper by summarizing its main findings and pointing some possible future prospects in DA domain -notably application of the DA selection strategies to the class of mixing DA methods.

Data augmentation by erasing part of the image
Methods in this group rely on removing part of the image either by masking it out [11,84] or replacing with certain noise [50].
The main motivation behind Cutout is better utilization of the entire context of the image, which is especially important when dealing with partial occlusion. Due to masking a particular region of the image in the input layer, the network needs to recreate the missing information based on the context. Conceptually, the technique is similar to dropout [40] and can be thought of as an extension of dropout to the input layer. However, there are two key differences: firstly, Cutout is applied exclusively to the input layer (not to all layers as dropout is) and secondly, it drops regions of the input image, rather than individual pixels. While there are variants of dropout that work with continuous regions, e.g. DropBlock [15], they are applied to randomly selected regions, different in every layer, so, unlike in Cutout, the key regularization mechanism in this case relies on randomness.
The method uses just one hyperparameter, the size of the filter. The authors of [11] claim to have considered one more hyperparameter -the shape of the filter, but as the size turned out to be much more important, all experiments are focused on size optimization of the cutout, with a square shape assumed by default. Based on the experiments, the relation between model accuracy and filter size has a parabolic shape -the accuracy increases with an increase of the filter size, up to an optimal point after which masking out too much information deteriorates the resulting accuracy.
An important point to make is that Cutout's occlusion ratio (a proportion of pixels masked to all pixels) can vary even for filters of the same size due to various possible positions of the center point of the cutout region, which is selected randomly. This diversity of the effective filter size is crucial for achieving high performance. An alternative approach is to always select a center point of the filter far enough from the image edges, so as the entire mask would fall within the image, but apply the method with 50% probability [11].
Another erasing augmentation approach, Random Erasing [84] is a deep dive into Cutout and its high level description is very similar to that of Cutout. However, Random Erasing is a more flexible approach and transforms the image in a slightly different manner. The method admits rectangular shape of the masking patch and its effective application depends on 3 (instead of 1 as in Cutout) hyperparameters: the erasing probability p, the range of area ratio S e /S ∈ [s l , s h ] (1) between the masked region and the whole image, which controls the patch size, and the range of aspect ratio r e ∈ [r 1 , r 2 ] (2) between height and width of the masked region (3), which controls the patch shape.
Additionally, the method fills in the erased region differently to Cutout. Each pixel's value within the masked region is replaced with a randomly selected value between 0 and 255 (other less successful approaches are discussed in [84]).
Furthermore, the entire patch area must not extend beyond the original image range, so the following patch selection process (1)-(4) is repeated until an appropriate patch is found.
H e ← S e × r e , W e ← S e r e (3) where S is the image size, S e , r e , H e , W e are the area, the aspect ratio, the height and the width of the filtering rectangle, respectively, and x e , y e indicate the upper left corner of this rectangle. The effects of Cutout and Random Erasing are compared in Figure 2.
Generally, Random Erasing yields better results than the baseline for all reasonable parameter settings, however, the final parameter values need to be optimized for a given data set. For instance, for CIFAR-10, it is recommended in [84] that the values of p, s h and r 1 be equal to 0.5, 0.4 and 0.3, respectively. The method is generally not the best choice for a standalone DA technique. It is inferior, for instance, to random flipping [62] or random cropping [62], although when added on top of any of them, as a complementary augmentation method it improves classification accuracy.
The next erasing method, Patch Gaussian [50], is a hybrid of Cutout and a Gaussian noise augmentation procedure. Adding Gaussian noise to the image region selected by Cutout improves performance of the trained model on corrupted images, in addition to its efficient dealing with partial occlusion. The motivation behind Patch Gaussian is twofold: (a) Cutout is generally able to increase the model's accuracy but without ensuring its robustness, while (b) addition of Gaussian noise increases robustness but hurts accuracy of the model. The aim of Patch Gaussian is to get the best of both worlds and combine high accuracy on a held-out set of images with the robustness to corrupted images. The method is parameterized by a patch size (same as in Cutout) and the maximum standard deviation of Gaussian noise. Its effects are visualized in Figure 3 and compared with application of Gaussian blur. In order to confirm that high accuracy and robustness to noise are indeed caused by application of Patch Gaussian augmentation and are not just a result of stacking data augmentation techniques, in ablation study presented in [50] Patch Gaussian was compared with an application of Cutout followed by the Gaussian noise addition, as well as with either of these two techniques applied alone to half of the batches. Among these approaches Patch Gaussian achieved the smallest error on corrupted data while maintaining high accuracy on noncorrupted test samples.
An interesting approach is a combination of Patch Gaussian with regularization strategy, or another DA technique. Using Patch Gaussian together with any of the following strategies: weight decay [16], label smoothing [67], Drop-Block [15], or AutoAugment [7] leads to decreasing of the error on corrupted data whilst making the error on non-corrupted data just marginally higher than the baseline.
The genre of erasing methods is relatively small since researchers quickly shifted their attention towards methods described in section 3 that mix images in a patch-wise manner. Patch-wise mixing has similar effects to erasing part of the original image, but without reducing the training signal provided by each image. In case of mixing methods a masked part of the image is replaced by a part of another image, thus providing a stronger training stimulus.

Data augmentation by image mixing
Image mixing DA methods rely on blending two input images and their corresponding labels according to the following equations: where x 1 , x 2 are original input images, y 1 , y 1 are one-hot label encodings, λ is a mixing ratio, B is a mixing mask matrix suitable for both pixel-wise and patch-wise mixing and I is an identity matrix of the same dimensionality as B. denotes element-wise matrix multiplication operation. The vast majority of approaches described in this section are built around equations (5)-(6) and mainly differ by the method of λ selection and construction of matrix B. Figure 4 presents a map of the mixing methods, indicating for each of them the publication date, certain key characteristics and relations to other methods. In the remainder of this section the methods presented in Figure 4 are described and discussed in more detail.
Mixup constructs new training samples according to equations (5)-(6) by means of the same weighted mean of the images and their labels, i.e. the entire matrix B is populated with λ. The underlying assumption of Mixup is that linear Fig. 4. Image mixing DA methods presented on a time scale, with key characteristics and dependencies indicated. Dotted regions separate methods in which mixing takes pixel-wise form (Pixel-wise) from those with spatial mixing (Patch-wise), and those with mixing applied not to a pair of images, but either just one image and its transformed version or more than 2 images (Other than 2 images). Directed lines indicate inspirations (dotted lines) or direct extensions (solid lines) of the methods. Each method appears once except RICAP which in a patch-wise manner mixes 4 images, hence appears in both Patch-wise and Other than 2 images groups.
interpolation of feature vectors should lead to an adequate linear combination of the associated labels. This linear combination of images / classes is controlled by λ, e.g., λ = 0.5 leads to averaging the images and their corresponding labels, while λ ∈ {0, 1} preserves one of the original images and its label.
The authors of Mixup additionally checked whether it is more beneficial to sample image pairs from the entire data set vs. sampling them solely from observations belonging to the same class, and whether is it preferred to apply weighted average (5)-(6) in the input layer vs. its application in subsequent layers. The results presented in Table 1 prove that it is more beneficial to mix images selected at random rather than within the same class and that mixing images in the input is more advantageous than mixing their latent representations.
Although [82] is considered to be a founding paper in the image mixing augmentation research line, historically, there were three papers published prior  Table 1. Part of the ablation study presented in [82]. Experiments were performed on CIFAR-10 data set and measured the median test error over the last 10 epochs (the rightmost column). The ERM denotes Empirical Risk Minimization which is the baseline approach in this study. The first experiment checks whether images in a pair should be selected at random (RP) or should rather represent similar observations (KNN). The next one verifies whether it is more beneficial to apply data augmentation procedure to the latent representation (Layers 1-5) or to the input images (RP result in the first experiment).
to Mixup that tackled the problem of mixing images [10,31,43] but approached it from different angles.
Smart Augmentation (SA) [43], the first of these methods, attempts to learn the way of mixing that minimizes the loss. This learning-based approach is quite specific since the vast majority of the current augmentation papers focus on applying effective, yet arbitrary chosen, mixing techniques. SA employs two separate networks: the first one (called Augmentor) is responsible for mixing two or more images, while the other one (Target) is responsible for image classification. Selection of images for mixing is performed randomly and is limited to observations from the same class. Parameters of both networks are updated based on the loss of the Target network. The loss of the Augmentor network is furthered extended with an MSE-based comparison of the mixed image with randomly selected image (from the same class) other than the ones used to create the mixed sample. Conceptually, the approach is flexible and architectures of both networks can vary as long as the Augmentor network returns the output of the same size as the input. Once the training is completed the Augmentor is discarded.
Mixing images in a learned Feature Space is proposed in the second of pre-Mixup papers [10]. The authors did not propose any particular name for their method and referred to it as "Data set augmentation in Feature Space". For the sake of brevity, we coined the name Feature Space to this method. In Feature Space augmentation, latent features are generated using a sequence-to-sequence model which enables cross domain application of the method, e.g. to both text and image classification. The set of operations performed on those latent features encompasses adding a random noise, as well as interpolation and extrapolation of features from different observations. Technically, the method uses a sequence autoencoder (a stacked LSTM [45] with encoder and decoder layers) to generate the training features. Whilst method execution an image is propagated through the encoder network and subsequently its hidden representation is either augmented with Gaussian noise or mixed with a similar observation. The mixing process consists in finding K nearest neighbors in the feature space with the same class label as the selected image. For each pair of hidden representations (of a given image and one of its K nearest neighbours) the mixing is performed according to one of the following equations: where c j is the vector representing latent features of a given image, c k is its neighboring vector in the latent feature space, and λ ∈ [0, 1] controls the degree of interpolation (7) and extrapolation (8), respectively. It is suggested in [10] that extrapolation should be preferred over interpolation as it is able to create samples that display higher variability.
The third DA approach that appeared prior to Mixup, SamplePairing [31], is a relatively straightforward technique that constructs a new data sample by a pixel-wise averaging of two randomly selected images. Unlike Mixup it does not consider the label of the second image and simply assigns the label of the first image to the newly created mixed training sample.
In the image classification task SamplePairing is used as a pre-training mechanism which is turned on and off at regular intervals before being completely disabled at the end of the pre-training phase. The effects of Mixup and Sam-plePairing are compared in Figure 5. Both methods mix images in a pixel-wise manner: Mixup with a user defined mixing ratio, in this case 0.7 and Sample-Pairing with a fixed ratio of 0.5. Another approach related to Mixup is Between-Class learning (BCL) [70], published in the same year. BCL was initially proposed in the context of mixing sound waveforms [71] and then adjusted to image domain [70]. A major distinction between Mixup and BCL is that the latter mixes images from different classes only, whereas Mixup selects two images at random, which makes it possible to mix two images from the same class.
There are two BCL implementations proposed in [70]. The first one follows eq. (5) and the other one (named BC+), instead of (5) applies eq. (9), which takes into account characteristics of the image, i.e. pixel-wise image mean and standard deviation: where λ is the mixing ratio, σ 1 , σ 2 are pixel-wise standard deviations, and µ 1 , µ 2 are pixel-wise means, respectively. Both methods apply eq. (6) to create mixed label encoding. During CNN training the goal of the network is to predict the newly created mixed label encoding which is equivalent to predicting the mixing ratio used to create the mixed sample. For this task standard categorical cross entropy loss function is replaced with Kullback-Leibler divergence, which proved to be more effective in the ablation study presented in [70]. Based on the ablation experiments it can also be concluded that arbitrary choosing one of the two possible labels for the mixed sample, instead of their linear combination, worsens the performance. Furthermore, while mixing examples from different classes renders better results, the performance also improves (compared to the baseline) if mixed images belong to the same class. Similarly to Mixup study [82], [70] found that application of the mixing process in the input layer is more beneficial than its usage in subsequent layers. In the latter case, for deeper CNN layers mixing may even deteriorate performance.
Going back to the Mixup method, subsequent related research was mainly concentrated along two axes: direct improvements of the method [19,73] (discussed in section 3.1), and questioning the efficacy of pixel-wise mixing [42,65,72,75,80] (discussed in section 3.2).

Direct extensions of Mixup
AdaMixup [19] was the first direct extension of Mixup. The method attempts to learn a better than Mixup mixing distribution to avoid the so-called manifold intrusion problem, i.e. a situation in which a synthetically created image coincides with a class which is different from any of the two classes assigned to images being mixed.
AdaMixup extends Mixup by adding 2 neural networks called Intrusion Discriminator (ID) and Policy Region Generator (PRG). The first one is responsible for predicting whether or not the resultant mixed sample would lead to manifold intrusion and the role of the latter one is to propose the mixing policy. All three networks are trained jointly using the following loss function: where L D , L D and L intr are standard loss, data space regularization loss and intrusion regularization, respectively. H, ϕ and {π k } denote the main classifier, ID and PRG, respectively. Another direct extension of Mixup is Manifold Mixup [73], which is motivated by an observation that Mixup produces sharp decision boundaries in latent image representations. Manifold Mixup addresses this issue by performing image mixing operations in hidden layers with latent feature representations. More precisely, the method proposes to select randomly one layer in a network, either hidden or input, and then process two random mini-batches of data until reaching the selected layer, at which point the two mini-batches are mixed according to (5). Afterwards, the processing is continued until the output layer is reached where the loss is calculated and all network parameters are updated according to the gradients. If randomly selected layer happens to be the input one, the method is equivalent to Mixup.
Manifold Mixup achieves comparative or higher performance than Mixup and AdaMixup. Note that the results are somewhat contradictory to the conclusion presented in [82] that mixing should be applied in the input layer.

Patch-wise extensions of Mixup
The other stream of Mixup follow-up papers, which question the efficacy of mixing pixels linearly, share many properties with the augmentation methods focused on occluding parts of the image, described in section 2. A primary approach here is Mixed-Example method [65] which tests various patch-wise mixing approaches to conclude that they work comparably well to pixel-wise methods and even excel them in certain settings.
An underlying assumption of Mixed-Example [65] is that linear interpolations represent just a small subset of mixing operations that can potentially be used for data augmentation. Consequently, the following patch-wise mixing schemes are proposed in [65]: -Vertical Concat -a method where bottom and upper parts of the new image come from two different images. This effect can be implemented with eq. (5) where B has 1s in the bottom part and 0s in the upper part or vice versa; -Horizontal Concat -same as Vertical Concat, but in reference to the left and right parts of the new image; -Mixed Concat -a combination of both the above methods in which a new image is constructed from 4 patches and patches on each diagonal come from the same source image; -Random 2x2 -a randomized version of Mixed Concat where the source of each of the 4 patches is independently selected at random (either the first or the second image). Random 2x2 implements an idea similar to RICAP method described below in this section, which combines 4 images and each patch comes from a different image; -Random Square -a method in which a randomly chosen square from one image is placed on the other image. Random Square is similar to CutMix method described below in this section; -Random Column Interval -a generalization of Horizontal Concat where a randomly selected vertical stripe is picked in one image and pasted onto the other image; -Random Row Interval -an analogous generalization of Vertical Concat; -Random Columns/Random Rows -a further generalization of Random Interval approaches in which a subset of rows/columns, respectively is chosen in one image and pasted onto another image; -Random Pixels -a new image is created based on pixels sampled uniformly from both images; -Random Elements -a new image is created based on color channels values of the respective pixels sampled uniformly from both images. Mixed-Example paper [65] additionally proposes several strategies which combine pixel-wise and patch-wise mixing, e.g. a blend of Vertical Concat and Mixup, where Mixup is applied to one of the input images and followed by Vertical Concat procedure.
Several subsequent papers in this line of research either build on [42,72,75] or explore in-depth [80] one of particular augmentation methods presented in [65]. CutMix [80] mixes images by occluding part of one image with a patch extracted from the other image. SmoothMix [42] and Attentive CutMix [75] extend CutMix by either smoothing sharp edges in the resultant image (SmoothMix ) or by using a specific way of patch selection (Attentive CutMix ). Saliency Mix [72] attempts to achieve the same properties as Attentive CutMix but using saliency information. All four above-mentioned methods are discussed in more detail below and the effects of their application are illustrated in Figure 7   CutMix [80] is motivated by the Cutout method (described in section 2), however, unlike Cutout which simply clears out part of the image with possible loss of relevant information, CutMix introduces new information to the resultant image extracted from the other image it is mixed with. The method is regarded as a strong alternative to Mixup.
A patch size in CutMix is proportional to the image size and calculated according to the following equations: where r w and r h are the width and the height of the patch, respectively, W and H are the width and the height of original images and λ is the mixing ratio. According to [80] the method should be applied in the input layer (to raw images) and its application to hidden layer representations degrades performance.
SmoothMix [42] is a direct extension of CutMix that attempts to avoid sharp edges, i.e. unnatural changes in pixel values around the implemented patch.
SmoothMix creates a new observation according to eqs. (5)-(6) with a specific form of matrix B which has 0s at the edges of a patch and 1 in its center. B implements a smooth transition from the edge pixels towards the patch center, i.e. its elements change gradually from 0 to 1 along with moving further away from the edges and approaching the central region of the patch. Both, circular and rectangular patches were tested in [42] with the final preference for a circular one.
Attentive CutMix [75] is another extension of CutMix. The method divides both images into patches (typically a 7 × 7 grid) and in one of the images selects some number of patches that activate the most the respective class-related output. Next, the method pastes those most-representative patches onto the other image in the respective places. The label of the new image is calculated based on the proportion of the pixels corresponding to each of the two source images. Attentive CutMix addresses one of the problems with CutMix, i.e. susceptibility to copying a non-representative part of the image (e.g. background) and consequently producing an example with a noisy label. Attentive CutMix, on the contrary, puts special attention to selection of the meaningful and classrepresentative patches to be transferred to the resultant image. One disadvantage of Attentive CutMix, compared to other discussed methods, is the requirement of having a pretrained feature extractor that is used for selecting the most relevant patches.
The idea of selecting the most representative patches is also implemented in SaliencyMix method [72] which uses saliency information to find the most representative pixels and then, similarly to CutMix, selects a patch of a size determined by λ. In SaliencyMix the patch is either centered on the most salient pixel (if the entire patch fits within the image) or keeps this pixel within the patched area (otherwise). The method follows equations (5)-(6) and is implemented by setting the values of matrix B to 1 within the patch area (selected based on the saliency peak region) and to 0 elsewhere. Figure 8 presents an example of saliency map and the effect of SaliencyMix application.
In [72] four saliency calculation methods were considered, relaying either on statistical approaches [1,27,52] or a learning based approach [57], and eventually decided to use the statistical saliency method from [52] as it offered slightly better classification accuracy (with SaliencyMix ) and, additionally, is invariant to image size, as opposed to learning based methods which are limited by the input size of the architecture used. Various placements of the salient patch of the image were also tested, including placing the patch on salient / non-salient / corresponding part of the other image, with a conclusion that the choice of a corresponding place is the best option in terms of regularization.
The following two sections consider methods that deviate from certain baseline assumptions that were commonly considered in the methods discussed so far. Section 3.3 presents methods that does not follow equation (5) [29,36] and section 3.4 discusses the methods that use the number of images other than two to produce a mixed sample [25,32,35,68,87].

Beyond equation (5)
The next method cannot be definitively classified as either pixel-wise or patchwise. Puzzle Mix [36] is motivated by the same principles, as the previouslydescribed Attentive CutMix and Saliency Mix methods, namely ensuring that the resultant image contains patches that are relevant for both classes. Similarly to Saliency Mix this goal is achieved through utilization of the saliency information [63] to determine the most important patches in both images, though Puzzle Mix additionally performs an optimal transportation of these most salient parts that would maximize their exposure in the resultant augmented image. This results in placing a relevant patch in the target image not in the place that corresponds to its location in the source image but in a destination that contains low saliency features compared to the rest of the target image. The overall goal is achieved through jointly seeking an optimal mixing mask and an optimal placements for the patches coming from both source images in the augmented image. Consequently, instead of (5) the following equation is used for creating a mixed sample in Puzzle Mix : where B is a mask with b ij ∈ [0, 1] and 1 , 2 are n×n transportation matrices.
i , i = 1, 2 encodes the placement of patches extracted from the source images x i , i = 1, 2 in the augmented sample. The mixing ratio λ is defined as λ = 1 n ij b ij . Since there are two goals in Puzzle Mix the problem is solved via iterative alternating of the following two phases: (1) minimization of B and (2) simultaneous optimization of 1 and 2 . Although certain improvements of both CPU and GPU implementations, as well as down-sampling strategies for mask optimization are proposed in [36], the method still requires substantially more computational resources than previous DA techniques, e.g. Mixup or CutMix. On the other hand, whenever Puzzle Mix is compared with other methods in the same experiment setup, it achieves higher accuracy. Numerical results are presented in section 5.
The other method that departs from eq. (5) is SnapMix [29]. The method is dedicated to fine-grained image classification problem, in which classes differentiate by details only like, for instance, in CUB [74], Stanford Cars [78] or FGVC Aircrafts [51] datasets. Consequently, SnapMix assumes that visual features indicative for a given class may take a small part of the image, in which case traditional methods like Mixup [82] or CutMix [80] would not be effective. An example of SnapMix application is depicted in Figure 9. SnapMix modifies CutMix in the two following aspects: the way of label vector calculation for the mixed sample, and independent selection of the size and placement of the patch in each of the two input images (the patch is not automatically pasted in the corresponding location of the target image).
SnapMix method can be expressed as follows: where B λ 1 and B λ 2 are two binary masks containing random box regions with the area ratio λ 1 and λ 2 , respectively, and T θ is a function that maps the patch from x 2 onto the patch in x 1 .
In order to calculate the label for the augmented image (13), the class activation map (CAM) [85] is used. CAM is a transformation applied on top of the last convolutional layer of the network so as to point locations of the classdiscriminative features. The output of CAM is normalized to get Semantic Percent Map (SPM) using eq. (14): The label of the image created according to (13) is finally calculated as follows: where p 1 , p 2 ∈ [0, 1] are partial labels assigned to classes corresponding to the class of images 1 and 2, respectively.
An interesting aspect of the method is the lack of a constraint that partial labels should sum up to 1. This way it is possible to indicate in the label that as a result of the operation one image has become relatively more / less relevant than previously (for instance, when the patch masked a discriminative feature in that image).
Another relevant SnapMix feature is the ability to offer more effective augmentations as the training process progresses. This is due to the fact that CAM works on top of the classifier, and therefore, becomes more accurate along with the classification improvement, making the resulting augmented samples more effective.

Mixing other than 2 images
Until now all described methods used 2 images to create a mixed sample. The next group of approaches [25,32,35,68,87] veered off in the other direction. RI-CAP [68] combines 4 images in a pixel-wise fashion while AugMix [25] and Style Augmentation [32] use one image in the augmentation process. MixStyle [87] on the other hand, relies only on the per channel statistics, like mean and standard deviation calculated from several hidden layers of the second image, disregarding the input pixels. A potential breakthrough is proposed in Co-mixup [35], one of the youngest methods, that performs mixing of the entire mini-batch rather than each pair separately. All five methods are briefly discussed below.
RICAP augmentation process consists of the three following steps. Firstly, four images are randomly selected -one for each of the four parts of the new image: upper left, upper right, lower left and lower right. Secondly, the images are randomly cropped with respect to a boundary position point, i.e. a point that determines the area a given image will occupy in the newly created augmented image. The boundary point BP = (w, h) is calculated using the following equations: where I x and I y are width and height of the original image, respectively. For selecting the BP coordinates the ratios w and h ale sampled from Beta distribution parameterized by β ∈ (0, ∞). Based on BP the cropping sizes The new class label is calculated as a combination of four original one-hot encoded labels weighted by the relative size of the area assigned to each of the four images in the newly created image. An example of RICAP application is presented in Figure 10. The next two methods apply mixing procedure to the original image and its transformed version(s). AugMix [25] starts off with application of traditional augmentation operations (e.g. translation, shear, rotation, etc.) to the input image. More precisely a set of k (k = 3 by default) augmentation chains, each composed of 3 operations, are selected and applied independently to k copies of the original image. The resulting k images are mixed together linearly with random weights. Next this augmented image is mixed linearly with the original non-augmented image. AugMix creates 2 images in the above-described manner.
In order to enforce a consistent embedding across diverse augmentations of the same input image a consistency loss in the form of Jensen-Shannon (JS) divergence [61] is included in the model loss function (17): where M = (p orig + p aug1 + p aug2 )/3 and p orig , p aug1 , p aug2 are posterior distributions of the original sample and its augmented variants.
Additionally, two alternative approaches were tested in [25] that use respectively 1 or 3 augmented samples to calculate the consistency loss. The former setup performed the worse whereas the gains from the latter one were marginal and did not justify the related computational complexity increasing. The results of AugMix application are visualized in Figure 11.
The other single-image method, Style Augmentation [32], randomly changes the texture, contrast and colors of the image while preserving the objects' shapes and the semantic meaning of the image. This is achieved by using two neural networks: P and T , described in detail in [14]. Network P is trained on image styles and provides style embeddings which are subsequently used by network T that performs style transfer operation.
In practice, for the sake of computational efficiency, Style Augmentation does not operationally use network P to provide a style embedding from a randomly sampled image, but instead randomly samples a style embedding vector directly from a distribution with mean and covariance matching those used for training network P . Such a random sampling (from appropriate distribution) simulates the process of choosing an image from the training data set and calculating its style embedding, without computational load related to its actual calculation.
Additionally, in order to control the strength of the augmentation process, a randomly sampled embedding vector is linearly mixed with a style embedding of the input image. The effective embedding vector z used for style transfer is therefore defined in the following way: where P is the style predictor network, c is the input image, and µ, Σ are the mean vector and covariance matrix of embeddings from the training set. An example of Style Augmentation application is presented in Figure 12. Another method, MixStyle [87], attempts to achieve a similar goal as Style Augmentation but using a different approach to style transfer. The method is dedicated to the problem of Domain Generalization (DG), i.e. construction of classifiers robust to domain shift, able to generalize to unseen domains. To this end MixStyle does not mix pixels but instance-level feature statistics of the two images. These statistics extracted from early layers of a CNN are calculated in the following manner: where γ mix , β mix are the mix feature statistics, λ is the instance-wise weight sampled from Beta(β, β) distribution parameterized with β ∈ (0, ∞) and µ(x i ), σ(x i ), i = 1, 2 are mean and standard deviations computed for the set of all elements in each color channel, in each instance. Once the mixing statistics are calculated, the mixed sample is defined as: Equation (22) is inspired by the arbitrary style transfer [30] and is applied with probability 0.5. Otherwise, the instance is not augmented. It is recommended in [87] to apply MixStyle to multiple lower level layers which yields the best performance. The optimal combination of layers depends on the final task and including the last layer always deteriorates the performance. An interesting observation is that if multiple layers are selected the same sample can be processed with different mixed statistics as the shuffling process that selects pairs of samples is independent in each layer. This property constitutes additional regularization mechanism in MixStyle.
The final method in this area, Co-Mixup [35], lifts the idea of mixing from the level of pairs of images to the level of entire training mini-batches. The method, similarly to Saliency Mix, uses saliency information to generate mixed samples but at the same time looks at the whole mini-batch to encourage diversity among the constructed mixed samples. Since Co-Mixup works on the entire mini-batch equations (5)- (6) are no longer valid and the mixing process is described as follows: where z j ∈ L m×n for j = 1, ..., m and L m×n = { l L |l = 0, 1, ..., L} is a discretized mask of dimensions equal to the number of images in the mini-batch (m) and the number of regions into which the image is divided into (n), for optimization purposes (see below). x B is the mini-batch and g : R m×n → R n returns a column-wise sum of matrix z j x B for j = 1, ..., m . As z j is a 2D matrix, one can interpret the k th column of z j (z j,k ∈ L m ) as the mixing ratio for m inputs from a mini-batch at the k th area.
Additionally an optimization procedure is applied as the method aims to maximize the exposed saliency, maintain local smoothness within the images (adjacent locations in the mixed sample are similar to one another) and also encourage diversity among the constructed mixed samples [35].

Comparison of DA methods based on particular properties
In this section the aforementioned mixing methods are compared on the basis of certain aspects which refer to their operational properties, efficacy and computational complexity. The key characteristic features of the methods are summarized in Table 2.

4.1
Where is the augmentation applied?
All methods apply the DA procedure in the input layer except three: Feature Space, Manifold Mixup and MixStyle (cf. the first column of Table 2). Feature Space uses additional encoder-decoder network to create image embeddings which are then mixed. Manifold Mixup is a direct extension of Mixup with the mixing mechanism applied to a pool of eligible layers that includes both input and hidden layers. The layer in which mixing is performed is selected randomly.
MixStyle does not actually mix images but the instance-level feature statistics extracted from early layers of a CNN.

How is the augmentation applied?
Generally, there are three ways in which the DA techniques that we discuss are applied. The first one, which includes Smart Augmentation, AdaMixup, Attentive CutMix, Saliency Mix and Style Augmentation, is to rely on an auxiliary mechanism to aid the augmentation process. In the case of Smart Augmentation a CNN based augmentation model precedes the classification network and is trained together to optimize the effects of augmentation for a given problem.
In AdaMixup an additional network is used to predict whether or not mixing a given pair of images will result in manifold intrusion. In Attentive CutMix a CNN is utilized to detect the regions that are most representative for a given class, whereas Saliency Mix achieves the same objective by using the saliency information. Style Augmentation applies a style transfer network to change visual attributes of an image without changing its meaning.
In the second group, encompassing all but two of the remaining methods, an augmentation technique follows a certain rule or procedure which is applied in some randomized way. These rules differ in terms of complexity and the number  Table 2. Comparison of data augmentation techniques with respect to particular baseline properties: where, how and in which form the augmentation is applied, whether or not it mixes labels or utilizes a specific loss function, how many images take part in a single augmentation and what is the computational complexity of the method. Iinput layer, Hhidden layer, AMauxiliary mechanism (either network or other), Ooptimization, Rrule, CEE -Categorical Cross Entropy, Athere is no significant computational overhead, Brequires either special training process, multiple evaluations or an auxiliary component that incurs additional computational cost.
of hyperparameters, however the main dividing line is between methods whose application results in pixel-wise mixing and those which lead to patch-wise mixing. This aspect is further discussed in the following section.
The third set of methods is represented by Puzzle Mix and Co-Mixup. The former attempts to find an optimal way of mixing a given pair of images by solving an optimization task with two objectives. The first objective is to find an optimal mask, i.e. decide how much of the image should be concealed with the other image in a given image region. The second goal is to find optimal locations for the patches extracted from original images in the mixed image, in order to maximize the exposed saliency.
The other method, Co-Mixup, works on the whole mini-batch and attempts to optimize the exposed saliency, as well as encourage diversity among created (augmented) samples.
Both methods bear some resemblance to Saliency Mix yet the key difference is that Saliency Mix uses saliency directly to make a decision while Puzzle Mix and Co-Mixup utilize this information to steer the optimization process. A summary of how is augmentation applied in each method is presented in the second column of Table 2.

Is the resulting mix pixel-wise or patch-wise?
Probably the most fundamental division of mixing augmentation methods is whether they output pixel-wise or patch-wise combination of the images (cf. the third column of Table 2). A distinction between pixels-wise and patch-wise mixing was clarified in section 1.2. Please recall that the term pixel-wise is used for mixing images using pixel-wise weighted average and patch-wise for mixing images spatially, by means of taking patches from original images and joining them together.
Pixel-wise methods are specifically well suited for dealing with corrupted images and adversarial attacks whereas patch-wise augmentations are useful in the context of partial occlusion and weakly supervised object localization. A detailed performance comparison of both types of methods is presented in section 5.
Smart Augmentation, SmoothMix, Puzzle Mix and Co-Mixup are the exceptions from this division as they combine aspects of both types of mixing. Smart Augmentation uses an auxiliary network to learn the optimal way of mixing and the resulting approach can be either pixel-wise or patch-wise. SmoothMix constructs a graded mixing mask in a way that at the center of the mask and at the edges of the image the resulting mixing is patch-wise as it uses just one image. As we transition from the center of the mask to the edge, mixing becomes pixel-wise as the mask gradually changes its values from 1 to 0. Puzzle Mix also exhibits properties of both pixel-wise and patch-wise approaches as it is steered by the saliency information which can result in the patch containing just one image for salient patches and a mixture of two images for less salient patches. Co-Mixup is also guided by saliency information and can exhibit both pixel-wise and patch-wise properties in certain patches of the augmented image.

Does the augmentation technique mix labels?
Label smoothing has proven to be a very effective regularization technique in the presence of label noise [16,67]. Maximizing log p(y|x) when y actually represents an incorrect label can be harmful and one way to prevent this is explicit modeling of label noise. Label smoothing regularizes the model by replacing the 0 and 1 targets with the targets /(k − 1) and 1 − , where k is the number of output classes.
Mixing techniques that combine images from various classes together with mixing their corresponding labels also benefit from this regularization property. The majority of DA techniques mix labels. The ones that do not could be divided in 3 distinctive groups: early methods, methods designed to mix samples from the same class only, and methods that need just one image to work.
SamplePairing, the only representative of early methods that we consider, was designed in a way to assign just one label (from one of the images being mixed) to the augmented image. This resulted in a complicated training procedure that requires the DA to be alternately turned on and off some number of times, and this idea was not continued in subsequent research.
The methods designed and tested only in the context of mixing images from the same class are Feature Space and Smart Augmentation.
The last genre encompasses AugMix, Style Augmentation and MixStyle which all use the content of just one image during the mixing process, hence there is no room for label mixing. AugMix mixes an image with its versions processed with traditional (label preserving) techniques and Smart Augmentation mixes an image with its own version transformed using the style transfer algorithm. In MixStyle the second image is technically required however the method does not refer to its pixel-based (raw) representation but considers the instance-level feature statistics that correspond to visual features.
A taxonomy of the methods based on the label mixing property is summarized in the fourth column of Table 2.

Does the augmentation technique use standard loss function?
Most of DA techniques described in this survey use standard loss function during training, as presented in the fifth column of Table 2. There are four exceptions: AdaMixup introduces a separate element in the loss function (10) responsible for penalizing manifold intrusion (a situation when an augmented sample corresponds to a class different from any of the two image classes). AugMix extends the standard loss function to ensure that augmented samples are as close to original ones as possible (17). Between-Class learning employs KL-divergence without specifying reasons other than better experimental outcomes [70]. Finally, Smart Augmentation uses an auxiliary Augmentor network whose error is added to the standard loss during training.

How many images are used to create an augmented image?
Amongst discussed methods all but four use 2 images to obtain an augmented sample (see the second to last column of Table 2). Out of those four methods, two use more than 2 images: RICAP, which mixes four images in a patch-wise fashion by cropping them and afterwards patching into one image and Co-Mixup which works with the entire training mini-batch at once. AugMix and Style Augmentation mix an image with a transformed version of itself.

What is the computational complexity of the augmentation technique?
The two key factors contributing to the computational complexity of the considered methods are: (1) whether an augmentation uses a simple rule or a relies on more complex approach, as well as, (2) whether there are any additional components or specific training process that is required for the method to work. Based on the above, the methods can be roughly divided into the two following groups (cf. the last column of Table 2).
The functional overheads of particular methods related to using the abovementioned additional components are summarized below. -Attentive CutMix [75] -utilizes an auxiliary network to propose important regions that should be included in the mixing process (an additional forward step of that network is required). -Puzzle Mix [36] and Co-mixup [35] -both require calculation of saliency information and utilize an optimization procedure for optimal mixing that incurs additional computational burden. -AugMix [25] -even though applies standard augmentations and simple mixing rule, because of the modified loss function requires more than one forward step to calculate the JS divergence. -Saliency Mix [72] requires additional step of saliency calculation -SnapMix [29] requires two forward steps to calculate CAM which is required in the mixing process. Generally speaking, it can be roughly estimated that -depending on the augmentation method -the use of the above-listed functional components introduces between a few percents to a few hundred percents of computational time overhead compared to the baseline (straightforward augmentation methods).

Experimental evaluation on standard benchmarks
This section summarizes the accuracy results of both erasing and mixing methods on popular benchmark sets. Additionally, the methods are compared based on their relative improvement over baseline results.
We start in section 5.1 with performance analysis on classification task on clean data using 3 data sets. Next, the methods are compared on classification of corrupted images in section 5.2, and in the context of adversarial examples in section 5.3. The final comparison is made in the problem of weakly supervised object localization and for the partial occlusion task -section 5.4. The next two sections address the possibility of application of analysed augmentation methods to image-related tasks other than classification (section 5.5) and to modalities other than images (section 5.6). Section 5.7 discusses the robustness of augmentation methods to parameter selection.
Please note that quantitative comparisons of the methods presented in this section rely on experiments that compared at least 3 different methods using a common architecture. The only exception from this rule are the tables presenting top results (Tables 3, 4 and 5), which are compiled based on all available outcomes.

Evaluation on clean data
Quantitative assessment of the methods on clean data is presented on three widely-used benchmarks: CIFAR-10, CIFAR-100 and ImageNet.
Unless stated otherwise, a reference to baseline result, labeled as "NO DA", would mean the outcomes of the same network architecture, the same training conditions and pre-processing of the input data, but with no use of analyzed data augmentation.
CIFAR-10 CIFAR-10 data set [37] consists of 60 000 color images of size 32x32 grouped into 10 classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck), with 6 000 images per class. The set is composed of 50 000 training samples and 10 000 test ones. There are 1 000 images per class in the test subset. Examples of CIFAR-10 images are depicted in Figure 13.  Figure 14 presents performance results of Mixup, CutMix and AttentiveCut-Mix on CIFAR-10 grouped by networks of the same type but with varying complexity. The following architectures are compared: ResNet [22], DenseNet [28] and EfficientNet [69], each of them in several realizations. The best overall result is achieved by EfficientNet-B7 and Attentive CutMix, with an error of 4.14% (compared to 5.05% baseline).
Looking at the left panels of the figure, one can conclude that DA improves the accuracy regardless of particular architecture type and complexity. In all three left panels the error decreases when gradually more complex architectures are used (top to bottom) and with an application of more advanced DA methods (right to left). It is generally assumed that Mixup, as the initial method in the area, is the least advanced, followed by CutMix, developed based on a certain criticism of Mixup, and AttentiveCutMix which is an extension of CutMix.
Another way to look at the results is from the perspective of a relative improvement over the baseline (the right panels). A general observation is that using DA with more complex types of architectures yields lower relative boost, on average equal to around 22%, 21% and 11%, respectively for ResNet, DenseNet and EfficientNet. However, the relative effects of DA vary substantially within each architecture type. The highest relative advantage is achieved by the most complex model (ResNet-152) in ResNet group, but for the other two architectures the highest boost is observed for DenseNet-201 and EfficientNet-B1, respectively, which are not the most complex ones. Figure 15 shows the results of experiments grouped around the same baselines (i.e. particular architectures) for a wider selection of augmentation methods. The following architectures are considered: two ResNets of the same complexity trained differently than in Figure 14 (coming from two different papers [65,72] with slightly varying experiment designs), two Wide ResNets [68,72], a specific case of DenseNet [68] where the compression factor for both bottleneck and transition layers is smaller than one, two Pyramidal ResNets [68,80] with different numbers of convolutional layers and different widening factors, as well as Shake-Shake network [68]. Generally, similar trends can be observed as in the case of groups of architectures. Simpler models benefit relatively more from data augmentation, however, in terms of absolute figures they still yield higher errors. All in all, every DA technique is able to improve over the baseline, with patch-wise methods achieving slightly better results. For both ResNet-18 architectures best result were accomplished by patch-wise (Saliency Mix ) method, for PyramidNet-200-240 by CutMix (a patch-wise mixing approach), and for the remaining architectures RICAP, which is also a patch-wise mixing method, performed better or equally good than its competitors. The results additionally confirm that erasing methods (Cutout and RandomErasing) are slightly inferior to the mixing ones.  Table 3. Top-5 overall best outcomes on CIFAR-10.
The last comparison, presented in Table 3, lists top-5 combinations of an architecture and an augmentation method for CIFAR-10 found in the literature. The list is led by two versions of the Shake-Shake model [13].
CIFAR-100 CIFAR-100 data set [38] is defined in a similar way to CIFAR-10, except that it is more fine grained and has 100 classes. Each class contains 600 images, divided in 500 training and 100 test samples. Fifty randomly selected images with corresponding classes are depicted in Figure 16.   [75]. Each panel presents a particular type of architecture (ResNet, DenseNet, EfficientNet) with complexity of the network increasing from top to bottom. Left panels present an absolute error value and the right ones a relative improvement over the baseline ("NO DA" results). Figure 17 is a CIFAR-100 analogue of Figure 14 with the same range of tested architectures and DA methods. Its aim is to investigate how the network's complexity impacts the relative benefit of using particular DA techniques. The absolute errors (left panels) are much higher than in the case of CIFAR-10, because CIFAR-100 is a much more challenging data set (there are only 500 training samples per each of 100 fine-grained categories).
Interestingly, for CIFAR-100 a relative boost (right panels) is generally much smaller than in the case of its less complex counterpart (cf. Figure 14). This difference is most probably a direct consequence of smaller amount of training data available for each class compared to CIFAR-10.
Certain differences can also be observed in the average relative improvement across considered architecture types. While for CIFAR-10 the highest relative gain was observed for the smallest model (ResNet), in the case of CIFAR-100 the biggest relative improvement is noted for the DenseNet network. On average, the use of augmentation methods resulted in a relative gain of around 9.4%, 10.1% and 7.1%, respectively for ResNet, DenseNet and EfficientNet. Similarly to CIFAR-10 architecture-based trends (within each panel) vary significantly.   [23], several ResNet and WideResNet architectures [81] and Pyramidal ResNet [20]. This time models that relatively benefit most from DA application are not the simplest ones. Furthermore, it can be observed that application of certain augmentation techniques (AugMix and Cutout) actually deteriorates accuracy for some architectures.  Table 4. Top-5 overall best outcomes on CIFAR-100. Table 4 presents top-5 combinations of model architecture and augmentation method found in the literature. The best accuracy is achieved either by highly complex models (Pyramidal ResNet, Shake-Shake) joint with one of the founding methods (Mixup, BCL or CutMix ) or their extensions (SmoothMix ), or by applying one of the most recent data augmentation techniques, i.e. Puzzle Mix, with less complex architecture.
ImageNet The most comprehensive benchmark that we consider is ImageNet [9], which contains 1.2 million training samples and 50 000 validation ones, divided into 1 000 categories. The size of the images varies with the average of 469x387 pixels. In the experiments reported in the literature the images are usually cropped to 256x256 pixels or 224x224 pixels, depending on the architecture used. Example images are presented in Figure 19.  [35,36,68,72]. A predominant observation from the figure is that for ImageNet, a far more complex data set than the CIFAR sets, the relative advantage of applying data augmentation is much lower. The average relative increment is between 5 − 6% and the maximum gain equals 10.5% for Puzzle Mix method and ResNet-50 model. Table 5 presents top-5 combinations of model architecture and augmentation method. Similarly to the CIFAR sets the leaders are the most advanced architectures (ResNeXt-101 64*4d and ResNeXt-101) combined with base mixing techniques, followed by certain less complex architectures paired with more recent mixing methods.

Evaluation on corrupted images
In this section we verify the efficacy of models trained with DA techniques on corrupted test data. Experimental evaluation is performed on CIFAR-100-C [24], which is obtained from CIFAR-100 by applying 15 different corruptions which are divided into four major categories: noise, blur, weather changes, and digital corruptions. The effects of applying these corruptions are presented in Figure 21.     [25] and the next ones from [36] and [42], respectively.
ResNet). It can be observed that on corrupted data, regardless of particular architecture, certain data augmentations (Cutout, Mixup, CutMix, SmoothMix ) do not offer substantial improvement compared to the baseline results.
When leaving aside Puzzle Mix (Aug), a method that consistently outperforms its competitors is AugMix which was explicitly designed for dealing with corrupted data. A runner-up technique, which is also substantially better than the remaining methods, is Puzzle Mix. Not surprisingly, the strongest overall approach is an application of Puzzle Mix on top of AugMix in which a pair of images, after being processed with AugMix, is mixed using Puzzle Mix. This combination, denoted as Puzzle Mix (Aug) in the figure managed to further decrease the classification error down to the level of 29.9%, which is the lowest result on CIFAR-100-C reported in the literature.

Evaluation on adversarial examples
While this area is not the mainstream of image augmentation research, it is worth to mention that some augmentation methods were also evaluated against adversarial examples. An adversarial attack used most often in this context is Fast Gradient Sign Method (FGSM) employed as a white-box attack. In this case the gradient of the loss function is analyzed to find the most effective way of perturbing the image so as to mislead the classifier. Similarly to the case of image corruption adversarial examples are used exclusively at test time. Figures 23 and 24 present the FGSM error rates from the experiments reported in [36] and [72] regarding CIFAR-100 and ImageNet, respectively. Generally, the results are inconclusive as for CIFAR-100 pixel-wise mixing methods (Mixup, Manifold Mixup, AugMix ) are superior, whereas for ImageNet the best performing method is Saliency Mix, a patch-wise mixing approach. It should be noted that in [73] visibly lower FGSM error rates for Mixup (59.3%) and  [77]. The left column presents absolute error values and the right one a relative improvement over the baseline. All results come from [36] which offers a broad comparison of data augmentation methods, in both clean data and adversarial attacks scenarios. Fig. 24. FGSM error rates on ImageNet for ResNet-50 architecture trained using three data augmentation techniques (Cutout, Mixup and CutMix ). The left column presents absolute error values and the right one a relative improvement over the baseline. All results come from [72].
Manifold Mixup (55.0%) were reported for CIFAR-100 and PreAct ResNet-18 network, however with no details regarding the experiment setup.
The method that clearly stands out in terms of accomplished results is Puzzle Mix (adv) [36], which is a variant of Puzzle Mix employing adversarial training. At the same time, this is the only method that requires substantially more computational power compared to the "NO DA" reference case.

Evaluation on weakly supervised object localization and partial occlusion tasks
In this section we discuss the efficacy of DA techniques in the problems of weakly supervised object localization (WSOL) and partial occlusion of the target object, which are frequently encountered in the image classification or object detection settings. In both contexts a desired property of the model is to make prediction based on a wide array of class-relevant visual features and not just the most distinctive features. In WSOL this property helps in finding a fragment of an image that entirely covers a given object, and in the context of partial occlusion supports the network in inferring the correct class based on the remaining, less distinctive features. Utilization of augmented data in WSOL is discussed in [29,35,68,72,80] in reference to CutMix, RICAP, Saliency Mix, SnapMix and Co-Mixup, respectively. In all cases the results are assessed using Class Activation Mapping [85] already mentioned in section 3.3. CAM creates a heat map of the same size as input images, indicating regions in the original image that contributed most towards a certain class prediction. CAM requires a CNN model to possess a global average pooling layer to obtain the spatial average of the feature maps prior to the output layer, so as to visualize the most class-contributive pixels.
The results indicate that augmented data indeed broadens the regions used by the model to infer the correct class [35,68,80] with Saliency Mix being more discriminative against the image background. The only exception is SnapMix, which was designed for fine-grained classification and limits the regions activated by CAM to parts of the image that are relevant for a particular fine-grained class (e.g wings instead of a whole bird).
RICAP was tested in [68] also in object detection task. The experiment was conducted on MS COCO (Microsoft Common Objects in Context) set [48] using YOLOv3 architecture [58]. MS COCO is a large set of images presenting common objects with associated bounding boxes and segmentation maps, accompanied by the respective captions. In the reported experiments only information about the bounding boxes was used.
RICAP was adjusted so as not to interfere with the area in which an object was located. The target bounding boxes where restricted to the cropped regions. This way the model was no longer able to benefit from partial labels (each object had one class assigned to it) but could learn partial features (only part of an object might fall into the cropped region, so the network had to learn to make predictions based on the remaining part). This adjusted version of RICAP slightly improved the mean Average Precision (mAP) over the baseline (YOLOv3 without augmentation) from 51.3% to 52.7% [68].

Application to image-related tasks other than classification
The canonical tasks in Computer Vision are image categorization, object localization, object detection and semantic segmentation. These tasks can be further divided into specific cases: -Fine grained categorization -in which differences between classes refer to small visual features. -Categorization under domain shift -in which a model is trained on data from a distribution other than the one applied at inference stage. -Categorization with federated learning -in which a model is trained on data that is scattered among many devices as opposed to being collected in one place. -Categorization of corrupted images -which refers to images distorted by certain noise, blur, weather conditions, digital processing, etc. -Categorization of adversarial images -which refers to images that were purposefully modified to confuse the model. When it comes to application of certain data augmentation techniques to various tasks, there are two key properties: (1) whether the mixing is performed pixel-wise or patch-wise and (2) how many images are mixed. Consequently the methods could be divided into 3 following groups: -Group A -Pixel-wise or mixed Pixel-wise and Patch-wise augmentations that work on 2 or more images: Smart Augmentation [43] Methods from group A are limited to categorization task due to their underlying property of mixing images-pixel wise. This leads to certain regions of the image representing more than one class and renders application of these methods to other tasks difficult (e.g. what should be done with a bounding box for a part of the image that is mixed?). Augmentations in group B directly address categorization and localization tasks and can be further adjusted to object detection and segmentation by proper handling of an additional information associated to the task (e.g. RICAP [68] method limits the bounding box to the area corresponding to the selected patch).
Augmentation approaches from group C, which work on a single image, can in principle be applied to all tasks.
On a more detailed level, there are certain methods that were either developed with a particular problem in mind or were studied in the context of a specific problem and perform well on it.

Application to modalities other than images
The main focus of this survey is data augmentation for images, however there are also other modalities that could be considered for application of the analyzed augmentation methods: text and audio. In this section we point the methods that could potentially be extended to other modalities, and present examples of such applications already reported in the literature.
When it comes to applying image augmentation methods to other modalities the following 3 groups could be distinguished: -Group A -Mixup-like methods (Pixel-wise mixing) that work on 2 or more images and do not utilize any complex mixing mechanism, i.e. Mixup [82], , Co-mixup [35] and Patch Gaussian [50].
Methods from group A can be applied to other modalities without any adaptation as long as the same size of the input objects is ensured. In the context of audio it means having the same length and the same spectrum of frequencies, and for the text data, the same size of vector embeddings. Examples of successful applications of the methods from group A to other modalities are presented below in this section.
For methods from group B their application to modalities other than images is technically possible, however, not yet empirically tested. Such a mixing would potentially signify specific modality-depending aspects, e.g. spatial mixing of embeddings of different sentences or pasting a part of a voice spectrogram into another one.
A potential application of the methods from group C to other modalities would require introducing major changes to their design and operation, as they are inherently related to image data. Some methods from this group utilize image saliency information [29,35,36,72], other use image specific data transformations, like style transfer or rotation [25,32,87]. Yet another ones, utilize architectures dedicated to processing the image data [19,43,75].
So far several examples of applying image augmentation methods to other modalities have been proposed in the recent literature. Most notably the Mixup method was applied to speaker classification task [88] and to sentence classification [34]. Also Mixup like method were applied to rare event detection based on sound data [5] and to text classification [18].
Even though application of augmentation methods to non-image data seems to be scarce, we are convinced that this area offers many flourishing research directions related to both adaptation of the methods from groups B and C, as well as invention of the entirely new augmentation approaches devoted to particular data modalities and addressing their specificity.

Robustness to parameter selection
The majority of the methods presented in the survey rely on no more than 3 hyper-parameters and are relatively robust to their selection or change, as well as the change of the data set.
The method's robustness to the problem change (same task different data set) is mostly correlated with the way the augmentation method is applied (cf. Section 4.2). For the rule based methods, the change of a problem does not require any hyper-parameter changes. For other methods, however, one can face some difficulties with their application to other data sets. Attentive CutMix [75] utilizes an additional feature extractor that might not be available for all problems. Puzzle Mix [36] and Co-mixup [35] employ an optimization procedure that requires some assumptions on how to simplify the optimization problem which might differ between problems (e.g. wideness of the image may impact how the grid of location for the optimization process is created). The next method, Style Augmentation [32], was shown to be sensitive to hyper-parameter change in the ablation study. In case of this method both strength of regularization via style transfer as well as ratio of augmented samples to non-augmented ones greatly impact the method's accuracy. Yet another method that we believe is hard to apply directly to other data sets is Smart Augmentation [43] -one of the older approaches that employs a dedicated problem-specific network responsible for mixing images. Training of this network, in addition to computational overhead, is generally difficult and may potentially lead to poor mixed samples that would impact the accuracy of the target network.

Selecting the best data augmentation strategy
Up to now we have discussed newest mixing augmentation methods whose utilization improved the state-of-the-art results in many visual tasks, for instance image classification (either clean [70,80] or corrupted [25,36]), object detection [68] or WSOL [68,80]. While application of gradually more advanced and more complicated methods proven successful, an alternative approach is to consider an ensemble of traditional augmentation methods and pick-up those of them that are particularly well suited to a given task and/or experiment setup.
Traditional data augmentation techniques, e.g. image scaling, translation or rotation are generally effective in improving accuracy of DL image classifiers, however an impact of a particular augmentation depends on characteristics of the data set and the task at hand, which poses certain limitations to their effective application. Moreover, the number and diversity of traditional augmentation methods prohibit using them all at once, as such a situation would heavily slow down the training process and actually deteriorate the accuracy. Hence, the ability to limit the number of considered augmentation options for a given problem (task, data set, imposed constraints, etc.) is an important issue. In this section several automated approaches to effective selection of the optimal subset of traditional augmentation techniques (for a given task and dataset) are summarized and evaluated. These methods will be referred to as data augmentation policy selection (DAPS) approaches.

Black Box methods
A seminal DAPS method is AutoAugment [7] which is composed of two major components: the search algorithm and the search space. The search engine (an RNN controller) samples from the search space the DA policy S defined as the following triple: (image augmentation operation, probability of its application, magnitude of the operation). The controller is trained with Reinforcement Learning Proximal Policy Optimization algorithm [60] based on the reward signal from the auxiliary model (e.g. a CNN model in the case of image classification) which measures the efficacy of the selected policy in improving model generalization. After completion of the policy exploration phase the auxiliary models are discarded and the best policies found are used for target model training. The choice of RL as the training algorithm is arbitrary and is inspired by automated architecture search techniques [89]. Actually, any other suitable technique, e.g. augmented random search or evolutionary strategy could be used in its place. In order to cast the problem of selecting S into a discrete search space, the augmentation methods' probabilities and magnitudes are discretized to uniformly spaced values.
In the search for the optimal DA policy only a fraction of observations from the original data set are used, for instance 4 000 and 1 000 in the case of CIFAR-10 and SVHN [54], respectively. The underlying principle of AutoAugment is that an optimal DA policy for a given data set would not change if the entire data set was considered.
The best policies found for the two above-mentioned data sets are different and clearly linked to the image content. For CIFAR-10 the vast majority of best augmentation policies are focused on color-based transformations since within the respective classes (e.g. airplane or truck) the diversity introduced by colorrelated augmentations coincide with real-life cases where one object can be of various colors. In SVHN the best augmentations are geometric transformations since for street numbers categorization they seem to be more important than color, which is actually irrelevant if only the number is readable.
An interesting question is whether augmentation policies are transferable between data sets. To this end the best AutoAugment policies found for SVHN were subsequently applied in the training process of the networks solving classification problems on 5 other data sets (Oxford 102 Flowers [55], Caltech-101 [44], Oxford-IIIT Pets [12], FGVC Aircraft [51] and Stanford Cars [78]). In all cases using policies learned on SVHN resulted in performance increase over the baseline [7].
In the ablation experiments application of an optimal policy found by Au-toAugment was compared with the use of the same subset of augmentation techniques, but with random probabilities and magnitudes, as well as with random policy sampling. In both cases AutoAugment proven superior.
The main limitation of the method is its extensive computational cost. In order to generate sufficient number of training signals for the RNN controller, thousands of auxiliary models need to be trained with sampled augmentation policies before the final out-of-sample accuracy is achieved. All the remaining methods presented in this section efficiently address this problem while keeping the results at comparable level.
The first method that bases upon the idea of AutoAugment and matches its accuracy is Population Based Augmentation (PBA) [26]. PBA produces dynamic augmentation policy schedule and combines random search with evolutionary strategy to significantly decrease computation time.
The difference between the policy (constructed in AutoAugment) and the policy schedule (in PBA) lies in the time range for which the optimal augmentation policy is selected. In AutoAugment the optimal policy is selected once for the entire training time. In PBA it is chosen for each epoch, leading to a sequence of optimal policies, i.e. a policy schedule. The underpinning idea of PBA is that diversification of policies applied at different training stages is advantageous for the final accuracy.
The main motivation of [26] is to demonstrate that state-of-the-art results (comparable to those of the AutoAugment application) can be achieved with significantly lower computational cost. As a backbone PBA uses Population Based Training algorithm (PBT) [33] that selects a subset of augmentation techniques independently for each epoch. The schedule learned so far is used as a starting point. In other words, at any given epoch all candidate policies share the common pool of policies selected historically.
Initially PBT trains in parallel an ensemble of randomly initialized models with the same architecture. At certain intervals the performance of these models is evaluated on a validation set. At this point two mechanisms are applied: exploitation and exploration. Exploitation consists in replacing the weights of 25% of the worst-performing models with the weights from the top-25% models. Exploration relies on randomly perturbing the hyperparameters of the models with replaced weights, in order to extend the hyperparameter space search. PBA uses exactly the same hyperparameters as AutoAugment, i.e. a list of triples: (transformation type, probability of its application, its magnitude).
Similarly to AutoAugment the quest for an optimal DA policy is performed on a reduced data set. In the experiments presented in [26] the policy schedule is learned only once using WideResNet architecture, and is subsequently applied to training other models (based on other architectures). Since DA method applied in a given epoch is not linked to any particular architecture, the only relevant difference from the perspective of the DA process is the number of required training epochs. The training schedule is adjusted proportionally by stretching it out (shrinking) in case a particular architecture requires longer (shorter) training than WideResNet. For instance, an optimal schedule for 2 epochs consisting of a set of augmentations A and B, could be stretch out to 4 epochs by repeating each of these sets, leading to a schedule A, A, B, B.
Fast AutoAugment (FAA) [47] is another approach indicating AutoAugment as its inspiration. FAA addresses the problem of AutoAugment computational complexity by using a more effective search strategy based on density matching. Instead of repeatedly training auxiliary models with different augmentations in order to select the best-performing one, FAA searches for augmentation policies that minimize categorical cross-entropy loss on validation set. The process relays on treating the augmented data as missing data points and selecting the bestperforming augmentations using the Tree-structured Parzen Estimator (TPE) algorithm [4].
FAA search space is similar to that of AutoAugment and PBA as the method uses a list of transformation operations, each of them described by its probability and magnitude. The goal is to find a set of optimal policies that can be used for the final model training. The key difference is that TPE enables searching over continuous space so neither probabilities nor magnitudes need to be discretized.
The search process consists of the following steps. First, the data set is split into K pairs (D A is the data that will be used to evaluate different augmentations, for k = 1, . . . , K. Next, the model is trained from scratch using D A set. Since in order to evaluate a new policy only augmentation and prediction is required, there is no need to train the model again from scratch.
Once the above exploration phase is completed, top-N policies are selected as the optimal augmentation policies for a given data set. In the final step, the model is trained using the entire training data set and the optimal policies selected across K folds.
All three above-described methods (AutoAugment, PBA, FAA) are based on the assumption that a set of augmentation policies that will result in high model accuracy can be found based on a smaller proxy task. Therefore, all of them limit the size of the training set used to search for the optimal augmentation policies.

Reducing the search space
The sole method in this area is RandAugment [8], which questions an assumption that a search for optimal policies over the entire solution space can be effectively performed with a smaller proxy task. This disbelief is based on the experiments showing that the optimal augmentation magnitude depends on the size of a training set and the size of a model. Consequently, RandAugment postulates significant reduction of the search space so as to make the selection of an optimal augmentation policy a feasible task.
To this end it is proposed in [8] to jointly optimize magnitudes of all augmentation operations by means of a single parameter called distortion magnitude. Furthermore, it is proposed to uniformly set the probability of all augmentation operations, thus reducing the search space even further. The resulting algorithm depends on just two parameters: the number of transformations applied to an image and the magnitude distortion parameter indicating regularization strength.
In the experiments performed on CIFAR-10 a family of WideResNet architectures with various values of the widening parameter, responsible for the number of convolutional filters, hence complexity of the network, were trained on data sets of various sizes. The first one tested 7 WideResNet models with various widening factors, and the other one considered 8 different sizes of the training set, between 1 000 and 45 000 images sampled randomly from CIFAR-10 with fixed WideResNet architecture with widening factor = 10. Both evaluations showed that more complex networks and/or bigger data sets require stronger regularization to achieve full classification potential. This finding is somewhat counterintuitive in reference to the training set size where a common belief is that smaller training sets require more regularization. A hypothetical explanation presented in [8] is that strong augmentation on small data set might result in high noise to signal ratio.

Making the augmentation pipeline differentiable
Methods in this section transform augmentation operations, which up to now were intrinsically non-differentiable, to differentiable ones, thus allowing joint optimization of the data augmentation pipeline and classification network.
Chronologically, the first paper in this area is Faster AutoAugment [21] that builds on and enhances Fast AutoAugment approach [47] described in section 6.1. The method adjusts the policy search pipeline so as to make it fully differentiable and consequently enable gradient based optimization. This is achieved by implementing the following three concepts: (1) approximate gradient calculation of discrete image operation [3], (2) making the operation selection process differentiable thanks to the adaptation of neural architecture search methods [49], and (3) expanding the idea presented in [47], where the validation loss on the augmented sample was minimized, to GAN approach with augmentation policy treated as a generator and a critic network [2].
Faster AutoAugment also adapts the foundations of DAPS, i.e. organization of the augmentations into policies and sub-policies, as well as the way the experiment is structured (policy search on smaller sample, model training on full data). Since the method transforms the search for optimal parameters into an optimization task solving which is more time-effective, it can be applied to larger search spaces. It is concluded in [21] that increasing the number of sub-policies or the number of operations in each sub-policy is increasing the performance.
Differentiable Automatic Data Augmentation (DADA) [46] relaxes the DA policy selection problem to differentiable one using (continuous) Gumbel-Softmax distribution that approximates samples from the categorical distribution.
Technically, DADA approaches the tasks of sub-policy selection and augmentation parameter selection by sampling from a categorical distribution and Bernoulli distribution, respectively and then relaxes this optimization problem to differentiable one using Gumbel-Softmax. Eventually, joint optimization of augmentation parameters and classification network weights is carried out using the RELAX estimator [17]. Finally, the bi-level optimization of DA parameters and classification network weights is carried out according to the following equations: subject to ω * (d) = arg min where, d = {α, β, m, φ} represent probability of selecting a sub-policy (α), probability of applying a transformation (β), magnitude of the transformation (m), and RELAX network parameters (φ), respectively. ω represents parameters of the classification network. In order to solve the bi-level optimization problem (24)-(25) ω and d are optimized alternately through gradient descent. As in AutoAugment the policies are searched for on a reduced data set, although using only one network training limited to 20 epochs.
The next method, Adversarial AutoAugment [83] generates adversarial images as augmented samples. The method performs training on full data set, getting rid of the smaller proxy task, and returns a dynamic policy schedule which is updated during training.
Technically, Adversarial AutoAugment uses two neural networks: Task Network (TN) and Policy Search Network (PSN). PSN is responsible for sampling policies that should be applied to training data. It is implemented as a one-layer LSTM [45] with 100 neurons in hidden layer and an embedding size of 32. TN is trained on data augmented with the sampled policies. Each image from the training batch is processed multiple times using different policies and all these transformed versions are added to the training batch. Based on the parts of the TN loss function associated with each sampled policy better policies are generated. PNS parameters are updated using the REINFORCE algorithm [76]. Adversarial AutoAugment modifies a default policy search space by skipping policy probability and considering continuous policy magnitudes: where TN and PSN are indicated with F and A, respectively and are parameterized by w and θ. The optimization problem is formulated as a min-max game: TN tries to train a network and PSN attempts to increase the loss by providing harder augmentations, also making TN more robust. Such a setup as it conforms to the current state of the training process and provides augmentation with lower severity at the initial training phase (to allow for quick learning of the main patterns) and increases the severity in subsequent training once the network learned all simple patterns.
The high level setup of Adversarial AutoAugment is similar to that of GANs in that one network tries to outplay the other one. Yet the method is inherently different as it works on existing images by applying a transformation from a predefined list and does not synthesize new images from noise, as GANs typically do.
Policies learned using Adversarial AutoAugment transferred to other problem give comparable or better results than AutoAugment, however the best results are achieved when the policies are searched for independently, focusing on a given data set [83].
Another method in this group, MetaAugment [86], challenges one of the well established DA assumptions that an augmentation transformation is selected for the entire data set. MetaAugment evaluates a fit of the selected augmentation operation to the image and assigns a weight based on the significance of this fit. The weight is calculated by Augmentation Policy Network (APN) and passed on to Task Network (TN) to calculate the weighted loss of the augmented training image.
APN takes as its input an embedding of a transformation function together with deep features extracted from the image with TN and outputs a weight to adjust the augmented image loss computed by TN. The goal of APN is to improve performance of TN on a validation set via adjusting the weights of the losses. APN is implemented as a 1-hidden layer MLP that takes the aforementioned embedding and deep features as its input, which is followed by a fully-connected layer of size 100 with RELU activations and an output layer with one sigmoid neuron. TN can be implemented as any of the standard CNN architectures that minimizes the weighted training loss using the weights provided by APN: subject to ω * (θ) = arg min ω L(X tr , ω, θ) where θ and ω are parameters of APN and TN and X tr and X val denote training and validation sets, respectively. MetaAugment proposes an additional mechanism, the transformation sampler (TS), working in tandem with APN and TN. TS samples transformations according to a probability distribution estimated in the outputs of APN that reflects the overall effectiveness of the transformation for the entire dataset. As the distribution is estimated based on APN, it evolves over the training process and provides gradually better recommendations.
One of the MetaAugment disadvantages is computational overhead. The method requires 3 forward and backward passes in TN which takes 3 times longer than in a typical training scenario. Its clear advantage, on the other hand, is the ability to fit the transformation to a given sample. While this fitting is not performed explicitly because the image and transformation are sampled independently, it is forced by assigning the weight to the image, which denotes the impact of this image on the loss function that is used to update TN weights.
The most recent method in this area [53] tries to jointly learn the optimal data augmentation parameters while training the end task model. The method will be henceforth referred to as ABO (Data Augmentation with Online Bi-level Optimization). ABO employs two neural networks, a CNN Classification Network (CN) and an MLP Augmentation Network (AN). The training process is divided into two streams. In the inner loop AN predicts augmentation parameters, based on which the image is augmented, and fed afterwards into CN to calculate the loss. In the outer loop a feedback signal for AN is created: subject to ω * = arg min where A θ is AN parameterized by θ, and CN is parameterized by ω. Joint training of AN using back propagation is possible thanks to the following features. Firstly, AN is trained on the validation set (cf. eq. (30)) to improve generalization properties of CN. Secondly, in order to enable gradient calculation the AN error on the validation set is calculated using CN (eq. (30)) and subsequently used to update parameters of AN. Thirdly, an online approximation of the bi-level optimization is proposed to enable updating of AN parameters at each training step. Fourthly, differentiable augmentation operations from Kornia library [59] are used.
The online approximation of the bi-level optimization is used to overcome the problematic setup used in many previous methods [7,8,26] where optimization process treats the two objectives separately as black-box problems. In ABO the weights of AN are updated after each step in reference to CN outputs: where G (t) is the gradient of the training loss at iteration t. AN is implemented as an MLP with input and output of size n, being the number of hyperparameters to optimize, and two hidden layers with n and 10n neurons, respectively. Bigger architectures were also tested but it was found out empirically that the size of AN does not have significant impact on the accuracy of the classifier [53].
Contrary to Adversarial AutoAugment [83] and MetaAugment [86] where subsequent training resulted in more adequate change of the method's parameters, in the case of ABO the transformations applied at the beginning of the training are stronger and approach identity towards the end of the training.

Quantitative evaluation
We quantitatively compare the results of application of nine DAPS methods presented in sections 6.1-6.3 on the same three benchmark sets that were used in evaluation of previously introduced DA methods. Methods are compared between each other and against selected erasing and mixing approaches described in sections 2 and 3, respectively.
A summary of results on CIFAR-10 presented in Figure 25 shows that application of DAPS methods yields on average better results than utilization of  [7,8,21,26,46,47,53,83,86] adopted the same experiment design. advanced pixel-wise or patch-wise DA approaches. The only exception is Manifold Mixup, which slightly outperforms the majority of DAPS solutions, albeit the supporting evidence comes from one experiment only. Adversarial AutoAugment and MetaAugment, two methods that introduce changes not aimed solely at reducing AutoAugment computation time, reign among DAPS methods.
The above observations are also confirmed in experiments with more complex data sets. In the case of CIFAR-100 (see Figure 26), again all DAPS methods fare very well -with the average error on WideResNet-28-10 architecture slightly below 17. At the same time Puzzle Mix (a DA approach) achieves the best overall score. Likewise, on ImageNet ( Figure 27) the results of majority of DAPS methods are close to each other except the two leading approaches (Adversarial AutoAugment and MetaAugment) which outperform the rest. The results are generally better than those of DA methods, except Puzzle Mix which demonstrates a slight upper-hand over all but the two above-mentioned methods.
A general observation is that the more complex and bigger the data set is the smaller relative improvement over the baseline, stemming from DAPS usage, can be expected. Depending on the architecture the relative improvement varies between 24% and 51% for CIFAR-10, between 5% and 26% for CIFAR-100, and between 1% and 14% for ImageNet. Based on the results we conclude that the  [7,8,21,26,46,47,53,83,86] adopted the same experiment design. An open question, which we believe is worth investigation is application of DAPS methods to more complex DA techniques, going beyond traditional augmentations. The results presented in Figures 25-27 suggest that inclusion of more complex erasing or mixing methods into DAPS pool of augmentation techniques might lead to their efficacy improvement beyond the state-of-the-art MetaAugment and Adversarial AutoAugment solutions.

Comparison of DAPS methods based on particular properties
This section collates all DAPS approaches introduced above in reference to their key aspects summarized in Table 6.
The DAPS baseline algorithm One of the key factors differentiating DAPS methods is an algorithm used to address the task of finding best augmentation parameters (cf. row "Algorithm" in Table 6). AutoAugment uses a Reinforcement Learning based approach in which many networks have to be trained on different versions of augmented data in order to generate a strong enough learning signal for the RNN controller. PBA employs Evolutionary Strategy metaheuristic in which a population of models trained on different versions of augmented data is maintained and gradually ameliorated through application of evolutionary operators. FFA relies on Density Matching algorithm which consists in initially training the classifier on non-augmented data and then verifying which transformations minimize the cross-entropy loss on validation set. RandAugment uses a simple grid search approach on a predefined list of hyper-parameters.
The other five methods share a common approach that consists in making the data augmentation pipeline differentiable. Faster AutoAugment actually follows an approach similar to FAA, as it uses Density Matching to optimize augmentation parameters. However, instead of searching for the best augmentation policy parameters, it simply learns them from the data using differentiable data augmentation pipeline. DADA formulates the policy search problem as a Monte Carlo sampling problem. Adversarial AutoAugment replicates the data set n times, applies different augmentations to each replica and calculates part of the loss associated with that replica. MetaAugment tries to find an optimal augmentation for each sample by means of reformulating the problem as sample reweighting problem. The most recent work, ABO, introduces an online approach to solving bi-level optimization problem.
Fixed or varying in time policy schedule Following the intuitive reasoning presented in [26] an optimal augmentation policy (i.e. the one with best generalization properties) depends on the advancement of the training process. However, among older methods (up to 2019), PBA [26] is the only one that uses a variable policy (policy schedule), and all other methods employ a fixed policy approach. As for the recent methods (2020-2021), 3 out of 5 (Adversarial Autoaugment, MetaAugment and ABO) employ dynamic policies. Furthermore, in the RandAugment paper [8] the authors tested different schedules of distortion parameter M (a constant magnitude, a random magnitude, a linearly increasing magnitude, a randomly selected magnitude with increasing upper bound) but concluded that a constant global distortion value yields comparable results, so  Table 6. Comparison of DAPS methods along key differentiating axes. The following abbreviations are used: RL -Reinforcement Learning, ES -Evolutionary Strategy, DM -Density Matching, GS -Grid Search, BO -Bi-level Optimization , MC -Monte Carlo gradient estimate, AT -Adversarial Training , diffaugmentations made differentiable and SR -Sample Re-weighting. Values in rows "Proxy task" and "Final task", as well as the value in row "Target models" for RandAugment are specific for CIFAR-10 data set.
there is no point of increasing the method's complexity. Schedule characteristics adopted by particular methods are summarized in row "Dynamic" of Table 6. Figure 28 presents example outputs of a single policy for DAPS method for the same image that was used throughout the survey. The key factor determining the output of a given policy is whether the method utilizes a fixed or varying in time augmentation policy schedule.
The policies that use fixed schedule will apply the same augmentations throughout the entire training process, whereas for varying in time schedule the policy can change as the training progresses. Another aspect of the policy is the augmentation strength -there are 3 such levels illustrated in Figure 28. Again, for a fixed policy the strength remains the same throughout the training process, while for a varying one it can change along with a training progress. Computational complexity of the method An assessment of the methods' complexity based on the reported GPU hours would not be adequate as the presented solutions rely on various graphic cards, training set sizes, etc. For this reason, we decided to present two computational complexity parameters in Table 6: "Epochs" -which is the number of times the algorithm goes through the entire training data set as reported in the respective papers (these values are not directly comparable due to different sizes of the training sets) and "Epoch norm." (normalized epochs) -which is a crude estimation of epoch-data-set-size complexity, that assumes that the same number of training epochs (200) and the same amount of data are considered by each method. Based on this rough estimation it can be concluded that the most computationally effective methods are MetaAugment, Faster AutoAugment and DADA, followed by FAA and PBA. At the other extreme there is AutoAugment which is orders of magnitude more time demanding than the remaining approaches.
Search space The majority of DAPS methods consider discrete parameter spaces. The three exceptions search continuous spaces as they use either Density Matching (FAA, Faster AutoAugment) or a separate Augmentation Network that works on augmentation embeddings (MetaAugment).
Among all methods only RandAugment significantly reduces the search space by restricting it to 10 2 possible options. Furthermore, Adversarial AutoAugment removes the probability parameter of each operation and hence reduces the search space to 10 22 elements. The remaining methods consider much bigger spaces: with 10 32 elements in the case of AA, FFA, Faster AutoAugment, DADA and MetaAugment, and 10 61 for PBA, which uses the same number of parameters but additionally searches for an optimal schedule of the policies.
A summary of search space characteristics are presented in rows "Space type" and "Space size" of Table 6.
Proxy task This aspect is connected to the baseline algorithm employed by each method, described in section 6.5. Three non-differentiable methods use a proxy task for finding the optimal data augmentation policy. The complexity of proxy differs between AA, PBA and FAA but in each case additional models need to be trained. More details are presented below in section 6.5. The only method in this group that trains the final classifier models right from the start is RandAugment.
Among differentiable methods there are two (Faster AutoAugment and DADA) that use a separate augmentation parameter search phase, although both of them use just one auxiliary model. The remaining differentiable methods do not use proxy task as they jointly train the augmentation pipeline and the target model.
Rows "Proxy task" and "Final task" in Table 6 show the number of observations used respectively to solve the proxy task and the final task (with the augmentation strategy selected based on proxy).

Number of trained models
RandAugment is the sole method that trains several final classifiers -the number of them depends on the data set. The remaining methods train just one final classifier, since at this stage the best policy is already selected or is optimized jointly with the target model training.
The target classifier is always trained on the full available training data, while auxiliary models are trained only on a fraction of this data. In case of AA, PBA, Faster Autoaugment and DADA between 1 000 and 4 000 observations are used in auxiliary training, depending on the data set, and in the case of FFA the training set is divided into a number of equal-size parts, each of them used to train one auxiliary model. The respective numbers of auxiliary models when dealing with CIFAR-10 are presented in the last two rows of Table 6. One more thing worth mentioning is n-time multiplication of the training set size of the target model in Adversarial AutoAugment -one instance per each DA.

Conclusions
Data augmentation in image classification task is mainly applied to increase the size of the training data set and make the model more robust by creating variations of the images that, although procedurally generated, resemble real world test case settings. DA is a well know regularization mechanism helpful in preventing over-fitting of the learning process. Additionally, scarce availability of annotated training data is considered one of the biggest impediments in DL applications to narrow domains (e.g. particular business problems) or those requiring high level expertise. In such scenarios DA can play a critical role in increasing the size of the training data without increasing the cost of manual creation of the annotated data.
Mixing DA methods and DAPS approaches for image categorization is a relatively new research area, therefore the vast majority of the papers discussed in this survey were published between 2018 and 2021. Methods created with the image classification problem in mind gradually become explored in the context of other data modalities, for instance an application of Mixup to text data [18,34], or become adapted to particular use cases like semantic segmentation [56], federated learning [79] or fairness [6].
The properties and design of DA method determine its impact on the augmented image and are crucial aspects in deciding which DA method fits best the task at hand. Among mixing methods, the pixel-wise approaches (e.g. Mixup) work better with noise (corrupted images or incorrect labels) while the patchwise ones (e.g. CutMix ) are better suited to the task of partial occlusion or weakly supervised object localization problem. Patch-wise methods (nonetheless with some adjustments required, like the adaptation of RICAP described in section section 5.4) are also effective in the object detection task. Additionally, there are DA methods designed specifically for a particular problem or setting, like AugMix which is devoted to the case of corrupted images.
In reference to data augmentation policy search (DAPS) methods, a general observation is that the more complex and bigger the data set is the smaller relative improvement over the baseline, stemming from DAPS usage, can be expected. In other words, for complex and diverse data application of existing DAPS approaches is still limited, possibly due to considering only simple, traditional DA techniques in the developed policies.
A similar observation is also valid in the case of individual DA approaches where the experiments have proven that the relative gain from DA application is smaller for more complex data sets.
An open question is whether mixing raw input images is more effective than mixing their latent feature-based representations. While some related discussion had already taken place in the literature, the outcomes are yet inconclusive. It seems reasonable to assume that the advantage of either of these two approaches may depend on a particular DA method applied.

Possible future directions
Generally, within mixing DA methods one can observe two major tendencies: (1) joining approaches that demonstrate different capabilities into coherent, though more complex, synergetic methods and (2) extending well established methods by combining them with certain statistical image-based information, so as to achieve better properties and higher accuracy of such hybridized approaches. A motivating example of the former direction could be a combination of Puzzle Mix [36] and AugMix [25] that achieves better results on corrupted images than any of the constituting methods alone. Premiere examples of the latter trend are, for instance, Patch Gaussian [50] that joins the idea of Cutout [11] with addition of Gaussian noise to make the method work well on both clean and corrupted data, or Puzzle Mix that combines the idea of mixing with an information on saliency of visual features to expose the most discriminative parts of two mixed images in the augmented sample.
We suspect that unless some major breakthrough occurs, the predictable future of data augmentation lies in further hybridization, with more and more methods joint together to optimize the final accuracy or to address some niche problem.
Another promising direction, which we believe will be explored in the near future, is image-based selection of DA techniques that aims at finding an optimal augmentation for each sample. One of possible realizations of this idea has been proposed in MetaAugment approach [86]. There is also room for optimization based solutions that combine learning the optimal DA selection and parametrization with the target model training. Such a bi-level approach has been recently proposed in ABO method [53].
In the field of DAPS methods a promising, though yet under-explored research avenue is considering more recent, well-established DA techniques thus extending the search space beyond traditional transformations (e.g. affine or color related transformations) which may potentially lead to improved performance.