Introduction

An inseparable part of every statistical data analysis is the discussion of the uncertainties that characterize the data and the model’s predictions. The first type of uncertainty is essential when formulating a model, building the likelihood, etc., while the other type reflects the model’s predictive abilities. Indeed, providing the model’s predictions with uncertainties allows assessing how confident the model is. A detailed error analysis of the model’s predictions enables one to investigate the quality of the model, particularly verifying whether the model over- or under-fits the data. The latter aspect is crucial if one makes predictions outside the data domain.

Deep learning (DL) methods1 proved their excellence in many domains of life2. Indeed, DL systems are utilized in image recognition and classification3 as well as speech recognition4 problems, or even in the game GO5. DL systems help optimize complex computational systems6, extracting non-trivial information hidden in big data7. DL systems are used in high-risk applications, such as autonomous driving8 or medical image analysis9,10,11. In high-risk applications, the knowledge of how certain the system’s predictions are is decidedly of great importance.

There is no doubt that proper estimation of uncertainties of a DL system’s prediction’s is crucial to control the validity of its model and applicability. Therefore, many groups have recently been studying this problem—for a recent review, see Gawlikowski et al.12. As many model parameters define a DL system, optimizing it requires immense computational power. Hence the successful statistical approach to the DL system, one which takes into account various sources of uncertainties, should be simple to implement and, at the same time, should not decrease the efficiency of the resulting DL model.

The methods for estimating the uncertainties for (shallow) neural networks (NN) were developed before the origin of deep learning. One of the directions was to use the Bayesian statistics that offer a consistent approach to error analysis for NNs13. Unfortunately, the Bayesian tools are not readily applicable to DL systems. For instance, the Bayesian technique developed by MacKay14 utilizes the Laplace approximation, which is simple in implementation and fast in execution for the shallow NN, but very difficult to implement for DL systems. Indeed, in MacKay’s approach to predicting the model uncertainties, the hessian matrix must be inverted, which is impossible to perform in the case of a DL system due to too many model parameters. Other Bayesian methods utilize the Monte Carlo (MC) chain algorithms15. Again, using this type of approach for the DL is difficult due to the massive number of model parameters. Nevertheless, the Bayesian-inspired, as well as the non-Bayesian statistical techniques for DL, have been developed recently16,17. They enable DL users to perform error analysis. Many proposed approaches are based on either the MC dropout or the bootstrap technique. The MC dropout18 is an example of the Bayesian-inspired approach. In contrast, the bootstrap technique is an excellent example of the non-Bayesian method19, which statisticians have successfully exploited for years. Presently, it is widely adapted for shallow and deep NN20.

This paper aims to consider one of the practical (microbiological) applications of DL systems—a problem of counting microorganisms on images of Petri dishes. Our goal is to perform the error analysis, which will allow us to understand the validity of the counting system and let us propose the significant model’s improvements. Indeed, our main achievement is a proposal of the self-normalization mechanism, implemented as additional modules in the network architecture. A counting system with self-normalization modules works more efficiently, accurately, and in the broader domain than the vanilla model.

There are many different approaches to object counting on images. An obvious way to face this type of problem is to build a regression model21,22. The biggest advantage of this method is that it is enough to label each image with the number of objects in it. If a dataset includes more detailed annotation with a bounding box for every object in an image, it is feasible to leverage detectors for object counting23,24,25. Eventually, an autoencoder can be used to estimate the density map (DM) based on a given image, which can be later integrated to obtain the number of objects26,27,28,29,30. It requires objects to be labeled with the coordinates of their center. The latter two methods performed better on the dataset we worked on in our experiments. The main goal of this study is to perform a comprehensive statistical analysis for a deep learning solution. Thus, we decided to focus on the density map approach because the architecture of autoencoders is usually less complex than recent neural networks for object detection, and the methods introduced in this paper are more transparent.

Counting microbial colonies on Petri dishes is an essential step in a microbiological laboratory to evaluate the cleanliness of the samples. Traditionally, this task is done manually or semi-automatically using traditional computer vision methods31,32. However, recent studies prove that a DL-based methodology accelerates the process23,25,29,33,34,35. We shall utilize the DM method to predict the number of microbial colonies on a Petri dish. The DM system is defined by U\(^2\)-Net DL model36. To test the DM models, we utilize the AGAR (Annotated Germs for Automated Recognition) dataset23, which includes several thousand pictures of Petri dishes with five species of microbial colonies grown.

Figure 1
figure 1

SNDM network: U\(^2\)-Net extended by the normalization bypass-layers (blue) that output \(\beta\) normalization parameter. Note that \(\beta\) parameter multiplies last layers of the U\(^2\)-Net that creates the output density map.

Full size image

The vanilla U\(^2\)-Net network accurately predicts the number of microbes up to about 60 colonies but underestimates the counts above. It also overestimates its prediction for some samples with just a few colonies. We adapt two known statistical approaches to perform an error analysis of the U\(^2\)-Net predictions. Namely, we implement the bootstrap and MC dropout frameworks, respectively. The analysis of the model’s uncertainties indicates that the main problem with the performance of U\(^2\)-Net is not the localization of microorganisms itself, but rather the normalization of density maps for a large range of possible outcomes (from 0 to 100 colonies on a Petri dish). Thus, we propose a modification of the U\(^2\)-Net, called Self-Normalized Density Map (SNDM) architecture, which includes extra normalization layers significantly improving the system’s performance in the previously failing domain. Moreover, the SNDM works as efficiently as detector-base approaches, such as Faster37 and Cascade38 R-CNN.

The paper is organized as follows: in second section the DM and U\(^2\)-Net architecture is introduced and the AGAR dataset is described; third section reviews two statistical approaches for neural networks designed to estimate the uncertainties, namely, bootstrap and MC dropout; in fourth section we present our numerical results. The SNDM is introduced, and its performance is studied in detail. Conclusions are drawn in the last section.

Density map and microbiological data

U\(^2\)-net as density map

The DM, denoted by \(\mathscr {M}\), transforms the given input image, \(\mathbf {x}\), into a density map target

$$\begin{aligned} \mathbf {y} = \mathscr {M} (\mathbf {x}). \end{aligned}$$
(1)

Note that \(\mathscr {M}\) is parametrized by weights and hyperparameters, which we intentionally omit in notation because they are not relevant to our discussion. In our approach, the total number of objects in the image is obtained by summing over the entries (the pixels) of the density map \({\textbf {y}}\).

The convolutional neural network (CNN) has been designed to face problems such as image recognition, image classification39,40, and segmentation. Hence the CNN-based models are also proposed to transform the image into a density map. Among several types of CNNs, architecture U-Net-type41 is the most successful in reproducing the density map of the objects30. In the present paper, we discuss the results of the analysis in which the U\(^2\)-Net36 is utilized.

The U\(^2\)-Net has a form of a two-level nested U-shaped structure as presented in Fig. 1. The top-level is built of eleven U-Net blocks, connected by pooling (encoder branch) or upsampling layers (decoder branch), with additional concatenation connections. On the bottom level, each U-Net block itself also has an encoder-decoder structure, within additional residual connections in the U-Net block.

The analysis is performed on the AGAR dataset, including images of Petri dishes with bounding box annotations for each colony. To generate density maps, we assume that the center of objects lies in the center of the bounding box, and corresponding coordinates are added to an empty map in the form of single pixels. Next, the map is blurred with a Gaussian filter, which leads to a smooth density map normalized so that the sum over all pixels gives the total number of colonies. In that way we create the annotation map for each input image. The autoencoder is trained to transform a real image into a density map. Note that each density map is normalized so that the sum over all pixels gives the total number of objects. Hence the network predicting the density map for an input picture also provides information about the total number of objects.

Figure 2
figure 2

Exemplary images of Petri dish from the higher-resolution subset of the AGAR dataset that was used during the numerical experiments. From the left: empty dish with no colonies, dish containing 16 colonies of B. subtilis, dish containing 9 colonies of E. coli, and dish containing 72 colonies of S. aureus. Samples differ in numbers of colonies, types of microbial species cultured, and image acquisition setups—detailed information can be found in Supplementary Material for Majchrowska et al.23.

Full size image
Figure 3
figure 3

Histogram of the number of annotated microbial colonies per image in the training (left), validation (center), and test (right) subsets.

Full size image

AGAR dataset

As mentioned before, we utilize the AGAR dataset23 to test the properties of the U\(^2\)-Net in the MC dropout and bootstrap approaches. It is a huge microbiological database including 18 000 high-resolution images of Petri dishes with cultured colonies of 5 standard microbes taken under diverse lighting conditions. Professional microbiologists manually annotated microbial colonies by precisely labeling each colony using a bounding box. AGAR is diverse data collected to automate microbial colony counting. The data covers edge cases, including highly crowded plates with overlapping colonies, cases with extensive and tiny colonies, and cases where colonies are hardly visible, e.g., when located close to the dish edge. Typical plates captured in the various setups (three different illumination conditions) with microbial colonies of different shapes and sizes are shown in Fig. 2.

Note that AGAR mainly contains images with fewer than 50 colonies, about \(85\%\) of samples (see Fig. 3), or even empty dishes, which are essential for testing the models’ tendency to generate false-positive counts.

Uncertainties in DM

Some methods of estimation of uncertainties in the predictions of the DM, for the U-Net architecture, have recently been discussed by Eaton-Rosen et al.42. Our paper focuses on two approaches: bootstrap and MC dropout adapted for U\(^2\)-Net. We briefly review both of them in the following two subsections.

Bootstrap method

A bootstrap method for error estimation is one of the oldest and the most popular methods of statistics19. An extensive review can be found in the work of Hastie et al.43, whereas Tibshirani44 discusses the application of the bootstrap technique in the neural network analyses. Two types of bootstrap approaches are distinguished, namely, pairs sampling and residual sampling algorithms. We adapt the first option.

The main idea of the bootstrap is to train the same model on some number of data subsets sampled (with replacement) from the original set. Then, as a result, one obtains the ensemble of models. In the inference mode, one makes predictions of all models in the ensemble. The bootstrap model prediction is then given by the mean over the model predictions in the ensemble. Additionally, the variance over the models’ outcomes gives \(1\sigma\) uncertainty.

The procedure of taking the average over the models is robust to overfitting. In our applications, we consider \(B=20\) bootstrap training subsets. The same network architecture is trained for each subset, and the best model is selected using the validation subset. If the noise of the data is random and not correlated (from sample to sample), then the total error obtained for the ensemble of models is B times smaller than the error of one model. For the detailed explanation, see Bishop45. Note that the bootstrap approach is not a Bayesian approach, but if it is combined with early stopping, it is interpreted as approximate inference46.

The known strengths of the bootstrap approach are the following:

  • It is simple in implementation and relatively fast in execution, i.e., it is usually enough to run the analysis for \(B=10\) to 20.

  • The obtained model is given by the ensemble of networks of the same architecture but with a different configuration of network parameters.

  • The predictions of the bootstrap model are similar to the Gaussian process (GP) in data range20. Indeed, the GP model is defined by some density distribution used to sample the model. The best model prediction is given by the average over the outcomes of the model samples.

The mean overall models give the response of the bootstrap model. Therefore the bootstrap model should not overfit the data. Indeed, a single fit obtained for a particular subset of the data might be overtrained, but the mean over the models is not. Hence the bootstrap model should be characterized by a good generalization ability.

The main steps of the bootstrap approach:

  1. (i)

    consider the training data set \(\mathscr {D}_{train}\) with the total number of samples equals N;

  2. (ii)

    obtain B subsets \(C_i \subset \mathscr {D}_{train}\) with an equal number of images. \(C_i\) is obtained by sampling from the data \(\mathscr {D}\) (with replacement) b images. Note that in our experiments \(B=20\), moreover \(b \approx 0.63 \,N\) as it is recommended in the literature43;

  3. (iii)

    consider network \(\mathscr {M}\) and train it on \(C_i\) data subset, as the result one gets the model \(\mathscr {M}_i^B \equiv \mathscr {M}(C_i)\);

  4. (iv)

    to make the predictions for given input \(\mathbf {x}\) compute the outcome of each network \(\mathscr {M}_i^B\) and take the mean, namely:

    $$\begin{aligned} \overline{\mathscr {M}}(\mathbf {x}) = \frac{1}{B}\sum _{i=1}^B \mathscr {M}_i(\mathbf {x}), \end{aligned}$$
    (2)
  5. (v)

    to estimate the uncertainty for the prediction of the network calculate the variance:

    $$\begin{aligned} \Delta ^2 \mathscr {M}(\mathbf {x}) = \displaystyle \frac{1}{B} \sum _{i=1}^B\left( \overline{\mathscr {M}}(\mathbf {x}) - \mathscr {M}_i(\mathbf {x})\right) ^2, \end{aligned}$$
    (3)

    where \(\sqrt{\Delta ^2 \mathscr {M}(\mathbf {x})}\) is interpreted as \(1\sigma\) uncertainty.

MC dropout

In 1995, Neal proved that a class of neural networks with a single hidden layer and an infinite number of units converge to the Gaussian process15. Moreover, Rasmussen and Williams47 studied Gaussian process methods in supervised learning. In both cases, the Bayesian statistics stand a background for all derivations. It is believed that the GP approach allows accessing the model prediction uncertainties with reasonable accuracy.

Motivated by the Bayesian statistics as well as the success of GP techniques in the estimation of model uncertainties, Gal and Ghahramani studied the dropout technique18,48. They showed that a deep neural network with a dropout layer after each weight layer can be understood as the Gaussian process.

The main idea of the approach is to keep dropout layers active in the training and inference modes. It means that, for a given input \(\mathbf {x}\), every prediction is computed with a slightly different configuration of active units. Therefore, to obtain the MC dropout prediction for a given neural network, one has to run it N-times. Then, the mean over the predictions gives the best value while the variance estimates the uncertainty of the predictions.

Figure 4
figure 4

Dependence of the predicted number of colonies on ground truth for the test data set within the DM model (right), and self-normalized DM model (left). Note that for ideal detection every blue dot representing a single Petri dish image should lay on the \(y=x\) orange line. Uncertainties, denoted by black bars, in the bootstrap and MC dropout approaches are given in top and bottom rows, respectively.

Full size image
Figure 5
figure 5

Dependence of the counting uncertainty on mean colony size calculated for the test subset within both models. Uncertainty values are averaged on intervals of mean colonies size.

Full size image
Figure 6
figure 6

Dependence of the counting uncertainty per single colony on coverage factor calculated for the test subset using both models.

Full size image

The main steps of the MC dropout:

  1. (i)

    Consider a network \(\mathscr {M}_{dropout}\);

  2. (ii)

    Train the \(\mathscr {M}_{dropout}\) on the data set \(\mathscr {D}_{train}\). The training is run with regularization—we apply Adam algorithm with weight decay \(\mu =0.0005\). Note that during the training the dropout layers are active.

  3. (iii)

    To make a prediction for given input vector \(\mathbf {x}\), run the network \(\mathscr {M}_{dropout}\), \(r=20\) times, keeping dropout layers active. Then the dropout network prediction is given by the

    $$\begin{aligned} \overline{\mathscr {M}}_{dropout}(\mathbf {x}) = \frac{1}{r}\sum _{i=1}^r \mathscr {M}_{dropout}^i(\mathbf {x}), \end{aligned}$$
    (4)

    where \(\mathscr {M}_{dropout}^i(\mathbf {x})\) denotes the i-th run prediction of the network. The \(1\sigma\) uncertainty is given by the square root of the variance:

    $$\begin{aligned} \Delta ^2 \mathscr {M}_{dropout}(\mathbf {x}) = \displaystyle \frac{1}{r} \sum _{i=1}^r\left( \overline{\mathscr {M}}_{dropout}(\mathbf {x}) - \mathscr {M}_{dropout}^i(\mathbf {x})\right) ^2. \end{aligned}$$
    (5)
Figure 7
figure 7

Examples of density maps for two exemplary input images from the test subset (with 16 and 72 labelled colonies), together with their uncertainties per pixel: the DM model (rows 1st and 3rd), and the SNDM model (rows 2nd and 4th). In the first case DM model predicts \(17\pm 0.99\), while SNDM predicts \(16\pm 0.44\) colonies. In the second case DM predicts \(63\pm 3.32\), and the SNDM predicts \(71\pm 1.57\) colonies. Uncertainty maps calculated using MC dropout method.

Full size image
Figure 8
figure 8

Dependence of the normalization parameter \(\beta\) on the number of colonies. Uncertainties of the \(\beta\) parameter calculated using bootstrap method (left) and MC dropout (right). Results for the test subset.

Full size image

Numerical results

This section examines the properties of the U\(^2\)-Net architecture as a density map predictor. The results of these studies allow us to propose the self-normalization mechanism in the U\(^2\)-Net model. We justify the relevance of this approach in the following sections.

To study the U\(^2\)-Net properties, we consider the higher-resolution subset of the AGAR dataset, represented by RGB images of Petri dishes with a resolution of about \(4000\times 4000\) pixels. We limit our discussion to images containing no more than 100 colonies. Because, from the microbiological point of view, the accuracy of automatic counting is less relevant for more populated samples, i.e. determining whether the sample is contaminated or not is the most important.

The analyzed data is divided into three subsets: training (5656 images), validation (706 images), and test (706 images) data subset. As introduced earlier, we consider two types of statistical approaches for estimating uncertainties in predictions of deep neural networks: the bootstrap and the MC dropout. In both methods, we exploit the same network architecture shown in Fig. 1. Moreover, the same training scheme is adapted for both models—the Adam optimization algorithm with learning rate of 0.0001, weight decay equals 0.0005, and the size of the minibatch of two. The training takes at least 100 epochs.

We ran the computations on NVIDIA TITAN RTX GPU with 24 GB of memory. The training time was about 40 h for MC dropout. The bootstrap experiments took longer, and the training lasted about four days, performed in parallel on five GPUs.

Performance of vanilla U\(^2\)-Net as a density map predictor

To access the quality of both discussed models, we compare the predictions of the network versus the ground truth computed on the test data set; see the left column in Fig. 4. Each predicted number of colonies is given with \(1\sigma\) uncertainty. Note that both statistical models work well for the samples with the number of colonies smaller than 60. However, for images with the number of colonies lower than 10, the DM count is overestimated. Predictions with tiny uncertainties characterize the MC dropout for these cases. Consequently, the model disagrees with the data within the \(1\sigma\) confidence level. In contrast, the bootstrap model predictions have larger uncertainties. Hence, the model agrees with the ground truth in the \(1\sigma\) level. From that perspective, the bootstrap approach works better than the MC dropout.

Table 1 Average uncertainties of predictions for the test dataset within the DM and the SNDM models in the bootstrap and MC dropout approaches (as presented in Fig. 4).
Full size table

The detailed examination of our results shows that the uncertainty of the model’s predictions depends on the mean size of colonies for a given dish. It is illustrated in Fig. 5 (left panel). Interestingly, for the images with smaller colonies, the uncertainties for the bootstrap approach are larger than for the dropout model. However, when the size of the colonies increases, the effect is the opposite.

We noted that the uncertainties of the model’s prediction (normalized to the number of colonies) weakly depend on the dish coverage (the fraction of the dish covered by colonies), as shown in Fig. 6 (left panel). It is interesting to note that the MC dropout uncertainties are systematically larger than for the bootstrap approach, see Table 1, where the average (over the test dataset) uncertainties are given.

In Fig. 7, we showed four panels. Each contains an input picture, a map of annotations, network prediction, and a map of uncertainties computed for each output pixel. We show the results for two dishes that contain 16 and 72 bacterial colonies, respectively. The DM prediction agrees with the ground truth in the first case (first panel). In contrast to the second case (third panel), the DM underestimates the count. However, in the latter case, the DM correctly reproduces spots representing the objects in the image, but the most significant uncertainties are consistently observed in the spot positions. Therefore, we conclude that the DM reconstructs the map of spots (objects) correctly but has a problem finding the correct normalization for the images with a larger number of colonies. Therefore, we propose a modification of the network architecture so that the network can correct by itself the normalization of the output.

Self-normalized DM (SNDM)

To propose the normalization module in the network, we must explain how the loss function is constructed in the U\(^2\)-Net model.

Let us denote by \(E_{\mathrm {U}^2}\) the loss function U\(^2\)-Net. It is a function of the data and model parameters. From the statistical perspective, it refers to the logarithm of the likelihood function. In practice, \(E_{\mathrm {U}^2}\) has a quite complicated structure36:

$$\begin{aligned} E_{\mathrm {U}^2} = \sum _\mathrm {images} \sum _{M=0}^6 \mathscr {L}( P_M,P^G) \end{aligned}$$
(6)

where \(P_M\) and \(P^G\) denote predicted density map by M-th U-Net block and the ground truth, respectively.

The \(\mathscr {L}_M\) is the standard binary cross entropy function:

$$\begin{aligned} \mathscr {L}(P,P_G) = - \sum _{i,j}^{H,W} \left[ {P_G}_{i,j} \ln P_{i,j} + (1 - {P_G}_{i,j}) \ln (1- P_{i,j}) \right] , \end{aligned}$$
(7)

where i, j denotes the pixel indexes in the figure of the width W and height H.

The U\(^2\)-Net generates six output maps, denoted by \(S_{1,\dots ,6}\), from the encoder (\(S_6\)) and five from the decoder branch (\(S_{1,\dots ,5}\)), respectively. Outputs from smaller blocks (\(S_{2,\dots ,6}\)) are next upsampled to fit the size of the output map. Note that \(S_1\) already has the proper \(1008\times 1008\) size. After upsampling (we still denote them by \(S_i\)), each of them gives the density map and \(P_i = \sigma (S_i)\), where \(\sigma\) is the sigmoid activation function. All maps \(P_{1..6}\) contributes to the loss function. Moreover, \(S_{1..6}\) maps are fused with a concatenation operation followed by a \(1\!\times \!1\) convolution layer (giving \(S_0\)). Then \(P_0=\sigma (S_0)\) is the final density map output of the U\(^2\)-Net. The \(P_0\) contributes also to the loss function.

Our idea is to allow the network to correct the normalization of output pixels in \(S_i\) maps. Since every output pixel ranges from 0 to 1, one can not simply re-scale the \(P_i\)-outputs by some normalization number. Therefore, we propose the rescaling of pixels normalization in \(S_i\) map. We use the property that sigmoid is a linear function on neighborhood, namely, \(\sigma (x) \approx = \frac{1}{2} + \frac{x}{2} +\dots\). Hence, the rescaling of the sigmoid argument changes the total normalization of the sigmoid almost linearly in this range. In practice, we rescale each \(S_i\) output by a factor \(\beta\), so

$$\begin{aligned} P_i = \sigma (S_i) \rightarrow \sigma (\beta S_i). \end{aligned}$$
(8)
Figure 9
figure 9

Dependence of \(\beta\) parameter on dish coverage, together with mean colony size indicated by the color of points. The points for different ranges of colony size (see the legend) cluster in subsets that show approximately linear dependence between \(\beta\) and coverage. Additionally, a linear function is fitted in each range, with a slope that decreases with colony size. Results obtained within MC dropout for the test subset.

Full size image

The normalization value \(\beta\) might depart from one, and it can be a function of the distribution of colonies, the total coverage, etc. Therefore, we assume that \(\beta\) depends on the input image and is an additional network outcome. To limit the number of parameters that define \(\beta\) we connect it with the smallest U-Net block of U\(^2\)-Net, the encoder part, see Fig. 1. In the rest of the paper, the U\(^2\)-Net architecture with our modification will be called a self-normalized density map (SNDM).

In principle, the system should work without the need for corrections, \(\beta =1\), hence we expect that \(\beta\) should take values around one. To keep the control \(\beta\)-dependence, we add to the loss term a penalty term:

$$\begin{aligned} E_{\mathrm {U}^2} \rightarrow E_{\mathrm {U}^2}+\sum _{images}\frac{1}{2}\left( 1 - \beta \right) ^2. \end{aligned}$$
(9)

In general \(\beta\) can take any positive number, but, in practical applications, we assume that \(\beta = \tilde{\sigma }(\dots ) = \lambda \sigma (\dots )\) to speed up the process of training. It guarantees that normalization ranges from 0 to 1.5. In our experiments, we tested a sigmoid with \(\lambda =2\) and \(\tilde{\sigma }=ReLU\). However, the most optimal results have been obtained for the sigmoid activation function with \(\lambda =1.5\).

In Fig. 4 (right column of panel), we plot the predictions of SNDM versus ground truth. We see that modifying U\(^2\)-Net architecture by introducing the self-normalization module significantly improves the network’s predictions. Indeed, the SNDM works well for images containing less than 60 objects and samples with the number of objects larger than 60. Interestingly, in the improved model, the bootstrap and MC dropout predictions are in agreement. The estimated uncertainties for both approaches became similar, see Figs. 5 and 6 (right panel), as well as Table 1. Moreover, the bootstrap uncertainties for SNDM are reduced with respect to the original model.

Figure 10
figure 10

Density maps of one of the outliers from Fig. 4 predicted with the SNDM. The model predicts \(29\pm 7.30\) colonies, while the sample contains 77 colonies of two types: 19 P. aeruginosa (big ones), and 58 C. albicans (small ones). The blue boxes represent all the colonies’ ground truth position and size. Uncertainties calculated using the bootstrap method.

Full size image

The normalization parameter \(\beta\) depends on the features of the input image, such as the number of colonies. It is illustrated in Fig. 8. For both statistical approaches, \(\beta\) varies from 0.96 to 1.06 with the tendency to take lower values for less crowded images. On the other hand, for the images containing more than 50 objects, \(\beta\) varies less. It is also worth mentioning that \(\beta\) takes a wide range of values for a given number of colonies, which varies with the number of colonies, which translates to coverage and other features like mean colony size. It is visible in Fig. 9 where we plot the dependence of \(\beta\) on the dish coverage. When we group the samples by the mean colony size (indicated by different colors of points in Fig. 9), one can observe that \(\beta\) almost linearly depends on coverage. Moreover, the slope of the fitted lines increases gradually with the mean size of colonies.

Figure 7, shows density maps together with pixel uncertainties for two input pictures with 16 and 72 objects, respectively. The predictions for the SNDM are shown in the second and the fourth panel, respectively. We see that SNDM outperforms the original U\(^2\)-Net. Indeed, the SNDM accurately predicts the number of colonies for both images.

Eventually, we note that, overlapping objects is one of the difficulties the counter system must face. In Fig. 7, we see an example of colonies that overlap. In this case, the amplitude of density maps is naturally enlarged. But the interesting thing is that also uncertainties have higher values there—see the pixel uncertainty maps in the last column of Fig. 7. Therefore, we may conclude that these areas have a more substantial impact on counting uncertainties. However, if one compares DM and SNDM pixel uncertainty in the overlapping regions, then in the SNDM case, uncertainties are slightly reduced compared to DM.

The detailed inspection of the results presented in Fig. 4 shows that there are input figures for which the SNDM network overshoots. In Fig. 10 we plot the predicted map with uncertainties for such a case. We see that the network’s main problem is the presence of colonies of different sizes on the same dish. It complicates the inference of the normalization parameter, which also depends on the size of the colonies, see in Fig. 9. The blue boxes in Fig. 10 represent ground truth labels showing the precise position and size of the colonies. It should be also noted that the pixel uncertainties for small colonies in the small box area are much larger than for big colonies located in bigger boxes.

Table 2 Counting metrics, mean absolute error (MAE) (Eq. 10) and symmetric Mean Absolute Percentage Error (sMAPE) (Eq. 11), for the DM and SNDM model (in MC dropout) in comparison with other counting methods that use popular R-CNN object detectors: Faster37 and Cascade38, which are discussed by Majchrowska et al.23.
Full size table

We close the discussion of the numerical result by comparing the SNDM model predictions with the other popular counting systems. In one of our previous papers23,25 we discussed some of them. For the comparison, we choose two of the most successful approaches (in our experiments) to count AGAR microbiological colonies, namely, Faster R-CNN detector37 and Cascade R-CNN detector38. Note that, for both models, the algorithm first performs the detection of interesting objects and then counts them.

To quantitatively compare various models predictions, we consider a standard Mean Absolute Error:

$$\begin{aligned} \mathrm {MAE} =\frac{1}{N}\sum _{i=1}^N|n_i - \tilde{n}_i| , \end{aligned}$$
(10)

and less common in usage, symmetric Mean Absolute Percentage Error:

$$\begin{aligned} \mathrm {sMAPE} = \frac{100\%}{N}\sum _{i=1}^N\frac{|n_i - \tilde{n}_i|}{|n_i + \tilde{n}_i|}, \end{aligned}$$
(11)

where N is a number of all samples, \(n_i\) is true count of microbe colonies present on i-th image, and \(\tilde{n}_i\) is predicted number of colonies. Note that the sMAPE shows the relative error with respect to the overall number of colonies.

Both metrics are obtained for the higher-resolution subset of the AGAR dataset and the four models: Faster and Cascade detectors, as well as DM and SNDM in the MC dropout approach. The metrics are given in Table 2. We see that counting by detection gives better results than the standard DM model. But the SNDM gives comparable or even slightly better counting results than the detectors. Further discussion of other deep learning detectors, e.g., well-known YOLO, can be found in our previous paper24. Note that preparing annotations for the density map models are less expensive than for detector models. Moreover, the training of the detector systems is more resources-demanding and time-consuming than the density map models. Therefore, for the data for which the density map approaches work with the same or better efficiency as the detectors, the density map systems as SNDM are recommended.

Summary

We have shown that the detailed analysis of the uncertainties in the deep neural network model leads to a deeper understanding of its limitations. It allows us to assess how uncertain the predictions are. More profound knowledge of the model allowed us to modify the DM model by introducing the Self-Normalization mechanism. The SNDM significantly improves the DM predictions. Indeed, before the modification, the model underestimated the number of colonies for highly populated dishes. In contrast, for some images with a small number of colonies, the DM model overestimated the counts. The SNDM network proposed by us can correctly count in both mentioned domains. Its performance is comparable with the other counting systems, such as Faster and Cascade R-CNN detectors. Eventually, we note that the bootstrap and the MC dropout predictions and the estimate of uncertainties in the SNDM are in statistical agreement, which was not observed in the original model.