Keywords

1 Introduction

In an outdoor computer vision system like a surveillance video system, several atmospheric interferences such as rain streak, snow, etc. affect the video contents and features[1, 2]. In challenging weather, these unwanted interferences degrade the performance of video content analysis (VCA) algorithm such as scene analysis [3], event detection [4], object detection [5] and tracking [6] of various computer vision system [7]. Rain streak removal in the video (RSRV) has lots of importance in several computer vision applications i.e. driverless car, surveillance camera, traffic surveillance, and other relevant applications.

The RSRV has recently got lots of attention in the computer vision research area due to its new challenging applications. Various numeric methods have been proposed to remove rain streaks in a video sequence to increase the visibility of video content [8,9,10]. Garg et al. [8] initially introduced to an RSRV technique with a comprehensive analysis of the visual characteristic of the rain streak in a video sequence. Since then many techniques have been proposed for the RSRV tasks and achieved good results in various rain conditions. An extensive summarization of video-based rain streak removal is included in [9]. Chen et al. [10] have been proposed an RSRV technique for the highly dynamic scenes. Considering the directional properties of rain streak, a tensor-based rain streak removal method has been proposed in [11]. In Wei et al. [12], authors have modelled the rain streak stochastically using a mixture of Gaussian technique while Li et al. [13] proposed multiscale convolutional filters to separate different size of rain streaks.

In recent time, a technique using spatiotemporal appearance (STA) properties of rain streak has been proposed to remove rain streak in a video sequence [14]. The method counts moving objects including rain appearances at the pixel level for a number of frames and assumes that if the counts of the appearance of moving objects are within a mid-range then the pixel is identified as a rain pixel, otherwise, the pixel is identified as a background or foreground object (not rain). Normally, the pixel is a part of the background if the count is zero or the pixel is a part of the foreground object if the count is very high. This method is very successful to remove rain streak but it loses some parts of the moving object especially the short appearance part of the foreground. In this technique, authors have assumed that the appearance of moving object in a pixel exists for several frames. Sometimes, the appearance of moving objects in some pixels exists for a very short time only e.g. one or two frames and these pixels are excluded by the STA technique as a moving object. Figure 1(a) shows the generated mask of frame 72 of a video sequence “Traffic” by applying the STA properties of the rain streak. In this figure, the yellow marked pixels of moving object are missed by this method. The appearance of moving objects in these pixels exist for current few frames only. The corresponding original frame is given in Fig. 1(b).

Fig. 1.
figure 1

Moving objects mask generated by STA and Photometric Correlation

Full size image

In this paper, we address this problem and proposed a new method which uses the photometric correlation properties of rain streak in a video sequence. Photometric features already introduced in Garg et al. [8], based on their observation that a rain streak appears brighter than the background they have been proposed two photometric constraints. Here they consider the background as stationary, the candidate pixels which may include a rain streak should comply with the equation below

$$ \Delta I = I_{n} - I_{n - 1} = I_{n} - I_{n + 1} > c, c = 3 $$
(1)

where In is the current frame image, In-1 and In+1 are the adjacent previous and next frame respectively. C is a threshold which indicates the minimum intensity change due to rain streak. In Eq. (1) authours have used a future frame, that is not possible for real-time application. The second photometric constraint proposed in Garg et al. [8] is

$$ \Delta I = - \beta I_{b} + \alpha , \beta = \frac{\tau }{T} , \alpha = \tau \overline{E}_{d} $$
(2)

where authors consider Ib= In-1, and photometric constraint \( \beta \) is calculated as \( \left( {0 < \beta < 0.039 } \right) \). Since the maximum value of \( \alpha = 3.5 \times 10^{ - 3} \), where \( \overline{E}_{d} \) is the average irradiance. \( \beta \) involved with camera exposure time T and time \( \tau \) that a drop projects onto a pixel. \( \tau \) depends on the physical properties of the rain. \( \tau \) can vary with the various size of rain streak. The range of \( \tau \) obtained as \( 0 < \tau < 1.18 \). Camera exposure time T also can vary in different video sequence captured by a different camera. In this paper, we propose a novel algorithm by combining the STA properties of rain streak and the photometric correlation of rain streak with the background to improve the recovered moving objects in rain-free videos.

We have applied the proposed algorithm on four different video sequences including real and synthetic rain streak to observe the performance and compare them with the state-of-the-art methods. The overall performance of the proposed algorithm is significantly better than the state-of-the-art methods for both real and synthetic rain streaks.

2 Proposed Method

This section will describe the details of the proposed methodology. We divide our methodology into four sections and describe separately. This method has been proposed by combining the photometric features of rain streak with the STA property of rain streak. In this method, firstly we have applied background modelling to separate background and foreground including the rain of a video frame. The STA properties of rain streak are used to separate moving objects and rain streaks from the foreground of the video frame. In parallel, we have applied the photometric correlation of rain streak to separate the moving objects and rain streaks from the rainy frame. Then, we have fused the output results of both different features of the rain streak and generated a rain-free frame of the video sequence.

2.1 GMM Background Modelling and Foreground Generation

GMM is a well-known algorithm in the field of computer vision. We use this algorithm to generate the background of every frame and subtract the background to find the foreground of the frame. Generally, the GMM technique is used in video processing to detect moving objects. Here, all moving substances (including rain streak) in a video has been considered as moving objects and separated from the frame to generate background. In this technique, each pixel is modelled independently by a mixture of K Gaussian distributions (usual setting K = 3) [15, 16]. In our proposed technique, we assume that at time t, the value of kth Gaussian intensity = \( \eta_{k,t} \), mean = \( \mu_{k,t} \), variance = \( \sigma_{k,t}^{2} \), and weight in the mixture = \( \omega_{k,t} \), so that \( \sum\limits_{k = 1}^{K} {\omega_{k,t} } = 1 \). In the first step of our experiment, the parameters are initialized as follows: standard deviation (\( \sigma_{k}^{{}} \)) = 2.5, weight (\( \omega_{k} \)) = 0.001 and learning rate, \( \alpha \) = 0.1. α is used for balancing the contribution between current and previous values and the value is 0 < α<1 [17].

In the second step, after initializing the parameters, the current pixels are used to match with kth Gaussian for every new observation if the condition \( \left| {X_{t} - \mu_{k,t} } \right| \le 2.5\sigma_{k,t} \) is satisfied against existing models, where \( X_{t} \) is the new pixel intensity at time t. If a model matches, the Gaussian model will be updated as follows:

$$ \mu_{k,t} \leftarrow (1 - \alpha )\mu_{k,t - 1} + \alpha X_{t} $$
(3)
$$ \sigma_{k,t}^{2} \leftarrow (1 - \alpha )\sigma_{k,t - 1}^{2} + \alpha (X_{t} - \mu_{k,t} )^{T} (X_{t} - \mu_{k,t} ); $$
(4)
$$ \omega_{k,t} \leftarrow (1 - \alpha )\omega_{k,t - 1} + \alpha , $$
(5)

and the weights of other Gaussians models are updated as

$$ \omega_{k,t} \leftarrow (1 - \alpha )\omega_{k,t - 1} $$
(6)

In the third step, the values of weights are normalized among all models in such a way that \( \sum\limits_{k = 1}^{K} {\omega_{k,t} } = 1. \) On the other hand, if a model fails to match, then a new model is introduced with initial parameter values. If it has already crossed the maximum allowable number of models, based on the value of weight/standard deviation, a new model substitutes the existing model. At each frame, we generate a background frame using the mean value of the stable model. The model is considered stable if the ratio of weight and the standard deviation is the minimum compared to other models of the pixel. We use the background frame to find rain streak and other foregrounds and as well to generate the rain-free video in the proposed method.

Initially, we have generated foreground by subtracting the background from the input frame,

$$ F_{n} = \left| {I_{n} - B} \right| $$
(7)

where F is foreground image of the nth frame and I is the original image of the nth frame. This image contains rain streak and moving objects.

2.2 STA Properties of Rain

The rain streak appearance is temporary in a frame. This temporary appearance property has been used to separate rain streak and moving objects from the foreground in [14]. To apply this property, authors first apply an intensity threshold 20 [15, 18, 19] to generate a binary image of foreground contains rain streak and moving objects. To separate rain streak and moving objects, a mask has been generated using STA properties of rain streak.

$$ M_{n} = \mathop \sum \limits_{i = n}^{n - m} F_{i} ;(i = n,n - 1,n - 2, \ldots n - m) $$
(8)

Where M is the mask for nth frame generated by summing the binary foreground images F of m number adjacent frame. The mask contains value 0 to m for every pixel and this value represents the appearance value of moving objects and rain in a pixel. İf an appearance value in a pixel is more than 20% of the maximum value m, this pixel is considered as moving object. Using this phenomenon, moving objects and rain streak can be separated in a frame.

2.3 Photometric Correlation

The rain streaks produce a positive fluctuation of the intensities in the pixels of a frame. The rain streak intensity depends on the brightness of the drop as well as the background scene radiance [8]. The fluctuation of positive intensity for rain streaks is not very high, because the rain streak intensity includes the background radiance. On the other hand, the moving object intensity does not depend on the background radiance. Thus, the moving objects can produce a high-intensity fluctuation in the pixels of a frame. Here we have applied this different photometric property of the rain streak and moving objects to separate them. We observe that moving objects produce intensity fluctuation significantly high in most of the scenario. Here we used an intensity correlation for each pixel which is obtained between background B and separated foreground F.

$$ \gamma = \frac{F(i,j)}{B(i,j)} $$
(9)

Where \( \gamma \) is photometric correlation constraint. The candidate pixels which may contain the moving objects should comply with the condition \( \gamma > 1 \). Otherwise, it will consider as rain streak or background.

2.4 Rain Free Video Generation

Here we have fused the output of both features and generate an object mask for the processing frame. To generate rain-free video we have used a background image and input image of the processing frame. The moving objects have been generated from the input image and the other part of the frame have been generated from the background image of the processing frame.

3 Experimental Results

We have applied the proposed algorithm on four different video sequences including two synthetic rain streak and two real rain streak. For synthetic rain streak dataset, we have compared results in objectively and subjectively as ground truth is known in these cases while the results of the real rain streak datasets have been compared in subjectively only. Here we have compared the performance of the proposed method with two other methods [12, 14]. We have obtained the results of these methods by implementing their software on the datasets. The software of Wei et al. [12] has been downloaded from their website [20] whereas we have collected the software of [14] by personal communication.

3.1 Results on Videos with Synthetic Rain Streaks

To evaluate the performance of the proposed method on video sequences with synthetic rain streak, we have used two video sequences with heavy rain and dense rain respectively. The first video sequence named “truck” has been downloaded from CDNET dataset [21] which contains a moving truck and externally added heavy rain streak. Figure 2 shows the visual performance comparison of the state-of-the-art methods with the proposed method in one of the important frame (Frame 65) of the video sequence. In Fig. 2, the red circle indicates the rain streak removal performance by all methods respectively. The proposed method outperforms Wei et al’s method [12] whereas the method in [14] performs better than the proposed method by a small number of rain streaks. However, the method in [14] losses more part of the moving object than the proposed method. The rectangle indicates the loss of moving objects in the frame.

Fig. 2.
figure 2

Rain removal results on video sequence with synthetic rain.

Full size image

Figure 3 shows the performance comparison in objectively. The calculated PSNR value of every frame of the video sequence “truck” for all methods are shown in the graph. The PSNR graph in Fig. 3 shows that Wei et al. [12] perform better than the proposed method for the first few frames. The proposed method outperforms all other methods after the first 30 frames. The proposed algorithm processes input video frames as it comes in, typically without storing input frames of the video and the learning process is also real-time. Thus, the proposed algorithm needs the first few frames to learn and then delivers a better result for the rest. That is the main reason why the proposed method struggles for the first few frames. The advantage of this algorithm is that it can serve in a real-time application with the consideration of required processing time.

Fig. 3.
figure 3

Comparison graph of PSNR calculated against ground truth of full video sequence with synthetic rain streak in frame level.

Full size image

Figure 4 shows the subjective performance comparison of the state-of-the-art methods with the proposed method for the video sequence “Highway”. This video sequence includes a moving car and externally added dense rain streaks. The proposed method shows better performance in subjective comparison. The circle indicates the rain streaks those are removed by the proposed method and [14] but missed by Wei et al’s method [12]. The rectangle marked area shows the distortion of moving objects by the RSRV process. The method in [14] provides the maximum distortion of the moving objects.

Fig. 4.
figure 4

Rain removal results on video sequence with synthetic rain.

Full size image

The objective performance of the synthetic video sequence is listed in Table 1. Table 1 represents the PSNR and SSIM value of the frame that is shown in Figs. 2 and 4 for visual comparison. Thus, we can evaluate the methods in terms of objective and subjective in the same manner. For the frame shown in Fig. 2, the proposed method outperforms the state-of-the-art method. On Fig. 4, the method in [14] and the proposed method shows almost the same PSNR value which is higher than Wei et al. [12]. In terms of SSIM, the proposed method outperform all the state-of-the-art methods.

Table 1. Performance comparison of the different methods on video with synthetic rain in terms of PSNR and SSIM against ground truth
Full size table

3.2 Results on Videos with Natural Rain Streaks

To observe the performance on a video sequence with natural rain streaks, we show the experiments on two different video sequences. One of them includes moving objects and the other does not have any moving object. The rain streaks are varied from moderate to dense in these different video sequences.

In Fig. 5, we represent the experimental results of the video sequence denoted as ‘traffic’ [22] by implementing both methods. Input video includes two moving objects, one moving car and one pedestrian with dense rain. The subjective comparison of the proposed method shows better performance compared to the state-of-the-art methods. The proposed method outperforms Wei et al. [12] and [14] by removing rain streak and maintaining moving objects in a better way.

Fig. 5.
figure 5

RSRV methods performance comparison of a video sequence with real rain streaks.

Full size image

Figure 6 clearly shows the performance comparison of all methods on the video sequence denoted as ‘wall’ [22] of size 288 × 368 × 171 in terms of subject quality. The video sequence ‘wall’ consists of a regular pattern in the background. The input frame represents moderate rain streaks. The proposed method and [14] both outperform Wei et al.’s method [12] for almost complete rain streak removal.

Fig. 6.
figure 6

RSRV methods performance comparison of a video sequence with real rain streaks where no moving objects exist.

Full size image

4 Conclusion

This paper proposed a new algorithm combining STA properties of rain streak and photometric features to remove rain streak in videos. This method uses photometric correlation of rain streak with background intensity as a photometric feature. This method can separate moving objects and rain streak from the foreground of the video frame. This method successfully exploits the duration trend of the rain and the moving objects through the STA features and the amount of pixel intensity variations of the rain and moving objects and then combines them to separate rain and other moving regions for better performance. The results on real and synthetic rains in videos reveal that the proposed method outperforms the state-of-the-art methods by removing rain and keeping other moving objects in better quality. In future, we will investigate a wider range of moving objects and rains.