Vision-based outlier detection techniques in automated surveillance: a survey and future ideas

Abstract

Outlier detection is one of the emerging study topics influenced by video annotation. An outlier is anything odd or irregular that deviates from the norm. Outlier detection is subjective since it is influenced by a variety of contextual circumstances. Human mobility patterns and classification are explored in this survey to combine it for further anomaly detection. There is a possibility of noise interference in automated surveillance, which disturbs a video stream and makes it difficult to identify details, reducing its accuracy. The presence of such frequent noise-interference raises the error rate, particularly in real-time processing models. Aside from that, Motion Camouflage must be addressed in intelligent surveillance to obtain a clear view of the frame. This paper offers a focused survey, highlighting the three major issues in video anomaly detection: the inadequate utilization of Motion Patterns, the prolonged Noise Interference that affects accuracy and leads to an increased error rate, and the elevated false-alarm rate due to Motion Camouflage. A brief analysis of these techniques, as well as their limitations, is provided. This paper focuses on probable challenges in the field of video-based outlier detection in automated surveillance and ways to mitigate those challenges. Moreover, this survey draws attention to the promising directions of research.

Appendix

This section contains supplementary material with a significant outlook towards some of the mathematical work relevant to the research scope of this survey.

1.1 A. SC2Net

According to [67] Sparse coding (SC) has demonstrated effectiveness in uncovering semantic information from noisy and high dimensional data. The given method develops a novel variant of l1-solver by introducing the adaptive momentum vectors into ISTA to enable per parameter updates and encapsulate the historical information in optimization. Given a data matrix $\textbf{X}=\left[ \textbf{x}_{1}, \textbf{x}_{2}, \cdots , \textbf{x}_{n}\right] \in \mathbb {R}^{d_{x} \times n}$, sparse coding aims to learn a dictionary $\textbf{B}=\left[ \textbf{b}_{1}, \textbf{b}_{2}, \cdots , \textbf{b}_{n}\right] \in \mathbb {R}^{d_{x} \times \bar{d}_{s}}$ that is used to generate sparse codes $\left[ \textbf{s}_{1}, \textbf{s}_{2}, \cdots , \textbf{s}_{n}\right] \in \mathbb {R}^{d_{s} \times n}$ for the input data X. The optimization problem can be formulated as follows:

$$\begin{aligned} \begin{aligned} \min _{\textbf{S}, \textbf{B}}&\sum _{i}\left\| \textbf{x}_{i}-\textbf{B} \textbf{s}_{i}\right\| _{2}^{2} \\ \text{ s.t. }&\left\| \textbf{s}_{i}\right\| _{0} \le k, \text{ and } \left\| \textbf{b}_{j}\right\| ^{2} \le 1, j=1, \cdots d_{s} \end{aligned} \end{aligned}$$

(10)

The above optimization is hard to solve due to the non-convexity of the $\ell _{0}$ norm. Therefore, it is often relaxed to the following problem with the $\ell _{1}$ norm,

$$\begin{aligned} \begin{aligned} \min _{\textbf{S}, \textbf{B}}&\sum _{i}\left\| \textbf{x}_{i}-\textbf{B} \textbf{s}_{i}\right\| _{2}^{2}+\lambda \left\| \textbf{s}_{i}\right\| _{1}^{2} \\ \text{ s.t. }&\left\| \textbf{b}_{j}\right\| ^{2} \le 1, \text{ and } j=1, \cdots d_{s}. \end{aligned} \end{aligned}$$

(11)

To solve (11), a conventional way is to alternatingly optimize B and S, which correspond to two optimization procedures: dictionary learning and sparse approximation. Specifically, by fixing S, (11) reduces to the following $\ell _{2}$ constrained optimization problem,

$$\begin{aligned} \begin{array}{cl} \min _{\textbf{B}} &{} \Vert \textbf{X}-\textbf{B S}\Vert _{F}^{2} \\ \text{ s.t. } &{} \left\| \textbf{b}_{i}\right\| ^{2} \le 1, \text{ and } i=1, \cdots d_{s}. \end{array} \end{aligned}$$

(12)

The above problem is the well-known ridge regression problem, which has a closed-form solution. By fixing B, (12) reduces to the sparse approximation problem which aims to represent the input x by a linear combination of B as follows,

$$\begin{aligned} \min _{\textbf{s}} \sum _{i}\left\| \textbf{x}_{i}-\textbf{B} \textbf{s}_{i}\right\| _{F}^{2}+\lambda \left\| \textbf{s}_{i}\right\| _{1} \end{aligned}$$

(13)

The updating formula can be mathematically expressed as

$$\begin{aligned} \textbf{s}^{(t)}=s h_{(\lambda \tau )}\left( \textbf{s}^{(t-1)}-\tau \nabla g\left( \textbf{s}^{(t-1)}\right) \right) \end{aligned}$$

(14)

where the shrinkage function is defined as ${\text {sh}}_{(\lambda \tau )}(\textbf{s})=$ ${\text {sign}}(\textbf{s})\left( |\textbf{s}|-\lambda _{\tau }\right) _{+}$. The solution of (14) can be achieved through the following update rule,

$$\begin{aligned} \begin{aligned} \textbf{s}^{(t)}&=s h_{(\lambda \tau )}\left( \textbf{s}^{(t-1)}-\tau \left( \textbf{B}^{T}\left( \textbf{B} \textbf{s}^{(t-1)}-\textbf{X}\right) \right) \right) \\ {}&=s h_{(\lambda \tau )}\left( \textbf{W}_{e} \textbf{s}^{(t-1)}+\textbf{W}_{d} \textbf{x}\right) \end{aligned} \end{aligned}$$

(15)

where, $\textbf{W}_{e}=\textbf{I}-\tau \textbf{B}^{T} \textbf{B}$ and $\textbf{W}_{d}=\tau \textbf{B}^{T}$.

Note that (15) could be treated as a simple RNN. In other words, the $\ell _{1}$-oriented optimization algorithm can be reformulated as a model-based optimization algorithm.

1.2 B. AnomalyNet

AnomalyNet [68] is an optimization network which simultaneously achieves sparse representation and dictionary learning using a novel LSTM network (termed SC2Net). Several algorithms have been proposed to optimize neural networks by incorporating the “momentum” into the dynamics of stochastic gradient decent (SGD). These methods have shown promising performance in improving the robustness and convergence speed of SGD. Borrowing the high-level idea of these optimization methods, adaptive momentum vectors i(t), f(t) are introduced to ISTA at the time step t as follows,

$$\begin{aligned} \left. \begin{array}{r} \begin{array}{l} \tilde{\textbf{c}}^{(t)}=\textbf{W}_{e} \textbf{s}^{(t-1)}+\textbf{W}_{d} \textbf{x} \\ \textbf{c}^{(t)}=\textbf{f}^{(t)} \odot \textbf{c}^{(t-1)}+\textbf{i}^{(t)} \odot \tilde{\textbf{c}}^{(t)} \\ \textbf{s}^{(t)}={\text {sh}}_{(\lambda \tau )}\left( \textbf{c}^{(t)}\right) \end{array} \end{array}\right\} \end{aligned}$$

(16)

where $\odot$ is the element-wise product of the vectors. Here, $\tilde{\textbf{c}}^{(t)}$, $\textbf{c}^{(t)}$, and $\textbf{c}^{(t-1)}$ denotes the linear combination at current, previous and combined iteration levels respectively. In adaptive ISTA, c(t) accumulates all the historical information with different weightssimilar to the diagonal matrix containing the sum of the squares of the past gradients. In the equations, i, f and s indicate the input gate , forget gate and output gates respectively. Following the above notations, the updating rule in ISTA can be equivalently expressed to $\textbf{s}^{(t)}={\text {sh}}_{(\lambda \tau )}\left( \tilde{\textbf{c}}^{(t)}\right)$. SLSTM does not have “output gate” like the vanilla LSTM. The SLSTM unit is achieved by rewriting above equation as follows:

$$\begin{aligned} \left. \begin{array}{r} \begin{array}{l} \textbf{i}^{(t)}=\sigma \left( \textbf{W}_{i s} \textbf{s}^{(t-1)}+\textbf{W}_{i x} \textbf{x}\right) \\ \textbf{f}^{(t)}=\sigma \left( \textbf{W}_{f s} \textbf{s}^{(t-1)}+\textbf{W}_{f x} \textbf{x}\right) \\ \tilde{\textbf{c}}^{(t)}=\textbf{W}_{e} \textbf{s}^{(t-1)}+\textbf{W}_{d} \textbf{x}, \\ \textbf{c}^{(t)}=\textbf{f}^{(t)} \odot \textbf{c}^{(t-1)}+\textbf{i}^{(t)} \odot \tilde{\textbf{c}}^{(t)} \\ \textbf{s}^{(t)}=h_{(\textbf{D}, \textbf{u})}\left( \textbf{c}^{(t)}\right) \end{array} \end{array}\right\} \end{aligned}$$

(17)

where W denotes the weight matrix (e.g. $\textbf{W}_{i s}$ is the weight matrix from the input gate to the outputs), $\sigma (\textbf{x})=\frac{1}{1+e^{-\textbf{x}}}$, $h_{(\textbf{D}, \textbf{u})}=\textbf{D}(\tanh (\textbf{x}+\textbf{u})+\tanh (\textbf{x}-\textbf{u}))$ where, $\textbf{u}$ and $\textbf{D}$ denote a trainable vector and diagonal matrix, respectively. It uses smooth and differentiable nonlinear activation function named “Double tanh” instead of the shrinkage function to address the vanishing gradient problem.

1.3 C. MOCAP

In Automatic Human Motion Capture i.e MOCAP [18] all training and testing motion sequences are converted to their motion strings. A test motion string is compared with all training motion strings individually, using the suffix array technique. The i th test string has f codeword indices in its motion string while the j th training string has g codeword indices. The codeword indices in these two strings are identical and the largest common sequence is of length l. For the i th test string, the following two metrics are calculated with respect to a given motion category k.

• 1. Max-parameter: the average of MLMs of all training motions in category k;

• 2. Sim-parameter: the average of the similarity product of all training motions in category k.

For the k th category, we have

$$\begin{aligned} M A X_{i}^{k}&=\frac{\sum _{j \in k} M L M_{i, j}}{\sum _{j \in k} \underline{1}(j \in k)} \\ S I M_{i}^{k}&=\frac{\sum _{j \in k} S R P_{i, j}}{\sum _{j \in k} \underline{1}(j \in k)} \end{aligned}$$

where $\underline{1}(\cdot )$ is the indicator function.

Alternatively, for each test motion, all categories are pitted against each other, one-on-one, and the category with the higher value for the parameter is considered. All category-specific votes are aggregated together and, then, normalized to get the soft scores. That is,

$$\begin{aligned} \begin{aligned} {\text {Vote}}_{i, M A X}^{k}&=\sum _{\forall m \ne k} \underline{1}\left( M A X_{i}^{k}>M A X_{i}^{m}\right) \\ {\text {Vote}}_{i, S I M}^{k}&=\sum _{\forall m \ne k} \underline{1}\left( S I M_{i}^{k}>S I M_{i}^{m}\right) \end{aligned} \end{aligned}$$

(18)

The full body level-n codebooks that offer the best classification results are short-listed for fusion to yield the final decision.

1.4 D. Camouflaged object detection

In [37], a constructive approach is presented by characterizing entity-texture and statistical modeling of Camouflaged Images in texture smoothing conditions. Camouflage image is a combination of Camouflaged object image which is a target and Camouflage pattern image in general background. The Camouflaged object image intensities are modified by a predefined threshold value and its binary representation which is treated as a new target image. The correlation peaks $\alpha ^{\wedge 2}$ and $\beta ^{\wedge 2}$ between Scene and Normalized basis Images are evaluated to detect the target in the camouflaged image S. When the Camouflaged image is a linear combination of basis images ($\textrm{f}_{\textrm{o}}^{\wedge }\textrm{x}$ and $\mathrm {f*}^{\wedge }\textrm{x}$) the normalization of basis images with its autocorrelation is $\textrm{f}_{\textrm{o}}^{\wedge }\textrm{x}$ and $\mathrm {f*}^{\wedge }\textrm{x}$. The Correlation peaks from normalized basis images are.

$$\begin{aligned} \begin{array}{c} \alpha ^{\wedge 2}=\left[ \textrm{S}(\textrm{x}) * \textrm{f}_{\textrm{o}}^{\wedge }(\textrm{x})\right] ^{2} \text{ and } \\ \beta ^{\wedge 2}=\left[ \textrm{S}(\textrm{x}) * \mathrm f * ^{\wedge }(\textrm{x})\right] ^{2} \end{array} \end{aligned}$$

(19)

These correlations peaks obtained from the camouflage image with basis images of the target and its binary image are considered as filter coefficients to identify the target in the camouflaged image. The filter equation is given with correlation peaks is:

$$\begin{aligned} C(x)=\frac{\left[ S(x) * f_{0}(x)\right] ^{2}}{[S(x) * {f_*} (x)]^{2}-\sqrt{N}\left[ S^{2}(x) * {f_*} (x)\right] } \end{aligned}$$

(20)

If the degree of Camouflage is high enough in the case of single or multiple Camouflaged objects the modeling of the target will not be able to detect the object due to intensity variation and random textures. There is the need for some texture (regular or irregular) models to detect the Camouflaged objects. The cluster shade (CS) and cluster prominence (CP) gives the characteristic summary of GraylevelCM (GLCM). For object detection the following set of statistics are found to be suitable and their use is found to be satisfactory in both the cases of with and without texture smoothing illustration. The feature considered completely depends on normalized GLCM at different offsets.

$$\begin{aligned} \left. \begin{array}{r} \text{ GraylevelCM } =G C(i, j)=\frac{C(i, j)}{\sum _{i, j=1}^{M-1} C(i, j)} \\ \text{ Contrast } =\sum _{i, j=1}^{M}(i-j)^{2} G C(i, j) \\ \text{ CS } =\sum _{i, j=1}^{M}\left( i-D_{x}+j-D_{y}\right) ^{3} G C(i, j) \\ \qquad \begin{array}{r} \text{ CP }=\sum _{i, j=1}^{M}\left( i-D_{x}+j-D_{y}\right) ^{4} G C(i, j) \\ \end{array} \end{array}\right\} \end{aligned}$$

(21)

where,$\begin{array}{c} D_{x}=\sum _{i, j=1}^{M} i G C(i, j) and D_{y}=\sum _{i, j=1}^{M} j V(i, j). \end{array}$ Target extracted in this approach completely depends on the selection of seed block in image i.e., in texture terminology primitive element extraction. This can be done by the normalized GLCM based characteristics summation like contrast, cluster shade and cluster prominence.