Real-time visual content description system based on MPEG-7 descriptors

Abstract

This paper presents a real-time Visual Content Description System (VCDS) based on MPEG-7 descriptors. In our approach the system’s structure is divided into two parts, the first of which is the extraction of the descriptors using the VCDS. The second part uses the descriptors’ values in a particular search algorithm. We propose here original solutions for both parts. The proposed system architecture could be used for real-time video indexing and retrieval, content summarization, content delivery, surveillance, personalized services, etc. The descriptor extractor IP core, which is part of the VCDS, implements four MPEG-7 visual descriptors and was designed for ASIC implementation in CMOS 0.35 μm, which is a novel solution for a real-time content description problem. The proposed hardware architecture splits the computational burden into several threads, so that calculations are made simultaneously in order to improve the system’s speed. These methods make the hardware implementation of the most computationally demanding modules of the system more time- and power-efficient. Four different variations of the basic hardware architecture are discussed. New search algorithms based on the VCDS responses are also proposed. Experimental results demonstrate the effectiveness of the hardware architectures, and the new approach to similarity-based searching methods.

Appendix: Object matching techniques

In this section we describe three object matching techniques used in our experiments. The matching process inputs are always two color images: I ^′, I ^′′. The particular object extracted from an image i is referred to as $O_{i}^{\prime}$. Object properties are indicated as follows:

• .Vol—object’s area (volume);

• .SCD—object’s SCD;

• .EHD—object’s EHD;

• .x, .y—the coefficients of i-th object’s mass center.

1.1 Combined matching technique

The combined matching technique relies on a two-step image similarity measurement. The steps are called griddles, because of their specific task (each one filters similar images based on different measurement techniques). The first griddle passes those images which have a sufficient number of similar features.

Assume that n, m are the number of objects in images I ^′, I ^′′ respectively. Then we have:

$$ \left\{\exists i,j \ : \ D\left(O^{\prime}_{i},O_{j}^{\prime\prime}\right)\leq t_1,t_2 \right\}\Rightarrow \left\{s=s+1\right\} \label{eq:com-match-1-griddle} $$

(8)

where

$$ D\left(a,b\right) \leq t_1, t_2 \Leftrightarrow \left\{\left[ L_1\left(a.SCD,b.SCD\right)\leq t_1 \right]\wedge\right. \left.\left[L_1\left(a.EHD,b.EHD\right)\leq t_2 \right] \right\}\\ \label{eq:com-match-dist} $$

(9)

and i = 1,..,n j = 1,..,m. s is the number of similar features on both of the images.

The second one is based on calculations of the combined distance which takes into account SCD, EHD and vertical placement of the objects on the images:

$$ \left\{\left[s\geq 0.6n\right] \vee \left[s\geq0.9m\right]\right\}\Rightarrow \left\{\forall i,j \left[\mathrm{min}\left(L_1\left(O^{'}_{i}.SCD,O^{''}_{j} .SCD\right)\right)\right] \ : \ \mathrm{d}\left(O^{'}_{i},O^{''}_{j}\right) \right\} $$

where

$$ \mathrm{d}\left(a,b\right) = \left\{1.5L_1\left(a.SCD,b.SCD\right)\right. + \left. 4L_1\left(a.EHD,b.EHD\right) + 8|a.y-b.y|\right\}\\ \label{eq:com-match-dist-final} $$

(10)

is the distance that allows us to rank images passed through first griddle.

1.2 Similar objects matching technique

This method is quite similar to the previous one. We have modified the similarity measure used in the first step—when two objects are matched as similar, the respective bits in an additional vector are set in order to exclude the objects from the remainder of the similarity measurement process.

Assume that n,m are the number of objects in images I ^′, I ^′′ respectively. We initialize a vector $M\in\Im^{1\times n}, \ \left\{ \forall i \ : \ M\left(i\right)=0 \right\}$.

The first griddle works as follows:

$$ \left\{\left[\exists i,j \ : \ D\left(O^{'}_{i},O_{j}^{''}\right)\leq t_1,t_2\right] \wedge \left[M\left(i\right)=0\right] \right\}\Rightarrow \left\{s=s+1, M\left(i\right)=1\right\} $$

The distance D is computed in the same manner as in the previous matching method (8). Note that in this technique, the placement of the objects is not taken into account. For further distance calculations we assume

$$ \left\{\left[s\geq 0.6n\right] \vee \left[s\geq 0.9m\right]\right\}\Rightarrow \left\{\forall i,j \left[\mathrm{min}\left(L_1\left(O^{'}_{i}.SCD,O^{''}_{j} .SCD\right)\right)\right] \ : \ \mathrm{d}\left(O^{'}_{i},O^{''}_{j}\right) \right\} $$

where

$$ \mathrm{d}\left(a,b\right) = \left\{1.5L_1\left(a.SCD,b.SCD\right) +\right. \left. 4L_1\left(a.EHD,b.EHD\right)\right\} \label{eq:sim-match-dist-final} $$

(11)

is the distance that allows us to rank images passed through first griddle.

1.3 The biggest object matching technique

The biggest object matching technique differ considerably from the other techniques presented. It does not contain complex similarity matching methods—the idea is to find the two biggest objects in the reference image and to calculate the distance between the selected objects from the first image, and the most similar objects from the second image. The assumption that we must use is that n,m ≥ 2. Then we have:

$$\begin{array}{rrr} \left\{\exists i,j \left[\mathrm{max}\left(O_{i}^{'}.Vol\right)\right.\right. \wedge \\ \left.\left.\mathrm{min}\left(L_1\left(O_{i}^{'}.SCD,O_{j}^{''} .SCD\right)\right)\right] \ : \ d(O_{i}^{'},O_{j}^{''})\right\} \wedge\\ \left\{\exists l,k \left[\mathrm{max}\left(O_{l}^{'}.Vol\right)\right.\right. \wedge \\ \left.\left.\mathrm{min}\left(L_1\left(O_{l}^{'}.SCD,O_{k}^{''} .SCD\right)\right)\right] \ : \ d(O_{l}^{'},O_{k}^{''})\right\} \wedge\\ \left(l\neq i\right)\wedge \left(l\neq k\right)\Rightarrow 0.5\mathrm{d}\left(O^{'}_{i},O^{''}_{j}\right)+0.5\mathrm{d}\left(O^{'}_{l},O^{ ''}_{k} \right) \end{array} $$

Where the distance d is computed as follows:

$$ \mathrm{d}\left(a,b\right) = L_1(a.SCD,b.SCD) \label{eq:big-match-dist-final} $$

(12)