Fuzzy Motion-Adaptive De-Interlacing with Smart Temporal Interpolation

Abstract

The performance of a motion-adaptive de-interlacing algorithm is directly dependent on the quality of the spatial and temporal interpolators. As shown in Chapter 4, the inclusion of an edge-dependent algorithm clearly improves the results in moving areas of the image, whereas small improvements are obtained in zones where the level of motion is inferior since a combination of the spatial and temporal interpolators is carried out in these cases. Similarly, the performance of the motion-adaptive de-interlacing algorithm relies partially on the quality of the temporal interpolator in pixels with an intermediate level of motion and the dependence is complete in static areas. The aim of this Chapter is to improve the temporal interpolator used in the fuzzy motion-adaptive de-interlacing algorithm developed in the previous Chapters of this book. The temporal technique used in the de-interlacing process could be selected according to the origin of image sequences. In this sense, pull-down material is increasingly combined with interlaced video such as computer generated moving and/or animated imagery or scrolling texts. Unfortunately, knowledge of the picture repetition pattern is not included in the transmission, and this information is highly relevant for de-interlacing. For instance, de-interlacing is perfectly performed by weaving the correct fields of pull-down material. This algorithm is the best due to its simplicity, however, if the correct field is not selected, an annoying artifact known as ‘feathering’ appears (see Figure 5.1). Different detectors have been proposed in the literature to identify the field-pairs originated by the pull-down process from the same film image. The primitive approaches perform a global identification for the entire field, that is, a control signal is activated to denote the presence of film material. One of these detectors, called zero-vector matching detectors, try to match the zero motion vector on a previous field. This kind of detectors normally use two kinds of signals: a first one to calculate the frame similarity and a second one to measure the field similarity. Based on the analysis of both similarity metrics, control signals are generated, which indicate the mode of the video signal, i.e. video or film, and the type and phase of the film mode to determinate the image’s position in the 3:2 or 2:2 pull-down pattern. This approach is described in [1], and it is used by the majority of current film detectors. However, there are some drawbacks in the performance of these conventional film detectors since the sum of absolute differences and its comparison with an unique threshold value is not suitable. Firstly, an unique sum does not properly measure the level of motion. For example, in the case that the majority of parts in the image are static but there are moving parts of small dimensions, a low value of the global sum is achieved. This ignores the moving areas and, hence, feathering effect appears in this moving area. A second disadvantage is the determination of the threshold value. Its selection is extremely difficult since the properties of the image are very variable. Other approaches try to analyze the frame characteristics to find out the repetition pattern [2]-[5]. Among them, several proposals have been reported in the literature to identify jagged edges in frames due to ‘feathering’ effect [2],[3]. Other detectors study the location of edges in the frames [4]. The idea is that, if two frames are similar, edges should be at the same spatial position. Therefore, the analysis of edges position can reveal the picture repetition pattern. Finally, a motion vector based approach has been proposed in [5]. The sum of the length of the motion vectors should indicate if two fields are identical or not, since fields from the same frame should provide a similar value of the sum. In recent literature, the research works in the area of film-detection can be categorized into two groups. A first one is focused on the increase of the robustness in the detection of pattern repetition. This is especially relevant since wrong mode decisions can produce annoying artifacts. In [6], the number of wrong film mode decisions are reduced by using a new motion detection scheme called the ‘Arrow’ detector. It reduces the problem of interpreting vertically detailed areas as motion. The method proposed in [7] also obtains a considerable robustness improvement by using a layered structure that separates the film-mode source from interlaced TV signal to guarantee the correction of weaving sequence. The second group of research works is focused on the development of film-mode detectors that work locally. This kind of algorithms is very demanded due to the high increase of TV material that combines images from different origins in a single field, which is known as hybrid material. None of the techniques previously cited can locally deal with hybrid material, because all of them detect a single mode for the entire field. The proposal described in [8] can assign different modes in a single field. This algorithm firstly implements an image segmentation process and, after that, one mode decision is made for each detected object. In [9], an adaptive interpolation is performed by weighting between field insertion and other interpolation algorithm for de-interlacing purpose. The weighting factors are the output of a fuzzy system, and they are obtained by analyzing as inputs of the system the intra and inter-field signal differences concerning the current pixel along a set of pre-determinated directions. Based on the idea of film-mode detectors, an effective temporal interpolator is presented in this Chapter. The aim is to detect the most adequate combination of neighboring pixels in the temporal domain, so that, weaving is applied with the correct field when picture repetition appears. Since this strategy is locally performed, the algorithm can deal with any kind of sequences: film, hybrid, and video sequences. This Chapter is organized as follows: Section 5.1 describes the new approach, which is able to adapt the strategy of interpolation to the presence of repeated areas in consecutive fields. The advantages of including the new temporal interpolator are shown in Section 5.2 by de-interlacing video, hybrid, and film sequences. These results are compared with those obtained with the previous versions of the algorithm in Section 5.3, whereas Section 5.4 summarizes a comparison with MC de-interlacing algorithms. Finally, some conclusions are expounded in Section 5.6.