Deep learning-based importance map for content-aware media retargeting

Abstract

We introduce a deep learning-driven framework for creating an adaptably applicable importance map (A2R-Map) that can be integrated with existing image and video retargeting operators. A conventional retargeting algorithm uses a heuristic approach to seek an off-the-self algorithm used into their retargeting system. The extracted importance map of the image does not match the characteristics of the input image; therefore, it affects the retargeting results and limits the performance of the retargeting method. Our designed framework attempts to minimize the artifacts/distortions caused by inappropriate energy, e.g., the shrunk phenomenon in warping-based results and carving-through-object distortion in the seam carving-based approach. Our proposed framework focuses on capturing sensitive distortion regions and activating their energy to solve this challenge. We verify the effectiveness of our proposed scheme by plugging it in three typical retargeting methods: seam carving-based, warping-based for image, and video retargeting. Extensive experiments and evaluations are conducted on two widely used databases. On the one hand, A2R-Map significantly reduces the time of importance map generation in retargeting systems to $\sim 9$ times compared to the baseline saliency map. On the other hand, our A2R-Map achieves improvement over the baseline methods with an average of 11% and 9% in terms of image and video quality, respectively. The experimental results and evaluations demonstrate that our strategy for A2R-Map substantially outperforms the previous works and significantly boosts the visual quality of video/image retargeting.

Appendices

Appendix A: More comparisons

Apart from the comparisons with content-aware retargeting approaches, we further exhibit our results competing with a semantic-aware retargeting approach [60] in Fig. 17. In this figure, besides [60] (PM), we further show the results from other five retargeting methods: seam carving (SC) [3] and its improved version (ISC) [61], patch-based warping (PW) [6], saliency-based mesh parametrization (SMP) [23], multi-operator (MOR) [4]. These results are obtained from [60]. We can observe that our result outperforms the compared results. If the carving distortions occur in SC, ISC, and SMP, a linear-like phenomenon falls in PW, MOR, and PM (e.g., the green door). Meanwhile, our result does not have such phenomena and appears in a balanced structure compared to the input image. Figure 18 exhibits the performance of our A2R-Map in terms of enlarging. In this experiment, we enlarge images to 25% of width.

Fig. 12

Comparison with Multi-operator and Cui et al. [21]

Fig. 13

Left to right: input image, NIF energy map, SC + NIF, A2R-Map, SC + A2R-Map

Fig. 14

Comparison with CarvingNet

Fig. 15

Comparison with WSSDCNN

Fig. 16

Comparison with Cycle-IR

Fig. 17

Comparisons with content-aware and semantic-aware retargeting approaches

Fig. 18

Enlarging results. In each pair, left: input image, right: enlarged one

Appendix B: List of notations

Symbol	Definition
$\mathcal {I}$	Input image
SC	Seam carving operator
OMap	The energy map generated by TFS-Net
BMap	The energy map generated by (4)
A2R-Map	The final importance map generated by our model
SOD	Salient Object Detection
TFS-Net	The network we proposed to generate OMap
TFS	Feature Sharing Session module
AFS	Adjacent-layer Feature Sharing module
$\mathcal {X}^i$	Feature maps at layer $i^{th}$
$\mathcal {X}_u$	Feature maps at upper layer
$\mathcal {X}_l$	Feature maps at lower layer
$\mathcal {A}^s$	The source image/video in general
$\mathcal {A}^t$	The target image/video $\mathcal {A}^s$ after retargeting process
$\mathcal {P}$	A certain resizing operator
$\mathcal {R}$	A retargeting system using operator $\mathcal {P}$ to resize $\mathcal {A}^s$ and output $\mathcal {A}^t$
$\mathcal {M}$	An off-the-shelf method that $\mathcal {R}$ uses to define the importance in the input $\mathcal {A}^s$