Image and Video Retargetting by Darting

Abstract

This paper considers the problem of altering an image by imperceptibly adding or removing pixels, for example, to fit a differently shaped frame with minimal loss of interesting content. We show how to construct a family of convex programs that suitably rearrange pixels while minimizing image artifacts and distortions. We call this “darting” on analogy to a tailor’s darts—small edits are discreetly distributed throughout the fabric of the image. We develop a reduction to integer dynamic programming on edit trellises, yielding fast algorithms. One- and two-pass variants of the method have O(1) per-pixel complexity. Of the many edits that darting supports, five are demonstrated here: image retargetting to smaller aspect ratios; adding or moving or removing scene objects while preserving image dimensions; image expansion with gaps filled by a rudimentary form of texture synthesis; temporal video summarization by “packing” motion in time; and an extension to spatial video retargetting that avoids motion artifacts by preserving optical flow.