Video Extrapolation in Space and Time

Abstract

Novel view synthesis (NVS) and video prediction (VP) are typically considered disjoint tasks in computer vision. However, they can both be seen as ways to observe the spatial-temporal world: NVS aims to synthesize a scene from a new point of view, while VP aims to see a scene from a new point of time. These two tasks provide complementary signals to obtain a scene representation, as viewpoint changes from spatial observations inform depth, and temporal observations inform the motion of cameras and individual objects. Inspired by these observations, we propose to study the problem of Video Extrapolation in Space and Time (VEST). We propose a model that leverages the self-supervision and the complementary cues from both tasks, while existing methods can only solve one of them. Experiments show that our method achieves performance better than or comparable to several state-of-the-art NVS and VP methods on indoor and outdoor real-world datasets. (Project page: https://cs.stanford.edu/~yzzhang/projects/vest/.)

Appendices

A Architecture Details

The architecture used for the MPI encoder is specified in Table 4.

Table 4. MP2 is max pooling with stride 2, Up2 is nearest-neighbor upsampling with scale 2, + is concatenation. Reshape transforms a tensor with $C \times D$ channels into C channels, and D is merged to the batch dimension, and ReshapeBack is the reverse operation. All layers up till up1b use ReLU activation and the layers for conv1, conv2 and conv3 use LeakyReLu with a negative slope 0.2. There is no activation following the very last layer. All layers use Instance Norm for activation normalization and Spectral Norm for weight normalization.

B Implementation Details

To have a better gradient flow, similar to Tucker et al. [32], we add a harmonious bias 1/i to the alpha channel prediction, so that $w_i$ from Eq. (11) becomes uniformly 1/D during initialization. We also add an identity bias to $f^\theta$ such that each MPI plane is associated with zero motion during initialization.

In all experiments, we set the number of MPI planes to be $D = 16$. The depth values for MPI planes are linear in the inverse space, with $d_1 = 1000$ and $d_D = 1$.

Table 5. Ablation on the number of MPI planes D. Increasing the plane count improves the performance but also increases the training time. We adopt $D=16$ in the main paper since further increasing D results in diminishing returns.

C Training Details

1.1 C.1 KITTI

Since videos from KITTI are taken by stereo cameras with fixed relative poses, the depth scale is consistent across scenes and therefore we set it to be a constant $\sigma = 1$. We use $\lambda _{1}^\text {space} = 1000$, $\lambda _\text {spec}^\text {space} = 100$, $\lambda _{1}^\text {time} = 1000$, and $\lambda _\text {perc}^\text {time} = 10$. We use Adam Optimizer [9] with an initial learning rate 0.0002, which we exponentially decrease by a factor of 0.8 for every 5 epochs. We train our model for 200K iterations on two NVIDIA TITAN RTX GPUs for about two days. During training, we apply horizontal flip with 50% probability and apply color jittering as data augmentation.

1.2 C.2 RealEstate10K

We train our model for 200K iterations on one NVIDIA GeForce RTX 3090 GPU, which takes about one day. We use $\mathcal {L}_1^\text {space} = 10, \mathcal {L}_\text {perc}^\text {space} = 10, \mathcal {L}_1^\text {time} = 10, \mathcal {L}_\text {perc}^\text {time} = 0$. We use Adam Optimizer [9] with a constant learning rate 0.0002.

1.3 C.3 Ablations on the Number of MPI Planes

To study the effect of the number of MPI planes, we perform an ablation study on the KITTI [6] dataset with resolution $128\times 384$. As shown in Table 5, a small number of MPI planes ($D=4$ or 8) results in degraded model performance. Further increasing the number of planes from 16 to 32 results in marginal performance gain, with a cost of $2.1\times$ slower training time. Therefore, we use $D=16$ for all other experiments.

Table 6. Results of next-frame prediction on CATER [7]. Our model achieves better performance compared to PredRNN [36].

1.4 C.4 Modeling Dynamic Scenes

To test whether our method is able to model more dynamic scenes, we test our method on CATER [7], a dataset of scenes with 5–10 individually moving objects. We show a quantitative comparison with a video prediction baseline PredRNN [36]. As shown in Table 6, our model achieves better performance across all three metrics.

Qualitatively, our method makes temporal prediction consistent with the ground truth object motions on this dataset. In Fig. 12, the model correctly recovers the purple object and the gold object occluded by the blue cone. Our model effectively handles object occlusions by warping from neighboring pixels with similar RGB values.

Fig. 12.

Model prediction on an example scene with occlusion. (a) and (b) are two historical frames as model inputs, (c) and (d) are the predicted and ground truth next frame, respectively. Top-left corners of subfigures are zoomed-in views for occluded regions.

1.5 C.5 Discussions

While we focus on demonstrating the possibility of simultaneous extrapolation in both space and time, specific modules can be further optimized for each task. For example, it is possible to improve the dynamic scene representation to better handle video prediction with long horizons or highly complex motion, or to synthesize novel views with a large viewpoint change.

In the meantime, while our method is designed for natural scenes with many potential positive impacts such as interactive scene exploration for family entertainment, like all other visual content generation methods, our method might be exploited by malicious users with potential negative impacts. We expect such impacts to be minimal as our method is not designed to work with human videos. In our code release, we will explicitly specify allowable uses of our system with appropriate licenses. We will use techniques such as watermarking to label visual content generated by our system.