Deep Long Term Prediction for Semantic Segmentation in Autonomous Driving

Abstract

Temporal prediction is an important function in autonomous driving (AD) systems as it forecasts how the environment will change and transform in the next few seconds. Humans have an inherited prediction capability that extrapolates a present scenario to the future. In this paper, we present a novel approach to look further into the future using a standard semantic segmentation representation and time series networks of varying architectures. An important property of our approach is its flexibility to predict an arbitrary time horizon into the future. We perform prediction in the semantic segmentation domain where inputs are semantic segmentation masks. We present extensive results and discussion on different data dimensionalities that can prove beneficial for prediction on longer time horizons (up to $2\,\textrm{s}$). We also show results of our approach on two widely employed datasets in AD research, i.e., Cityscapes and BDD100K. We report two types of mIoUs as we have investigated with self generated ground truth labels (mIoU$^{seg}$) for both of our dataset and actual ground truth labels (mIoU$^\textrm{gt}$) for a specific split of the Cityscapes dataset. Our method achieves $57.12\%$ and $83.95\%$ mIoU$^{seg}$, respectively, on the validation split of BDD100K and Cityscapes, for short-term time horizon predictions (up to $0.2\,\textrm{s}$ and $0.06\,\textrm{s}$), outperforming the current state of the art on Cityscapes by $13.71\%$ absolute. For long-term predictions (up to $2\,\textrm{s}$ and $0.6\,\textrm{s}$), we achieve $37.96\%$ and $63.65\%$ mIoU$^{seg}$, respectively, for BDD100K and Cityscapes. Specifically on the validation split of Cityscapes with perfect ground truth annotations, we achieve $67.55\%$ and $63.60\%$ mIoU$^\textrm{gt}$, outperforming current state of the art by $1.45\%$ absolute and $4.2\%$ absolute with time horizon predictions up to $0.06\,\textrm{s}$ and $0.18\,\textrm{s}$, respectively.

A Supplementary Material

Table 4. Re-ordering of semantic classes in BDD100K

1.1 A.1 Qualitative Results

In this section, we show the qualitative results of our method on sequences of BDD100K, $\mathcal {D}^\mathrm {BDD-MOTS}_\textrm{val}$ and Cityscapes, $\mathcal {D}^\mathrm {CS-vid}_\textrm{val}$. In Fig. 5, we show the qualitative results of prediction method on a sequence of $\mathcal {D}^\mathrm {BDD-MOTS}_\textrm{val}$. We can observe that, for increasing time steps, i.e., $\varDelta {t}=\{1, 2, 5, 10\}$, the prediction worsens for dynamic objects. This can be inferred from the increase in white regions in the absolute difference estimation visualizations (bottom row) defined as $\hat{\textbf{d}}_{t}=|\hat{\textbf{m}}_{t}-\overline{\textbf{m}}_{t}|$. Also, majority of the predictions are incorrect in the boundary segregation of different classes, i.e., where car pixel occupancy meets road occupancy, where sidewalk occupancy meets road occupancy.

Specifically for the Cityscapes dataset, in Fig. 6, the predictions (middle row) are at par with their corresponding 20$^\textrm{th}$ frame ground truth annotations (top row) for $\varDelta {t}=\{1, 2, 5\}$, obtained from the dataset. The semantic class boundaries are so well captured with noise supression in predictions (middle row). However, for $\varDelta {t}=10$, we can see that the prediction focuses more on predicting the static classes than the finer dynamic classes boundary details, i.e., visible from the missing sidewalk in the left region of $\hat{\textbf{m}}_{t+10}$ that is visible in the ground truth annotation $\overline{\textbf{m}}_{t+10}$ (in pink) of Fig. 6.

Fig. 5.

Output predictions for a sequence of the BDD100K validation split, $\mathcal {D}^\mathrm {BDD-MOTS}_\textrm{val}$. The top row depicts the pseudo ground truth $\overline{\textbf{m}}_{t}$, $\overline{\textbf{m}}_{t+1}$, $\overline{\textbf{m}}_{t+2}$, $\overline{\textbf{m}}_{t+5}$, $\overline{\textbf{m}}_{t+10}$ generated by PSANet. In the middle row, we show the input semantic segmentation $\overline{\textbf{m}}_{t}$ along with the predictions $\hat{\textbf{m}}_{t+1}$, $\hat{\textbf{m}}_{t+2}$, $\hat{\textbf{m}}_{t+5}$, $\hat{\textbf{m}}_{t+10}$ from the prediction network. The bottom row portrays the absolute difference $\hat{\textbf{d}}_{t+1}$, $\hat{\textbf{d}}_{t+2}$, $\hat{\textbf{d}}_{t+5}$, $\hat{\textbf{d}}_{t+10}$, between the ground truth and prediction frames.

Fig. 6.

Output predictions for a sequence of the Cityscapes validation split, $\mathcal {D}^\mathrm {CS-vid}_\textrm{val}$. The top row depicts actual 20$^\textrm{th}$ frame ground truth annotations available in the dataset for $\overline{\textbf{m}}_{t}$, $\overline{\textbf{m}}_{t+1}$, $\overline{\textbf{m}}_{t+2}$, $\overline{\textbf{m}}_{t+5}$, $\overline{\textbf{m}}_{t+10}$. In the middle row, we show the input semantic segmentation $\overline{\textbf{m}}_{t}$ along with the predictions $\hat{\textbf{m}}_{t+1}$, $\hat{\textbf{m}}_{t+2}$, $\hat{\textbf{m}}_{t+5}$, $\hat{\textbf{m}}_{t+10}$ from the prediction network. The bottom row portrays the absolute difference $\hat{\textbf{d}}_{t+1}$, $\hat{\textbf{d}}_{t+2}$, $\hat{\textbf{d}}_{t+5}$, $\hat{\textbf{d}}_{t+10}$, between the ground truth and prediction frames.

Fig. 7.

A semantic segmentation input mask $\overline{\textbf{m}}_{t}$ from $\mathcal {D}^\mathrm {BDD-MOTS}_\textrm{val}$ showing different semantic classes. The left white encircled region portrays the proximal occurence of class sidewalk ($s=2$) near to class road ($s=1$). Similarly, the right white encircled region portrays the proximal occurrence of class road ($s=1$) near to class car ($s=14$).

1.2 A.2 Prediction Invariance on the Ordering of Semantic Classes

Table 5. Confusion matrix on BDD100K: scores of all the ${S}=19$ classes in BDD100K with original ordering. The classes with highest true score are highlighted in red and the cells with second highest true score are marked in light red.

Table 6. Confusion matrix on BDD100K: scores of all the ${S}=19$ classes in BDD100K with first re-ordering. The classes with highest true score are highlighted in red and the cells with second highest true score are marked in light red.

Table 7. Confusion matrix on BDD100K: scores of all the ${S}=19$ classes in BDD100K with second re-ordering. The classes with highest true score are highlighted in red and the cells with second highest true score are marked in light red.

We investigate the behavior of our model when 1-channel inputs are fed to our predictor network, i.e., the generated pseudo ground truth $\overline{\textbf{m}}_{t} \in \mathcal {S}^{H \times W \times 1}$ for the BDD100K dataset $\mathcal {D}^\mathrm {BDD-MOTS}$. The semantic segmentation mask $\overline{\textbf{m}}_{t}$ contains class indices $s \in \mathcal {S}=\{1, 2, ..., S\}$, where $S=19$. Here, each semantic class corresponds to a specific class index s, e.g., $s=0$ for class road, $s=12$ for class person, etc according to the scene. For instance, if we consider a small region in a semantic segmentation mask, we usually find a certain semantic class pixels in close proximity with another semantic class pixels, i.e., class road pixels ($s=1$) almost always occurs adjacent to class car pixels ($s=14$) and class sidewalk pixels ($s=2$) almost are always adjacent to class road pixels ($s=1$) as can be seen in Fig. 7. To investigate the proposed method’s performance and robustness when the original class orientation is re-ordered, we conducted some experiments by shuffling the class indexes in the generated pseudo ground truth frames $\overline{\textbf{m}}_{t}$. For instance, now the same scene would contain class road ($s=5$) adjacent to class car ($s=16$) and class sidewalk ($s=9$) adjacent to class road ($s=5$). Note that, the semantic classes remain the same, just the class indices are shuffled randomly. In Table 4, we can see the original class order along with the re-ordered class indices where semantic classes are marked by their actual defined colors in $\mathcal {D}^{BDD-MOTS}$. can This is an important investigation to prove that our predictor model still learns the proximal relationship between the semantic classes instead of the numerical class indices occupancy, i.e., our model perfectly learns that class road pixels are most likely to occur near class car pixels and vice-versa, irrespective of their class indices value. Hence, we performed experiments by re-ordering the class indices of $\mathcal {D}^\mathrm {BDD-MOTS}$ in such a way that the classes that occurred near to each other in terms of class index distance, e.g., road ($s=1$) and sidewalk ($s=2$) are now placed further apart, e.g., road ($s=5$) and sidewalk ($s=9$) as can be seen in Table 4.

Table 5 shows the confusion matrix for the original class order of $\mathcal {D}^\mathrm {BDD-MOTS}$. The confusion matrix represents how each class in the prediction is confused and interpreted with respect to all the classes present in the ground truth and vice-versa. It can be observed that, every class is predicted well with the highest score for itself (see diagonal) except class rider ($s=13$) which is predicted as class person ($s=12$) with a score of 0.49 which is obvious as rider fits into the broader category of person after all. Similarly, class motorcycle ($s=18$) gets confused for class road ($s=1$) with a score of 0.33. This could be simply attributed the fact that the class road heavily overpowers the pixel distribution in all scenes whereas the class motorcycle has very minimal occupancy in most of the scenes. Now, in Table 6, we can see the confusion matrix for the first re-ordering of classes. In Table 6, it can be observed that, the predictor still confuses class rider ($s=17$) for class person($s=11$) with a score of 0.34 and class motorcycle ($s=19$) for class road ($s=5$) with a score of 0.38. Similarly, in Table 7, we can see the confusion matrix for the second re-ordering of classes. We can see that the predictor once again interprets class rider ($s=5$) as class person ($s=13$) with a score of 0.37 and class motorcycle ($s=11$) as class road ($s=8$) with a score of 0.38. It can be inferred that the predictor regardless of class index ordering, behaves exactly the same for all the class predictions. Thus, the predictor can be safely labeled as invariant to the class ordering.