Real-Time Light Field Path Tracing

Abstract

Light field rendering and displays are emerging technologies that produce more immersive visual 3D experiences than the conventional stereoscopic 3D technologies, as well as provide a more comfortable virtual or augmented reality (VR/AR) experience by mitigating the vergence–accommodation conflict. Path tracing photorealistic synthetic light fields in real time is extremely challenging, since it involves rendering a large amount of viewpoints for each frame. However, these viewpoints are often spatially very close to each other, especially in light field AR glasses or other near-eye light field displays. In this paper, we propose a practical real-time light field path tracing pipeline and demonstrate it by rendering a $6\times 6$ grid of 720p viewpoints at 18 frames per second on a single GPU, through utilizing denoising filters and spatiotemporal sample reprojection. In addition, we discuss how the pipeline can be scaled to yield higher-quality results if more parallel computing resources are available. We also show that our approach can be used to simultaneously serve multiple clients with varying light field grid sizes, with the quality remaining constant across clients.

Appendices

Appendix A: Discard Percentages

Figure 5 illustrates the average discard percentages per viewpoint after spatial reprojection in Sponza, for the five tested viewpoint configurations. Overall, the average discard percentages confirm the intuition that rendering a middle viewpoint is slightly better in terms of spatially discarded pixels than rendering the top left viewpoint, specifically for the furthest reprojected viewpoints in the corners. Adding a second rendered viewpoint clearly lowers the amount of discarded pixels compared to the single-camera setups. However, path tracing more viewpoints means much more computational effort compared to rendering only a single viewpoint, so Fig. 5 indicates that the reduction in the discard percentages is simply not enough to justify path tracing an additional viewpoint when striving for real-time performance.

Fig. 5.

Viewpoint-wise spatial discard percentage heatmaps for Sponza, for the different tested configurations. The path traced viewpoints are marked with camera icons. Note that each heatmap has a different range.

Appendix B: Comparison Images

Figures 6 and 7 illustrate the results for all scenes, showing the original 1 spp input for the middle viewpoint, the denoised + reprojected results for the synthesized top left viewpoint, and the reference 4096 spp top left viewpoint. Overall, the visual quality is high enough to be usable in various real-time applications; the most visible denoising artifacts are the oversmoothing of shadows, and a moderate amount of blur. Moreover, the blockwise nature of BMFR can still be seen in the Eternal Valley result (Fig. 6d), as temporal accumulation has not yet had time to fully compensate for it. Similarly, temporal accumulation has not yet removed some of the residual noise in Figs. 7c–f.

Fig. 6.

Results for frame 15 of Sponza (left column) and Eternal Valley (right column) after denoising and reprojecting the middle viewpoint onto the top left viewpoint of the $6\times 6$ grid.

Fig. 7.

Results for frame 6 of Warehouse (left column) and frame 8 of Bistro Interior (right column) after denoising and reprojecting the middle viewpoint onto the top left viewpoint of the $6\times 6$ grid.

In general, narrow strips along the top and left borders exhibit a higher amount of noise, as they were not visible in the middle viewpoint, and thus did not benefit from the denoising. The effect of discards due to the fast camera motion is also evident in the disoccluded areas of Bistro Interior (Figs. 7d and 7f). These are further visualized in Fig. 8, which shows a selection of the results in Figs. 6 and 7 before the discarded areas have been filled in, and with the discards shown in red. These artifacts could be mitigated for example by path tracing more samples at the affected locations, or by leveraging the denoise-after-reprojection pipeline option if parallel resources are available. Applying deep learning based temporal rendering techniques (see [9]) can also be an attractive option in that case.

The temporal quality of the results can be seen in the supplementary video, which features all four test scenes. Overall, the temporal behaviour is relatively stable, thanks to the temporal accumulation. However, the accumulation does also involve a tradeoff between introducing ghosting (too much reuse) and not being able to suppress fireflies (too little reuse), as highlighted in the video for Eternal Valley. As for the quality in the disoccluded areas, the single-frame examples discussed above are also representative of the full video sequences.

Fig. 8.

The denoising + reprojection results corresponding to Fig. 6c, Fig. 6f, Fig. 7c and Fig. 7f, before the discarded reprojection locations (visualized in red) have been filled in with path tracing. (Color figure online)