Pascal VR Tech

NVIDIA recently launched our Pascal architecture with the GeForce GTX 1080 and 1070 GPUs. Pascal introduces a couple of new technologies aimed specifically at improving the Virtual Reality experience (VR). Single Pass Stereo addresses the geometry end of the pipeline, allowing both views of a stereo scene to be efficiently rendered in a single pass. Lens Matched Shading meanwhile addresses the pixel end of the pipeline, providing a distortion that gives a higher shading rate near the image center, efficiently providing higher visual detail where it is most needed for VR views. In this article, I will give a brief overview of the technologies and a light introduction to the programming required.

Single Pass Stereo is a simple concept: the vertex shader can output positions with two x co-ordinates. This new technology allows it to simultaneously output both positions for the two viewports that make up a stereo view. (I will discuss vertex shaders for simplicity but Single Pass Stereo can apply to all geometry stages, including tessellation and geometry shaders.) A special semantic is used to define the right eye position: NV_X_RIGHT.

struct VSOutput
{
	float4 Pos : SV_POSITION;
	float4 X_Right : NV_X_RIGHT; // Custom Semantic for Single Pass Stereo
}

VSOutput VSMain (float4 Pos : POSITION)
{
	VSOutput output;
	output.Pos     = mul(mul(mul(Pos, World), View),      Projection);
	output.X_Right = mul(mul(mul(Pos, World), ViewRight), Projection).x;  // Only scalar x is used.
}

It’s necessary to make one API call to configure the rendering state. NvAPI_D3D_SetSinglePassStereoMode tells the driver how to configure the left and right viewports. They can be side-by-side in one render target or the first two slices of a render target array. In the side-by-side case, the shader needs to additionally output a viewport mask; in the case of render target arrays, the viewport mask is not needed and the GPU will automatically provide the slice indices. Details are in our programming guide which is part of our VRWorks download.

single pass

Single Pass Stereo uses custom semantics: NV_VIEWPORT_MASK and NV_X_RIGHT. Consequently, shaders using these semantics have to be compiled with functions from NVIDIA’s NVAPI library. We have a programming guide for Single Pass Stereo that covers the details, also in VRWorks. I won’t copy-and-paste the API reference here. The basic message is that it’s just another shader compilation function and a trivial drop-in replacement for Direct3D’s CreateVertexShader, CreateGeometryShader, or CreateDomainShader.

Lens Matched Shading is a hardware feature in Pascal that complements our existing Multi-Resolution technology by adding an extra transform per viewport. Both technologies address a problem with many VR headsets: they warp the rendered image with a lens between the screen and the eye. Here is an Oculus DK2 lens:

lens matched shading

The displayed image has to be warped to compensate for the optical distortion of the lens. The resulting front-buffer barrel distortion is instantly familiar to anyone who has developed a VR program:

radial distortion

I won’t go into the details of the radial distortion here because it is nothing new and it’s well explained elsewhere. I quite like eVRydayVR’s video on YouTube. The important point for our discussion is how the distortion alters the perceived shading rate. The scene is first rendered to a typical rectangular render target. The VR software – such as the Oculus Rift SDK – then distorts the image by radially pulling the pixels in from the edges. In diagram form, it looks like this:


On the left, in pink, is the render target or back-buffer that is rendered by the app; on the right is the same area after distortion. Circles in the original remain circles after the distortion but their spacing changes. The effect on an image is like this:



The important result is that edge pixels in the original are squished together by the distortion and the final, displayed image is effectively super-sampled at the edges (give or take the quality of the resampling kernel). Said another way, the shading rate is higher at the edges than in the center of the image. I have highlighted a couple of areas in the image. Note the effect on the arch in the bottom left of the image, compared to the center:



This is bad because we didn’t ask for super-sampling of the image and a lot of shading is wasted. It’s doubly bad because the user’s eye is typically directed at the image center and the distortion produces a lower shading rate in the center, where detail is most needed. Techniques like foveated rendering actually strive to produce the opposite effect: a higher shading rate in the center and lower in the user’s peripheral vision. A higher shading rate at the edges is a double waste. This is a graph of the shading rate – the number of pre-distortion pixels that contribute to one final pixel on the display surface. The rate reaches as high as 5 at the corners:


foveated rendering

Multi Resolution Shading is a technique in our VRWorks library that addresses the VR shading rate problem. Multi Res Shading uses Viewport Multicast and has been supported since the launch of NVIDIA’s Maxwell cards. See Nathan Reed’s presentation. Multi Resolution Shading splits each eye’s view into a 3x3 grid of viewports and reduces the shading rate in the edge and corner viewports. This definitely works to reduce the number of rendered pixels and reduce shading waste. However, it is only a piecewise linear approximation to the ideal solution. Multi-resolution shading is based upon two NVIDA technologies that have been supported since the introduction of our Maxwell architecture: Viewport Multi-Cast and Fast Geometry Shaders. I won’t go into more detail here – refer to Nathan’s presentation or Alexey’s Multi-Projection SDK Programming Guide.



Lens Matched Shading allows us to continuously vary the shading rate by adding an extra transform of clip-space positions, prior to the perspective division. It’s not a general transform, but rather a linear modification of the w co-ordinate:

w’ = w + Ax + By

It is restricted to a linear transform because that allows the rest of the graphics pipeline to continue to work as usual: raster, z-buffer, etc. The visual effect of the warp is something like increasing the perspective effect closer to the screen edges, as you’d expect when messing with w:



You need four viewports because the A and B coefficients need to switch signs in each clip space quadrant. The modification of w always needs to be positive but x and y switch signs.



The VR lens position and barrel distortion are not centered about the render target center but are offset. So the required coefficients are not symmetrical. However, they must match along the viewport edges or you would get discontinuities. The required setup is more like this:



I did ask myself: surely the switch of x and y signs could be implemented with a couple of abs functions, rather than four viewports:

w’ = w + A|x| + B|y|

But it doesn’t work where two vertices of a triangle fall into different quadrants. I hacked up a DX11 sample to experiment with warping and it demonstrates the problem nicely (you can trivially try this yourself). It has a single viewport and my only modification is the addition of the warp to the VS. My hacky warp uses abs(x) and abs(y). Look at the yellow beam near the top of the screen. It is not warped correctly because the end vertices fall in different screen quadrants– it should bend in the middle and has become detached from its supporting geometry. Compare to the grey concrete wall behind it.



The wall and beam are parallel in the model. This is the same view without my hacky warp. If you compare this to my warped image, then using a single viewport is clearly very broken. Four viewports are required.



The warp is a simple, linear transformation of w – isn’t that what a vertex shader is used for? My hacked Cascaded Shadow Map sample is just a DX11 sample with a trivially modified VS. Yes you could use a VS, but… the vertex shader cannot transform w independently for each eye, not without reverting to a pass per eye and defeating Single Pass Stereo. Recall that Single Pass Stereo gives you two x co-ordinates, but not two w co-ordinates. So using a VS would require a pass per eye and the technique inherently requires four viewports per eye. Without hardware support for Single Pass Stereo, Lens Matched Shading and Viewport Multicast , you would require eight passes over the scene (and that’s not counting other passes that are typically numerous in sophisticated, modern game engines). CPU scene traversal or geometry throughput would quickly become the bottleneck. Hence, we implement the transform in fixed function hardware and we set the A and B coefficients with an NVAPI call. Again, details are in our programming guide for Lens Matched Shading.

One final point: compare my screenshot of the Cascaded Shadow Maps (CSM) sample against the warped Sponza one. Note the difference in the image edges? Modifying clip space w does not transform the clip planes at the screen edges. The clip planes do not get any special treatment and remain at the render target edges. So my hacky CSM screenshot is still wasting fill because we only need the central octagon, as in the Sponza image. The application must explicitly address the clipping itself. We find that prefilling the depth buffer with minimum depth is the most efficient solution.

My colleague Dmitry has written about another application of the Lens Matched Shading warp: Perspective Surround. That use case requires no change in the clip planes at the screen edges. Hence, the application has to choose to implement the clipping required for VR warping.

I hope that I’ve managed to provide a useful introduction to our new VR technologies: Single Pass Stereo and Lens Matched Shading. They each help to make rendering the VR experience far more efficient. If you would like more information, our VRWorks suite of APIs and libraries has a wealth of further resources to help integrate these technologies into engines. We have a high-level programming guide for the Multi-Projection SDK with plenty more discussion. There are also lower-level programming guides for Single Pass Stereo and Lens Matched Shading. And we have code samples demonstrating the use of the APIs. We are working on integrations for Unreal Engine 4 and Unity. Watch our developer web site for updates.

References

  1. Multi-res VR
  2. Programming guides and samples are in the VRWorks package
  3. Oculus Rift distortion rendering doc, Video
  4. Dmitry’s Perspective Surround blog (Coming Soon)