Fixing Ray-traced Shadow Content Issues with NVIDIA RTX-Dev

Custom NVIDIA RTX branches for Unreal Engine 4 (RTX-Dev) may substantially help your development effort by providing enhanced compatibility with certain types of common content. While UE4 has substantially improved its rendering capabilities with ray tracing, ray tracing effects often have expectations on how the scene was constructed.

Today, real-time content, such as games, is often authored without these constraints, relying on expectations from the world of rasterization. RTX-Dev works to bridge the gap by allowing the ray-traced effect to behave more akin to the raster version.

Shadow comparison

Shadows provide an excellent example of this. Comparing the ray-traced and raster shadows in the Realistic Rendering demo scene offers insight into challenges that can arise from assumptions of raster shadow map behavior.

Realistic apartment scene with light streaming through a sliding door and casting shadows in the scene.
Figure 1. A Realistic Rendering sample with raster shadows.
In this version, the curtains appear darker, the highlight is missing from the floor, and an odd shadow has appeared on the balcony outside.
Figure 2. The same Realistic Rendering sample with default ray-traced shadows.

Immediately, you see substantial differences. Several more shadows are cast in the ray-traced version, helping to ground things better. However, the sunlight streaming in the window and hitting the floor is suspiciously missing. The reflection on the wood floor is visible, but not the diffuse shading from the sun directly striking it. Additionally, the floor on the patio is much dimmer.

Here’s how you can fix the problems with RTX-Dev:

  • Single-sided objects
  • Translucency
  • Occlusion
  • Culling

Single-sided objects

The patio floor is the first big clue. The sun is clearly being blocked from a substantial distance away. Carefully looking at the scene, a large city backdrop sits outside the patio, and it lies between the sun and the patio. The backdrop is just a simple plane. In raster shadows, potentially shadow-casting objects are single-sided by default, whereas ray tracing treats them as physical objects and records both front and back face hits. This can easily be disabled by setting CVar r.RayTracing.Shadows.EnabledTwoSidedGeometry to 0.

The image appears no different from the previous iteration.
Figure 3. The Realistic Rendering sample with single-sided geometry shadows.


Unfortunately, single-sided geometry is not enough to resolve this issue. Ray tracing reverses the direction of testing for occlusion. Raster shadows start from the light, where ray-traced shadows start from the point in the scene. Although both are culling back faces, the shadow map is seeing the back face of the backdrop, while the ray tracing occlusion test is seeing the front face. Ray tracing produces a more accurate penumbra when it knows the closest shadowing surface, which is a front face.

This is where the first RTX-Dev enhancement helps resolve the problem. RTX-Dev allows the direction used for occlusion to be selectable by CVar r.RayTracing.OcclusionDirection. Setting this variable to 1 results in it culling front faces rather than back faces, so that it matches the culling done in shadow maps.

The image now has proper shadowing on the balcony.
Figure 4. A Realistic Rendering sample with the occlusion cull direction set to cull front faces.

Now, the patio has proper sunlight striking it, because the backdrop is no longer casting a shadow over everything when ray tracing is enabled. Unfortunately, there is still no sun streaming through the sliding door.


The problem is that translucent surfaces default to casting no shadows in rasterization, but they do default to casting shadows in ray tracing. Truthfully, they should cast some amount of shadow, but UE4 does not yet support partial coverage shadows. While this is easily fixed by disabling the Cast Ray Traced Shadows checkbox on the material, it could be tedious to perform this operation on all transparent materials in a game to achieve the original raster behavior.

To help resolve this challenge, RTX-Dev offers a CVar variable to exclude all translucent materials from casting ray-traced shadows, to match the raster default behavior.

The image appears the same except that a highlight now appears on the floor in front of the transparent sliding glass door.
Figure 5. A Realistic Rendering sample with translucent materials not casting shadows.

Now, the light clearly streams in through the glass and strikes the floor. However, the curtains still are not backlit as in the original raster image.


Looking closely at the scene geometry, the curtains are the sort of non-physical geometry that you often encounter in real-time graphics. They are a single layer of triangles marked as double-sided with a special subsurface scattering material.

It turns out that in ray tracing, the curtains are essentially casting shadows on themselves. When ray tracing looks to cast the ray toward the light, the normal is pointing away from the light, so it is interpreting the curtain as being on the back side of an object casting shadow on itself. Therefore, a ray is never even cast toward the light. However, the special subsurface material means you must treat the curtain as two sides together and evaluate light leaking through the curtain.

To solve this, RTX-Dev has added the ability to disable the normal culling feature used for casting occlusion rays with materials requiring transmission. The CVar r.RayTracing.Shadows.CullTransmissives variable allows disabling the offending behavior.

Figure 6. Shading models used in Realistic Rendering. The curtains are using subsurface scattering.
The image is now similar to the original raster image, but with better detail.
Figure 7. A Realistic Rendering sample with normal culling disabled for scattering materials.

Finally, here’s an image where the ray tracing results are compatible with the expected behavior from when the scene was developed with rasterization in mind. The result is still not the same. The ray-traced version has substantially more detail and better grounding of objects. It just no longer suffers from odd differences tied to subtle behavior differences between ray tracing and raster shadows.

Next steps

The RTX-Dev branches provide the tools to achieve this goal without altering the content, saving extra work when enabling ray tracing on previously authored content.

If these options sound like something that could help your project, try out the RTX-Dev branches. For more information, see NVIDIA Unreal Engine Custom Branches.

No Comments