GPU Gems

GPU Gems is now available, right here, online. You can purchase a beautifully printed version of this book, and others in the series, at a 30% discount courtesy of InformIT and Addison-Wesley.

The CD content, including demos and content, is available on the web and for download.

Chapter 10. Cinematic Lighting

Fabio Pellacini
Pixar Animation Studios

Kiril Vidimce
Pixar Animation Studios

In this chapter, we present a simplified implementation of uberlight, a light shader that expresses the lighting model described by Ronen Barzel (1997, 1999). A superset of this model was developed over several years at Pixar Animation Studios and used for the production of animated movies such as the Walt Disney presentations of the Pixar Animation Studios films Toy Story, A Bug's Life, Monsters, Inc., and Finding Nemo.

Our Cg implementation is based on Barzel's original approach and on the RenderMan Shading Language implementation written by Larry Gritz (Barzel 1999). Further details about this lighting approach and its uses in movie production can be found in Apodaca and Gritz 1999 and Birn 2000.

10.1 Introduction

Lighting is an important aspect of computer cinematography, in which lights and shadows are used to convey mood and support storytelling (Calahan 1999). Although realism remains an important aspect of computer-generated imagery, lighting directors constantly cheat the physics of light to support the artistic depiction of animated movies. Performing these tricks on a real-world set is a daunting task that often requires hours of setup.

Freed of the limitations of real physics, developers of computer cinematography have been devising lighting models that let artists illuminate scenes intuitively. The lighting model presented in this chapter is an adaptation of the model developed over the last decade at Pixar Animation Studios and used in the production of most of our movies. See Figures 10-1, 10-2, and 10-3.

Figure 10-1 Barn Lights in .

Figure 10-2 Cookies Contribute to a Window Effect in .

Figure 10-3 Lighting Conveys Mood in .

10.2 A Direct Lighting Illumination Model

The shader we present in this chapter models only the shaping and controls of the light sources that illuminate the scene; it doesn't cover the intricacies of how to model the surface details and the light reflection behavior. (Some examples of interesting surface behaviors can be found in Apodaca and Gritz 1999.)

In general, the illumination model used in our movie production performs two kinds of operations, similar to the pseudocode shown here.

color illuminationModel()
{
  Compute the surface characteristic For each light
  {
    Evaluate the light source Compute the surface response
  }
}

First, we compute the surface shading information by performing various texture lookups, interpolating values over the mesh, and computing procedural patterns. Then we loop over each light source that illuminates the object and compute its contribution. We do this by evaluating first the light color and then the surface response to the illumination of each light.

In this chapter, we present a simple shader that computes the contribution of only one light for a plastic reflection model. Extending it to a more general solution for multiple lights and better-looking surfaces is left as an exercise for the reader.

Our lighting model provides artists with control over various aspects of illumination: selection, color, shaping, shadowing, and texturing.

10.2.1 Selection

Each object in the scene can selectively respond to each individual light. This powerful characteristic of our lighting model lets the artist turn off lights on objects when additional light is creating an undesired effect. It also lets artists add extra lights that create a desired effect in a specific location without affecting the rest of the scene.

10.2.2 Color

A light's most noticeable properties are the color and the intensity that describe its emission. Similar to the OpenGL fixed-function lighting, our implementation provides separate weights for the ambient, diffuse, and specular illumination effects that artists can separately tweak. One of the most important aspects of our lighting model is that these terms can be freely changed per-object, letting the artist light entire sets with a small number of lights.

10.2.3 Shaping

To control regions of a scene that are illuminated by a light, real-world cinematographers commonly employ spotlights and rectangular lights (known as barn doors) to shape the light distribution. Our lighting model generalizes on these concepts by providing two types of shaping:

Omni light: This light specifies a near and far truncation distance (the light shines with full intensity inside each) and two adjustable, smooth drop-off zones around these edges.
Barn shaping: Barn shaping is a generalized truncated pyramid whose cross section is a superellipse with adjustable soft edges. The width and height of each of the two superellipses can be controlled separately to allow a wider range of shapes. The use of a superelliptical cross section lets us continuously vary the shape of the light, from a circle to a square (and to even more exotic, star-like shapes). Furthermore, the pyramid can be freely sheared to allow window and doorway effects. Variations of the shape of the cross section of the pyramid are illustrated in Figures 10-4 and 10-5.

Figure 10-4 Barn Door Shapes

Figure 10-5 Barn Door Edge Sizes

10.2.4 Shadowing

Shadowing is an important aspect of our lighting model; shadows are probably the attributes that artists cheat most often. Shadows are tweaked not only for speed considerations, but also for the ability to control each little aspect of the shadow's look, which is so important in defining the overall mood of a movie. For example, compare the strong shadows in film noir movies with the almost invisible ones in musicals.

As for lighting intensity, artists decide which objects cast shadows and which ones receive them. Also, the lighting designer can cheat shadow positions by moving them in relationship to the light origin. For example, she can allow bright highlights in a character's eyes while making sure that the shadow does not cross the character's face.

Darkness

One of the biggest problems of a direct lighting model is that shadows tend to be too dark. This happens because most of the indirect illumination that naturally occurs in the environment is never computed. To mimic reality in our model, we've created lights that can be adjusted to change the density of shadows, by letting some light propagate through the objects in a scene. In our implementation, we use the diffuse contribution of the surface to color the shadow region. We believe this is better than using an ambient term, because our method lets us maintain those nice gradients that make the shadow believable. Later in the chapter, we elaborate on this topic.

Hue

The hue of a shadowed area in the real world is slightly different from that of a nearby unoccluded region—for example, notice the slight bluish tint of outdoor shadows on a bright, sunny day. To mimic this effect, we allow the artist to change the shadow color. Slight variations of the shadow hue can make the difference between a good-looking shadow and a fake-looking one. In practice, you should think of shadow casters "spraying" receivers with the shadow color, which is commonly black. See Figure 10-6 for images with different shadow colors.

Figure 10-6 Variations in Shadow Colors

Reflection

One important caveat concerns highlights. When computing the surface contribution in the shadow area, we use only the diffuse response to obtain those nice gradients seen in outdoor environments, but we don't want to see highlights in the shadow region, because we are cheating diffuse interreflections. To achieve this effect, we simply switch off the specular contribution in the shadow regions. This little adjustment is just one example of how light changes the reflection behavior of surfaces. Tweaks like these are used widely in movie production to achieve that specific look we hope viewers will love.

Shadow Maps

Of the various techniques used to implement shadows, we use shadow maps in our model, for their simplicity and flexibility—and because we use them often in our movies. Although the shader in this chapter is based on such an algorithm, we encourage the reader to experiment with other shadow algorithms. The important aspect of the shader is not how we decide if a pixel is in shadow, it's how we use this information.

Shadow Blurring

The most important aspect missing from our implementation is shadow blurring. Artists often adjust the softness of shadow edges in order to cheat area lights or simply to get the particular look that the director wants. Blurring shadow edges is particularly hard to do efficiently. Various techniques are available, but presenting them is outside the scope of this chapter.

10.2.5 Texturing

Finally, we added projective texture support to allow a wide variety of effects, such as slide projectors and fake shadows from off-screen objects. These tricks are known in the movie production world as cookies; they are also used in game production, but less often. While game developers use texture projection for shape, coloring, and shadowing effects, movie creators tend to use soft cookies to enrich visual details and to add special effects, such as off-camera shadow casters or strangely shaped lights.

10.2.6 Results

Figures 10-7 and 10-8 illustrate the use of the uberlight to illuminate the head of a character from Pixar's short film Geri's Game. The surface of the model is flat and plastic-like, and we don't apply any material-related textures on it. By using a simpler surface model for the object, we can better emphasize the various effects we can obtain by using just the light shader with different parameters. The proper combination of light and surface modeling brings this character to life, as in the original Pixar short.

Figure 10-7 Lighting Geri

Figure 10-8 Lighting Styles

10.3 The Uberlight Shader

Listings 10-1 and 10-2 show the source code of the uberlight shader, based on the one by Larry Gritz in Barzel 1999.

Example 10-1. The Vertex Program for an Uberlight-Like Shader

void uberlight_vp(varying float4 Pobject
                  : POSITION, // Vertex position in object space
                    varying float3 Nobject
                  : NORMAL,
                    // Vertex normal in object space
                    varying float3 VertexColor
                  : COLOR0, // Vertex color
                    uniform float4x4 ModelViewProj,
                    // ModelViewProj matrix
                    uniform float4x4 ObjectToWorld,
                    // ObjectToWorld matrix
                    uniform float4x4 ObjectToWorldIT,
                    // Inverse transpose of the
                    // ObjectToWorld matrix
                    uniform float4x4 WorldToLight,
                    // Light space
                    uniform float4x4 WorldToLightIT,
                    // Inverse transpose of light
                    // space to transform normals
                    uniform float4x4
                        WorldToShadowProj, // Light space concatenated with
                    // the projection matrix used for
                    // the shadow. This defines
                    // shadow space.
                    uniform float3 CameraPosInWorld,
                    // Camera position
                    // in world space
                    uniform float ShadowBias,
                    // Shadow bias
                    out float4 HPosition
                  : POSITION,
                    // Rasterizer position
                    out float3 CameraPosInLight
                  : TEXCOORD0, // Camera position
                    // in light space
                    out float3 Plight
                  : TEXCOORD1,
                    // Interpolated position
                    // in light space
                    out float3 Nlight
                  : TEXCOORD2,
                    // Interpolated normal
                    // in light space
                    out float4 ShadowUV
                  : TEXCOORD3,
                    // Shadow UV
                    out float3 Color
                  : COLOR0)
// Surface color
{
  // Compute coordinates for the rasterizer
  HPosition = mul(ModelViewProj, Pobject);
  // Compute world space pos and normal
  float4 Pworld = mul(ObjectToWorld, Pobject);
  float3 Nworld = mul(ObjectToWorldIT, float4(Nobject, 0)).xyz;
  // Compute the position of the point in light space
  CameraPosInLight = mul(WorldToLight, float4(CameraPosInWorld, 1)).xyz;
  Plight = mul(WorldToLight, Pworld).xyz;
  Nlight = mul(WorldToLightIT, float4(Nworld, 0)).xyz;
  // Compute the U-V for the shadow and texture projection
  float4 shadowProj = mul(WorldToShadowProj, Pworld);
  // Rescale x, y to the range 0..1
  ShadowUV.xy = 0.5 * (shadowProj.xy + shadowProj.ww);
  // When transforming z, remember to apply the bias
  ShadowUV.z = 0.5 * (shadowProj.z + shadowProj.w - ShadowBias);
  ShadowUV.w = shadowProj.w;
  // Pass the color as is
  Color = VertexColor;
}

Example 10-2. The Fragment Program for an Uberlight-Like Shader

// SHADER PARAMETERS ================================================
// Superellipse params
struct SuperellipseShapingParams
{
  float width, height;
  float widthEdge, heightEdge;
  float round;
};
// Distance shaping params
struct DistanceShapingParams
{
  float near, far;
  float nearEdge, farEdge;
};
// Light params
struct LightParams
{
  float3 color;
  // light color
  float3 weights; // light weights (ambient, diffuse, specular)
};
struct SurfaceParams
{
  float3 weights;
  // surface weights (ambient, diffuse, specular)
  float roughness; // roughness
};
// BRDF/LIGHT INTERACTION ===========================================
// Compute the light direction
float3 computeLightDir(float3 Plight)
{
  // Spot only
  return -normalize(Plight);
}
// Ambient contribution of lit
float ambient(float3 litResult)
{
  return litResult.x;
} // Diffuse contribution of lit
float diffuse(float3 litResult) { return litResult.y; }
// Specular contribution of lit
float specular(float3 litResult) { return litResult.z; }
// SUPERELLIPSE =====================================================
float computeSuperellipseShaping(
    float3 Plight,
    // Point in light space
    bool barnShaping,
    // Barn shaping
    SuperellipseShapingParams params) // Superellipse shaping params
{
  if (!barnShaping)
  {
    return 1;
  }
  else
  {
    // Project the point onto the z == 1 plane
    float2 Pproj = Plight.xy / Plight.z;
    // Because we want to evaluate the superellipse
    // in the first quadrant, for simplicity, get the right values
    float a = params.width;
    float A = params.width + params.widthEdge;
    float b = params.height;
    float B = params.height + params.heightEdge;
    float2 pos = abs(Pproj);
    // Evaluate the superellipse in the first quadrant
    float exp1 = 2.0 / params.round;
    float exp2 = -params.round / 2.0;
    float inner =
        a * b * pow(pow(b * pos.x, exp1) + pow(a * pos.y, exp1), exp2);
    float outer =
        A * B * pow(pow(B * pos.x, exp1) + pow(A * pos.y, exp1), exp2);
    return 1 - smoothstep(inner, outer, 1);
  }
} // DISTANCE SHAPING =================================================
float computeDistanceShaping(
    float3 Plight,
    // Point in light space
    bool barnShaping,
    // Barn shaping
    DistanceShapingParams params) // Distance shaping params
{
  float depth;
  if (barnShaping)
  {
    depth = -Plight.z;
  }
  else
  {
    depth = length(Plight.z);
  }
  return smoothstep(params.near - params.nearEdge, params.near, depth) *
         (1 - smoothstep(params.far, params.far + params.farEdge, depth));
}
// MAIN =============================================================
float4 uberlight_fp(float3 CameraPosInLight
                    : TEXCOORD0, // Camera position in light space
                      float3 Plight
                    : TEXCOORD1,
                      // Interpolated position in light space
                      float3 Nlight
                    : TEXCOORD2,
                      // Interpolated normal in light space
                      float4 ShadowUV
                    : TEXCOORD3, // Shadow UV
                      // SURFACE PROPERTIES ------------------------
                      float3 SurfaceColor
                    : COLOR0,
                      // Surface color
                      uniform SurfaceParams Surface, // Other surface params
                      // (weights, roughness)
                      // LIGHT PROPERTIES --------------------------
                      uniform LightParams Light, // Light properties
                      // SHAPING -----------------------------------
                      // Choose between barn shaping (superelliptic pyramid)
                      // and omni shaping
                      uniform bool BarnShaping,
                      uniform SuperellipseShapingParams SuperellipseShaping,
                      // Superellipse
                      // shaping
                      uniform DistanceShapingParams DistanceShaping,
                      // Distance
                      // shaping
                      // DISTANCE FALLOFF --------------------------
                      // COOKIES AND SHADOWS -----------------------
                      uniform sampler2D Shadow,
                      // Shadow texture
                      uniform float3 ShadowColor,
                      // Shadow color
                      uniform sampler2D Cookie,
                      // Cookie texture
                      uniform float CookieDensity) // Cookie density
{
  // TRANSFORM VECTORS TO LIGHT SPACE ---------------------
  // Compute the normal in light space (normalize after vertex
  // interpolation)
  float3 N = normalize(Nlight);
  // Compute the light direction
  float3 L = computeLightDir(Plight);
  // Compute the view direction (vector from the point to the eye)
  float3 V = normalize(CameraPosInLight - Plight);
  // Compute the half-angle for the specular term
  float3 H = normalize(L + V);
  // COMPUTE THE TEXTURE PROJECTION - COOKIE
  float3 cookieColor = tex2Dproj(Cookie, ShadowUV).xyz;
  Light.color = lerp(Light.color, cookieColor, CookieDensity);
  // COMPUTE THE SHADOW EFFECT ---------------------------
  // Get the amount of shadow
  float shadow = tex2Dproj(Shadow, ShadowUV).x;
  // Modify the light color so that it blends with the shadow color
  // in the shadow areas
  float3 mixedLightColor = lerp(ShadowColor, Light.color, shadow);
  // COMPUTE THE ATTENUATION DUE TO SHAPING --------------
  float attenuation = 1;
  // Contribution from the superellipse shaping
  attenuation *=
      computeSuperellipseShaping(Plight, BarnShaping, SuperellipseShaping);
  // Contribution from the distance shaping
  attenuation *= computeDistanceShaping(Plight, BarnShaping, DistanceShaping);
  // APPLY TO SURFACE ------------------------------------
  // Here you should substitute other code for different
  // surface reflection models. This code computes the lighting
  // for a plastic-like surface.
  // Lighting computation
  float3 litResult = lit(dot(N, L), dot(N, H), Surface.roughness).xyz;
  // Multiply by the surface and light weights
  litResult *= Surface.weights * Light.weights;
  // Compute the ambient, diffuse, and specular final colors.
  // For the ambient term, use the color of the light as is
  float3 ambientColor = Light.color * SurfaceColor * ambient(litResult);
  // For the diffuse term, use the color of the light
  // mixed with the color in the shadow
  float3 diffuseColor = mixedLightColor * SurfaceColor * diffuse(litResult);
  // The specular color is simply the light color times the specular
  // term, because we want to obtain white highlights regardless of the
  // surface color. Our shadows won't be fully black, so we want to
  // make sure that the highlights do not appear in shadow.
  float3 specularColor = mixedLightColor * shadow * specular(litResult);
  // Compute the final diffuse color
  float3 color = attenuation * (ambientColor + diffuseColor + specularColor);
  // Compute the diffuse color
  return float4(color, 1);
}

10.4 Performance Concerns

10.4.1 Speed

We can easily speed up the uberlight shader by replacing certain analytic computations with texture lookups. The textures are generated by discretely sampling the functions of computations we want to avoid. As long as the textures have a high-enough resolution (potentially re-creating them based on the scene and camera parameters), the quality of the image can still be very good. Because of production and quality demands in the world of offline rendering, this kind of optimization is rarely performed.

10.4.2 Expense

The most expensive code in this shader is the computation of the light's shape-based attenuation. We can construct a texture map to evaluate the superelliptical shaping for a given set of barn parameters and then use the shadow texture coordinates to look up the barn map contribution in light space as a projective texture. The use of a barn map dramatically reduces the number of shader instructions. Plus, it can more than double the speed of shading (as measured on NVIDIA's Quadro FX 2000 board).

10.4.3 Optimization

When neither the camera nor the objects move in the scene, we can also optimize camera-dependent and scene-dependent shading components of the shader (such as the distance-based shaping). This is a typical usage scenario for a lighting artist who is modifying lighting parameters to light a frame in a given shot. When the artist replaces both the superellipse and the distance-based shaping with two texture lookups, the modified shader performs more than three times faster than the original one.

If you choose this approach to optimize your shaders, consider using a high-level language to create these maps on the fly. In our proprietary multipass interactive renderer, we define the creation of these maps as separate passes. Once created, the pass results are cached and constantly reused. Only when the parameters that affect these maps change are the passes marked as dirty, queued for reevaluation, and once again cached.

10.5 Conclusion

Our lighting model is a simple attempt to provide a comprehensive set of lighting controls that covers most effects used daily by lighting artists. Although our implementation covers a wide variety of effects, many more can be added (and are indeed added daily) to allow the artist more flexibility and expressiveness. Examples of such controls are found in Barzel 1999. Readers may extend our source code examples to cover these and other algorithms.

The lighting controls presented covered only part of the look for which the light source is responsible. When you are developing a full illumination model, be aware that the surface-reflection characteristic of a surface is also important; this property is what distinguishes the appearance of the materials in the scene.

10.6 References

Apodaca, Anthony A., and Larry Gritz, eds. 1999. Advanced RenderMan: Creating CGI for Motion Pictures. Morgan Kaufmann.

Barzel, Ronen. 1997. "Lighting Controls for Computer Cinematography." Journal of Graphics Tools 2(1), pp. 1–20. Available online at http://www.acm.org/jgt/papers/Barzel97

Barzel, Ronen. 1999. "Lighting Controls for Computer Cinematography." In Advanced RenderMan: Creating CGI for Motion Pictures, edited by Anthony A. Apodaca and Larry Gritz. Morgan Kaufmann. Code for the chapter was provided by Larry Gritz.

Birn, Jeremy. 2000. Digital Lighting and Rendering. New Riders Publishing.

Calahan, Sharon. 1999. "Storytelling through Lighting: A Computer Graphics Perspective." In Advanced RenderMan: Creating CGI for Motion Pictures, edited by Anthony A. Apodaca and Larry Gritz. Morgan Kaufmann.

Copyright

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and Addison-Wesley was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers discounts on this book when ordered in quantity for bulk purchases and special sales. For more information, please contact:

U.S. Corporate and Government Sales
(800) 382-3419
corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact:

International Sales
international@pearsoned.com

Visit Addison-Wesley on the Web: www.awprofessional.com

Library of Congress Control Number: 2004100582

GeForce™ and NVIDIA Quadro^® are trademarks or registered trademarks of NVIDIA Corporation.
RenderMan^® is a registered trademark of Pixar Animation Studios.
"Shadow Map Antialiasing" © 2003 NVIDIA Corporation and Pixar Animation Studios.
"Cinematic Lighting" © 2003 Pixar Animation Studios.
Dawn images © 2002 NVIDIA Corporation. Vulcan images © 2003 NVIDIA Corporation.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. Printed in the United States of America. Published simultaneously in Canada.

For information on obtaining permission for use of material from this work, please submit a written request to:

Pearson Education, Inc.
Rights and Contracts Department
One Lake Street
Upper Saddle River, NJ 07458

Text printed on recycled and acid-free paper.

5 6 7 8 9 10 QWT 09 08 07

5th Printing September 2007