Diffuse and Normals Research

Before embarking on my Normal, Specular and Diffuse maps; I looked further into how each of them affects my model, and why they are good to use.

(Edit: 06/05/2013)
Nigel very kindly lent me his book; "Digital Texturing and Painting" - The 2nd Chapter; "Reference Materials, Textures, and Practical Stuff" ('Sunny Versus Cloudy') was excellent in helping me identify the right kind of reference images to use for my diffuse map.

Demers, Owen 2001. Digital Texturing and Painting. Edition. New Riders.

(Reference generated with: harvardgenerator.com)

The following notes are excerpts from;

http://www.chrisalbeluhn.com/Normal_Map_Tutorial.html
http://wiki.polycount.net/NormalMap?action=show&redirect=Normal+Map

What are Normals?

The pixels of the normal map each store a normal, a vector that describes the surface slope of the original high-res mesh at that point. The red, green, and blue channels of the normal map are used to control the direction of each pixel's normal.
When a normal map is applied to a low-poly mesh, the texture pixels control the direction each of the pixels on the low-poly mesh will be facing in 3D space, creating the illusion of more surface detail or better curvature. However, the silhouette of the model doesn't change.

Tangent-space normal map

A tangent-space normal map. image byEric Chadwick

Predominantly-blue colors. Object can rotate and deform. Good for deforming meshes, like characters, animals, flags, etc. 

 Maps can be reused easily, like on differently-shaped meshes. 
 Maps can be tiled and mirrored easily, though some games might not support mirroring very well. 
 Easier to overlay painted details. 
 Easier to use image compression. 
 More difficult to avoid smoothing problems from the low-poly vertex normals (see Smoothing Groups and Hard Edges). 
 Slightly slower performance than an object-space map (but not by much).

Object-space normal map

An object-space normal map.

image by Eric Chadwick

Rainbow colours. Objects can rotate, but usually shouldn't be deformed, unless the shader has been modified to support deformation.
 Easier to generate high-quality curvature because it completely ignores the crude smoothing of the low-poly vertex normals.
 Slightly better performance than a tangent-space map (but not by much).
 Can't easily reuse maps, different mesh shapes require unique maps.
 Difficult to tile properly, and mirroring requires specific shader support.
 Harder to overlay painted details because the base colors vary across the surface of the mesh. Painted details must be converted into Object Space to be combined properly with the OS map.
 They don't compress very well, since the blue channel can't be recreated in the shader like with tangent-space maps. Also the three color channels contain very different data which doesn't compress well, creating many artifacts. Using a half-resolution object-space map is one option.

RGB Channels

Shaders can use different techniques to render tangent-space normal maps, but the normal map directions are usually consistent within a game. Usually the red channel of a tangent-space normal map stores the X axis (pointing the normals predominantly leftwards or rightwards), the green channel stores the Y axis (pointing the normals predominantly upwards or downwards), and the blue channel stores the Z axis (pointing the normals outwards away from the surface).

The red, green, and blue channels of a tangent-space normal map.

image byEric Chadwick

If you see lighting coming from the wrong angle when you're looking at your normal-mapped model, and the model is using a tangent-space normal map, the normal map shader might be expecting the red or green channel (or both) to point in the opposite direction. To fix this either change the shader, or simply invert the appropriate color channels in an image editor, so that the black pixels become white and the white pixels become black.

Some shaders expect the color channels to be swapped or re-arranged to work with a particular compression format. For example the DXT5_nm format usually expects the X axis to be in the alpha channel, the Y axis to be in the green channel, and the red and blue channels to be empty.

Tangent Basis

Tangent-space normal maps use a special kind of vertex data called the tangent basis. This is similar to UV coordinates except it provides directionality across the surface, it forms a surface-relative coordinate system for the per-pixel normals stored in the normal map.

Light rays are in world space, but the normals stored in the normal map are in tangent space. When a normal-mapped model is being rendered, the light rays must be converted from world space into tangent space, using the tangent basis to get there. At that point the incoming light rays are compared against the directions of the normals in the normal map, and this determines how much each pixel of the mesh is going to be lit. Alternatively, instead of converting the light rays some shaders will convert the normals in the normal map from tangent space into world space. Then those world-space normals are compared against the light rays, and the model is lit appropriately. The method depends on who wrote the shader, but the end result is the same.

Unfortunately for artists, there are many different ways to calculate the tangent basis: 3ds Max, Maya, DirectX 9, NVMeshMender, Eric Lengyel, a custom solution, etc. This means a normal map baked in one application probably won't shade correctly in another. Artists must do some testing with different baking tools to find which works best with their output. When the renderer (or game engine) renders your game model, the shader must use the same tangent basis as the normal map baker, otherwise you'll get incorrect lighting, especially across the seams between UV shells.

The xNormal SDK supports custom tangent basis methods. When a programmer uses it to implement their renderer's own tangent basis, artists can then use Xnormal to bake normal maps that will match their renderer perfectly.

The UVs and the vertex normals on the low-res mesh directly influence the coloring of a tangent-space normal map when it is baked. Each tangent basis vertex is a combination of three things: the mesh vertex's normal (influenced by smoothing), the vertex's tangent (usually derived from the V texture coordinate), and the vertex's bitangent (derived in code, also called the binormal). These three vectors create an axis for each vertex, giving it a specific orientation in the tangent space. These axes are used to properly transform the incoming lighting from world space into tangent space, so your normal-mapped model will be lit correctly.

When a triangle's vertex normals are pointing straight out, and a pixel in the normal map is neutral blue (128,128,255) this means that pixel's normal will be pointing straight out from the surface of the low-poly mesh. When that pixel normal is tilted towards the left or the right in the tangent coordinate space, it will get either more or less red color, depending on whether the normal map is set to store the X axis as either a positive or a negative value. Same goes for when the normal is tilted up or down in tangent space, it will either get more or less green color. If the vertex normals aren't exactly perpendicular to the triangle, the normal map pixels will be tinted away from neutral blue as well. The vertex normals and the pixel normals in the normal map are combined together to create the final per-pixel surface normals.

Shaders are written to use a particular direction or "handedness" for the X and Y axes in a normal map. Most apps tend to prefer +X (red facing right) and +Y (green facing up), while others like 3ds Max prefer +X and -Y. This is why you often need to invert the green channel of a normal map to get it to render correctly in this or that app... the shader is expecting a particular handedness.

When shared edges are at different angles in UV space, different colors will show up

along the seam. The tangent basis uses these colors to light the model properly.

image byEric Chadwick

When you look at a tangent-space normal map for a character, you typically see different colors along the UV seams. This is because the UV shells are often oriented at different angles on the mesh, a necessary evil when translating the 3D mesh into 2D textures. The body might be mapped with a vertical shell, and the arm mapped with a horizontal one. This requires the normals in the normal map to be twisted for the different orientations of those UV shells. The UVs are twisted, so the normals must be twisted in order to compensate. The tangent basis helps reorient (twist) the lighting as it comes into the surface's local space, so the lighting will then look uniform across the normal mapped mesh.

When an artist tiles a tangent-space normal map across an arbitrary mesh, like a landscape, this tends to shade correctly because the mesh has a uniform direction in tangent space. If the mesh has discontinuous UV coordinates (UV seams), or the normal map has large directional gradients across it, the tangent space won't be uniform any more so the surface will probably have shading seams.

Using Bevels

Bevels/chamfers generally improve the silhouette of the model, and can also help reflect specular highlights better.

However bevels tend to produce long thin triangles, which slow down the in-game rendering of your model. Real-time renderers have trouble rendering long thin triangles because they create a lot of sub-pixel areas to render.

Bevels also balloon the vertex count, which can increase the transform cost and memory usage. Hard edges increase the vertex count too, but not when the edge also shares a seam in UV space. For a good explanation of the vertex count issue, see Beautiful, Yet Friendly.

Using hard edges with matching UV shells tends to give better performance and better cosmetic results than using bevels. However there are differing opinions on this, see the Polycount thread "Maya transfer maps help".

Edited Vertex Normals

If you use bevels the shading will be improved by editing the vertex normals so the larger flat surfaces have perpendicular normals. The vertex normals are then forced to blend across the smaller bevel faces, instead of across the larger faces. See the Polycount thread Superspecular soft edges tutorial chapter 1.

Bending normals on bevelled models.

From the tutorial Shading techniques Superspecular soft edges

image by Paolo Oliverio

Baking

The process of transferring normals from the high-res model to the in-game model is often called baking. The baking tool usually starts projecting a certain numerical distance out from the low-poly mesh, and sends rays inwards towards the high-poly mesh. When a ray intersects the high-poly mesh, it records the mesh's surface normal and saves it in the normal map.

To get an understanding of how all the options affect your normal map, do some test bakes on simple meshes like boxes. They generate quickly so you can experiment with UV mirroring, smoothing groups, etc. This helps you learn the settings that really matter.

Solving Pixel Artifacts

Random pixel artifacts in the bake.

image byEric Chadwick

If you are using 3ds Max's Render To Texture to bake from one UV layout to another, you may see stray pixels scattered across the bake. This only happens if you are using a copy of the original mesh in the Projection, and that mesh is using a different UV channel than the original mesh.

There are two solutions for this:

Add a Push modifier to the copied mesh, and set it to a low value like 0.01.

- or -

Turn off Filter Maps in the render settings (Rendering menu > Render Setup > Renderer tab > uncheck Filter Maps). To prevent aliasing you may want to enable the Global Supersampler in Render Setup.

See also #Anti-Aliasing.