Custom HLSL ShaderEffect on DIRECTX9

CNA snapshot 3b43bd86  ·  Deep Dives › Renderers  ·  source links pinned to 3b43bd86

On DIRECTX9, CNA’s Direct3D 9 renderer, a ShaderEffect is HLSL source compiled at run time with D3DCompile(). This page states that contract precisely: which shader models and entry points are compiled, how uniform names are resolved, what the SpriteBatch path feeds the shader, and why a custom effect does not reach a 3D draw. It is for anyone writing custom HLSL for CNA on Windows. Microsoft’s own compiled effects are a separate route (compiled effects).

Compile model

The renderer creates the effect renderer and compiles immediately when both source strings are non-empty; construction never throws for a compile error. IsEffectValid() says whether the program compiled and GetCompileErrorEXT() returns the compiler's message (D3D9EffectRenderer.cpp). DIRECTX9 does not override GetShaderDialectEXT(), so it reports the shared default, Unknown: a caller cannot discover from the device that HLSL is expected, and GLSL handed to it simply fails to compile. Compiling needs a real d3dcompiler_47.dll; under Wine that is the reason CNA’s own effect tests pin a separate prefix (Direct3D evidence). The public API is on Shader effects and the per-renderer language table on Shaders: what executes where.

Names are resolved through the constant table

After both stages compile, the effect renderer parses each stage’s constant table (CTAB), the compiler-assigned map from a uniform name to its registers, which genuinely differs per compiled shader. A SetUniform… call uploads its value to that name’s float4 registers in every stage that declares it; a name that no stage declares is ignored. SetUniformInt is converted to a float, because the float4 register file is what the effect renderer writes (an HLSL int constant is therefore declared as a float in practice). DeclareUniformBlockEXT is ignored. The CTAB parser itself is checked against real D3DCompile()/D3DDisassemble() output by DirectX9_ConstantTable (DirectX9 examples CMake).

The SpriteBatch path

When a ShaderEffect is passed to SpriteBatch::Begin, the custom shaders replace Microsoft’s SpriteEffect shaders for that flush and receive the same 24-byte sprite vertex the stock path draws, bound with the renderer’s stride-24 vertex declaration (D3D9SpriteBatch.cpp):

The stock path receives XNA’s MatrixTransform with Direct3D 9’s half-pixel correction baked in; a custom shader does not. Instead the renderer writes the current viewport width and height to a uniform named vpSize before the effect is applied, and the vertex shader must turn pixel positions into clip space itself (and apply its own half-pixel offset if it needs texel-exact sampling). SpriteBatch binds the sprite’s texture through its own texture binding before the custom shaders are bound. Changing the effect flushes the pending batch first, so sprites queued for one effect are never redrawn with another. A compiled XNA effect passed to SpriteBatch::Begin takes a separate route when CNA_DIRECTX9_COMPILED_EFFECTS is on.

The 3D draw path

DIRECTX9 stops at SpriteBatch. Its 3D draw dispatch has no branch for a custom effect: a Direct3D 9 ShaderEffect is consumed by SpriteBatch, and DrawPrimitives and DrawIndexedPrimitives are dispatched through the stock-effect paths (D3D9EffectDraw.cpp), which select one of the five XNA stock effects from the draw’s parameters. What a 3D draw produces with a custom effect applied on DIRECTX9 was not traced for this page. For custom 3D shaders on Windows, use the stock effects, or Microsoft’s compiled effects in a build with the default-OFF CNA_DIRECTX9_COMPILED_EFFECTS option, on DIRECTX9; the next section says why SDL_GPU is no alternative there at this snapshot.

A shader for the SpriteBatch path

This pair turns a texture into its colour inverse through SpriteBatch. Only the names matter for the uniform: the constant table tells the renderer where vpSize lives. It was written against the contract above for this page and was neither compiled nor run; the C++ half uses only the public ShaderEffect and SpriteBatch API (ShaderEffect.hpp).

const char* vertexHlsl = R"(
float2 vpSize;   // written by the renderer (viewport width, height)
struct VSIn  { float3 pos : POSITION0; float4 col : COLOR0; float2 uv : TEXCOORD0; };
struct VSOut { float4 pos : POSITION;  float4 col : COLOR0; float2 uv : TEXCOORD0; };
VSOut main(VSIn input) {
    VSOut output;
    float2 ndc = (input.pos.xy / vpSize) * 2.0 - 1.0;
    output.pos = float4(ndc.x, -ndc.y, 0.0, 1.0);
    output.col = input.col;
    output.uv  = input.uv;
    return output;
})";

const char* pixelHlsl = R"(
sampler2D texSampler : register(s0);
float4 main(float4 col : COLOR0, float2 uv : TEXCOORD0) : COLOR0 {
    float4 source = tex2D(texSampler, uv);
    return float4(1.0 - source.rgb, 1.0);
})";

ShaderEffect invert(device, vertexHlsl, pixelHlsl);
if (!invert.IsEffectValid())
    throw std::runtime_error("custom HLSL did not compile: " + invert.GetCompileErrorEXT());

SamplerState pointClamp = SamplerState::PointClamp;
SpriteBatch batch(device);
batch.Begin(SpriteSortMode::Deferred, BlendState::Opaque,
            &pointClamp, nullptr, nullptr, &invert);
batch.Draw(redTexture, destination, Color::White);
batch.End();

Drawn over a solid (255,0,0,255) texture, the expected result is (0,255,255,255). A check of that value would prove more than compilation: the vertex declaration, the renderer-written vpSize, the sampler-0 texture path, the pixel shader and the readback all take part in one public draw. CNA’s own DirectX9_SpriteBatch_CustomEffect and DirectX9_EffectRenderer registrations exercise this path with their own shaders.

Custom shaders on the other Windows path

SDL_GPU, the default renderer on Windows, does not accept HLSL, and at this snapshot its ShaderEffect is compiled out of every Windows build: no runtime path turns shader source into the bytecode SDL’s Direct3D 12 GPU driver consumes, so CustomEffects answers false there (evidence tiers). A portable effect for the GL family is GLSL; a Windows game that needs custom HLSL selects DIRECTX9.

Read in this order

1. D3D9EffectRenderer.cpp: compile, constant table and uniform upload.
2. D3D9SpriteBatch.cpp: the custom-effect branch of FlushBatch.
3. D3D9EffectDraw.cpp: the stock-effect 3D dispatch.
4. ShaderEffect.hpp: the public surface.