Custom HLSL ShaderEffect on the Direct3D renderers

CNA snapshot 009d40f5  ·  Deep Dives › Renderers  ·  source links pinned to 009d40f5

On the three Direct3D renderers a ShaderEffect is HLSL source compiled at run time with D3DCompile(). This page states the contract precisely: which shader models and entry points are compiled, how uniform names are resolved, what the SpriteBatch and 3D draw paths feed the shader, how textures are bound, where Direct3D 11 and Direct3D 12 differ, and where DIRECTX9 stops. It also records how this contract changed, because earlier descriptions of a fixed, name-ignoring layout no longer apply. It is for anyone writing custom HLSL for CNA on Windows.

Compile model

All three create the effect renderer and compile immediately when both source strings are non-empty; construction never throws for a compile error. IsEffectValid() says whether the program compiled and GetCompileErrorEXT() returns the compiler's message. DIRECTX11 and DIRECTX12 declare HLSL as their shader dialect (GetShaderDialectEXT()), so a caller can choose the right payload, and GLSL, which is what a portable effect usually carries for the GL family, fails as a compile error rather than being accepted silently; DIRECTX9 declares no dialect. The public API is on Shader effects and the per-renderer language table on Shaders: what executes where.

Names are resolved by reflection

On DIRECTX11 and DIRECTX12, D3DProgramReflection.cpp reflects both compiled stages and records, per variable name, the constant-buffer slot, byte offset, size, class and element count. The SetUniform… setters then write through those locations:

• A name that no stage declares is ignored; a name declared in both stages is written to every location. A trailing [0] is canonicalised away, so an array may be addressed by its bare name.
• SetUniformInt writes integer bits into an int variable; the float, vector, matrix and array setters write floats (SetUniformMat4 takes column-major data). SetUniformFloatArray, …Vec2Array, …Vec3Array and …Mat4Array are implemented.
• Up to 14 constant buffers (b0–b13), 16 shader resources and 16 samplers are represented; a program whose reflected core resources cannot be represented fails validation with a message instead of drawing with missing bindings.
• vpSize is written by the renderer: SpriteBatch sets it to the current viewport width and height before the effect is bound (SetViewportSizeEXT).

On DIRECTX9 the effect renderer parses each stage's constant table and uploads a named value to that name's float4 registers; SetUniformInt is converted to a float, because that is the register set Direct3D 9 shaders read. DeclareUniformBlockEXT is ignored by all three.

The SpriteBatch path

When a ShaderEffect is passed to SpriteBatch::Begin, the vertex the shader receives is SpriteBatch's own 32-byte vertex, bound with a three-element input layout:

SpriteBatch binds the sprite's texture and sampler to t0/s0 itself; the effect leaves slot 0 alone unless the caller explicitly set a texture there, and binds every other reflected slot from the effect's own textures. The vertex shader must therefore turn pixel positions into clip space itself, with vpSize.

The 3D draw path

An applied ShaderEffect also drives DrawPrimitives and DrawIndexedPrimitives on the DXGI renderers. Before the draw the renderer writes the effect's World, View and Projection matrices by those names, and builds (and caches) an input layout that matches the vertex shader's input signature to the bound VertexDeclaration. When the signature cannot be matched, or the effect is not a valid effect of this renderer, the draw throws NotSupportedException with a message saying which; it does not draw with a guessed layout. A shader that declares float3 POSITION, a tangent or any game-specific stride is therefore usable as long as the declaration supplies those semantics.

DIRECTX9 is narrower. Its draw dispatch has no branch for a custom effect: a Direct3D 9 ShaderEffect is consumed by SpriteBatch, and a 3D draw is dispatched through the stock-effect paths. What a 3D draw produces with a custom effect applied on DIRECTX9 was not traced for this page; use the DXGI renderers for custom 3D shaders.

Where Direct3D 11 and Direct3D 12 stop being the same

• DIRECTX11 binds the compiled vertex and pixel shaders, the input layout and the constant buffers as separate context state. Each bind maps every reflected constant buffer with D3D11_MAP_WRITE_DISCARD, copies the current values and attaches the buffers to the same b slots in both stages.
• DIRECTX12 has no independent shader bind. The effect builds a root signature from its reflected constant-buffer, texture and sampler counts through the renderer's root-signature cache, and a draw asks the shared pipeline-state cache for a PSO keyed by the effect's shaders and the current state: vertex layout, topology, blend, depth-stencil and rasterizer state, the bound render-target formats and sample count, and the depth format. Uniform values stay in the reflection until a consuming draw snapshots them into that frame's constant range, so changing a uniform between two draws in one frame is safe. Every PSO, root signature and buffer a custom draw references is retained until the frame's fence completes.

The consequence is that one DIRECTX12 effect object may be used with an MRT set, an MSAA target or another colour format: each combination gets its own cached pipeline state. Earlier descriptions said the Direct3D 12 effect built one PSO up front for a single-sample R8G8B8A8 target with blending, depth and culling disabled; that is no longer the code.

A shader that works on both DXGI renderers

This pair turns a texture into its colour inverse through SpriteBatch; with reflection the constant-buffer layout no longer has to match a fixed byte layout, only the names matter. The C++ half was syntax-checked against the TARGET headers (see the note below); the HLSL was read, not compiled.

const char* vertexHlsl = R"(
struct VSIn  { float2 pos : POSITION0; float2 uv : TEXCOORD0; float4 col : COLOR0; };
struct VSOut { float4 pos : SV_Position; float2 uv : TEXCOORD0; float4 col : TEXCOORD1; };
cbuffer CB : register(b0) { float4 vpSize; };   // written by the renderer (width, height)
VSOut main(VSIn input) {
    VSOut output;
    float2 ndc = (input.pos / vpSize.xy) * 2.0 - 1.0;
    output.pos = float4(ndc.x, -ndc.y, 0.0, 1.0);
    output.uv  = input.uv;
    output.col = input.col;
    return output;
})";

const char* pixelHlsl = R"(
Texture2D texSampler : register(t0);
SamplerState texSamplerSampler : register(s0);
struct PSIn { float4 pos : SV_Position; float2 uv : TEXCOORD0; float4 col : TEXCOORD1; };
float4 main(PSIn input) : SV_Target {
    float4 source = texSampler.Sample(texSamplerSampler, input.uv);
    return float4(1.0 - source.rgb, 1.0);
})";

ShaderEffect invert(device, vertexHlsl, pixelHlsl);
if (!invert.IsEffectValid())
    throw std::runtime_error("custom HLSL did not compile: " + invert.GetCompileErrorEXT());

SamplerState pointClamp = SamplerState::PointClamp;
SpriteBatch batch(device);
batch.Begin(SpriteSortMode::Deferred, BlendState::Opaque,
            &pointClamp, nullptr, nullptr, &invert);
batch.Draw(redTexture, destination, Color::White);
batch.End();

Drawn over a solid (255,0,0,255) texture, the expected result is exactly (0,255,255,255). A check of that value proves more than compilation: the input layout, the renderer-written vpSize, the b0 binding, the t0/s0 resource path, the pixel shader and the readback all took part in one public draw. The shared ShaderEffect_ReflectionContract fixture in the DirectX parity inventory covers the reflection rules for both renderers; on the Linux loop DIRECTX12 runs it on a forced-headless device and reads the off-screen back buffer.

What changed, and why it matters for older shaders

An earlier version of this contract ignored the name argument entirely. Both DXGI renderers stored uniforms in one 128-byte block in which the setter decided the byte offset: vpSize at 0, one matrix at 16–79 from SetUniformMat4, one shared vector slot at 80–95 from any vector setter, one scalar at 96–99 (an integer stored as a float), padding after that; array setters and extra texture slots were no-ops, and the effect served SpriteBatch only. The completed contract (DX-223) replaced that with reflection. A shader written for the old layout, with padding vectors placing a colour at byte 80, still compiles, but its values now arrive only if the names it declares are the names the game passes to the setters.

Read in this order

1. D3DProgramReflection.hpp: the limits and setters.
2. D3D11EffectRenderer.cpp: compile, bind and the input-layout cache.
3. D3D12EffectRenderer.cpp: root signature and PSO acquisition.
4. DirectX11Renderer.cpp and DirectX12Renderer.cpp: the customEffectRequested branch of the extended draw.
5. ShaderEffect.hpp: the public surface.