Tutorial 52: Writing Custom Shaders (ShaderEffect)

CNA — C++ XNA 4.0 reimplementation

Two different custom-effect paths

XNA's custom-shader story was HLSL compiled by the content pipeline to a D3D9 Effect Framework binary, then handed to new Effect(device, effectCode). CNA implements that constructor and the XNB EffectReader for XNA/FNA Effect Framework bytecode:

// Microsoft/Xna/Framework/Graphics/Effect.hpp
Effect(GraphicsDevice& device, const std::vector<SharpRuntime::bytecs>& effectCode);

That compatibility path is real but not universal: it works on 11 of the 14 renderer identities — FNA3D always, and 10 more (the three EasyGL identities, SDL_GPU, VULKAN, WEBGPU, SOFTWARE, DIRECTX9, DIRECTX11, METAL) behind eight default-OFF CNA_*_COMPILED_EFFECTS options. Other renderers report GraphicsCapability::CompiledEffects false. It accepts the compiled Effect Framework binary, commonly stored as .fxb or inside XNB; at run time it does not compile HLSL .fx source and does not accept DXBC or MGFX. (At build time the cna-content tool can compile .fx source to an XNB through an external legacy fxc; see Tutorial 150 and Tutorial 128.) This tutorial covers the other path: hand-authored, renderer-native ShaderEffect. The two are gated by different capabilities — CompiledEffects for bytecode, CustomEffects (plus ExecutesShaderEffectSourceEXT()) for source — and FNA3D, for example, runs any .fxb yet cannot run a source ShaderEffect at all.

The constructor takes source text, not a path

This is the single most important fact on this page. ShaderEffect takes three arguments, and the two strings are the shader source code itself:

// Microsoft/Xna/Framework/Graphics/ShaderEffect.hpp
CNAEXT ShaderEffect(GraphicsDevice& device,
                   const std::string& vertSrc,   // contents of the vertex shader (not a file path)
                   const std::string& fragSrc);  // contents of the fragment shader (not a file path)

The header comment spells out "not a file path" for both parameters. Passing "Content/shaders/tinted" will not load anything — that string is handed to the renderer as shader source, fails to compile, and leaves you with an invalid effect.

The constructor does not throw on a compile failure. This is how you find out:

CNAEXT [[nodiscard]] bool IsEffectValid() const;

Treat a false here as a hard error during development. To learn why, read the compiler's log with GetCompileErrorEXT() (empty when the shader compiled or when the renderer keeps no log) or the structured GetShaderDiagnosticsEXT(). The failure is deliberately not an exception: on several renderers CustomEffects is true while GLSL source is never compiled at all, and throwing would turn a documented capability boundary into a crash.

CNAEXT [[nodiscard]] std::string GetCompileErrorEXT() const;
CNAEXT [[nodiscard]] std::vector<CNA::ShaderDiagnosticEXT> GetShaderDiagnosticsEXT() const;

A minimal working shader pair

The most direct place to put source is a raw string literal. This pair is GLSL ES 3.00, which is what OPENGLES3 and WEBGL2 compile (see the dialect table below for the other renderers), and it matches CNA's SpriteBatch vertex layout: location 0 is the position (SpriteBatch supplies x, y and a layer depth; the shader reads the first two), location 1 the texture coordinate, location 2 the vertex colour. SpriteBatch itself sets the projection uniform on the effect's program.

#version 300 es
precision highp float;

layout(location = 0) in vec2 aPos;
layout(location = 1) in vec2 aTexCoord;
layout(location = 2) in vec4 aColor;

out vec2 TexCoord;
out vec4 Color;

uniform mat4 projection;

void main()
{
    gl_Position = projection * vec4(aPos, 0.0, 1.0);
    TexCoord = aTexCoord;
    Color = aColor;
}

#version 300 es
precision mediump float;

in vec2 TexCoord;
in vec4 Color;

out vec4 FragColor;

uniform sampler2D texture1;

void main()
{
    vec4 t = texture(texture1, TexCoord);
    FragColor = vec4(t.r, 0.0, 0.0, t.a);   // keep only the red channel
}

The sampler2D texture1 uniform defaults to texture unit 0, which is where SpriteBatch binds the sprite texture — so for this shader no explicit sampler binding is needed at all.

Using it with SpriteBatch

The six-argument Begin() overload takes an Effect* as its last parameter. Pass the ShaderEffect there and it replaces the built-in sprite shader for that batch:

#include <cstdio>
#include "Microsoft/Xna/Framework/Graphics/ShaderEffect.hpp"
#include "Microsoft/Xna/Framework/Graphics/SpriteBatch.hpp"

using namespace Microsoft::Xna::Framework;
using namespace Microsoft::Xna::Framework::Graphics;

static const char* kVertSrc = R"(#version 300 es
// ... vertex source as above ...
)";

static const char* kFragSrc = R"(#version 300 es
// ... fragment source as above ...
)";

void Initialize() override
{
    Game::Initialize();
    auto& device = getGraphicsDeviceProperty();

    sb_ = std::make_unique<SpriteBatch>(device);
    fx_ = std::make_unique<ShaderEffect>(device, kVertSrc, kFragSrc);

    if (!fx_->IsEffectValid()) {
        // The GLSL did not compile. Do not draw with it; say why.
        std::fprintf(stderr, "ShaderEffect: %s\n", fx_->GetCompileErrorEXT().c_str());
    }
}

void Draw(const GameTime&) override
{
    auto& device = getGraphicsDeviceProperty();
    device.Clear(Color(0, 255, 0, 255));

    sb_->Begin(SpriteSortMode::Deferred, BlendState::AlphaBlend,
               nullptr, nullptr, nullptr, fx_.get());
    sb_->Draw(tex_, Rectangle(64, 64, 128, 128),
              Rectangle(0, 0, 1, 1), Color::White);
    sb_->End();
}

Build the effect once and keep it alive. Constructing one per frame recompiles the shader program every frame.

Setting uniforms

There is no Parameters[...] collection on ShaderEffect. Uniforms are set by name through a family of direct methods, each taking a const char* name:

Note the difference between the array rows: SetUniformFloatArray counts scalars, while the vec2, vec3 and mat4 setters count elements. All are documented that way in the header, and it is easy to get wrong. There is no SetUniformMat3 and no vec4-array setter. A matrix uniform is filled from a Matrix with Matrix::ToColumnMajor(float[16]).

Textures beyond unit 0 are bound with SetTexture, which takes a reference and has Texture2D, TextureCube (for samplerCube) and Texture3D (for sampler3D) overloads:

CNAEXT void SetTexture(int unit, Texture2D& texture);
CNAEXT void SetTexture(int unit, TextureCube& texture);
CNAEXT void SetTexture(int unit, Texture3D& texture);

Unit 0 is normally driven by the caller — SpriteBatch's own texture parameter, for instance — so these are for the extra units a custom shader samples directly, the equivalent of XNA's GraphicsDevice.Textures[unit] = tex.

Driving a real 3D draw call

ShaderEffect implements IEffectMatrices, the same interface every stock effect implements. World, View, and Projection are therefore set as properties, and CNA forwards them to the renderer, which binds them as uniforms of exactly those names on your compiled program — matching the naming convention the original XNA samples' own .fx sources already used.

auto* fx = dynamic_cast<ShaderEffect*>(fxBase_.get());
Viewport vp = device.getViewportProperty();

fx->setWorldProperty(world);
fx->setViewProperty(Matrix::CreateLookAt(Vector3(0.0f, 0.0f, 3.0f),
                                         Vector3::Zero,
                                         Vector3(0.0f, 1.0f, 0.0f)));
fx->setProjectionProperty(Matrix::CreatePerspectiveFieldOfView(
    MathHelper::PiOver4, vp.getAspectRatioProperty(), 0.1f, 100.0f));

// Bind the program first, then push this effect's own uniforms.
fx->Apply();
fx->SetTexture(0, whiteTex_);                 // takes a Texture2D&
fx->SetUniformVec3("uLightDir", 0.0f, 0.0f, 1.0f);
fx->SetUniformVec3("uDiffuseColor", 200.0f / 255.0f, 100.0f / 255.0f, 50.0f / 255.0f);

device.SetVertexBuffer(vb_.get());
device.setIndicesProperty(ib_.get());
device.DrawIndexedPrimitives(PrimitiveType::TriangleList, 0, 0, 4, 0, 2);

The matching vertex shader declares those three uniforms and an attribute layout that matches the vertex stride you are drawing with — here VertexPositionNormalTexture, stride 32:

#version 300 es
precision highp float;

layout(location = 0) in vec3 aPosition;
layout(location = 1) in vec3 aNormal;
layout(location = 2) in vec2 aTexCoord;

out vec3 vWorldNormal;
out vec2 vTexCoord;

uniform mat4 World;
uniform mat4 View;
uniform mat4 Projection;

void main() {
    vec4 worldPos = World * vec4(aPosition, 1.0);
    gl_Position = Projection * View * worldPos;
    vWorldNormal = mat3(World) * aNormal;
    vTexCoord = aTexCoord;
}

Where the source actually lives

The constructor wants strings, so the text has to come from somewhere. There are three practical options.

Raw string literals, as above, keep the shader in the binary: no runtime file dependency, no way for the asset to go missing. The cost is that changing a shader means recompiling the game, and no editor will syntax-highlight GLSL inside a C++ literal.

Read the file yourself and pass the result straight in. Nothing in ShaderEffect requires ContentManager involvement:

#include "System/IO/File.hpp"

const std::string vert = System::IO::File::ReadAllText("Content/shaders/lit3d.vert.glsl");
const std::string frag = System::IO::File::ReadAllText("Content/shaders/lit3d.frag.glsl");

ShaderEffect fx(device, vert, frag);

A .cnj descriptor lets ContentManager do the reading. CNA's content loader has a real Effect reader for this. The descriptor has exactly two shader fields, vertex and fragment, each naming a file relative to the content root (not to the descriptor's own folder; here the descriptor sits directly in Content/):

{
  "cnjVersion": 1,
  "type": "Effect",
  "vertex": "lit3d.vert.glsl",
  "fragment": "lit3d.frag.glsl"
}

Missing either field raises a ContentLoadException. Load it as a std::shared_ptr<Effect> — that, not ShaderEffect, is the type the reader is registered for — then downcast:

getContentProperty().setRootDirectoryProperty("Content");

std::shared_ptr<Effect> fxBase =
    getContentProperty().Load<std::shared_ptr<Effect>>("lit3d");

auto* fx = dynamic_cast<ShaderEffect*>(fxBase.get());
if (fx == nullptr || !fx->IsEffectValid()) {
    // Descriptor loaded, but the shader did not compile.
}

Renderers and the shading language

The header documents these strings as GLSL, and GLSL is what the GL family expects — but the constructor does not parse them. It forwards them to whichever renderer is compiled in, and renderers disagree in three quite different ways: some execute your shader, some refuse it loudly, and a few accept it and quietly draw without it.

Renderers that execute a ShaderEffect

Your shader genuinely runs on: the three EasyGL identities (OPENGLES3, OPENGL33, WEBGL2), VULKAN, SDL_GPU (in builds with libshaderc, or when you hand it SPIR-V), WEBGPU and DIRECTX11, and — for SpriteBatch effects written in Metal Shading Language — on METAL. DIRECTX9 and METAL compile your HLSL or MSL for SpriteBatch draws but do not declare source execution, so ExecutesShaderEffectSourceEXT() reads false on both even though the text is compiled.

The shading language differs

Renderers that refuse

• METAL refuses a 3D draw with a custom effect (NotSupportedException at the call site); its custom effects are a SpriteBatch facility, and an MSL source that does not compile gives an invalid effect carrying the Metal compiler’s message rather than an exception.
• FNA3D returns an invalid effect (a null effect renderer; CustomEffects is false, no exception). FNA3D's only shader entry point takes a compiled Effect Framework binary — use Effect(device, bytes) there.
• SDL_RENDERER and STUB also return an invalid effect: the 2D-only renderer has no programmable pipeline to speak of, and 3D calls throw there regardless.

The practical consequence, stated plainly: a single shader source string is not portable across CNA's renderers. Shipping on more than one shading language means authoring one source variant per language and selecting it — at build time with the CNA_RENDERER_<NAME> define CNA's configure emits (the three GL profiles share CNA_RENDERER_EASYGL, distinguished by CNA_GL_PROFILE_*), or at run time from the live device (Tutorial 151 builds exactly that). IsEffectValid() tells you whether the variant you picked compiled — but it cannot help you on HEADLESS or SOFTWARE, where nothing was attempted.

Pre-compiling GLSL to SPIR-V for VULKAN or SDL_GPU is an ordinary offline step:

glslc -fshader-stage=vert lit3d.vert.glsl -o lit3d.vert.spv
glslc -fshader-stage=frag lit3d.frag.glsl -o lit3d.frag.spv

Read those .spv files as bytes (System::IO::File::ReadAllBytes) and pass their contents as the two strings. (Vulkan GLSL needs explicit layout(location=..) and binding qualifiers; loose uniform declarations are not legal there, which is why ShaderEffect::DeclareUniformBlockEXT() exists.)

Typed code and packages (CNA_CNAEXT)

With CNA_CNAEXT=ON two more constructors exist: ShaderEffect(device, ShaderCodeEXT vertex, ShaderCodeEXT fragment), which names the language and entry point of each stage explicitly, and ShaderEffect(device, ShaderPackageEXT), which selects one vertex/fragment variant from a multi-language package for the live device. They throw std::invalid_argument for a bad stage/language/entry point and CNA::ShaderCompilationExceptionEXT when no variant is usable, and the query that decides is GraphicsDevice::SupportsShaderLanguageEXT(language, stage). Vulkan and SDL_GPU execute SPIR-V but their detailed capability profile reports the six ShaderDialect* features as unsupported (SPIR-V has no feature entry), so use that query, not the profile, for them. Tutorial 151 walks through choosing a payload per device.

Cloning

Clone() on a ShaderEffect recompiles a new renderer-side program from the same source strings rather than sharing the original's compiled program. This is a deliberate deviation from the stock effects, which share GPU state implicitly because CNA caches their pipelines globally by state rather than per instance. A ShaderEffect uniquely owns its compiled program, so cloning costs a real recompile — do not do it per frame.

Summary

• ShaderEffect is a CNAEXT extension for renderer-native shader source. The separate Effect(device, effectCode) compatibility path accepts XNA/FNA Effect Framework bytecode only on the 11 capable renderer identities.
• The constructor takes three arguments — the device, and the vertex and fragment shader source text. Not a file path.
• It does not throw on a compile failure. Check IsEffectValid(), and read GetCompileErrorEXT() for the reason.
• Uniforms are set with SetUniformMat4/Vec4/Vec3/Vec2/Float/Int/FloatArray/Vec2Array/Vec3Array/Mat4Array by name, and textures with SetTexture(unit, tex) (a reference). There is no Parameters collection.
• Call Apply() before setting uniforms — the portable order (the GL renderers also make the program current inside each setter).
• World/View/Projection come from IEffectMatrices and are bound as uniforms of those names automatically.
• Keep source in a raw string literal, read it yourself, or reference it from a .cnj Effect descriptor with "vertex" and "fragment" fields.
• The expected shading language is renderer-dependent: desktop GLSL on OPENGL33, GLSL ES on the ES/WebGL profiles, SPIR-V on VULKAN and SDL_GPU, WGSL on WEBGPU, HLSL compiled at run time on the Direct3D renderers, and Metal Shading Language on METAL (SpriteBatch only; a 3D draw with one throws). FNA3D returns an invalid effect, and SOFTWARE and HEADLESS accept it and never execute it.