Tutorial 125: Headless and Pixel Testing, and the XNA Oracle Corpus

CNA Tutorials  ·  Verification

Testing game logic is a solved problem — keep it away from GraphicsDevice and assert on it like any other C++. Testing rendering is the hard half, and it is where CNA has invested most: proving that a draw call produced the right pixels, on a machine with no GPU, with no human looking at the screen.

CNA does this in three tiers, and they are very unevenly sized. Understanding the proportions is the fastest way to understand how the project actually verifies itself.

That middle row is the one to notice. Hand-coded pixel assertions vastly outnumber golden-image comparisons in CNA, and that is a deliberate choice rather than an accident of history: an assertion that says “the pixel at (4,4) must be exactly (140,140,140), because the glyph bitmap's row 4 is on and the cell averaged to that grey” explains itself when it fails. A golden-image diff says only that 37 pixels changed.

HEADLESS and STUB: rendering nothing, on purpose

Every real CNA renderer needs a window and a graphics context to run at all, which makes them unsuitable for fast logic tests: a CI container with no display server cannot run them, and even with a virtual display, standing one up per test run is slow relative to what a logic test needs to prove.

Two renderers exist precisely to remove that dependency.

cmake -S . -B build-headless -DCNA_GRAPHICS_RENDERER=HEADLESS -DCNA_BUILD_TESTS=ON
cmake --build build-headless -j3

HEADLESS implements CNA's entire renderer interface without touching a GPU or a window — SDL's video subsystem is never initialised and no window is ever created — while doing real bookkeeping: argument validation, resource lifecycle tracking, draw-call and state-change counters. It needs no extra dependencies at all.

It has one runtime dial rather than three separate builds, selected by environment variable or programmatically before Game::Run():

CNA_HEADLESS_MODE=Fast ./build-headless/MyGameTests

STUB is the other end of the same idea: the smallest possible complete renderer implementation. Every method does nothing or returns a trivial value, it keeps no bookkeeping whatsoever, and it has no external dependencies. It is useful as a dependency-free build target and as a reference for anyone writing a new renderer.

GetBackBufferData: the technique that does the work

When you do need to know what was drawn, you read the back buffer and assert on it. There are three overloads:

void GetBackBufferData(Color* data, int elementCount);
void GetBackBufferData(Color* data, int startIndex, int elementCount);
void GetBackBufferData(const Rectangle* rect, Color* data, int startIndex, int elementCount);

The pattern CNA uses throughout looks like this. Note that the scene is composed into an ordinary RenderTarget2D first, so the test controls the exact resolution and never depends on the window manager giving it the size it asked for:

#include "Microsoft/Xna/Framework/Game.hpp"
#include "Microsoft/Xna/Framework/Graphics/GraphicsDevice.hpp"
#include "Microsoft/Xna/Framework/Graphics/RenderTarget2D.hpp"
#include "System/NotSupportedException.hpp"

using namespace Microsoft::Xna::Framework;
using namespace Microsoft::Xna::Framework::Graphics;

void Draw(const GameTime&) override
{
    auto& device = getGraphicsDeviceProperty();
    constexpr int kSize = 64;

    RenderTarget2D scene(device, kSize, kSize);
    device.SetRenderTarget(&scene);
    device.Clear(Color(140, 140, 140, 255));
    device.SetRenderTarget(nullptr);

    DrawTheThingUnderTest(scene);

    std::vector<Color> pixels(kSize * kSize);
    try
    {
        device.GetBackBufferData(pixels.data(), static_cast<int>(pixels.size()));
    }
    catch (const System::NotSupportedException&)
    {
        // HEADLESS / STUB have no back buffer to read. Skip, do not fail.
        std::exit(77);
    }

    const Color& c = pixels[4 * kSize + 4];
    Check(c.getRProperty() == 140 && c.getGProperty() == 140 && c.getBProperty() == 140,
          "cell centre reads the exact foreground colour");
}

Two conventions in that snippet are worth adopting in your own tests.

Exit code 77 means “skipped”. CNA registers its renderer tests with CTest's SKIP_RETURN_CODE set to 77, so a test that discovers it is running on a renderer without the capability it needs reports skipped rather than failing. That is what keeps one test source file usable across renderers with genuinely different capability sets. Tests that need real 3D guard on GraphicsDevice::SupportsCapability(GraphicsCapability::ThreeD) and skip the same way — though see Tutorial 101 first, because that query fails open and is not trustworthy on its own.

Choose coordinates you can justify. Pick a resolution where one source pixel maps onto one output pixel with no stretch or interpolation ambiguity, then assert at points whose expected value you derived by hand before running anything. A test that samples wherever the output happened to land is a test that will be adjusted until it passes.

Image comparison: the smaller tier

Where a scene is too complex to describe in a handful of point assertions, CNA falls back to comparing against a committed reference image — but sparingly. There are 17 golden PNGs in the tree, all for the shared GL implementation, covering the stock effects, PBR, blend and depth-stencil state, rasterizer cull mode, texture filtering and sprite rotation.

Seventeen is a small number next to more than 1,400 hand-written readback assertions, and that ratio is the point: golden images are used where they earn their keep, not as the default.

The XNA oracle corpus

The most interesting verification asset in the project is not a test of CNA against itself. It is a test of CNA against real XNA 4.0.

The corpus is 39 declarative scenes. Each is a small, line-oriented key=value text file — back-buffer size, clear colour, which of the five stock effects to construct, vertex data, light and fog settings, sprite draws. Two programs parse that same file identically: Oracle.cs, compiled against the genuine XNA 4.0 assemblies and run under Wine, and CnaOracleRender.cpp, built against CNA's real public Game/GraphicsDevice API. Each renders the scene to a PNG. The two PNGs are diffed.

Everything about the design is aimed at removing places for the harness to lie to itself:

• The scene is authored once. Both sides parse the same file, so the two implementations cannot drift apart through hand-transcription.
• Unknown keys are a hard error on both sides. A typo fails loudly instead of quietly changing what “a match” means.
• DXVK must be installed into the XNA side's Wine prefix too. Otherwise real XNA runs on WineD3D while CNA runs on DXVK, and the diff silently measures a driver difference and blames CNA for it.
• The 39 reference PNGs are committed. They were captured once, by hand, against the real runtime. A CTest re-renders every scene through CNA and diffs against them — without needing the XNA prefix at all.
• The diff tool is mutation-verified. A deliberately one-byte-mutated copy of a passing image is correctly reported as failing, and correctly passes again at tolerance 1 — so the tool is known to discriminate, not merely to print “PASS”.

python3 scripts/xna-diff.py xna_out.png cna_out.png --diff-out diff.png

“Pixel-exact” is a DIRECTX9 statement

The gap is not evidence that the other three renderers are broken. Direct3D 9 is the API real XNA itself ran on, and CNA's DIRECTX9 renderer executes through the same DXVK implementation the oracle does, so an exact match is achievable there in a way it simply is not across a different graphics API with different rasterization rules and different floating-point behaviour. What the table does establish is that the phrase “pixel-exact” belongs to one renderer, and that the corpus is a genuine, discriminating measurement rather than a rubber stamp.

It is also the strongest gate in the repository — and it runs in no CI workflow at all. If you are relying on it, you are running it yourself.

What one-renderer-per-build costs

CNA compiles exactly one renderer per build, chosen at configure time. There is no runtime switching. For verification, that has a direct consequence: exercising n renderers means n full configure, build and test cycles.

The numbers make the shape clear. CNA registers 1,621 renderer pixel, smoke and integration tests with CMake across the tree, spread over 42 renderer families — but any single build registers at most 293 of them, because only one family compiled. Which ones you get is entirely determined by your CNA_GRAPHICS_RENDERER value.

# One renderer, one build directory, one test run. Repeat per renderer.
cmake -S . -B build-gles3 -DCNA_GRAPHICS_RENDERER=OPENGLES3
cmake --build build-gles3 -j3
ctest --test-dir build-gles3 -L OPENGLES3 --output-on-failure

Most GPU tests need a real display

The renderer tests that create a window bake an explicit environment into their CTest registration: SDL_VIDEODRIVER=x11 together with a DISPLAY value. That value comes from the CNA_TEST_DISPLAY cache variable, which defaults to :0 — your real desktop session.

On a headless machine, point it at a virtual display instead:

Xvfb :99 -screen 0 1280x1024x24 &
cmake -S . -B build-gles3 -DCNA_GRAPHICS_RENDERER=OPENGLES3 -DCNA_TEST_DISPLAY=:99
cmake --build build-gles3 -j3
ctest --test-dir build-gles3 --output-on-failure

This is the practical reason the HEADLESS and STUB renderers exist: they are the only two that need none of this.

Applying this to your own game

1. Build your logic tests against HEADLESS. They run anywhere, need no display, and Validation mode will catch API misuse a real renderer would have rejected.
2. Compose into a RenderTarget2D at a resolution you chose, not the window's. One source pixel per output pixel removes every interpolation argument from your assertions.
3. Assert exact colours at hand-derived coordinates. Write down why each expected value is what it is, in a comment, before you run the test.
4. Catch NotSupportedException around the readback and skip, so the same test source survives a renderer switch.
5. Reach for a reference image only when point assertions genuinely cannot express the check — and diff at tolerance 0 unless you can write down why not.
6. Verify on the renderer you ship. Green on OPENGLES3 says nothing about WEBGL1, which really does lack MRT, instancing and Texture3D.

Where to go next

• Tutorial 99: Unit Testing CNA Game Logic
• Tutorial 101: Querying renderer capabilities — and why the query fails open
• Tutorial 72: Choosing a renderer
• Tutorial 79: Debugging CNA games
• Verification & Known Issues and XNA Compatibility reference