Tutorial 125: Headless and Pixel Testing, and the XNA Oracle Corpus

CNA Tutorials  ·  Verification

Testing game logic is a solved problem — keep it away from GraphicsDevice and assert on it like any other C++. Testing rendering is the hard half, and it is where CNA has invested most: proving that a draw call produced the right pixels, on a machine with no GPU, with no human looking at the screen.

CNA does this in three tiers, and they are very unevenly sized. Understanding the proportions is the fastest way to understand how the project actually verifies itself.

That middle row is the one to notice. Hand-coded pixel assertions vastly outnumber golden-image comparisons in CNA, and that is a deliberate choice rather than an accident of history: an assertion that says “the pixel at (4,4) must be exactly (140,140,140), because the glyph bitmap's row 4 is on and the cell averaged to that grey” explains itself when it fails. A golden-image diff says only that 37 pixels changed.

HEADLESS and STUB: rendering nothing, on purpose

Every CNA GPU renderer needs a window and a graphics context to run at all, which makes them unsuitable for fast logic tests: a CI container with no display server cannot run them, and even with a virtual display, standing one up per test run is slow relative to what a logic test needs to prove.

Two renderers exist precisely to remove that dependency without producing pixels. (A third, SOFTWARE, is a CPU rasteriser that needs no window either and produces pixels you can read back; it stays off-screen on windowing platforms.)

cmake -S . -B build-headless -DCNA_GRAPHICS_RENDERER=HEADLESS -DCNA_BUILD_TESTS=ON
cmake --build build-headless --parallel

HEADLESS implements CNA's entire renderer interface without touching a GPU or a window — SDL's video subsystem is never initialised and no window is ever created — while doing real bookkeeping: argument validation, resource lifecycle tracking, draw-call and state-change counters. It needs no extra dependencies at all.

It has one runtime dial rather than three separate builds, selected by environment variable or programmatically before Game::Run():

CNA_HEADLESS_MODE=Fast ./build-headless/MyGameTests

STUB is the other end of the same idea: the smallest possible complete renderer implementation. Every method does nothing or returns a trivial value, it keeps no bookkeeping whatsoever, and it has no external dependencies. It is useful as a dependency-free build target and as a reference for anyone writing a new renderer.

GetBackBufferData: the technique that does the work

When you do need to know what was drawn, you read the back buffer and assert on it. There are three overloads:

void GetBackBufferData(Color* data, int elementCount);
void GetBackBufferData(Color* data, int startIndex, int elementCount);
void GetBackBufferData(const Rectangle* rect, Color* data, int startIndex, int elementCount);

Four rules decide whether a readback works at all, all enforced by GraphicsDevice itself:

• The graphics profile must be HiDef. Under Reach — which is what an application that declares no profile gets — GetBackBufferData throws NotSupportedException. CNA’s own pixel-test base class selects HiDef explicitly.
• It reads the back buffer, not a render target, and it refuses to run while a render target is bound (InvalidOperationException). To read a RenderTarget2D, call its own GetData.
• The full-buffer overloads must be sized exactly: elementCount has to equal width × height of the back buffer, or you get an ArgumentException. Set the back-buffer size yourself so one source pixel maps to one output pixel, or use the rectangle overload to read a sub-region.
• A renderer with no pixels refuses. Ask which renderer you got rather than guessing.

The pattern CNA uses throughout looks like this (this variant compiles against the snapshot’s headers; DrawTheThingUnderTest is yours to write):

#include "CNA/GraphicsRendererType.hpp"
#include "Microsoft/Xna/Framework/Game.hpp"
#include "Microsoft/Xna/Framework/GraphicsDeviceManager.hpp"
#include "Microsoft/Xna/Framework/Graphics/GraphicsDevice.hpp"
#include "Microsoft/Xna/Framework/Graphics/GraphicsProfile.hpp"
#include "Microsoft/Xna/Framework/Rectangle.hpp"

#include <cstdio>
#include <cstdlib>

using namespace Microsoft::Xna::Framework;
using namespace Microsoft::Xna::Framework::Graphics;

class ReadbackGame : public Game
{
    static constexpr int kSize = 64;

public:
    ReadbackGame() : graphics_(this)
    {
        graphics_.setGraphicsProfileProperty(GraphicsProfile::HiDef);   // readback needs HiDef
        graphics_.setPreferredBackBufferWidthProperty(kSize);           // the size YOU chose,
        graphics_.setPreferredBackBufferHeightProperty(kSize);          // not the window manager's
    }

protected:
    void Draw(const GameTime&) override
    {
        auto& device = getGraphicsDeviceProperty();

        // HEADLESS and STUB have no back buffer to read. Skip (CTest code 77), do not fail.
        const CNA::GraphicsRendererType renderer = device.GetGraphicsRendererType();
        if (renderer == CNA::GraphicsRendererType::Headless ||
            renderer == CNA::GraphicsRendererType::Stub)
        {
            std::exit(77);
        }

        device.Clear(Color(140, 140, 140, 255));
        DrawTheThingUnderTest(device);

        // One pixel, through the rectangle overload.
        const Rectangle region(4, 4, 1, 1);
        Color pixel;
        device.GetBackBufferData(&region, &pixel, 0, 1);

        const bool ok = pixel.getRProperty() == 140 && pixel.getGProperty() == 140 &&
                        pixel.getBProperty() == 140;
        std::printf("[%s] cell centre reads the exact foreground colour\n", ok ? "PASS" : "FAIL");
        Exit();
        if (!ok) std::exit(1);
    }

private:
    GraphicsDeviceManager graphics_;
};

Two conventions in that snippet are worth adopting in your own tests.

Exit code 77 means “skipped”. CNA registers its renderer tests with CTest's SKIP_RETURN_CODE set to 77, so a test that discovers it is running on a renderer without the capability it needs reports skipped rather than failing. That is what keeps one test source file usable across renderers with genuinely different capability sets. Tests that need real 3D guard on GraphicsDevice::SupportsCapability(GraphicsCapability::ThreeD) and skip the same way — though see Tutorial 101 first for how far each renderer’s answer to that query can be trusted.

Choose coordinates you can justify. Pick a resolution where one source pixel maps onto one output pixel with no stretch or interpolation ambiguity, then assert at points whose expected value you derived by hand before running anything. A test that samples wherever the output happened to land is a test that will be adjusted until it passes.

Image comparison: the smaller tier

Where a scene is too complex to describe in a handful of point assertions, CNA falls back to comparing against a committed reference image — but sparingly. There are 17 golden PNGs in examples/golden/, all for the shared GL implementation, covering the stock effects, PBR, blend and depth-stencil state, rasterizer cull mode, texture filtering and sprite rotation. A second, much larger family exists for glTF: the conformance suite commits four sets of 140 goldens (EasyGL, Vulkan, Software and DirectX11), but each renderer is compared only with its own past output at tolerance 0, which catches regressions and non-determinism rather than disagreement with a reference renderer.

Seventeen general-purpose goldens — or under six hundred counting the glTF sets — is a small number next to roughly 815 readback call sites, and that ratio is the point: golden images are used where they earn their keep, not as the default.

The XNA oracle corpus

The most interesting verification asset in the project is not a test of CNA against itself. It is a test of CNA against real XNA 4.0.

The corpus is 39 declarative scenes. Each is a small, line-oriented key=value text file — back-buffer size, clear colour, which of the five stock effects to construct, vertex data, light and fog settings, sprite draws. Two programs parse that same file identically: Oracle.cs, compiled against the genuine XNA 4.0 assemblies and run under Wine (with DXVK, on Linux — not on Windows), and CnaOracleRender.cpp, built against CNA's real public Game/GraphicsDevice API. Each renders the scene to a PNG. The two PNGs are diffed. Every scene is 256×256 under the HiDef profile.

Two further sets of measurements sit beside the corpus but outside its denominator: seven unbound-texture scenes (64×64, in scenes/null-texture/) that measure what a stock effect samples from a texture that was never bound, and a 17-row channel-expansion table from FormatExpansionOracle.cs recording what real XNA returns for the channels a SurfaceFormat does not store. No script diffs them; unit tests hard-code the recorded values.

Everything about the design is aimed at removing places for the harness to lie to itself:

• The scene is authored once. Both sides parse the same file, so the two implementations cannot drift apart through hand-transcription.
• Unknown keys are a hard error on both sides. A typo fails loudly instead of quietly changing what “a match” means.
• DXVK must be installed into the XNA side's Wine prefix too. Otherwise real XNA runs on WineD3D while CNA runs on DXVK, and the diff silently measures a driver difference and blames CNA for it.
• The 39 reference PNGs are committed. They were captured once, by hand, against the real runtime. CNA's D3D9_XNA_Diff CTest re-renders every scene through the DIRECTX9 renderer and diffs against them — without needing the XNA prefix at all, though it still needs the Wine-and-DXVK side to run CNA's own Direct3D 9 build, and it exists only in a DIRECTX9 cross-build tree.
• The diff tool is mutation-verified. A deliberately one-value-off copy of a passing image is correctly reported as failing, and correctly passes again at tolerance 1 — so the tool is known to discriminate, not merely to print “PASS”.

python3 scripts/xna-diff.py xna_out.png cna_out.png --diff-out diff.png   # needs Pillow

It compares every RGBA channel of every pixel, prints one summary line such as PASS: 0/65536 pixels differ beyond tolerance=0, max per-channel delta=0, and exits non-zero on a mismatch. Tutorial 161 walks through rendering a scene and running the diff end to end.

“Pixel-exact” is a DIRECTX9 statement

The SOFTWARE, EasyGL and FNA3D rows are CNA-recorded, dated measurements, not current gates: the corpus scripts for them report differences rather than fail on them, except for the two aliased-line scenes: on EasyGL any difference fails the CTest, while the SOFTWARE CTest runs the same two scenes at tolerance 0 but fails only if one does not render (a difference is printed, not failed; CNA’s records list both as byte-exact). Treat the older EasyGL and FNA3D counts as possibly understated.

The gap is not evidence that the other renderers are broken. Direct3D 9 is the API real XNA itself ran on, and CNA's DIRECTX9 renderer executes through the same DXVK implementation the oracle does, so an exact match is achievable there in a way it simply is not across a different graphics API with different rasterization rules and different floating-point behaviour. What the table does establish is that the phrase “pixel-exact” belongs to one renderer on one stack — it is not a claim about Direct3D 9 on native Windows drivers — and that the corpus is a genuine, discriminating measurement rather than a rubber stamp.

It is also the strongest gate in the repository — and no CI workflow runs the whole corpus (the two-scene EasyGL line test is an ordinary CTest, whose CI status is unverified because the general job does not install Pillow). If you are relying on it, you are running it yourself.

Independent FNA evidence exists too, but it is a different, smaller thing: a manual harness that runs a real FNA build and diffs values (enums, state presets, packed vectors, viewport) against CNA's, plus FNA-recorded compiled-effect fixtures that CNA's tests replay. See Verification & Known Issues.

What one-renderer-per-build costs

CNA defaults to one renderer per build. CNA can opt into several with CNA_GRAPHICS_RENDERERS and select one before device creation, but renderer-specific configuration and dependencies still differ, and some test groups (the cross-renderer parity fixtures, the renderer-specific oracle programs) only exist for the renderer a tree was configured with. Separate build directories remain the clearest way to isolate a verification matrix.

The registration shape is configuration-dependent. Renderer, platform, audio implementation, host and feature options determine which pixel, smoke and integration tests are compiled and registered. In a multi-renderer build, several renderer-specific sets may be present; use ctest -N and names or labels to report the actual build rather than a tree-wide total.

# One renderer, one build directory, one test run. Repeat per renderer.
cmake -S . -B build-gles3 -DCNA_GRAPHICS_RENDERER=OPENGLES3
cmake --build build-gles3 --parallel
ctest --test-dir build-gles3 -R '^EasyGL_' --output-on-failure

Test names carry the family prefix (EasyGL_, Vulkan_, Software_, and so on), which is what -R selects here; only some registrations carry labels (ctest -L Parity, -L Software, -L DIRECTX9). The three GL-profile identities all belong to the EasyGL family, so OPENGLES3 registers EasyGL_ tests, not OPENGLES3_ ones.

Most GPU tests need a real display

The renderer tests that create a window bake SDL_VIDEODRIVER=x11 into their CTest registration. Where they open is decided by the CNA_TEST_DISPLAY cache variable, and its policy changed after alpha.1: it is now empty by default, so those tests inherit the DISPLAY of whoever launches them instead of forcing :0 — your real desktop session. Naming the live desktop is opt-in (-DCNA_TEST_DISPLAY=:0 together with -DCNA_TEST_ALLOW_LIVE_DISPLAY=ON; without the second flag a :0 value is reset to empty), and no test falls back to your Wayland compositor either.

On a headless machine, give them a virtual display, either by launching under one or by pointing the variable at one:

xvfb-run -a ctest --test-dir build-gles3 --output-on-failure      # inherit a virtual DISPLAY

# or a fixed Xvfb, honoured as before
Xvfb :99 -screen 0 1280x1024x24 &
cmake -S . -B build-gles3 -DCNA_GRAPHICS_RENDERER=OPENGLES3 -DCNA_TEST_DISPLAY=:99
cmake --build build-gles3 --parallel
ctest --test-dir build-gles3 --output-on-failure

On a workstation with a real GPU, tools/platform/run_gpu_tests_private.sh <build-dir> [ctest args] runs the tests on a private headless compositor with its own Xwayland, so nothing appears on your desktop; Tutorial 160 covers it, the focused per-module test targets and the bounded-memory runner.

This is the practical reason the HEADLESS and STUB renderers exist: they need none of this (and neither does the SOFTWARE CPU rasteriser, which stays off-screen).

Applying this to your own game

1. Build your logic tests against HEADLESS. They run anywhere, need no display, and Validation mode will catch API misuse a real renderer would have rejected.
2. Choose the resolution yourself: set the back-buffer size on GraphicsDeviceManager (or read a RenderTarget2D through its own GetData), and select the HiDef profile, which readback requires. One source pixel per output pixel removes every interpolation argument from your assertions.
3. Assert exact colours at hand-derived coordinates. Write down why each expected value is what it is, in a comment, before you run the test.
4. Skip when the renderer has no pixels (check GetGraphicsRendererType(), or handle the refusal), so the same test source survives a renderer switch.
5. Reach for a reference image only when point assertions genuinely cannot express the check — and diff at tolerance 0 unless you can write down why not.
6. Verify on the renderer you ship. Green on OPENGLES3 says nothing about METAL or DIRECTX9, which run different shader code and have their own refusals.

Where to go next

• Tutorial 99: Unit Testing CNA Game Logic
• Tutorial 160: Running Focused Tests, Bounded Runs and Parity Fixtures
• Tutorial 161: Diffing a Renderer Against the XNA Oracle
• Tutorial 101: Querying renderer capabilities — and why the query fails open
• Tutorial 72: Choosing a renderer
• Tutorial 79: Debugging CNA games
• Verification & Known Issues and XNA Compatibility reference