Oracles, tolerances and engagement gates

CNA snapshot c1c316b9  ·  Deep Dives › Diagnostics & verification  ·  source links pinned to c1c316b9

A test input and an observed output do not yield a verdict until something defines the expected result, and on some hosts the output is not even produced by the thing under test. This page covers both halves of that problem at snapshot c1c316b9: the oracles CNA uses (captured real-XNA images, a running FNA, independently derived tables, cross-renderer controls and CNA's own goldens), what authority and tolerance each carries, and the engagement gates that prove a Wine, virtual-display or browser run exercised the intended implementation rather than a substitute. It is for anyone who writes a pixel or value test, reads a green run from a translated host, or has to word a claim about correctness.

Choose the authority before the tolerance

CNA's oracles differ in what they can decide, and a tolerance only has meaning relative to the authority and the phenomenon measured. The hierarchy, strongest claim first:

Where XNA and FNA disagree, CNA's contributor rules (the owner decision of 2026-09-04 recorded in CLAUDE.md) make measured XNA behaviour, read from the shipped assemblies' IL, the tie-break; FNA remains the day-to-day reference because it is readable source. The per-file port checklist in CHECKLIST.md still carries the older "FNA wins" wording (see Known Issues). The decompiled XNA sources are evidence to audit rather than a specification: seventeen test files cite xna4-decomp paths in comments as the source of an expected value (for example ObjectEqualityContractTests.cpp and ResourceContentManagerTests.cpp), and no test or gate compares behaviour against that repository. The one script that reads it is the hand-run census generator audit_xna_runtime_surface.py, whose --reference-xml-dir defaults to xna4-decomp/dlls beside the CNA checkout: Microsoft's XML documentation and DLLs, not the decompiled C# (the generated census report records the directory it read). A worked case of the tie-break in practice is the packing rule on Rectangle, Point and Color: the genuine framework rounds float-to-byte conversions to nearest with ties to even where CNA, following FNA, had truncated.

Tolerance then follows the phenomenon. Zero is right for a byte-exact decode or a stable reference image; one channel value of rounding may justify a local, written exception; 40 on a whole-frame dump admits different hardware pipelines rasterising the same edge; 60 on a golden means only a coarse visual guard. Naming that strength is part of the verdict: "passed the oracle" is incomplete until the reader knows which oracle, which policy, and whether it actually executed.

The captured-XNA image corpus, briefly

tools/xna-oracle holds 39 declarative scenes and 39 matching 256×256 reference PNGs captured once, by hand, from the real XNA 4.0 runtime under Wine with DXVK. The C# runner's source is checked in; its compiled executable, the XNA runtime and the private 32-bit Wine prefix are not, and are needed only to regenerate a reference, never to compare a current CNA render with the committed corpus. One renderer-agnostic CNA program is built into four renderer trees (DIRECTX9, EasyGL, FNA3D, Software), and xna-diff.py compares absolute per-channel differences with a default tolerance of 0; it has no PSNR, SSIM or perceptual metric. What each registered command actually fails on differs by renderer: the DIRECTX9 gate fails on any differing pixel of any of the 39 scenes, the FNA3D command only on a scene that fails to render, and the EasyGL and Software commands cover two line scenes, with only the EasyGL one failing on a pixel difference. The full provenance, per-renderer records, vacuous-match history and scene-authoring procedure are on Using the XNA oracle as evidence; the method is taught in Tutorial 161.

Where a relaxation belongs

The governing rule is stricter than "looks similar": the header of xna-diff.py warns that raising the tolerance without a documented, scene-specific reason is how an authenticity project quietly becomes a parity project. The tool therefore offers only local relaxations (separate RGB and alpha bounds, a budget of raw differing pixels, an allowed rectangle), and at this snapshot no runner passes any of them. A renderer that does need one should follow a simple discipline: keep the policy beside the scenes, make every bound explicit and spatially limited, and have the runner check that the policy's scene set equals the corpus's scene set before comparing, so a new scene cannot inherit a relaxation silently and a stale row cannot linger.

Project-owned goldens answer a weaker question

PixelTestGame.hpp is the harness of the standalone renderer programs. It offers four check methods, all recorded in one PASS/FAIL stream that becomes the process exit code (a further hook, MarkFailedEXT(), fails the run without printing anything):

• ExpectPixel(label, rect, colour, tolerance = 0) reads back exactly one pixel (its rectangle must be 1×1, because GetBackBufferData rejects any other element count) and compares R, G and B only: the harness follows the examples' convention of ignoring alpha unless a test checks it explicitly.
• ExpectTrue(label, condition) records a non-pixel precondition ("the texture this test draws was actually created"), so a test whose setup failed reports that instead of checking pixels that no draw produced.
• Check(ok, label) is the same generic boolean assertion with its arguments the other way round, kept for checks such as "this render must differ from that one".
• CompareGoldenImage(label, rect, path, tolerance = 0) compares a region with a committed PNG, again on RGB, and prints the largest channel difference it used.

examples/golden holds 17 PNGs of 8×8 pixels, all captured from EasyGL and named after it. Fourteen EasyGL sources compare against them, and thirteen Vulkan registrations (for example Vulkan_BasicEffect_Golden) compile the same EasyGL sources and compare against the same files, with the source root as working directory because the golden paths are repository-relative. The tolerances actually passed at this snapshot span the whole range:

Only the canary that clears to solid blue is exact; the depth-write and sprite-rotation goldens accept 60. At that end a test catches gross shape or state regressions, not shading drift. That does not make it useless; it makes its contract different from the real-XNA corpus: a self-captured golden is a regression and cross-renderer oracle for CNA's chosen behaviour, not evidence that the behaviour matches XNA.

Setting CNA_UPDATE_GOLDEN makes CompareGoldenImage write the live render as the new reference instead of comparing, prints [UPDATE] and reports success. The header comment says "any non-empty value", but the code only tests whether the variable is present in the environment, so a variable that is set but empty (exported by a wrapper script, say) triggers the rewrite too; a green run with [UPDATE] lines compared nothing. Updating is therefore a review operation, never a way to make a failure green: the change should carry the image diff, the renderer, the driver and the reason the new output is correct, because regenerating a shared image silently changes what every consumer of it proves.

Point samples and the skip boundary

Single-pixel checks reach much further than goldens: 225 ExpectPixel( call sites in 49 C++ files, and 119 sources include the harness (lexical counts at c1c316b9). They are cheap and precise for clear colours, blend equations, scissor order or depth outcomes. Before constructing its game, the harness probes SDL video and returns 77 when no display is usable, which CTest reports as skipped. A green headless run can therefore mean that none of these comparisons executed, so a pixel-oracle report states the display used and the skip count, not only the exit status. The procedure for writing one is in Renderer example programs versus CTest.

What the pixel harness itself guarantees

Several properties of PixelTestGame.hpp decide what a pass from one of these programs can mean. None of them is visible in the test's own source:

• One frame. Draw() calls RunTest() on the first frame only, behind a latch, and then calls Exit(). Nothing that needs a second frame, an Update() sequence or a device reset can be observed through the harness. Tests of that kind hand-roll their own Game subclass and call ProbeGpuDisplayAvailable() themselves, returning kSkipExitCode when it is false.
• One pixel per ExpectPixel. The check reads its rectangle with GetBackBufferData(&region, &pixel, 0, 1), and GraphicsDevice::GetBackBufferData throws ArgumentException when the element count does not match the rectangle's area. The rectangle must therefore be 1×1. A larger one does not mean "any pixel in the region"; it raises an exception that ends the program.
• HiDef, always. RunPixelTest<TGame>() sets GraphicsProfile::HiDef before Run(), because back-buffer readback throws under the default Reach profile. It sets the profile on the fixture's GraphicsDeviceManager when the fixture has registered one, and otherwise calls SetGraphicsProfileEXT on the eagerly created device. A fixture meant to observe a Reach-profile refusal cannot rely on the harness default.
• Narrow skips. Besides the display pre-flight, exactly one exception becomes exit 77: one whose message contains VK_ERROR_SURFACE_LOST_KHR. A Vulkan swap chain raises that error on an X server without DRI3, after the display probe has already succeeded. Every other exception is rethrown on purpose, so a genuine pipeline defect ends the process and CTest reports a failure, not a skip.
• No SDL, no pre-flight. In a build without SDL the examples define CNA_EXAMPLES_NO_SDL. The probe then answers "available" without checking anything, and the selected platform's own window creation acts as the probe. A missing display therefore surfaces as a failure rather than a skip, which the header states is deliberate.
• Non-pixel failures. ExpectTrue(label, condition) and MarkFailedEXT() put a precondition or an expected exception into the same verdict. A test that only prints [FAIL] still exits 0.

Readback problems in these programs can be traced with CNA_BACKBUFFER_READ_TRACE. It makes GetBackBufferData print the requested rectangle and the back-buffer and viewport sizes to standard error (see Debugging). Checked by reading the harness and GraphicsDevice.cpp at c1c316b9; not executed.

FNA as a numeric reference

A separate chain runs C# against a live FNA.dll under Mono (tools/fna-reference), dumps CNA's side with cna_reference_dump (CnaReferenceDump.cpp) and compares the two JSON documents with compare-fna-reference.py. Its default categories are non-rendering APIs, PackedVector and Viewport (enums, state presets, packed-vector values, viewport project and unproject). Numbers compare with an absolute tolerance of 1e-4 (--tolerance), everything else must match exactly, and the walk is one-directional: every key FNA emits must exist on the CNA side, while CNA-only keys, such as extension members, are not reported. It exits 0 or 1, and it is a manual developer workflow, not a CTest and not in CI.

Its history is a useful counter-example to "a second implementation only matters for pixels". The first comparison found three defects in the harness itself before anything real (JSON numbers written with six significant digits, state classes dumped without inherited properties, enum members missing on the C++ side), and then exactly one genuine divergence: IndexElementSize used the semantic values 16 and 32 where FNA uses the ordinals 0 and 1. That was fixed on 2026-07-09 (CNA's enum now declares SixteenBits = 0 and ThirtyTwoBits = 1), so re-running the comparison would no longer report it. The example shows a second implementation challenging assumptions embedded in the public value surface, and why an oracle harness must be debugged before it is trusted.

Independent oracles inside the corpus

Not every expected value comes from an external runtime. Five recurring patterns keep CNA's own tests from grading themselves:

• Golden tables. PackedVectorGolden.md derives packed-vector values in Python from FNA's bit-packing formulas (round by adding 0.5, clamp, IEEE half floats via struct.pack), and the Golden* cases of PackedVectorTests.cpp check them with narrow tolerances. The genuine-framework packing oracle then outranks that table where the two differ (above).
• Recomputation ladders. The glTF oracle helpers in GltfOracleEXTTests.cpp dump accessors (L2) and vertex streams (L3) and evaluate world positions (L4) independently of the production routine, against values stored in each fixture's manifest (tolerance 1e-5). The helpers carry their own acceptance tests: deterministic dumps, enough structure to tell an offset error from a stride error, the manifest values for the control fixture, and proof that using the oracle does not change what the production import produces.
• Shared oracle headers. ViewSpaceFogRef.hpp holds canonical transformed-camera fog scenes and their analytically derived colours from the FNA fog formula; the EasyGL, SDL_GPU and Vulkan fog tests and a renderer-neutral contract test all include it, so the copies cannot drift apart.
• Byte-exact differentials. LzxDecoderDifferentialTests.cpp compares CNA's LZX output byte for byte with reference files produced by running FNA's unmodified LzxDecoder.cs under Mono, checked in with instructions for regenerating them; a fuzz suite beside it covers malformed input.
• Cross-renderer controls with a whitelist. Two renderer-agnostic programs dump a 64×64 RGBA8 back buffer: cross_renderer_diagnostic_scene.cpp (a vertex-coloured BasicEffect triangle, built for EasyGL, Vulkan, Software and WebGPU) and cross_renderer_2d_corpus.cpp (a fixed SpriteBatch corpus, built for EasyGL and Vulkan). The 2D corpus deliberately excludes every construct where two conforming renderers may legitimately disagree, and names them: linear filtering of a magnified sprite (kernel and edge behaviour unspecified), mip selection and mip-linear blending (not a shared contract), anisotropic filtering, MSAA and additive blending (capability-dependent; a 2D-only renderer may report them unsupported), rotation by a non-right angle (edge coverage is implementation defined) and non-opaque destinations (presentation-time alpha handling differs). cna_diag_compare then diffs two dumps per channel and exits 1 when the largest difference exceeds the tolerance argument (its own default, when none is given, is 1). The caller chooses the bound: compare-easygl-vulkan-diagnostic.sh, which runs both programs for EasyGL and Vulkan, defaults to 40, inherited from the older Software/EasyGL comparison. That 40 has never been exercised: the script's recorded first run (2026-09-06, llvmpipe on Xvfb) measured a largest difference of 1 on the triangle and a byte-identical 2D corpus, and passes at 2. Because a CNA build links one renderer by default, the two dumps come from two build directories, which is why compare-easygl-vulkan-diagnostic.sh is a script and not a CTest. Excluding unstable constructs is preferable to widening the tolerance until everything passes.

Engagement: proving the experiment happened

Some hosts return success while silently substituting a different runtime. Wine can fall back from DXVK to its own Direct3D, a browser page can load without reaching a frame, and a test binary can be listed without ever opening a device. On such hosts the first oracle is not the game's output but proof that the intended execution route engaged. A robust harness therefore returns two verdicts, an engagement verdict naming the runtime, renderer, device or browser path, and a behaviour verdict from the functional oracle, and only their conjunction is a pass.

The Wine wrapper as a verdict pipeline

The Direct3D wrappers follow one sequence: validate the arguments, select an isolated prefix through a renderer-specific variable, set Wine's diagnostic variables, refuse a prefix with no system.reg (printing the wineboot --init and dxvk-setup install commands instead of bootstrapping or mutating anything), run wine through tee, keep Wine's own exit status from PIPESTATUS, and only then inspect the captured log. A DXVK run must contain a DXVK: <version> line; without it the wrapper exits 3 whatever the program returned, because a passing executable through the wrong translation layer is a failed experiment. DIRECTX9 and DIRECTX11 share the physical prefix ~/.wine-cna-d3d11 through separate variables. The table of wrappers, markers and opt-outs is on Direct3D evidence: authenticity gates, and a marker proves which translator served the run, not how a Windows driver would behave.

Discovery needs a controlled escape hatch

A cross-compiled CnaTests.exe cannot run natively on the Linux host, so UnitTests.cmake sets the target's CROSSCOMPILING_EMULATOR to the Wine wrapper of the configured Direct3D identity, and GoogleTest discovery then runs --gtest_list_tests through it. Listing names creates no graphics device, so no engagement token can legitimately appear, and the emulator command therefore carries CNA_D3D9_SKIP_DXVK_GATE=1 or CNA_D3D11_SKIP_DXVK_GATE=1. Writing the bypass into the generated command, rather than teaching every wrapper to accept missing evidence, keeps it reviewable. Its reach is wider than discovery, though: the emulator is a property of the target, so the same gate-free wrapper also runs every discovered CnaTests case and every named route whose command is CnaTests (the file's own comment names CnaInputTests), on the stated grounds that most unit tests create no device either. In those trees, translation-layer engagement is proved by the separately registered renderer programs such as DirectX9_Smoke, which keep the gate; only programs that never open a device, such as DirectX9_Common and DirectX9_ShaderDispatch, opt out, each with a comment saying why. A CnaTests case that does create a device in a cross tree is not, by itself, evidence that DXVK served it.

Virtual displays are infrastructure, not evidence

A private Xvfb and the private Weston and Xwayland pair of run_gpu_tests_private.sh make headless presentation repeatable and keep tests off a developer's desktop. They do not prove physical display behaviour, vendor drivers, HDR or desktop composition, and a report must still say that a virtual display and a translation layer were used. Likewise a Windows executable returning zero under Wine does not become a native-Windows result. The environment belongs in the claim: compiler, executable format, Wine version, translation backend, GPU or software device, display server and prefix.

Browsers need an out-of-process verdict

A WebAssembly module's exit status is not observable from outside a browser, so a CNA browser test page publishes its verdict where a harness can read it: the C ABI's browser probe prints [FAIL] lines and a final === <passed>/<expected> PASS === summary to the console. run_browser_tests.mjs launches headless Chromium through Playwright, collects console lines and page exceptions, waits for that summary with a bounded deadline, and passes a page only when the summary is present, its two counts are equal and no [FAIL] line or page exception appeared. The page's own summary is the authority rather than an exit code, and a run that dies early never prints it, which is what makes its absence a failure. A page that reports its own checks still says nothing about composited pixels; the WebGPU parity route, run-webgpu-parity-test.sh, instead extracts the page's 64×64 back-buffer dump from the in-memory file system and diffs it against another renderer's dump with cna_diag_compare.

Serving is part of the environment. The runner serves the build directory itself over HTTP, bound to 127.0.0.1 on a port the operating system chooses, so two concurrent runs cannot collide on a fixed port; it never uses file://, from which the module cannot be fetched and whose origin rules differ from a deployed page. No workflow runs a browser renderer at this snapshot; the browser renderers are introduced in Tutorial 105, and the evidence each has is on web renderer evidence.

Hostility is any plausible false green

Wine fallback, a headless skip, a browser that started without a frame, a unit binary run through a gate-free wrapper and one renderer impersonating another look unrelated. They share one failure mode: the program appears healthy while the intended experiment never occurred. The transferable pattern is not any particular grep token but a procedure: identify how the host can substitute or skip the thing under test, then make evidence of non-substitution a first-class artifact of the run, with its own positive and negative fixtures (as DirectX11_DxvkGate provides for the DXVK wrapper; see Can a green gate go red?).

Evidence and limits

Read at c1c316b9 from the oracle tools, PixelTestGame.hpp, the golden sources and the Vulkan and EasyGL example CMake files, the FNA and LZX harnesses, the diagnostic corpus sources, the Wine, display and browser scripts and UnitTests.cmake. Counts of call sites, includes and tolerances are lexical. No oracle, Wine or browser run was executed for this page. That discovered CnaTests cases in a cross-compiled tree inherit the target's emulator follows CMake's documented handling of CROSSCOMPILING_EMULATOR and the comment beside it in UnitTests.cmake; it was not observed. The FNA divergence and its fix are CNA's own record (docs/fna-reference-harness.md), checked against the current enum values.