Oracles, tolerances and engagement gates

CNA snapshot 009d40f5  ·  Deep Dives › Diagnostics & verification  ·  source links pinned to 009d40f5

A test input and an observed output do not yield a verdict until something defines the expected result, and on some hosts the output is not even produced by the thing under test. This page covers both halves of that problem at snapshot 009d40f5: the oracles CNA uses (captured real-XNA images, a running FNA, independently derived tables, cross-renderer controls and CNA's own goldens), what authority and tolerance each carries, and the engagement gates that prove a Wine, Proton, virtual-display or browser run exercised the intended implementation rather than a substitute. It is for anyone who writes a pixel or value test, reads a green run from a translated host, or has to word a claim about correctness.

Choose the authority before the tolerance

CNA's oracles differ in what they can decide, and a tolerance only has meaning relative to the authority and the phenomenon measured. The hierarchy, strongest claim first:

Where XNA and FNA disagree, CNA's contributor rules (the owner decision of 2026-09-04 recorded in CLAUDE.md) make measured XNA behaviour, read from the shipped assemblies' IL, the tie-break; FNA remains the day-to-day reference because it is readable source. The per-file port checklist in CHECKLIST.md still carries the older "FNA wins" wording (see Known Issues). The decompiled XNA sources are evidence to audit rather than a specification: seventeen test files cite xna4-decomp paths in comments as the source of an expected value (for example ObjectEqualityContractTests.cpp and ResourceContentManagerTests.cpp), and no test or gate compares behaviour against that repository. The one script that reads it is the hand-run census generator audit_xna_runtime_surface.py, whose --reference-xml-dir defaults to xna4-decomp/dlls beside the CNA checkout: Microsoft's XML documentation and DLLs, not the decompiled C# (the generated census report records the directory it read). A worked case of the tie-break in practice is the packing rule on Rectangle, Point and Color: the genuine framework rounds float-to-byte conversions to nearest with ties to even where CNA, following FNA, had truncated.

Tolerance then follows the phenomenon. Zero is right for a byte-exact decode or a stable reference image; one channel value of rounding may justify a local, written exception; 40 on a whole-frame dump admits different hardware pipelines rasterising the same edge; 60 on a golden means only a coarse visual guard. Naming that strength is part of the verdict: "passed the oracle" is incomplete until the reader knows which oracle, which policy, and whether it actually executed.

The captured-XNA image corpus, briefly

tools/xna-oracle holds 39 declarative scenes and 39 matching 256×256 reference PNGs captured once, by hand, from the real XNA 4.0 runtime under Wine with DXVK. The C# runner's source is checked in; its compiled executable, the XNA runtime and the private 32-bit Wine prefix are not, and are needed only to regenerate a reference, never to compare a current CNA render with the committed corpus. One renderer-agnostic CNA program is built into four renderer trees (DIRECTX9, EasyGL, FNA3D, Software), and xna-diff.py compares absolute per-channel differences with a default tolerance of 0; it has no PSNR, SSIM or perceptual metric. What each registered command actually fails on differs by renderer: the DIRECTX9 gate fails on any differing pixel of any of the 39 scenes, the FNA3D command only on a scene that fails to render, and the EasyGL and Software commands cover two line scenes, with only the EasyGL one failing on a pixel difference. The full provenance, per-renderer records, vacuous-match history and scene-authoring procedure are on Using the XNA oracle as evidence; the method is taught in Tutorial 161.

Where a relaxation belongs

The governing rule is stricter than "looks similar": the header of xna-diff.py warns that raising the tolerance without a documented, scene-specific reason is how an authenticity project quietly becomes a parity project. The tool therefore offers only local relaxations (separate RGB and alpha bounds, a budget of raw differing pixels, an allowed rectangle), and at this snapshot no runner passes any of them. A per-scene policy table that once sat beside the scenes served a renderer family that no longer exists. The lesson it carries still applies to any current renderer that needs one: keep the policy beside the scenes, make every bound explicit and spatially limited, and have the runner check that the policy's scene set equals the corpus's scene set before comparing, so a new scene cannot inherit a relaxation silently and a stale row cannot linger.

Project-owned goldens answer a weaker question

PixelTestGame.hpp is the harness of the standalone renderer programs. It offers four check methods, all recorded in one PASS/FAIL stream that becomes the process exit code (a further hook, MarkFailedEXT(), fails the run without printing anything):

• ExpectPixel(label, rect, colour, tolerance = 0) reads back exactly one pixel (its rectangle must be 1×1, because GetBackBufferData rejects any other element count) and compares R, G and B only: the harness follows the examples' convention of ignoring alpha unless a test checks it explicitly.
• ExpectTrue(label, condition) records a non-pixel precondition ("the texture this test draws was actually created"), so a test whose setup failed reports that instead of checking pixels that no draw produced.
• Check(ok, label) is the same generic boolean assertion with its arguments the other way round, kept for checks such as "this render must differ from that one".
• CompareGoldenImage(label, rect, path, tolerance = 0) compares a region with a committed PNG, again on RGB, and prints the largest channel difference it used.

examples/golden holds 17 PNGs of 8×8 pixels, all captured from EasyGL and named after it. Fourteen EasyGL sources compare against them, and thirteen Vulkan registrations (for example Vulkan_BasicEffect_Golden) compile the same EasyGL sources and compare against the same files, with the source root as working directory because the golden paths are repository-relative. The tolerances actually passed at this snapshot span the whole range:

Only the canary that clears to solid blue is exact; the depth-write and sprite-rotation goldens accept 60. At that end a test catches gross shape or state regressions, not shading drift. That does not make it useless; it makes its contract different from the real-XNA corpus: a self-captured golden is a regression and cross-renderer oracle for CNA's chosen behaviour, not evidence that the behaviour matches XNA.

Setting CNA_UPDATE_GOLDEN makes CompareGoldenImage write the live render as the new reference instead of comparing, prints [UPDATE] and reports success. The header comment says "any non-empty value", but the code only tests whether the variable is present in the environment, so a variable that is set but empty (exported by a wrapper script, say) triggers the rewrite too; a green run with [UPDATE] lines compared nothing. Updating is therefore a review operation, never a way to make a failure green: the change should carry the image diff, the renderer, the driver and the reason the new output is correct, because regenerating a shared image silently changes what every consumer of it proves.

Point samples and the skip boundary

Single-pixel checks reach much further than goldens: 225 ExpectPixel( call sites in 49 C++ files, and 119 sources include the harness (lexical counts at 009d40f5). They are cheap and precise for clear colours, blend equations, scissor order or depth outcomes. Before constructing its game, the harness probes SDL video and returns 77 when no display is usable, which CTest reports as skipped. A green headless run can therefore mean that none of these comparisons executed, so a pixel-oracle report states the display used and the skip count, not only the exit status. The procedure for writing one is in Renderer example programs versus CTest.

What the pixel harness itself guarantees

Several properties of PixelTestGame.hpp decide what a pass from one of these programs can mean. None of them is visible in the test's own source:

• One frame. Draw() calls RunTest() on the first frame only, behind a latch, and then calls Exit(). Nothing that needs a second frame, an Update() sequence or a device reset can be observed through the harness. Tests of that kind hand-roll their own Game subclass and call ProbeGpuDisplayAvailable() themselves, returning kSkipExitCode when it is false.
• One pixel per ExpectPixel. The check reads its rectangle with GetBackBufferData(&region, &pixel, 0, 1), and GraphicsDevice::GetBackBufferData throws ArgumentException when the element count does not match the rectangle's area. The rectangle must therefore be 1×1. A larger one does not mean "any pixel in the region"; it raises an exception that ends the program.
• HiDef, always. RunPixelTest<TGame>() sets GraphicsProfile::HiDef before Run(), because back-buffer readback throws under the default Reach profile. It sets the profile on the fixture's GraphicsDeviceManager when the fixture has registered one, and otherwise calls SetGraphicsProfileEXT on the eagerly created device. A fixture meant to observe a Reach-profile refusal cannot rely on the harness default.
• Narrow skips. Besides the display pre-flight, exactly one exception becomes exit 77: one whose message contains VK_ERROR_SURFACE_LOST_KHR. A Vulkan swap chain raises that error on an X server without DRI3, after the display probe has already succeeded. Every other exception is rethrown on purpose, so a genuine pipeline defect ends the process and CTest reports a failure, not a skip.
• No SDL, no pre-flight. In a build without SDL the examples define CNA_EXAMPLES_NO_SDL. The probe then answers "available" without checking anything, and the selected platform's own window creation acts as the probe. A missing display therefore surfaces as a failure rather than a skip, which the header states is deliberate.
• Non-pixel failures. ExpectTrue(label, condition) and MarkFailedEXT() put a precondition or an expected exception into the same verdict. A test that only prints [FAIL] still exits 0.

Readback problems in these programs can be traced with CNA_BACKBUFFER_READ_TRACE. It makes GetBackBufferData print the requested rectangle and the back-buffer and viewport sizes to standard error (see Debugging). Checked by reading the harness and GraphicsDevice.cpp at 009d40f5; not executed.

FNA as a numeric reference

A separate chain runs C# against a live FNA.dll under Mono (tools/fna-reference), dumps CNA's side with cna_reference_dump (CnaReferenceDump.cpp) and compares the two JSON documents with compare-fna-reference.py. Its default categories are non-rendering APIs, PackedVector and Viewport (enums, state presets, packed-vector values, viewport project and unproject). Numbers compare with an absolute tolerance of 1e-4 (--tolerance), everything else must match exactly, and the walk is one-directional: every key FNA emits must exist on the CNA side, while CNA-only keys, such as extension members, are not reported. It exits 0 or 1, and it is a manual developer workflow, not a CTest and not in CI.

Its history is a useful counter-example to "a second implementation only matters for pixels". The first comparison found three defects in the harness itself before anything real (JSON numbers written with six significant digits, state classes dumped without inherited properties, enum members missing on the C++ side), and then exactly one genuine divergence: IndexElementSize used the semantic values 16 and 32 where FNA uses the ordinals 0 and 1. That was fixed on 2026-07-09 (CNA's enum now declares SixteenBits = 0 and ThirtyTwoBits = 1), so re-running the comparison would no longer report it. The example shows a second implementation challenging assumptions embedded in the public value surface, and why an oracle harness must be debugged before it is trusted.

Independent oracles inside the corpus

Not every expected value comes from an external runtime. Five recurring patterns keep CNA's own tests from grading themselves:

• Golden tables. PackedVectorGolden.md derives packed-vector values in Python from FNA's bit-packing formulas (round by adding 0.5, clamp, IEEE half floats via struct.pack), and the Golden* cases of PackedVectorTests.cpp check them with narrow tolerances. The genuine-framework packing oracle then outranks that table where the two differ (above).
• Recomputation ladders. The glTF oracle helpers in GltfOracleEXTTests.cpp dump accessors (L2) and vertex streams (L3) and evaluate world positions (L4) independently of the production routine, against values stored in each fixture's manifest (tolerance 1e-5). The helpers carry their own acceptance tests: deterministic dumps, enough structure to tell an offset error from a stride error, the manifest values for the control fixture, and proof that using the oracle does not change what the production import produces.
• Shared oracle headers. ViewSpaceFogRef.hpp holds canonical transformed-camera fog scenes and their analytically derived colours from the FNA fog formula; the EasyGL, SDL_GPU and Vulkan fog tests and a renderer-neutral contract test all include it, so the copies cannot drift apart.
• Byte-exact differentials. LzxDecoderDifferentialTests.cpp compares CNA's LZX output byte for byte with reference files produced by running FNA's unmodified LzxDecoder.cs under Mono, checked in with instructions for regenerating them; a fuzz suite beside it covers malformed input.
• Cross-renderer controls with a whitelist. Two renderer-agnostic programs dump a 64×64 RGBA8 back buffer: cross_renderer_diagnostic_scene.cpp (a vertex-coloured BasicEffect triangle, built for EasyGL, Vulkan, Software and WebGPU) and cross_renderer_2d_corpus.cpp (a fixed SpriteBatch corpus, built for Direct2D, EasyGL and Vulkan). The 2D corpus deliberately excludes every construct where two conforming renderers may legitimately disagree, and names them: linear filtering of a magnified sprite (kernel and edge behaviour unspecified), mip selection and mip-linear blending (not a shared contract), anisotropic filtering, MSAA and additive blending (capability-dependent; Direct2D reports them unsupported), rotation by a non-right angle (edge coverage is implementation defined) and non-opaque destinations (presentation-time alpha handling differs). cna_diag_compare then diffs two dumps per channel and exits 1 when the largest difference exceeds the tolerance argument (its own default, when none is given, is 1). The callers choose the bound: the Direct2D-versus-EasyGL 2D corpus passes 4, which its differential document argues is all that 1-2 LSB of tint or blend rounding can legitimately spread, and compare-easygl-vulkan-diagnostic.sh defaults to 40, inherited from the older Software/EasyGL comparison. That 40 has never been exercised: the script's recorded first run (2026-09-06, llvmpipe on Xvfb) measured a largest difference of 1 on the triangle and a byte-identical 2D corpus, and passes at 2. Because a CNA build links one renderer by default, the two dumps come from two build directories, which is why compare-easygl-vulkan-diagnostic.sh is a script and not a CTest. Excluding unstable constructs is preferable to widening the tolerance until everything passes.

Engagement: proving the experiment happened

Some hosts return success while silently substituting a different runtime. Wine can fall back from DXVK to its own Direct3D, a Proton lane can resolve a moving release, a browser page can load without reaching a frame, and a test binary can be listed without ever opening a device. On such hosts the first oracle is not the game's output but proof that the intended execution route engaged. A robust harness therefore returns two verdicts, an engagement verdict naming the runtime, renderer, device or browser path, and a behaviour verdict from the functional oracle, and only their conjunction is a pass.

The Wine wrapper as a verdict pipeline

The Direct3D wrappers follow one sequence: validate the arguments, select an isolated prefix through a renderer-specific variable, set Wine's diagnostic variables, refuse a prefix with no system.reg (printing the wineboot --init and dxvk-setup install commands instead of bootstrapping or mutating anything), run wine through tee, keep Wine's own exit status from PIPESTATUS, and only then inspect the captured log. A DXVK run must contain a DXVK: <version> line and a vkd3d-proton run a vkd3d-proton - applicationVersion: line; without it the wrapper exits 3 whatever the program returned, because a passing executable through the wrong translation layer is a failed experiment. DIRECTX9 and DIRECTX11 share the physical prefix ~/.wine-cna-d3d11 through separate variables, and DIRECTX12 has its own. The table of wrappers, markers and opt-outs is on Direct3D evidence: authenticity gates, and a marker proves which translator served the run, not how a Windows driver would behave.

Discovery needs a controlled escape hatch

A cross-compiled CnaTests.exe cannot run natively on the Linux host, so UnitTests.cmake sets the target's CROSSCOMPILING_EMULATOR to the Wine wrapper of the configured Direct3D or Direct2D identity, and GoogleTest discovery then runs --gtest_list_tests through it. Listing names creates no graphics device, so no engagement token can legitimately appear, and the emulator command therefore carries CNA_D3D9_SKIP_DXVK_GATE=1, CNA_D3D11_SKIP_DXVK_GATE=1 or CNA_D3D12_SKIP_VKD3D_GATE=1. Writing the bypass into the generated command, rather than teaching every wrapper to accept missing evidence, keeps it reviewable. Its reach is wider than discovery, though: the emulator is a property of the target, so the same gate-free wrapper also runs every discovered CnaTests case and every named route whose command is CnaTests (the file's own comment names CnaInputTests), on the stated grounds that most unit tests create no device either. In those trees, translation-layer engagement is proved by the separately registered renderer programs such as DirectX9_Smoke, which keep the gate; only programs that never open a device, such as DirectX9_Common and DirectX9_ShaderDispatch, opt out, each with a comment saying why. A CnaTests case that does create a device in a cross tree is not, by itself, evidence that DXVK or vkd3d-proton served it.

Direct2D: prefixes, a virtual display and a pinned Proton

Direct2D shows how the same idea is adapted to a different substitution risk. Its tests exercise Wine's built-in Direct2D, Direct3D 11 and DXGI rather than DXVK, so its registrations set the DXVK-gate opt-out, and the Wine launcher run-wine-direct2d.sh defaults to the developer's ordinary $HOME/.wine prefix; its comment calls that the compatible default and states that the launcher performs no prefix mutation (CNA_DIRECT2D_WINEPREFIX selects a dedicated one). Because an accumulated prefix is not reproducible, run-direct2d-fresh-wine-suite.sh runs the whole Direct2D CTest label against a prefix created in a temporary directory with exactly one declared dependency, wineboot --init: no winetricks package, no native DLL override, no DXVK. Evidence meant to be reproduced should prefer that route and record the prefix identity.

Every Direct2D run goes through run-direct2d-virtual-display.sh unless CNA_DIRECT2D_USE_HOST_DISPLAY=1, because Direct2D always presents to a real window. It validates the requested geometry (default 1280x800x24, CNA_DIRECT2D_XVFB_SCREEN), keeps the X authority and server state in a mktemp directory rather than $HOME, installs a cleanup trap for exit and interruption, and is reentrant, so a CTest run already started inside its display is reused instead of nesting a second server.

The Proton lane, run-proton-direct2d.sh, pins its runtime. direct2d-proton-pin.txt lists allowed Steam runtimes in priority order (Proton 9.0 (Beta), then Proton 8.0), the first installed one is used, and a missing pin file refuses the run. "Proton - Experimental" is refused unless CNA_DIRECT2D_PROTON_ALLOW_EXPERIMENTAL=1 marks a one-off investigation, because Steam upgrades it underneath a passing run and such a result must never be quoted as evidence. The runner publishes the runtime identity (Proton build, Wine build, and the DXVK version read from the shipped DLL with strings) to standard error and to an identity file, can stop after that with a dry-run variable, and refuses to operate on ~/.wine.

The DIRECTX12 Proton lane: route evidence, weaker reproducibility

run-proton-vkd3d.sh exists because a swap chain cannot be created through system Wine with overridden vkd3d-proton DLLs (the cause and the resulting fixture split are on Direct3D evidence: translators). Unlike the Direct2D lane it defaults to the moving "Proton - Experimental" installation, overlays vkd3d-proton's d3d12.dll and d3d12core.dll onto a Proton-managed prefix, and ends with exec python3 proton run, so it forwards only the child's exit status and applies no log-token gate; its diagnostic program writes results to a file because Proton does not reliably return a child's output. Both Proton runners refuse the ordinary ~/.wine prefix after a recorded incident on 2026-07-14, in which a wine call without an explicit prefix half-upgraded a developer's personal prefix and left it hung. A DIRECTX12 Proton pass is useful route evidence but weaker reproducibility evidence than the pinned Direct2D lane: engagement proof has to match the host's actual substitution risk rather than be copied mechanically from one wrapper to the next.

Virtual displays are infrastructure, not evidence

A private Xvfb, the private Weston and Xwayland pair of run_gpu_tests_private.sh and the Direct2D display wrapper make headless presentation repeatable and keep tests off a developer's desktop. They do not prove physical display behaviour, vendor drivers, HDR or desktop composition, and a report must still say that a virtual display and a translation layer were used. Likewise a Windows executable returning zero under Wine does not become a native-Windows result. The environment belongs in the claim: compiler, executable format, Wine or Proton version, translation backend, GPU or software device, display server and prefix.

Browsers need an out-of-process verdict

A WebAssembly module's exit status is not observable from outside a browser, so the HTML_DOM and SVG_DOM test pages publish their verdict on window (__cnaSmokeDone, __cnaSmokeResult, __cnaSmokePassed, __cnaSmokeExpected for the HTML_DOM smoke page; page-specific flags elsewhere). htmldom-browser-test.mjs launches headless Chromium through Playwright, waits for the flag with a bounded timeout (60 s by default), treats a page exception or a [FAIL] console line as a failure, and exits 0 only when the published result is 0, no failure line appeared and its screenshot checks passed. The screenshot is taken from outside the page through the browser's own compositor, decoded into a canvas and sampled at chosen points with a bounded RGB tolerance, which deliberately crosses out of renderer-owned state and back through Chromium's real composition result.

Serving is part of the environment. run-htmldom-browser-test.sh starts python3 -m http.server bound to 127.0.0.1, polls it with curl until it answers, and kills it from a cleanup trap; it never uses file://, from which the module cannot be fetched and whose origin rules differ from a deployed page. run-htmldom-test-suite.sh gives each of its six pages (smoke, pixel, stress, dispose, host-integration, memory) its own port from 8731 to 8736, so that the pages of one suite run do not share a port. That does not make two concurrent invocations of the same suite safe: both start with the same port, the wrapper discards the server's output, never checks that its own server bound the port, and its curl readiness poll is satisfied by whatever already answers there (or times out after five seconds and carries on), so a stale or foreign server can supply the page. Run one invocation per runner, or set CNA_HTMLDOM_TEST_PORT yourself. The SVG_DOM driver (svgdom-browser-test.mjs) adds structural assertions before it trusts pixels: the root is a real element in the SVG namespace, the SDL canvas is hidden, and every sprite-bearing child is an SVG element, which proves that the SVG implementation, not a look-alike image from another renderer, supplied the frame. The automated route is htmldom-ci.yml; the browser renderers are introduced in Tutorial 105.

Hostility is any plausible false green

Wine fallback, a stale Proton selection, a headless skip, a browser that started without a frame, a unit binary run through a gate-free wrapper and one renderer impersonating another look unrelated. They share one failure mode: the program appears healthy while the intended experiment never occurred. The transferable pattern is not any particular grep token but a procedure: identify how the host can substitute or skip the thing under test, then make evidence of non-substitution a first-class artifact of the run, with its own positive and negative fixtures (as DirectX11_DxvkGate provides for the DXVK wrapper; see Can a green gate go red?).

Evidence and limits

Read at 009d40f5 from the oracle tools, PixelTestGame.hpp, the golden sources and the Vulkan and EasyGL example CMake files, the FNA and LZX harnesses, the diagnostic corpus sources, the Wine, Proton, display and browser scripts and UnitTests.cmake. Counts of call sites, includes and tolerances are lexical. No oracle, Wine, Proton or browser run was executed for this page. That discovered CnaTests cases in a cross-compiled tree inherit the target's emulator follows CMake's documented handling of CROSSCOMPILING_EMULATOR and the comment beside it in UnitTests.cmake; it was not observed. The FNA divergence and its fix are CNA's own record (docs/fna-reference-harness.md), checked against the current enum values.