Using the XNA oracle as evidence

CNA snapshot 009d40f5  ·  Development › Human Takeover  ·  source links pinned to 009d40f5

CNA keeps a set of scenes and reference images captured from the genuine Microsoft XNA 4.0 runtime, and a small toolchain that renders the same scenes through CNA and diffs the pictures. It is the main place in the repository where rendering behaviour is measured on real XNA rather than reasoned (six one-question probes and a content-pipeline oracle sit beside it), which makes it the strongest evidence a maintainer can cite for a pixel-level claim and one of the easiest to over-read. This page explains what the oracle is, how a comparison is run and judged, what CNA's own records claim for each renderer, when to reach for it, how to add a scene, and where its evidence stops. The user-level tour is Verification: the oracle corpus and Tutorial 161.

What the oracle is

Three neighbours are easy to confuse with it. tools/xna-pipeline-oracle/ runs the genuine XNA Game Studio 4.0 Content Pipeline assemblies under Wine to record the pipeline's API and behaviour (its committed inventory lists 128 public types, 10 importers and 12 processors; the differential file holds 152 cases): it is about the build-time content tool, not rendering (content pipeline internals). The six spikes/xna-*-spike directories are one-question probes of real XNA. tools/fna-reference/ runs real FNA, not XNA, and feeds value comparisons and compiled-effect fixtures.

Where the reference images came from

From the real Microsoft XNA 4.0 runtime, executed under Wine with DXVK on Linux, not on Windows. The oracle README (tools/xna-oracle/README.md) requires DXVK to be installed into the same prefix as XNA: otherwise real XNA would run on WineD3D while CNA's DIRECTX9 path runs on DXVK, “and any diff would silently measure a driver difference”. CNA's records name the host: an AMD Radeon 780M through RADV, with D3DCAPS9 synthesized by DXVK (docs/d3d9-divergence-report.md, the caveat it says every result inherits). The header of the 2026-09-21 format table records Wine 10.0, DXVK 2.6.0 and RADV 25.0.7. The 39 corpus images were captured in July 2026, and no record of the Wine or DXVK versions used then was found in the README, plans or reports (hedged: absence from those files only). scripts/run-wine-dxvk9.sh asserts that a “DXVK:” log line appeared, so a silent fall-back to WineD3D fails the DIRECTX9 run instead of producing a misleading green.

The images were captured once, by hand, and are committed. scripts/run-oracle-corpus-diff.sh says never to edit a reference PNG any other way and never to widen its tolerance to turn a red comparison green. Regenerating a reference needs the real XNA runtime in a Wine prefix (or on Windows); nothing in CNA's own build produces one.

The corpus and its denominators

Keep three quantities apart: what exists, what a given command iterates (the denominator) and what was actually compared.

How a comparison is run and judged

scenes/NAME.scene --+--> Oracle.exe (real XNA, Wine + DXVK) --> reference/NAME.png   (committed once)
                    |
                    +--> cna_oracle_render_<renderer>  --> NAME.png (this run)
                                      |
        scripts/xna-diff.py reference/NAME.png NAME.png  -->  PASS | FAIL, exit 0 | 1

The recipe is in Tutorial 161 (display-free with SOFTWARE) and the README's build sections; the commands were read, not executed. The judge is scripts/xna-diff.py. It needs Pillow, opens both files as RGBA, fails on a size mismatch, and counts a pixel as differing when its RGB delta or its alpha delta exceeds the tolerance. --tolerance defaults to 0, so one channel value off in one pixel of 65,536 is a failure. It prints one summary line (PASS: 0/65536 pixels differ beyond tolerance=0, max per-channel delta=0), returns 0 or 1, and with --diff-out paints differing pixels red. Optional flags exist for separate RGB and alpha tolerances, a raw-difference budget and an allowed rectangle, but no script at this snapshot passes them. Its header warns that raising the tolerance without a documented per-scene reason is “exactly how an authenticity project quietly becomes a parity project”, and the oracle README records a mutation check: a copy off by one value fails at tolerance 0 and passes at 1.

python3 scripts/xna-diff.py <reference.png> <cna.png> [--diff-out diff.png] [--tolerance N]
scripts/run-oracle-corpus-diff-software.sh <cna_oracle_render_software> [tolerance] [scene ...]
scripts/run-oracle-corpus-diff-easygl.sh   <cna_oracle_render_easygl> [scene ...]
scripts/run-oracle-corpus-diff-fna3d.sh    <cna_oracle_render_fna3d> [tolerance]
scripts/run-oracle-corpus-multi.sh         <cna_oracle_render_*> "RENDERER;RENDERER"

An exact match can still be vacuous, and the oracle README records five ways it happened. (1) The first fog scene rendered uniformly white on both sides, an exact 0/65,536 that proved nothing, because fog was never applied. (2) The first environment-map scene claimed the non-Fresnel bucket but had run the Fresnel one, because its geometry made both identical. (3) The first single-bone skinning scene ran the four-bone bucket. (4) A 1×1 sprite texture stayed exact with the half-pixel offset removed, while the four-colour rotated and flipped scenes diverged by 4,800 pixels. (5) XNA's SaveAsPng wrote RGB 0 for an alpha-0 pixel where CNA's writer kept the raw RGB, so the images differed although the shader output was identical: the comparison is between encoded PNGs, not framebuffers. A scene is evidence only for what it discriminates.

Reading a red scene. Open the diff image: a few red pixels along a triangle edge are a different finding from a red rectangle. Fill rules and pixel-centre conventions legitimately differ at primitive edges, and a point-sampled texel boundary is a tie-break. That is why scripts/run-oracle-corpus-diff-fna3d.sh says the host's GL stack carries a corpus-wide divergence that is not renderer-specific: judge a scene against the recorded baseline (EasyGL's result on the same machine, for FNA3D). A scene where the renderer is worse than that baseline is a candidate defect; a renderer at the baseline is not necessarily broken.

What CNA's records claim, and what each gate enforces

Figures below are CNA's records, dated where CNA dates them, and were not re-run. The right-hand column is read from the runner scripts: it says which condition makes the registered command exit non-zero.

No GitHub workflow runs any of these (searched the workflow files for the scripts and CTest names). general-tests-ci.yml does not install Pillow, which the diff tool imports, so whether the EasyGL test runs there was not verified. CNA explains why the whole corpus is not a portable gate: the SOFTWARE script's header calls it “a measurement, not an exact-image gate”, and the CMake comment beside the registration says hardware floating-point and edge precision make universal exact identity “an invalid portable gate”. That also bounds what DIRECTX9's byte-exact record says about any other machine. Read the dated reports at their date: the 31-scene DIRECTX9 report lists six project-wide CNA-versus-XNA divergences, and NEXT.md records the per-pixel-lighting and specular one resolved on 2026-07-16, after the report.

When to reach for the oracle

Reach for it when a change alters public XNA behaviour that reaches pixels (stock effects, SpriteBatch, blend, depth and rasterizer state, texture formats): run the corpus for the affected renderer and compare per scene with the last recorded baseline rather than with a headline number. Reach for it also when two CNA contracts disagree about what XNA does, because neither side has measured anything. CLAUDE.md records the owner's 2026-09-04 rule that measured XNA behaviour, read from the shipped assemblies' IL, outranks FNA where they disagree; a real-XNA image is the pixel-level counterpart of that evidence. Four cases from CNA's records:

• Pixel centre. spikes/xna-pixel-center-spike/README.md: EasyGL_XnaPixelCenter demanded one convention, point-sampling legs that held seven renderers to it (at the time of the probe) demanded the other, and each was defended by a passing test. The probe found XNA uses integer pixel centres for a 3D quad (100/100) and half-integer centres for SpriteBatch; the 3D expectations were wrong and EasyGL was right. One residue is recorded as unexplained: at 8×4 to 21×13 XNA's sprites matched the half-integer rule for 256 of 273 pixels while CNA matched 273 of 273.
• Channel expansion. plans/plan_vulkan_parity.md VKPAR-0014: a renderer-neutral test (reasoned from documentation and FNA, “not measured”) and a Vulkan test pinned blue = 0 for a two-channel format, and neither had measured anything. tools/xna-oracle/FormatExpansionOracle.cs settled it: a missing colour channel samples as 1.0, a missing alpha as 1.0, with Alpha8 the exception. The neutral test was right and the Vulkan pin was wrong; the renderer now maps that channel with a view swizzle (ClassicSampledSwizzleEXT in VulkanRenderer.cpp).
• Unbound textures. GSC-0004 (plans/plan_graphics_shared_cleanup.md) measured that every stock effect reads an unbound texture as opaque black; a white fallback and a null view both fail the resulting unit tests.
• A bug the earlier scenes could not see. The sprite-sort scenes were the first to use layerDepth above zero, and exposed a DIRECTX9 projection that clipped every such sprite; the README records the fix and a mutation check that turned the scene red.

Do not reach for it for behaviour with no pixel (audio, input, storage, content loading), for questions the scene format cannot express (every scene has identity matrices; a non-identity perspective is untested), for comparing two CNA renderers with each other (the parity fixtures do that, against their own assertions), or for behaviour CNA's records found unobservable: SpriteSortMode.Immediate is not pixel-observable, SpriteSortMode.Texture has no deterministic expected image (per the README, FNA's texture sort uses an implementation-defined identity hash), and an attempt to observe DepthBias at magnitudes from 1.0 to ±1e8 produced no pixel change (D9-62, where CNA suspects a host limitation rather than a CNA defect).

How to add or update a scene

1. Decide the discriminator first. Predict the expected pixels by hand, add a positive control, and check that the scene would fail if the behaviour under test were absent (see the vacuous cases above). Copy a scene such as tools/xna-oracle/scenes/colored3d.scene and cite the task in its header comment.
2. Implement any new key on both sides in one change. An unknown key is an error in tools/xna-oracle/Oracle.cs and in tools/xna-oracle/CnaOracleRender.cpp; the XNA side must stay 4.0-era C#.
3. Capture the reference by hand. Build and run Oracle.exe in the XNA prefix with DXVK installed, per the README, and commit the PNG. This step needs the XNA prefix; without it you can render CNA's side but cannot add or change a scene.
4. Remember the glob. A new top-level scene joins every script's denominator, including D3D9_XNA_Diff, and a scene without a reference is a failure or a render-failure row. Put measurement-only scenes in a subdirectory, as null-texture/ does, and say so in the header.
5. Run it on DIRECTX9 if you can, and mutation-test. Break the code path on purpose and confirm the scene goes red, as the README does for sprite sorting and multi-texture batching.
6. Update every stated count. The README status, plan rows, CMake comments (the EasyGL example still says 31) and the pages of this site carry scene counts; the maintenance and pin policy says how they are re-pinned.

Limits of the evidence

• Representation is not behaviour. 331 of 331 XNA types and 3,627 of 3,627 documented members are represented (tools/audit_xna_runtime_surface.py); the oracle compares 39 scenes. A member existing says nothing about what it draws.
• Corpus size is not the denominator. 46 scenes and a 17-row table exist; 39 scenes enter a whole-corpus run, two enter each of the EasyGL and SOFTWARE line-coverage tests, and nothing enters CI.
• Host and driver. Every image and every DIRECTX9 result is Wine plus DXVK on one GPU; D3DCAPS9 is synthesized. Native Windows and real drivers are untested: D9-140 in plans/plan_dx9.md is open and marked needs_human, and its own text calls the oracle re-run on real Windows “the definitive result”.
• Tolerance. Zero wherever a script enforces exactness. A non-zero delta on another renderer calls for a per-scene reading and a written reason, not a wider tolerance.
• PNG, not framebuffer. Both images pass through an encoder, and the CNA side reads the back buffer while the XNA side reads a render target.
• Not a CI gate. The one exact whole-corpus check exists only in a DIRECTX9 cross-build tree that needs Wine and DXVK.
• Stale text. The README's closing paragraph still says remaining gaps need new CNA work although the specular and per-pixel-lighting scenes exist; read counts from the scene directory.

How to word a claim

A supportable sentence: “CNA's records report that DIRECTX9, through Wine and DXVK on an AMD Radeon 780M, matched all 39 scenes at tolerance 0; the last dated report covers 31; not repeated on native Windows; not in CI.” Unsupportable ones: “CNA is pixel-perfect against XNA”, “the corpus is a CI gate”, “a CTest name proves exactness” without reading the runner's exit conditions, or a per-renderer figure with no date, host and tolerance. Related reading: how to read CNA's plans and guards without trusting them, graphics evidence and known uncertainty.