Tutorial 161: Diffing a Renderer Against the XNA Oracle

CNA Tutorials  ·  CNA snapshot 009d40f5

Before you start — read Tutorial 125 for what the oracle corpus is, and Tutorial 160 for cloning the right branches and choosing a display for GPU tests. You need a CNA checkout on branch next (with sharp-runtime on its next branch beside it), Python 3 with Pillow (pip install pillow), and a Linux machine.

1. What you are about to do — and its limits

The XNA oracle corpus lives in tools/xna-oracle/. Each of its 39 scenes is a small text file describing one draw (256×256, HiDef, one effect, a handful of vertices or sprites). For every scene there is a reference PNG produced by the genuine Microsoft XNA 4.0 runtime, executed under Wine with DXVK on Linux. You render the same scene through a CNA renderer and diff the two PNGs pixel for pixel.

This tutorial uses the SOFTWARE renderer because it needs no window, GPU or display, so you cannot accidentally open windows on your desktop. The same recipe applies to EasyGL and FNA3D with their own targets (step 6).

2. Build the oracle renderer for your renderer

The scene interpreter for CNA’s side is one program, tools/xna-oracle/CnaOracleRender.cpp. It uses only CNA’s public Game, GraphicsDeviceManager, GraphicsDevice, stock-effect and SpriteBatch API, and is built once per renderer under a renderer-specific target name:

cd cna
cmake -S . -B build-software -DCNA_GRAPHICS_RENDERER=SOFTWARE
cmake --build build-software --target cna_oracle_render_software

The reference PNGs are committed in tools/xna-oracle/reference/, so you do not need XNA, Wine or a Windows machine to run CNA’s side of the comparison.

3. Read one scene

A scene is a line-oriented key=value text file; # lines are comments, and an unknown key is a hard error on both the XNA side and CNA’s side, so a typo cannot quietly change what “a match” means. This is tools/xna-oracle/scenes/colored3d.scene in full:

width=256
height=256
profile=HiDef
clearcolor=100,149,237,255
vertexcolor=true
lighting=false
primitive=TriangleList
vertex=-0.6,-0.6,0,255,0,0,255
vertex=0.0,0.7,0,0,255,0,255
vertex=0.6,-0.6,0,0,0,255,255

That is a BasicEffect with vertex colours and no lighting, one triangle, over a cornflower-blue clear, with World, View and Projection all identity. The scene format supports the five stock effects, fog, the four primitive types, all eight alpha-test comparisons and a SpriteBatch mode; the key reference is in tools/xna-oracle/README.md.

4. Render one scene and diff it

mkdir -p oracle-out

# CNA's side: scene in, PNG out (SDL_VIDEODRIVER=dummy mirrors what the corpus script sets for SOFTWARE)
SDL_VIDEODRIVER=dummy ./build-software/cna_oracle_render_software \
    tools/xna-oracle/scenes/colored3d.scene oracle-out/colored3d.png

# the comparison: reference first, CNA's image second
python3 scripts/xna-diff.py tools/xna-oracle/reference/colored3d.png oracle-out/colored3d.png \
    --diff-out oracle-out/colored3d-diff.png
echo "exit status $?"

cna_oracle_render_* takes exactly two arguments, <scene-file> <output-png>. xna-diff.py compares every RGBA channel of every pixel and prints one summary line, in one of these shapes (from the script’s source):

PASS: 0/65536 pixels differ beyond tolerance=0, max per-channel delta=0
FAIL: 37/65536 pixels differ beyond tolerance=0, max per-channel delta=64
FAIL: size mismatch -- xna=(256, 256) cna=(...)

(The numbers above illustrate the format; they are not results.) The exit status is 0 for PASS and 1 for FAIL. With --diff-out the tool writes a black image with every differing pixel painted red, and prints diff image written to .... Open it: a few red pixels along a triangle edge are a very different finding from a red rectangle.

5. Run the two-scene test CNA registers, then the whole-corpus measurement

CNA registers one small test for the software renderer: CTest Software_XnaLineCoverage runs the corpus script at tolerance 0 on the two line scenes colored_linelist_quad and colored_linestrip_quad, and nothing else. Read what it enforces before calling it an exact gate: the script (below) exits 1 only if a scene fails to render or has no reference PNG, so a pixel difference in either scene is printed as DIFF but does not fail the test. CNA’s registration comment calls it an exact regression and its records list both scenes as byte-exact (2026-09-11); the exit-status behaviour is read from the script, not executed. Add --verbose to ctest to see the MATCH/DIFF rows.

ctest --test-dir build-software -R Software_XnaLineCoverage --output-on-failure

The whole-corpus script for SOFTWARE is a measurement, not an exact-image gate (its own header says so): point-sampling decisions on an interpolated texel boundary and floating-point raster arithmetic can legitimately differ within Direct3D’s precision allowances. A scene that fails to render is nevertheless a hard failure.

scripts/run-oracle-corpus-diff-software.sh ./build-software/cna_oracle_render_software        # tolerance 0
scripts/run-oracle-corpus-diff-software.sh ./build-software/cna_oracle_render_software 1      # within one channel value
scripts/run-oracle-corpus-diff-software.sh ./build-software/cna_oracle_render_software 0 sprite_wrap_quad textured_quad

The usage is <renderer-exe> [tolerance] [scene ...]. It prints a two-column table (SCENE, RESULT) where a row is MATCH, DIFF followed by the diff tool’s line, RENDER-FAILED or NO-REFERENCE, then a summary line of the form === SOFTWARE vs real XNA 4.0 (tolerance=T): M/N matching, D differing, R not rendered ===. It exits 1 only if some scene did not render. It unsets DISPLAY and WAYLAND_DISPLAY and forces SDL_VIDEODRIVER=dummy.

For orientation only, CNA’s own plan records this measurement on 2026-09-11 as 18 of 39 byte-exact and 30 of 39 within one channel value, the same nine point-texture-boundary scenes outside 1 (CNA-recorded; your host may differ, and the site did not re-run it). The script globs scenes/*.scene non-recursively, so the seven unbound-texture scenes in scenes/null-texture/ and the 17-format channel-expansion table are not part of any run: they exist as recorded measurements that unit tests hard-code.

6. Other renderers and several at once

• EasyGL: scripts/run-oracle-corpus-diff-easygl.sh <cna_oracle_render_easygl> [scene ...] — tolerance 0, any difference exits 1. It needs a real window: it sets SDL_VIDEODRIVER=x11 and, if DISPLAY is not already exported, defaults it to :0, your live desktop. Export a virtual display first, or run it under tools/platform/run_gpu_tests_private.sh --exec ... as in Tutorial 160. CNA’s EasyGL exact gate is EasyGL_XnaLineCoverage (the same two line scenes).
• FNA3D: scripts/run-oracle-corpus-diff-fna3d.sh <cna_oracle_render_fna3d> [tolerance] (display default :99). CNA registers it as Fna3d_XNA_Oracle, but the script gates only on a scene failing to render; pixel differences are reported, not failed.
• Several renderers from one binary: scripts/run-oracle-corpus-multi.sh <cna_oracle_render_*> "RENDERER;RENDERER" renders the corpus once per named renderer (chosen per run through CNA_GRAPHICS_RENDERER) and diffs each against the same references at tolerance 0; a renderer not compiled into the binary is reported as skipped. CNA’s plan records verifying the script on OPENGLES3;OPENGL33 (2026-08-15): the OPENGLES3 leg reproduced the single-renderer EasyGL script at 9 passed and 30 failed of 39, and the plan gives no OPENGL33 total.

Dated whole-corpus measurements CNA has recorded, for scale: EasyGL 10 of 39 and FNA3D 10 of 39 on 2026-08-11 (Mesa llvmpipe), before later EasyGL pixel-centre and clip-depth fixes and not re-measured since (the 9-of-39 OPENGLES3 leg above is the only later count, and it also predates them), so treat them as possibly understated. The FNA3D script’s own header says the host’s GL stack “carries a corpus-wide divergence that is NOT renderer-specific”: a renderer near that baseline is at the established floor, not necessarily broken, and a scene where it is worse than EasyGL is the candidate defect.

7. Reading a failing scene

• Edges first. Look at diff.png. Rasterisation fill rules and pixel-centre conventions legitimately differ at primitive edges; CNA’s parity fixtures avoid sampling edges for exactly that reason.
• Texel boundaries. Point sampling exactly on an interpolated texel boundary is a tie-break; the real runtime and your renderer can pick different texels.
• The reference machine matters. The references came from real XNA running on DXVK (Direct3D 9 over Vulkan) on Linux. A different driver stack differs for reasons that are not CNA’s. That is also why DIRECTX9’s exact match is a statement about that stack, not about native Windows.
• Do not widen the tolerance to get green. If a difference is understood and permanent, the honest form is a per-scene, written-down policy, as CNA’s parity corpus script does — not a global number.

8. Where the references came from, and what you cannot do without them

tools/xna-oracle/Oracle.cs is compiled by the real csc.exe inside a 32-bit Wine prefix (~/.wine-cna-xna40) against the genuine Microsoft.Xna.Framework*.dll in the GAC, then run as wine Oracle.exe <scene> <out.png>. DXVK must be installed into that same prefix: otherwise real XNA runs on WineD3D while CNA’s Direct3D 9 path runs on DXVK, and the diff would silently measure a driver difference. That compiler targets .NET Framework 4.0-era C#, so Oracle.cs uses no modern syntax. The project’s reference machine was an AMD Radeon 780M through RADV. The exact commands are in the harness README; the site authors did not run them.

Consequences worth knowing:

• Adding or changing a scene means regenerating its reference PNG by hand against the real XNA runtime, and re-committing it. Never edit a reference in any other way. Most contributors cannot do this, so scene work belongs to someone with the reference machine.
• A second, smaller program, FormatExpansionOracle.cs, measured what real XNA returns for the channels a SurfaceFormat does not store (17 formats). Its result, reference/format-expansion/xna-format-expansion.txt, is a text table, not a scene.
• Six one-question probes under spikes/xna-*-spike/ (pixel-centre convention, colour clamps, multisample antialiasing and others) were measured the same way.

9. What a passing diff proves

• Proves: for that scene, on your machine, your renderer produced exactly the pixels the genuine XNA runtime produced on the reference machine.
• Does not prove: anything about scenes outside the corpus, other renderers, native Windows hardware, audio, input, content loading or API behaviour that never reaches a pixel. The 39 scenes are austere on purpose, so that a failure points at one behaviour.
• Is not automatic: no CNA workflow runs a whole-corpus oracle script. The whole-corpus exact gate, D3D9_XNA_Diff, exists only in a DIRECTX9 cross-build tree that needs Wine and DXVK.

Where to go next

• Tutorial 160: Running Focused Tests, Bounded Runs and Parity Fixtures — the renderer-neutral fixtures whose oracle is their own assertions.
• Tutorial 125: Headless and Pixel Testing, and the XNA Oracle Corpus
• Verification & Known Issues: the XNA 4.0 oracle corpus — the per-renderer table and its qualifiers.