CNA-VGAP-046: No workflow runs the XNA reference-image corpus comparisons other than the EasyGL line pair, or the FNA value-differential harness

CNA snapshot c1c316b9  ·  Known Issues › Verification gaps  ·  source links pinned to c1c316b9

D3D9_XNA_Diff, Fna3d_XNA_Oracle and Software_XnaLineCoverage are in no workflow's test selection, and the FNA value-differential harness is manual because it needs mono and a built FNA.dll.

Identifier

CNA-VGAP-046

Category

Verification gap

Subsystem

Testing & evidence

Status

Narrowed (partially fixed; describes only what survives)

Verified against

CNA c1c316b9 (c1c316b9c7a846ce8002809c151fcd1af14942c9)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

Yes: see Current tests

Affected contract

The XNA reference-image corpus CTests and the FNA value-differential harness (tools/fna-reference, cna_reference_dump, scripts/compare-fna-reference.py)

Expected behaviour

Reference-image comparisons that CNA cites as XNA-fidelity evidence are repeated automatically, and the FNA value comparison is run often enough that its recorded result is current.

Actual behaviour at TARGET

D3D9_XNA_Diff is registered only in a DIRECTX9 tree and Fna3d_XNA_Oracle only in an FNA3D tree; neither identity appears in any workflow. The platform, glTF and multi-renderer workflows that build SOFTWARE filter ctest to other tests, so Software_XnaLineCoverage never runs. EasyGL_XnaLineCoverage is reachable from the unfiltered run in general-tests-ci.yml, the only image-oracle CTest any workflow runs. The DIRECTX9 39-scene corpus record has not been repeated on native Windows. cna_reference_dump in Harnesses.cmake has no add_test by design (the comparison needs mono and a locally built FNA.dll), so the FNA value comparison is a manual developer step. The committed XNA value oracles for math, PackedVector and the content pipeline are not affected: they run as gtests inside CnaTests in CI. compare in compare-fna-reference.py walks the FNA keys only, which its docstring records as intentional because CNA has CNAEXT-only values; the consequence is that an unmarked extra value passes unnoticed.

Fixed since first recorded: general-tests-ci.yml now installs python3-pil, the Pillow that xna-diff.py imports, so that line pair can run there (AM4-272).

Source locations

• modules/renderers/directx9/examples/CMakeLists.txt — D3D9_XNA_Diff registration
• modules/renderers/fna3d/examples/CMakeLists.txt — Fna3d_XNA_Oracle registration
• scripts/xna-diff.py — PIL import
• cmake/Harnesses.cmake — cna_reference_dump target without a test
• scripts/compare-fna-reference.py — compare: iterates the FNA side only
• .github/workflows/general-tests-ci.yml — installs python3-pil for the EasyGL line-coverage image check (the fixed half)

Evidence

Checked by reading at c1c316b9; not executed.

Workflow selection checked by reading every ctest invocation under .github/workflows.

Independent re-verification: Checked by reading at c1c316b9; not executed. Workflow selection checked by reading every ctest and test-binary invocation under .github/workflows.

Re-verified at c1c316b9: b9b6fcbff (AM4-272) added python3-pil to general-tests-ci.yml; every other workflow's ctest selection still leaves out the D3D9, FNA3D and SOFTWARE oracle CTests. Not executed, and no workflow run was inspected.

Focused reproduction

No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.

Current tests

The oracle CTests exist and run locally; none runs in CI except the EasyGL line pair.

Regression test

A CI job per oracle-capable identity that runs its corpus CTest with Pillow installed (as general-tests-ci.yml now does for the EasyGL pair), and a two-way key comparison in compare-fna-reference.py.

Blast radius

XNA-fidelity claims backed by reference images or FNA values are as current as the last local run.

Workaround

No workaround is known.