CNA-VGAP-046: No workflow runs the XNA reference-image corpus comparisons or the FNA value-differential harness, and the one reachable image check runs in a workflow that installs no Pillow

CNA snapshot 009d40f5  ·  Known Issues › Verification gaps  ·  source links pinned to 009d40f5

D3D9_XNA_Diff, Fna3d_XNA_Oracle and Software_XnaLineCoverage are in no workflow's test selection, the one reachable image-oracle CTest runs in a workflow that installs no Pillow, and the FNA value-differential harness is manual because it needs mono and a built FNA.dll.

Identifier

CNA-VGAP-046

Category

Verification gap

Subsystem

Testing & evidence

Status

Open

Verified against

CNA 009d40f5 (009d40f5dd085c4e674d3479675fac84b12b3e0a)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

Yes: see Current tests

Affected contract

The XNA reference-image corpus CTests and the FNA value-differential harness (tools/fna-reference, cna_reference_dump, scripts/compare-fna-reference.py)

Expected behaviour

Reference-image comparisons that CNA cites as XNA-fidelity evidence are repeated automatically, and the FNA value comparison is run often enough that its recorded result is current.

Actual behaviour at TARGET

D3D9_XNA_Diff is registered only in a DIRECTX9 tree and Fna3d_XNA_Oracle only in an FNA3D tree; neither identity appears in any workflow. The platform, glTF and multi-renderer workflows that build SOFTWARE filter ctest to other tests, so Software_XnaLineCoverage never runs. EasyGL_XnaLineCoverage is reachable from the unfiltered run in general-tests-ci.yml, which installs no Pillow although xna-diff.py imports PIL (whether the runner image provides it is not recorded). The DIRECTX9 39-scene corpus record has not been repeated on native Windows. cna_reference_dump in Harnesses.cmake has no add_test by design (the comparison needs mono and a locally built FNA.dll), so the FNA value comparison is a manual developer step. The committed XNA value oracles for math, PackedVector and the content pipeline are not affected: they run as gtests inside CnaTests in CI. compare in compare-fna-reference.py walks the FNA keys only, which its docstring records as intentional because CNA has CNAEXT-only values; the consequence is that an unmarked extra value passes unnoticed.

Source locations

• modules/renderers/directx9/examples/CMakeLists.txt — D3D9_XNA_Diff registration
• modules/renderers/fna3d/examples/CMakeLists.txt — Fna3d_XNA_Oracle registration
• scripts/xna-diff.py — PIL import
• cmake/Harnesses.cmake — cna_reference_dump target without a test
• scripts/compare-fna-reference.py — compare: iterates the FNA side only

Evidence

Checked by reading at 009d40f5; not executed.

Workflow selection checked by reading every ctest invocation under .github/workflows.

Independent re-verification: Checked by reading at 009d40f5; not executed. Workflow selection checked by reading every ctest and test-binary invocation under .github/workflows.

Focused reproduction

No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.

Current tests

The oracle CTests exist and run locally; none runs in CI except, possibly, the EasyGL line pair.

Regression test

A CI job per oracle-capable identity that runs its corpus CTest with Pillow installed, and a two-way key comparison in compare-fna-reference.py.

Blast radius

XNA-fidelity claims backed by reference images or FNA values are as current as the last local run.

Workaround

No workaround is known.