CNA-VGAP-019: ModuleLinkClosure_* link-closure gates report CTest SKIP in every Ninja tree, so no Ninja-preset run and no CI job ever evaluates them

CNA snapshot 009d40f5  ·  Known Issues › Verification gaps  ·  source links pinned to 009d40f5

check_module_link_closure.py reads the make-only CMakeFiles/<target>.dir/link.txt and exits 77 when it is missing; the Ninja presets and every CI job either use Ninja or never run these tests, so CNA's module dependency contracts are not checked automatically.

Identifier

CNA-VGAP-019

Category

Verification gap

Subsystem

Testing & evidence

Status

Open

Verified against

CNA 009d40f5 (009d40f5dd085c4e674d3479675fac84b12b3e0a)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

Yes: see Current tests

Affected contract

The ModuleLinkClosure_* CTests registered by cmake/Tests/ModuleProbes.cmake (cna_add_module_probe and the CnaExtComposition, NetHasENet, NativeSdkFree_*, GraphicsNativeSdkFree and VulkanRendererClosure probes)

Expected behaviour

ModuleProbes.cmake describes each probe and its ModuleLinkClosure_* partner as "turning every module's real dependency closure into a permanent contract", and check_module_link_closure.py as "a permanent, mechanically checked contract": for example probe_math may link nothing but the math archive and Sharp Runtime, and the HEADLESS tree must show that renderer-neutral graphics needs no native graphics SDK. A contract of that kind is evidence only if some routine run evaluates it.

Actual behaviour at TARGET

The script reads <build>/CMakeFiles/<target>.dir/link.txt, which only the Makefiles generators write, and returns 77 ("SKIP: ... not found (non-Makefiles generator?)") when it is missing; ModuleProbes.cmake sets SKIP_RETURN_CODE 77 on every ModuleLinkClosure_ test. In CMakePresets.json, base-ninja (inherited by dev, unit, release-modules, multi-renderer and cnaext) and tests select Ninja; only the devices-*, macos, ios and web presets leave the default generator. In CI every workflow that configures CNA on Linux or Windows passes -G Ninja; the three jobs that keep the default generator (devices-tests.yml, apple-ci.yml, metal-macos-ci.yml) run CnaTests directly or filter ctest to ^Metal. general-tests-ci.yml, the only unfiltered ctest run, therefore reports every closure gate as skipped. The gates are sound; nothing automated exercises them.

Source locations

• scripts/check_module_link_closure.py — main - link.txt lookup and the exit-77 skip
• cmake/Tests/ModuleProbes.cmake — cna_add_module_probe and the SKIP_RETURN_CODE 77 applied to every ModuleLinkClosure_ test
• CMakePresets.json — base-ninja and tests presets (Ninja)
• .github/workflows/general-tests-ci.yml — -G Ninja configure and the unfiltered ctest run

Evidence

Checked by reading at 009d40f5; not executed. That a Ninja tree has no CMakeFiles/<target>.dir/link.txt is CMake generator behaviour (Ninja keeps link commands in build.ninja). All 20 files under .github/workflows were read for their generator and test selection. Since this was first reported the closure fleet grew (NativeSdkFree_*, GraphicsNativeSdkFree, VulkanRendererClosure), but every member uses the same script. WaylandLinkClosure.cmake, which inspects the built binaries with readelf, is generator-independent and not part of this gap.

Independently observed as a separate finding (merged): check_module_link_closure.py needs the Makefiles-only link.txt and exits 77 (SKIP) without it; the only unfiltered ctest in CI is a Ninja OPENGLES3 tree, which also registers neither the HEADLESS native-SDK-free nor the VULKAN closure gate.

Focused reproduction

Illustrative; not run for this entry.

cmake --preset unit && cmake --build cmake-build-unit
ctest --test-dir cmake-build-unit -R '^ModuleLinkClosure_' -V
# each entry prints: SKIP: .../CMakeFiles/<probe>.dir/link.txt not found (non-Makefiles generator?)

Current tests

The gates are the tests (ModuleLinkClosure_probe_math, _probe_storage and the rest). They do run in any Makefiles tree where someone runs ctest by hand, for example a devices-asan preset tree.

Regression test

Make the check generator-independent - read the probe's link command from build.ninja when link.txt is absent, or take the link fragments from CMake's file-API codemodel, which records them for every generator - and keep exit 77 only for a generator with neither. general-tests-ci.yml would then report each gate as passed or failed.

Blast radius

Evidence only: a new forbidden link edge (a module pulling in SDL, a renderer, networking or FFmpeg) would not be caught by any automated run. No shipped behaviour is affected.

Workaround

Evaluate the gates in a tree configured with -G "Unix Makefiles" and CNA_BUILD_TESTS=ON, then ctest -R '^ModuleLinkClosure_'.