CNA-VGAP-016: A plain ctest run of the tests preset is not a trustworthy result

CNA snapshot c1c316b9  ·  Known Issues › Verification gaps  ·  source links pinned to c1c316b9

The tests preset builds only CnaTests, yet ctest also runs the display-dependent smoke executables the same configuration registers and reports the unbuilt ones as failed, so CNA's own preset calls a direct CnaTests run the only authoritative result.

Identifier

CNA-VGAP-016

Category

Verification gap

Subsystem

Testing & evidence

Status

Narrowed (partially fixed; describes only what survives)

Verified against

CNA c1c316b9 (c1c316b9c7a846ce8002809c151fcd1af14942c9)

Evidence basis

Recorded by CNA: CNA's own recorded run, not repeated here

Tests touching this area

Yes: see Current tests

Affected contract

ctest over a build of the tests preset (CMakePresets.json)

Expected behaviour

Every registered test either builds with the target the preset builds or is skipped, and test cases are isolated enough to run as separate processes in parallel.

Actual behaviour at TARGET

The tests preset builds only CnaTests (its build preset names that one target), yet ctest also runs the many separately registered display-dependent graphics smoke executables that the same configuration registers (CNA_BUILD_EXAMPLES is on, the default renderer is OPENGLES3) and reports the unbuilt ones as failures. The preset description records this, and also still warns that cases sharing hard-coded /tmp fixture paths race across processes, and directs developers to run CnaTests directly. CI does not hit either problem: general-tests-ci.yml runs a plain ctest, but only after a full default build and serially; the other jobs select labels or run the test binary.

Fixed since first recorded: the race. gtest_discover_tests still runs every CnaTests case as its own process, but the audio fixtures that shared fixed directories (cna_cue_test, cna_wavebank_test) are written under a per-process scratch root (CnaAudioTest::FixtureRoot, GSC-0008), every test process gets its own per-user data root (AM4-162), and the socket-opening suites hold one CTest RESOURCE_LOCK (AM4-163). The fixed temp-directory names that remain in other suites are each used by a single case.

Source locations

• CMakePresets.json — tests configure preset description
• cmake/UnitTests.cmake — CnaTests composition and per-case registration
• modules/audio/tests/Microsoft/Xna/Framework/Audio/AudioTestScratch.hpp — FixtureRoot: one scratch directory per test process (the fixed half)

Evidence

Stated by CNA itself in CMakePresets.json; the registration was read in UnitTests.cmake at c1c316b9. Neither behaviour was reproduced here.

Independent re-verification: Stated by CNA itself in the tests configure preset description in CMakePresets.json, which cites plan_audio.md P9-BUILD-007; that record now lives in plan_audio20260717.md. The registrations were read in UnitTests.cmake and CMakeLists.txt at c1c316b9, and the audio fixture names still exist, under the per-process root described above. Neither behaviour was reproduced here.

Re-verified at c1c316b9: read the tests configure and build presets, AudioTestScratch.hpp, the RESOURCE_LOCK registration in UnitTests.cmake (d5a6b84f0) and tests/HarnessUserDataIsolation.cpp (d0bb0fe3a); the per-process audio fixture root dates from e5b911a85 (GSC-0008). Not executed.

Focused reproduction

No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.

Current tests

Not applicable: the gap is in how the suite is run.

Regression test

Register the display-dependent smoke executables only when they are part of the build (or label them so a preset can exclude them), and update the preset description, whose /tmp warning no longer applies.

Blast radius

Developers who run a bare ctest after cmake --preset tests; CI lanes that use labels or the test binary are unaffected.

Workaround

Run ./cmake-build-tests/CnaTests directly, or use ctest -L with a label.