CNA-VGAP-016: A plain ctest run of the tests preset is not a trustworthy result
Evidence basis: source-verified at the pinned commit; recorded by CNA's own run (not repeated here); tests exist (not executed for this page). Claims on this page were checked by reading the CNA source at commit 009d40f5; unless a sentence says otherwise, nothing here was built or executed. Nothing on this page was executed unless the Evidence section says so.
CNA's own tests preset records that ctest reports unbuilt display-dependent smoke executables as failed and races tests that share hard-coded /tmp fixture paths across processes, so only a direct CnaTests run is authoritative.
- Identifier
CNA-VGAP-016- Category
- Verification gap
- Subsystem
- Testing & evidence
- Status
- Open
- Verified against
- CNA
009d40f5(009d40f5dd085c4e674d3479675fac84b12b3e0a) - Evidence basis
- Recorded by CNA: CNA's own recorded run, not repeated here
- Tests touching this area
- Yes: see Current tests
- Affected contract
- ctest over a build of the tests preset (CMakePresets.json)
Expected behaviour
Every registered test either builds with the target the preset builds or is skipped, and test cases are isolated enough to run as separate processes in parallel.
Actual behaviour at TARGET
The tests preset builds only CnaTests (its build preset names that one target), yet ctest also runs the many separately registered display-dependent graphics smoke executables that the same configuration registers (CNA_BUILD_EXAMPLES is on, the default renderer is OPENGLES3) and reports the unbuilt ones as failures. gtest_discover_tests registers every CnaTests case as its own process without a resource lock, and several fixtures share fixed /tmp directory names (for example cna_cue_test and cna_wavebank_test in the audio tests), so they can race when ctest runs in parallel. The preset description records both and directs developers to run CnaTests directly. CI does not hit either problem: general-tests-ci.yml runs a plain ctest, but only after a full default build and serially; the other jobs select labels or run the test binary.
Source locations
CMakePresets.json— tests configure preset descriptioncmake/UnitTests.cmake— CnaTests composition and per-case registration
Evidence
Stated by CNA itself in CMakePresets.json; the registration was read in UnitTests.cmake at 009d40f5. Neither behaviour was reproduced here.
Independent re-verification: Stated by CNA itself in the tests configure preset description in CMakePresets.json, which cites plan_audio.md P9-BUILD-007; that record now lives in plan_audio20260717.md. The registrations were read in UnitTests.cmake and CMakeLists.txt at 009d40f5, and fixed /tmp fixture names still exist in the audio tests. Neither behaviour was reproduced here.
Focused reproduction
No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.
Current tests
Not applicable: the gap is in how the suite is run.
Regression test
Give each fixture that writes to /tmp a per-process directory, and register the display-dependent smoke executables only when they are part of the build (or label them so a preset can exclude them).
Blast radius
Developers who run a bare ctest after cmake --preset tests; CI lanes that use labels or the test binary are unaffected.
Workaround
Run ./cmake-build-tests/CnaTests directly, or use ctest -L with a label.
Related pages
The same subject is explained at several altitudes. These are the neighbouring pages at each one.
- Maintainer workflow
- Validation: commands
- Tests and validation
- Testing: the fast path · Test architecture: discovery
- Known issues
- Verification gap index