CNA-VGAP-047: Test strength is never measured automatically: no coverage build, no scheduled fuzzing campaign, and the Direct2D mutation checker only dry-runs

CNA snapshot 009d40f5  ·  Known Issues › Verification gaps  ·  source links pinned to 009d40f5

No workflow or CMake option enables coverage instrumentation, the libFuzzer-capable harnesses are not CTests and no job or schedule runs them (their large recorded campaigns were manual, one-time runs), and direct2d_mutation_check.py runs only as --dry-run in a manual workflow.

Identifier

CNA-VGAP-047

Category

Verification gap

Subsystem

Testing & evidence

Status

Open

Verified against

CNA 009d40f5 (009d40f5dd085c4e674d3479675fac84b12b3e0a)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

Yes: see Current tests

Affected contract

CNA's test-strength instruments: coverage, fuzzing, mutation checking

Expected behaviour

Instruments that CNA has built to measure its tests are run often enough to produce current numbers.

Actual behaviour at TARGET

No workflow and no file under cmake/ adds --coverage or configures gcov, lcov, gcovr or llvm-cov; cnaext-coverage.md records one --coverage build of 2026-08-18, for the graphics-ext layer only. Harnesses.cmake builds cna_compiled_effect_fuzzer, cna_xna_model_fuzzer and cna_xna_intermediate_fuzzer and states they are not registered as CTests; the C ABI fuzz sources are an object library only (CMakeLists.txt). CNA did run these harnesses by hand and records the results (plan_fx.md FX-051: about 3.08 million and 2.67 million coverage-guided executions with 41 crash classes fixed; plan_xnapipeline_parity.md XNAPP-290: a 1.16 million-document sanitizer run over the .x and FBX readers), but nothing repeats them, so those numbers age. In CI only deterministic fuzz-named GoogleTests run. d3d-windows-ci.yml (manual) runs direct2d_mutation_check.py with --dry-run, which checks only that each mutation's anchor still applies; the --run mode that rebuilds and requires the targeted test to fail is never automated.

Source locations

• cmake/Harnesses.cmake — fuzzer targets, not registered as CTests
• modules/c-api/CMakeLists.txt — cna_c_api_fuzz_targets object library
• docs/cnaext-coverage.md — the one recorded coverage build
• scripts/direct2d_mutation_check.py — --dry-run versus --run
• .github/workflows/d3d-windows-ci.yml — manual dry-run invocation

Evidence

Checked by reading at 009d40f5; not executed.

Independent re-verification: Checked by reading at 009d40f5; not executed. The regression test should be a scheduled coverage job, a scheduled short fuzzing run per harness from its committed corpus, and a scheduled --run of the mutation checker.

Focused reproduction

No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.

Current tests

Deterministic fuzz-named GoogleTests (for example the XNB container and LZX decoder fuzz tests).

Regression test

A scheduled coverage job, a scheduled short fuzzing campaign per harness with a corpus, and a scheduled --run of the mutation checker.

Blast radius

Confidence in test adequacy; no functional path is affected.

Workaround

No workaround is known.