CNA-VGAP-047: Test strength is never measured automatically: no coverage build, no scheduled fuzzing campaign and no mutation testing

CNA snapshot c1c316b9  ·  Known Issues › Verification gaps  ·  source links pinned to c1c316b9

No workflow or CMake option enables coverage instrumentation, the libFuzzer-capable harnesses are not CTests and no job or schedule runs them (their recorded campaigns were manual, one-time runs), and no source-mutation checker exists.

Identifier

CNA-VGAP-047

Category

Verification gap

Subsystem

Testing & evidence

Status

Open

Verified against

CNA c1c316b9 (c1c316b9c7a846ce8002809c151fcd1af14942c9)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

Yes: see Current tests

Affected contract

CNA's test-strength instruments: coverage, fuzzing, mutation checking

Expected behaviour

Instruments that CNA has built to measure its tests are run often enough to produce current numbers.

Actual behaviour at TARGET

No workflow and no file under cmake/ adds --coverage or configures gcov, lcov, gcovr or llvm-cov; the only recorded --coverage build, cnaext-coverage.md (2026-08-18), is marked by CNA as a historical record that does not describe the current tree. Harnesses.cmake builds cna_compiled_effect_fuzzer, cna_xna_model_fuzzer and cna_xna_intermediate_fuzzer and states they are not registered as CTests; the C ABI fuzz sources are an object library only (CMakeLists.txt). CNA did run these harnesses by hand and records the results (plan_fx.md FX-051: about 3.08 million and 2.67 million coverage-guided executions with 41 crash classes fixed; plan_xnapipeline_parity.md XNAPP-290: a 1.16 million-document sanitizer run over the .x and FBX readers), but nothing repeats them, so those numbers age. In CI only deterministic fuzz-named GoogleTests run. No tool mutates CNA's sources to check that a test then fails (the XNA sample sweep's semantic_mutation.py and tolerance_mutation.py mutate content files to test that tool's own classifier), and no workflow runs any mutation check.

Source locations

• cmake/Harnesses.cmake — fuzzer targets, not registered as CTests
• modules/c-api/CMakeLists.txt — cna_c_api_fuzz_targets object library
• docs/cnaext-coverage.md — the one recorded coverage build, marked as a historical record

Evidence

Checked by reading at c1c316b9; not executed.

Independent re-verification: Checked by reading at c1c316b9; not executed. The regression test should be a scheduled coverage job, a scheduled short fuzzing run per harness from its committed corpus, and a mutation-testing pass.

Focused reproduction

No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.

Current tests

Deterministic fuzz-named GoogleTests (for example the XNB container and LZX decoder fuzz tests).

Regression test

A scheduled coverage job, a scheduled short fuzzing campaign per harness with a corpus, and a mutation-testing pass.

Blast radius

Confidence in test adequacy; no functional path is affected.

Workaround

No workaround is known.