Test populations, counts and structural gates

CNA snapshot 009d40f5  ·  Deep Dives › Diagnostics & verification  ·  source links pinned to 009d40f5

CNA's test system is not a directory of independent unit-test programs. It is one aggregate GoogleTest binary with focused developer copies, about fifteen hundred renderer-example registrations, process-isolation helpers, a test whose pass condition is a compiler error, and dozens of script gates that check structure rather than behaviour. This page explains how to tell which layer produced a result, why "how many tests does CNA have?" has no configuration-free answer, what each static count at snapshot 009d40f5 actually counts, and how CNA turns project discipline (identity tables, dependency containment, generated inventories, mutation checks) into executable gates. It is written for anyone who reports a test result or adds a gate; the assembly mechanics are on Test architecture and change recipes and are not repeated here.

Which layer produced a result

Five kinds of executable evidence coexist, and each has a different registration path and a different failure mode. A report that does not say which one it came from cannot be interpreted.

Three layers of renderer gating

A renderer family's tests can disappear at three different points, and a green summary is silent about all three.

1. Source filtering. Before any object library exists, UnitTests.cmake drops renderer-local suites whose family is absent from CNA_RENDERER_IDENTITIES: the FNA3D suites unless FNA3D is compiled in, the EasyGL suites unless one of the five GL identities is, the WebGPU suites unless WEBGPU is. Membership in the compiled set, not the default identity, decides, so a multi-renderer CnaTests keeps the suites of every compiled family; inside a surviving file, the body is compiled only under CNA_RENDERER_<X> or CNA_RENDERER_PRESENT_<X>.
2. Registration conditions. Each family's examples/CMakeLists.txt decides which standalone programs are built and registered, usually only when that family is the configured renderer (or, for some families, a member of the compiled set).
3. Runtime pre-flight. A pixel program probes SDL_InitSubSystem(SDL_INIT_VIDEO) before constructing its game and returns kSkipExitCode (77) when no display is usable (PixelTestGame.hpp). CTest reports that as skipped, not failed.

The third layer is deliberate: it keeps a headless machine from turning the absence of a display into a renderer defect. The price is that a green CTest summary can contain no executed pixel comparison at all, so a report keeps the skip count and the display setup beside the pass count. The filter chain in the first layer is also part of the build contract rather than housekeeping, and parts of it assert themselves: with CNA_ENABLE_NET=OFF the file walks the surviving list and stops the configure with NET=OFF left disabled-module test selected if any network or GamerServices source slipped through; a test group without a focused-target name, or whose declared dependency target does not exist, is also a configure-time FATAL_ERROR. A glob or filter edit that would compile an un-linkable test therefore fails before compilation, not at link time.

Helpers that isolate, and helpers that only inspect

Some defects live in process-global state (SDL shutdown order, the audio mixer's global owner, the GamerServices dispatcher, network peers), so a GoogleTest case spawns a small helper and judges it, gaining a clean address space in which a crash or a wrong shutdown order is observable without poisoning later cases. Harnesses.cmake also defines executables that are tools, not tests: cna_xnb_audio_metadata_dump, cna_xwb_inspect, cna_reference_dump (the CNA half of the FNA value comparison) and cna_devices_microbenchmark, whose comment says it is deliberately not a CTest because its JSON-lines output is meant to be compared, not judged by exit code. The fuzz harnesses (cna_compiled_effect_fuzzer, cna_xna_model_fuzzer, cna_xna_intermediate_fuzzer) are likewise unregistered: they replay a corpus by default and become libFuzzer entry points only with an explicit option such as CNA_FX_FUZZER_ENTRY_POINT=ON. Their presence beside the harnesses does not put them into any test run; the governing question is always who invokes the target, and under which configuration.

Building and registering are two statements

A renderer example is built by one call (the family's own cna_<family>_test macro) and registered by a second (cna_register_renderer_test), and the two can sit under different conditions. At this snapshot the lexical counts are close in every family, which is worth knowing before assuming a large unregistered corpus:

The counts are lexical call sites in each family's examples/CMakeLists.txt (counted with grep at 009d40f5, not a configured inventory); a registration without a matching build call usually runs a script or a program built elsewhere. The label column is the practical warning: Vulkan, EasyGL and SDL_RENDERER label almost none of their registrations, so ctest -L EasyGL selects a handful of entries rather than the EasyGL corpus, and the name selector (-R '^EasyGL_') is the reliable one. Labels are also not family-scoped by design: GraphicsSmoke is attached by the graphics examples and by the EasyGL, Vulkan, WebGPU and SDL_RENDERER smoke entries alike.

When failing to compile is the pass

The CNAEXT surface rule ("a program restricted to XNA 4.0 must not reach an extension") is enforced by the compiler, not by a runtime assertion. Harnesses.cmake builds two programs with CNA_STRICT_XNA_API defined, which turns every CNAEXT marker into [[deprecated]], and with -Werror=deprecated-declarations, on GNU and Clang toolchains only:

• cna_strict_xna_api_check (StrictXnaApiSurfaceCheck.cpp) is an ordinary target; StrictXnaApiSurfaceCheck_Compile_Run runs it. It proves that code using only the XNA surface of Microsoft::Devices and the sensors compiles under the strict mode.
• cna_strict_xna_api_leak_check (StrictXnaApiSurfaceLeakCheck.cpp) is EXCLUDE_FROM_ALL, so no normal build ever compiles it. The test StrictXnaApiSurfaceLeakCheck_MustFailToCompile runs cmake --build <build> --target cna_strict_xna_api_leak_check --config $<CONFIG> as its command and carries WILL_FAIL TRUE: it passes only when that build fails.

The inverted polarity turns a namespace rule into executable evidence, and the positive twin keeps it honest: a strict mode so aggressive that nothing compiles would make the negative test pass for the wrong reason, which the positive test would then expose. Two limits follow by construction. WILL_FAIL accepts any build failure, so the diagnostic that proves the gate is the compiler naming the deliberate call, which is worth reading when the test's status changes; and both programs cover the Microsoft::Devices and sensor surface only, not the whole XNA namespace. The complementary idea at link level, fifteen minimal module consumers whose link lines are inspected for forbidden libraries, is described with its generator caveat on Module boundaries and their enforcement.

Discovery, selection, execution, verdict

Most apparent contradictions in test prose dissolve once a result is split into four stages, each with its own question and its own CNA-specific check.

A source can exist but be filtered out; a target can build but not be registered; a CTest can register but skip; a program can pass while running through the wrong graphics translation layer. Each of those is a different stage failing silently, and each is visible only if the report names the stage.

Six populations that must not be added together

At 009d40f5 static inspection gives the following counts. Each is exact for its definition and meaningless outside it; adding them, or choosing the largest as "the number of tests", mixes populations.

A test source can define many cases; a parameterised definition expands at run time; one executable becomes thousands of CTest entries; one standalone executable can exist with no registration. The reproduction commands, run from a checkout of the snapshot, make the definitions explicit:

# the glob population, a sorted union rather than a sum of directory totals -> 903
{ find modules -path 'modules/*/tests/*' -name '*.cpp' -type f
  find tests -name '*.cpp' -type f; } | sort -u | wc -l
# direct renderer-registration call sites -> 1437
grep -rE --include=CMakeLists.txt -c '^\s*cna_register_renderer_test\s*\(' modules/renderers \
  | awk -F: '{s+=$2} END{print s}'
# standalone example programs, and how many are GoogleTest sources -> 879, 0
find modules -path '*/examples/*' -name '*_test.cpp' | wc -l
find modules -path '*/examples/*' -name '*_test.cpp' -exec grep -l 'gtest/gtest.h' {} + | wc -l

The site's own inventory commands (ripgrep over test paths, then over declaration starts) are on Reproducing these numbers yourself.

Why there is no all-renderer total and no CTest total

A singular build enters one family; a multi-renderer build enters only an explicitly named, compatible set. Host gates then make a universal configuration impossible: the three Direct3D identities, DIRECT2D and GDI configure only for a Windows target, METAL only for Apple, and the five browser identities only under Emscripten. Summing every family's registration sites therefore produces a number that no supported build exposes as one CTest suite, and "all 25 renderers passed" is not a statement any single run can support.

Discovery adds a second barrier. PRE_TEST obtains case names by running CnaTests --gtest_list_tests when CTest reads the test files, so configure-time inspection cannot know the expanded list; conditional compilation, the renderer-present macros and parameter instantiation all change it. The authoritative count for a build comes from that built artifact and names its renderer and feature set; this site publishes none. Some disputed numbers, by contrast, can be settled by defining the noun: 25 public renderer identities and 21 implementation families are both correct, because five GL identities share the EasyGL family (see Renderer coverage is configuration-scoped).

Counts that only a run can give

Static inspection cannot establish: the CTest count of a configured and built tree; how many parameterised cases expand or actually run; which standalone registrations survive nested conditions; pass, failure, skip or timing totals; whether any pixel assertion reached a display; image-oracle divergence counts; or line, branch and function coverage. No workflow configures gcov, lcov, gcovr or llvm-cov, and no CMake option adds coverage instrumentation; the only line-coverage figure CNA publishes is a hand-recorded, single-module report from 2026-08-18 (see Coverage reports and what they measure). Files and reports named "coverage" otherwise describe API representation, source-to-suite mappings or feature checklists, and a percentage quoted without that qualifier misleads more than no percentage.

A reporting grammar

A durable result fits in one sentence when every noun is present. An execution result: "On commit X, the Y renderer build with features Z discovered N CTest entries; it ran R, skipped S and failed F on host H." A source audit uses a different sentence: "At commit X, this command found N declarations matching definition D." Comments rot; reproducible definitions can be rerun. CNA's own CMake comments and plans quote several renderer-registration and suite totals from different dates and configurations, which is exactly why the discipline is not to avoid numbers but to make every number carry its derivation, its configuration and its date.

Discipline expressed as tests

Many CNA regressions would compile and render a plausible frame while violating a dependency, selection, ABI or evidence contract. CNA therefore keeps structural gates beside the behavioural tests. They prove structure; the live suites still prove behaviour.

Registry and axis invariants

Renderer selection is checked in three directions at once. check_renderer_identities.py compares the public GraphicsRendererType enum, the CNA_GRAPHICS_RENDERER CMake list and the runtime registry against one canonical table, and follows each identity through to the accessor that must return its descriptor, because being spelled identically in two lists says nothing about whether a build can instantiate the renderer; it also checks the count stated in a handful of documents, since a number written into prose has no owner. Its companions check combination rules, that families never read the global renderer scalar and that runtime decisions stay behind descriptors. Platform and audio selection are independent tables with their own script-mode cases, and negative configure tests prove that host-inadmissible or mixed-SDL tuples fail with a useful message (the policy-test table is on Configuration and policy tests that need no GPU).

Containment gates

The SDL-free native platforms add gates that stop a direct SDL dependency from reappearing unnoticed. The source-gate step of platform-ci.yml runs, on the headless cell and again in the SDL-free X11 lane, sdl_inventory.py --check, sdl_classify.py --check (which fails on any SDL identifier that no contract-area rule classifies), renderer_sdl_audit.py --check, sdl_ratchet.py --check --strict and nonproduction_sdl_audit.py --check; the headless cell also runs check_contract.py (every contract header must stay in the SDL-free probe and carry documentation) and hot_path_lint.py. The two audits distinguish legitimate from forbidden use rather than banning a token everywhere. The hot-path lint flags a line only when two independent judgements agree (it is a platform call, with method names read from the contract headers, and it sits inside a per-pixel, per-vertex, per-sample or per-event loop), because such a move still reads correctly and passes every test, and shows up only as a frame-time regression on someone else's machine. The non-production audit gives tests and examples a finite, classified exception list with a checked-in per-file ceiling: a new file, an unclassified location or a higher count fails, removing a dependency always passes, and --update tightens the manifest to the new floor.

Generated inventories are executable claims

Several important totals are generated rather than maintained by hand: the XNA runtime census (331 types and 3,627 documented members, each with its semantic category), the Content Pipeline inventory, the C API's declaration, export, ABI-layout, coverage, limitation, consumer and release-gate records, the renderer and platform identity sets, and the CNB and XNB container checks. A generated record is still not automatically trustworthy. The C API coverage generator's module-scope table (generate_coverage_inventory.py) names every runtime module plus four deliberately out-of-scope ones, but has no entry for the design, diagnostics and inspector modules, and its scope validation refuses a module that nothing classifies; the checked-in coverage report and the release gate's "Not ready" verdict therefore describe a model that no longer covers every public module (see Known Issues). A command that reproduces a failure is stronger evidence than a checked-in percentage that nobody regenerated.

Can a green gate go red?

A gate earns trust when someone has watched it fail on purpose. CNA carries several such checks, and they differ in strength:

• direct2d_mutation_check.py applies one deliberate defect at a time to the Direct2D renderer. --dry-run only applies and reverts each mutation to prove that its anchor text still exists exactly once, the part that goes stale when a refactor silently turns a mutation into a no-op; --run rebuilds, runs the CTest meant to catch the defect and requires it to fail. The manual d3d-windows-ci.yml runs only the dry run; the full apply-build-expect-failure pass is a deliberate manual step. A dry run proves the wiring, not that any mutation was killed.
• verify-direct2d-debug-log.py turns the Direct2D debug-layer log into a gate that fails when the run never enabled the layer it claims to verify, logged an error or corruption message, or leaked an undocumented live object. Its --self-test replays committed positive and negative logs from tests/fixtures/direct2d/ (a clean hardware and a clean WARP log; dirty logs with an error message, a live texture, no debug layer and a truncated report) on Linux, and the same workflow runs that self-test before trusting the parser.
• The CTest DirectX11_DxvkGate runs test-run-wine-dxvk-gate.sh: a stub wine on PATH prints both marker spellings, then no marker (the wrapper must exit 3), then a marker with an application exit of 7 (which must be preserved). A log gate proves engagement only when both its positive marker and its negative cases are exercised.
• The golden-image comparison of PixelTestGame.hpp prints the largest channel difference it actually used on its [PASS] line, so a scene drifting from 2 to 59 under a tolerance of 60 is visible before the day it fails.
• The browser drivers fail on a page exception or a [FAIL] console line and require a named verdict object, so a page that never reached a frame cannot appear green (see browser verdicts).

Five states between a script and evidence

Four states should stay distinct: a script exists; CTest can register it; a workflow references it; and the workflow actually triggered with the environment the script needs. A fifth, that the intended implementation engaged, is often established only by a renderer token, a browser structure check, a native handle, a validation-layer marker or a translated-API log line. Three CNA examples show the gaps between the states. EasyGL_XnaLineCoverage is registered in an ordinary EasyGL tree that the unfiltered job in general-tests-ci.yml builds, but its diff tool imports Pillow, which that workflow does not install, so whether it runs there was not verified. The ModuleLinkClosure_* gates are registered on native test configurations and skip in every Ninja tree, which includes the tests preset, every preset built on the hidden Ninja base and the one unfiltered CI job; where a Makefiles tree exists, no workflow runs them through CTest (the details are on the generator caveat). The Direct2D mutation gate is referenced by a workflow that runs only its dry run and only on manual dispatch. Workflow presence is executable intent; a named run and its artifacts are execution evidence.

Gate-quality checklist

• It states one reviewable invariant.
• It refuses missing or empty inputs instead of passing over them.
• It has positive and negative fixtures, and someone has seen it fail.
• It records the configuration it evaluated (generator, renderer set, host).
• Its skips are explicit and counted, never folded into a pass.
• It is wired into the automation that is meant to run it, with the environment it needs.
• When modules move, its parser scope and the documents it checks change in the same commit.

The unifying principle is that a green result that was never produced is worse than a red one. The procedure for adding a behavioural regression test with the same properties is I need to add a regression test.

Evidence and limits

Everything above was read at 009d40f5 from UnitTests.cmake, Harnesses.cmake, the renderer families' example CMake files, the scripts and tools named, and the workflow files. The counts were made with find and grep over that tree and are lexical: none is a configured CTest inventory, an executed case count or a pass result, and nothing was configured, built or run for this page. The mutation, debug-log and gate self-tests were read, not executed; whether any workflow named here is currently green was not checked.