Tests, examples, presets and tools: what each verdict proves

CNA snapshot 009d40f5  ·  Deep Dives › Architecture & build  ·  source links pinned to 009d40f5

CNA's repository produces many kinds of green result: a configure that passes, a test binary that reports no failures, an example that exits 0, a script that prints OK, a preset that builds. Each of them establishes a different, usually narrower, fact than the word "passes" suggests. This page sorts CNA's test corpus, examples, presets and developer tools by what their verdict actually decides at snapshot 009d40f5, so that a maintainer or a reviewer can state a result without overclaiming it. The test machinery itself is described on Test architecture; the command-line tools a game author uses are on Command-line tools.

Five facts that must stay separate

A dependency was acquired; a target was built; a test was registered; it was executed (and not skipped); and an oracle was engaged that could have disagreed. These are separate facts, and every tool below establishes some of them and not the others. A dependency pin makes a build reproducible; it does not show that a test used that dependency. A built example shows that its code is reachable and links; it does not show correct pixels. A registered test in a ctest -N listing is not an executed one, and an executed test that exited 77 was skipped, which CTest reports separately and which is not feature evidence. The same coordinates appear in CNA's compatibility claims, described on Compatibility levels and the evidence vector.

The CnaTests corpus has no configuration-free size

CnaTests is assembled by cmake/UnitTests.cmake from module-owned test sources (modules/*/tests, the renderer families' and shared renderer helpers' tests, and the root tests/*.cpp), minus the minimal-link probes and the C API's own test programs, which have their own main(). The same per-module object groups also feed the 22 focused executables such as CnaStorageTests. Which sources enter depends on the configuration: feature options, the selected platform and audio values, the compiled renderer set (a renderer's own suites are kept whenever that renderer is compiled in, not only when it is the default), FFmpeg availability and networking all add or remove translation units. There is therefore no configuration-free "number of CNA tests". The site's source inventory (904 C++ test source files and 12,610 statically declared GoogleTest-family definitions) is a count of source text by a stated method, not of cases one binary instantiates or runs.

How the corpus is run matters as much as what it contains. The description of the tests preset in CMakePresets.json says that running the CnaTests binary directly is the authoritative way to run the suite, because ctest discovers each GoogleTest case and runs it as its own short-lived process, which reports every unbuilt display-dependent smoke executable as a failure and races tests that share hard-coded temporary fixture paths. The Apple workflow follows the same advice and runs the binary with a filter. CTest remains the right runner for named standalone gates and renderer executables; discovered cases run with the repository root as their working directory, because fixture paths are repository-relative. The detail is on Validation: the base matrix.

Examples are executable documentation, not one evidence tier

Module-owned examples range from tutorial programs through GPU readback tests, golden-image comparisons and host or browser harnesses to policy-only probes, and the 879 standalone *_test.cpp programs under the modules' examples/ directories are all of these at once. Neither the file name nor the registering directory says which kind a program is. Before citing one, record:

• whether the selected configuration compiles it at all (most are registered only for one renderer or platform);
• whether it runs automatically in some workflow or only when invoked by hand;
• which renderer, native driver and host actually engaged (CI's Linux GPU tests use Mesa's software drivers under Xvfb, not a physical GPU);
• what its oracle is: an exit status, structured values, pixels compared with a stored image, or a person looking at a window;
• what it does when the host cannot run it: skip with 77, refuse with a message, or fail.

Golden images need particular care about what they are compared with. The repository-level examples/golden/ directory holds 17 PNGs captured from CNA's EasyGL renderer. The EasyGL golden tests compare against them, and the Vulkan examples compile several of the same EasyGL example sources and compare against the same files. A pass there shows agreement with an earlier EasyGL capture, a regression and cross-renderer check; it is not a comparison with XNA. Comparisons with genuine XNA 4.0 output are a separate corpus, the oracle scenes under tools/xna-oracle (see Using the XNA oracle as evidence). A renderer-owned test executable and a shared reference image are two artifacts, and their provenance and update rules have to travel together: regenerating the image without recording which renderer, driver and revision produced it silently changes what every consumer of it proves.

Presets are named environments, not universal promises

CMakePresets.json defines 17 visible configure presets, a hidden base-ninja parent and 17 build presets. A preset proves exactly its declared cache values, target and commands:

• The web preset builds cna_house3d_demo with tests off. It is a WebAssembly build of one demo, not a browser conformance run, and it needs emcmake because it names no toolchain file.
• The tests build preset builds CnaTests; it does not build the standalone example executables that the same configuration registers with CTest.
• The dev, unit and release-* presets deliberately omit tests or demos, the C API, networking, FFmpeg and Draco, and tools/build/check_build_performance_policy.py (run by general-tests-ci.yml) fails if those closures regain them. A green dev build is therefore evidence about none of those features.
• The three devices-* presets set CNA_DEVICES=ON explicitly; their descriptions record that without it the whole CNA::Devices surface a preset named for devices promises would compile out silently. They select their sanitizer through CNA_SANITIZE and CNA_SANITIZE_OPTIMIZATION, not raw compiler and linker flags; an earlier arrangement that passed raw flags alongside the option is gone. Only devices-ubsan is used by a workflow.

Preset descriptions are documentation and can lag: the tests description still asks for a recursive submodule update and names only four submodules, while the CMake says the non-recursive form is correct (see Dependency acquisition). Hosted-workflow reality is audited separately on Test architecture: what CI covers.

Tools and what their verdicts decide

CNA's mechanical tools fall into distinct authority classes. Knowing the class tells you what a pass means before you read the tool.

The Wine launchers illustrate the last class. In a MinGW cross tree CnaTests is run through one of them, chosen by the default renderer, with an environment flag that skips the wrapper's DXVK or vkd3d-proton presence check, because listing tests or running plain unit tests creates no graphics device. The same wrapper run for a renderer test keeps the check, which requires the DXVK: <version> line that only DXVK's own logger prints, because a Direct3D 11 test that silently fell back to Wine's built-in WineD3D would otherwise pass while validating a different Direct3D implementation (see MinGW-w64 cross-build and Wine).

The cross-renderer smoke orchestrator

run-all-renderer-smoke-tests.sh runs the renderer smoke tests on a local machine, one renderer at a time. The smoke tests are the CTest entries labelled GraphicsSmoke, which run real game programs for a few frames: cna_house3d_demo --smoke 4 --depth-probe and cna_demo_2d --smoke 3, for example. The script has four ways to choose what it runs:

The per-renderer modes configure a tree only when its directory is missing, and they reuse an existing tree as it is. A tree that cannot be configured (without the Vulkan SDK or wgpu-native, for example) is reported as SKIPPED (configure failed). It is never counted as a pass, and it does not fail the run. Only a smoke test that ran and failed makes the script exit 1.

The multi mode cannot trust a single label, because ctest -L exits 0 when the label matches nothing. For each renderer it therefore lists the tree's labels and uses GraphicsSmoke or the renderer's own label, compared case-insensitively and ignoring underscores, but only if that label selects at least one test. Otherwise it reports SKIPPED (no smoke test). A renderer combination that does not configure fails the multi run outright.

The build step between configuring and testing is the script's weakest point. It builds with cmake --build <dir> -j4 -- -k, ignores the result and treats CTest as the only signal. A verdict can therefore come from binaries that the current source did not produce (see Known Issues). Nothing invokes the script: no workflow and no CTest registration references it. CNA's documentation mentions it only for the multi mode, in docs/runtime-renderer-selection.md. Its summary is evidence only for the build trees and sibling checkouts on the machine that ran it, and should be quoted with them. Checked by reading at 009d40f5; not executed.

A gate over command text: the Direct2D parallelism cap

verify-direct2d-parallel-jobs.sh is a structural validator whose subject is build commands rather than code. It scans ten named files: the Windows Direct3D workflow, the Direct2D example registrations, cmake/UnitTests.cmake, two Direct2D documents, the Direct2D plan and the four Direct2D launch scripts. It fails on any line that invokes cmake --build, ninja or make without a numeric --parallel or -j cap on the same line, and on any cap above 2. Prose that merely mentions such a flag is ignored. A listed file that no longer exists is itself a violation, so a rename cannot silently turn the gate into a no-op. The script runs before any build step in the Direct2D leg of d3d-windows-ci.yml, a workflow that is started only by manual dispatch. It therefore guards the text of those commands whenever that leg runs, not on every push. Run against the 009d40f5 tree for this page, it reports every scanned command capped at two jobs or fewer.

Reading a result before citing it

1. Name the configuration: preset or cache values, platform, audio, renderer set, generator and host.
2. Check registration with ctest -N in that tree; a test that is not registered there proves nothing about it.
3. Separate passed from skipped: exit 77 is a skip, and a whole suite whose cases all skipped can still be counted as passed by CTest.
4. Identify what engaged: which renderer and driver (a software rasteriser under Xvfb is not a GPU), which runtime (Node is not a browser), which sibling revisions (some CI lanes pin an older sharp-runtime).
5. Identify the oracle and its tolerance, and whether anyone has shown the gate failing on a deliberate mutation.

A statement that keeps acquisition, build, registration, execution and oracle engagement apart can be checked by someone else; a statement that merges them into "tested" cannot.

Evidence and limits

Read at 009d40f5 from cmake/UnitTests.cmake, cmake/Tests/ModuleProbes.cmake, CMakePresets.json, the renderer example CMake files, the scripts and tool directories named above and the workflow files. No tool, test or preset was run for this page; the classification is of what each tool can decide by construction, and the test-source counts are the site's recomputed inventory at this snapshot.