Testing handbook

CNA snapshot 009d40f5  ·  Development › Testing  ·  source links pinned to 009d40f5

CNA combines focused per-module tests, cross-module suites, renderer and platform conformance programs, configure-time and repository audits, sanitizer runs and host-specific CI. No single command proves every backend: what "the tests passed" means depends on the configured tree, the display the tests ran on and whether skipped entries were counted as passes. This page is the working handbook: the fastest useful local subset, the broader suite and its traps, running GPU and window tests without touching a live desktop, sanitizers, failure recipes, how CI is sliced and how to add a test. How the executables are assembled is on Test architecture and change recipes, and the change-to-test matrix is What to test after changing X.

Fastest useful local subset

The unit preset configures the STUB renderer with tests on and examples, networking, video, Draco and the C API off, so it compiles the framework and its unit tests without a GPU. A focused executable holds one module's tests and is built from the same object files as the aggregate CnaTests:

cmake --preset unit
cmake --build --preset unit-core --parallel
./cmake-build-unit/CnaCoreTests
./cmake-build-unit/CnaCoreTests --gtest_filter='<Suite>.*'

Focused binaries are developer iteration targets: they are excluded from the default build and are not extra CTest registrations, so ctest does not run them. Four have build presets (unit-core, unit-math, unit-content, unit-graphics); the rest are built by target name, for example cmake --build cmake-build-unit --target CnaAudioTests. The nearest target for a change is the one for the owning module:

A group with no remaining sources in a configuration gets no target: the unit preset turns networking off and builds no inspector, so CnaNetTests, CnaGamerServicesTests and CnaInspectorTests do not exist there. The generated test target index does not list these names, because they are created from variables (it shows one placeholder row for ${_cna_focused_test_target}); the names come from CNA_TEST_FOCUSED_TARGET_<group> in cmake/UnitTests.cmake. To see what a tree registered for CTest, run ctest --test-dir <build-dir> -N; the discovered CnaTests cases appear only once CnaTests itself has been built (by the GoogleTest module's PRE_TEST mode; not observed here).

Broader suite

cmake --preset tests
cmake --build cmake-build-tests --parallel        # everything: the tests preset has examples on
tools/platform/run_gpu_tests_private.sh cmake-build-tests --output-on-failure   # unfiltered, private display
ctest --test-dir cmake-build-tests --output-on-failure -L input                 # or a labelled subset

The tests preset writes cmake-build-tests (Ninja, Debug, OPENGLES3, tests and examples on); the pinned preset is authoritative, and a local override changes the directory, so use the one CMake prints. Three traps make an unfiltered ctest misleading:

• The build preset is narrower than the test set. cmake --build --preset tests builds only CnaTests. Many CTest entries are separate example or smoke executables, so an unfiltered ctest after that build reports missing binaries. CNA's own unfiltered CI job runs a full default build first. Either build everything, or select entries with -L or -R for targets you built.
• Window tests use whatever display you launch them from. CNA_TEST_DISPLAY is empty by default, so window-creating tests inherit the caller's DISPLAY: from a desktop session they open windows on the desktop. Run them through the private display wrapper below, or under a private X server as CI does (xvfb-run -a ctest …, or a fixed Xvfb display passed as -DCNA_TEST_DISPLAY=:99).
• Skips are not passes. See Skipped is not passed.

Select subsets by label or name (next section) rather than with a long filter string; the input subset, for instance, used to be a filter copied through documents and CI and is now one canonical filter behind ctest -L input. A green filtered run says nothing about an unfiltered one: the general CI job exists precisely because the specialised jobs ran only filtered subsets.

Selecting by label and by name

Labels are attached at registration and are not uniform across renderer families, so list what a tree carries with ctest --test-dir <build-dir> --print-labels before relying on one. Labels follow the current renderer spellings: -L DIRECTX9 matches, -L D3D9 matches nothing.

What the numbers mean

This snapshot has 904 C++ test source files (871 contain a counted macro) and 12,610 statically declared GoogleTest-family definitions, plus 879 standalone example pixel programs that are outside those figures; the generated index lists 239 syntactic add_executable and add_test declarations. None is an executed-case count or a pass rate, and no CTest total is derivable from the sources, so none is published here. The runs CNA itself recorded are summarised, with dates and configurations, on the verification page.

GPU and window tests

Never point graphical test suites at your live desktop by default. CNA's private display wrapper starts a headless Weston compositor with the GL renderer and a rootful Xwayland on a display number the X server chooses itself (warning when the private display has no DRI3), sets DISPLAY and WAYLAND_DISPLAY to those private servers, runs the tests and takes every process it started down again:

tools/platform/run_gpu_tests_private.sh <build-dir> --output-on-failure
tools/platform/run_gpu_tests_private.sh cmake-build-vulkan -R '^Vulkan_' -j6
tools/platform/run_gpu_tests_private.sh --exec <command> [arguments...]

Arguments after the build directory go to ctest; the --exec form runs one command (a test binary with a --gtest_filter, a demo, a stress run) in the same private environment for what CTest does not register. The wrapper refuses a build tree whose cache forces a display with CNA_TEST_DISPLAY (CTest's per-test environment would override the private one) and exits 77, ctest's skip code, when Xwayland or a compositor is unavailable. Xvfb often has no DRI3 and cannot present real Vulkan. The script's own header records a known limit: the Wine-based XNA interop and differential tests hang inside the private runtime directory, so exclude them (-E 'XnaPipelineGenuineRuntime|XnaDifferentialBuildTest') or run them separately. After a ctest run it also reads Testing/Temporary/LastTest.log and names every test that died on a Reach graphics-profile refusal before its assertions (tools/platform/profile_dead_tests.py): that is a test defect, not a renderer result.

A focused renderer test should select the renderer explicitly (in a multi-renderer build the CNA_GRAPHICS_RENDERER environment variable chooses at run time), have that API's diagnostics on, and keep the logs and artefacts from initialisation and presentation failures. Two output gates turn diagnostic lines into failures: the Vulkan renderer's [Vulkan Validation] messages and the OpenGL4 renderer's [OpenGL4 GL Error] messages (Output gates). For a large binary on a small machine, tools/tests/run_gtest_bounded.sh shards a GoogleTest binary so a limited number of tests share one process and reports a shard killed by a signal instead of a partial pass; its header records why (peak memory per 200 device-creating tests differed several-fold between renderers) and it composes with the wrapper's --exec form.

Which display a registered test uses is the CNA_TEST_DISPLAY cache value (cmake/TestDisplayPolicy.cmake). It is empty by default, so each window-creating test inherits the caller's DISPLAY; a value naming the live desktop (:0) is honoured only together with CNA_TEST_ALLOW_LIVE_DISPLAY=ON and is otherwise reset to empty; any other value, for example :99 for an Xvfb, is forced into the renderer tests' environment together with SDL_VIDEODRIVER=x11. Non-Windows registrations also carry a Wayland guard so an unset WAYLAND_DISPLAY cannot fall back to the live compositor. Clearing the per-test DISPLAY and adding the guard need CMake 3.28 or newer; on older CMake the tests keep an empty DISPLAY and the configure warns. Read at the pinned commit; not executed.

ASan and UBSan

The sanitizer presets are devices-asan, devices-tsan and devices-ubsan (Debug, OPENGLES3, CNA_DEVICES=ON, build preset target CnaTests; ASan at O0, TSan and UBSan at O1). Any configuration can also enable CNA_SANITIZE=address,undefined directly; ASan with TSan and TSan with MSan are refused, and MSVC and Emscripten are unsupported. List the presets with cmake --list-presets=all. Build and run the same focused reproducer first, then broaden. Only devices-ubsan is used by a workflow; the input and glTF workflows configure ASan and UBSan by hand on other renderers.

Sanitizer runtime ordering matters when foreign runtimes or GPU drivers are involved, and the honest way to handle a leak that is not CNA's is to record it, not to disable the check. tools/platform/lsan_x11_mesa.supp is the model: every entry names a leak that was reproduced without any CNA code, with the program and byte count that proved it. It suppresses Mesa's GLX client state on the software path and, more recently, Mesa's gallium screen state that only eglTerminate frees (CNA's lazily created default platform is deliberately never destroyed, so a process that draws through a standalone device exits without it). It deliberately does not suppress the RADV Vulkan ICD's small one-time allocation, which cannot be named without also matching genuine leaks; Vulkan sanitizer runs select lavapipe instead and record RADV as a driver finding. Use it with LSAN_OPTIONS=suppressions=tools/platform/lsan_x11_mesa.supp.

Failure recipes

Skipped is not passed

Exit code 77 is CNA's deliberate skip convention: a test that finds no display, adapter or helper program exits 77 and CTest reports it as skipped. The convention is applied broadly: once over the root-scope tests at the end of cmake/UnitTests.cmake, per module directory through cna_apply_skip_convention, and again by cmake/Tests/ModuleProbes.cmake for the link-closure tests it registers later. A skipped test is not feature evidence. Launchers use it too: the private wrapper, the X11 and Wayland private-server launchers, the WebGPU launcher and the shader-package generators (which exit 77 without shaderc). Also note that the module link-closure gates read CMakeFiles/<target>.dir/link.txt, which only Makefile generators write: in a Ninja tree (every preset that sets Ninja, and most CI jobs) they report skipped, so those contracts are evaluated only in a Makefiles tree.

CI is a matrix of slices

The repository has 20 workflow files defining 28 jobs: 17 trigger automatically on pushes and pull requests to next, develop and main (several with path filters), d3d-windows-ci.yml and gdi-windows-ci.yml are manual dispatch only, and content-pipeline-windows-ci.yml runs on pushes to its own branch plus manual dispatch. They cover general tests, platform and input, multi-renderer selection, Apple and Metal, Windows Direct3D and GDI, the content pipeline, the C API, sanitizers and web configurations. A green Linux unit job does not validate a Windows platform or the Metal backend; when you change a gated subsystem, identify and reproduce its owning workflow. The per-workflow table, with what each one gates and what it does not, is on Test architecture: what CI covers, and the user-level summary is What CI covers (and what it doesn't).

Adding tests

Place isolated behaviour beside the owning module: a .cpp under modules/<name>/tests/ mirroring the namespace path is collected by a CONFIGURE_DEPENDS glob and compiled into that module's object group, so it joins both CnaTests and the focused binary with no CMake edit. Use the top-level cross-module root for contract and conformance behaviour that spans modules, and the module's examples/ for a device-backed program. Prefer a minimal deterministic fixture; do not require a real desktop or device when headless behaviour is what is under test. Name the target and the test so CTest and CI filters can select it: a new suite whose name matches none of the tokens in the canonical input filter must be added to that one string in cmake/UnitTests.cmake, otherwise ctest -L input silently omits it, and a new glTF suite must fall into exactly one rung of the conformance ladder or a test in the content group fails. Add negative lifetime and unsupported-capability cases, not only the happy path. Two further rules come from the registration code: a source that needs its own main() or a helper process is registered as its own executable (the C API, the module probes and the harness programs all are), and a suite specific to a compiled-in family should guard its body on defined(CNA_RENDERER_<IDENTITY>) || defined(CNA_RENDERER_PRESENT_<IDENTITY>), as Direct2DRendererTests.cpp does: the PRESENT macro names every renderer compiled into the test build, while the plain identity macro names only the default, so guarding on it alone compiles the body to nothing for the other families of a multi-renderer build. The procedure is I need to add a regression test.

Source entry points: tests/ for shared assets and cross-module material, .github/workflows/ for the CI slices and run_gpu_tests_private.sh for the private display wrapper.