Test architecture and change recipes

CNA snapshot 009d40f5  ·  Development › Testing  ·  source links pinned to 009d40f5

A source file under tests/ does not automatically mean one standalone executable or one CI run. CNA's unit corpus is conditionally assembled into shared GoogleTest object groups; focused developer binaries reuse those objects while CTest discovers the aggregate. Renderer example programs, platform conformance, C ABI executables, configure-script gates, launcher-driven live suites and CI workflows are distinct evidence layers, and each proves something different. This page is the map of those layers for a maintainer who registers, selects or interprets tests: how CnaTests is built, what CTest names, labels and working directories mean, how GPU evidence stays honest, and what each workflow does and does not cover.

How CnaTests is assembled

cmake/UnitTests.cmake                     (when CNA_BUILD_TESTS=ON; needs vendor/googletest)
  → glob module-owned tests, renderer-owned tests and the shared top-level tests/*.cpp
  → remove standalone-main() files and sources unavailable in this configuration
  → group by physical owner: modules/<name>/tests → <name>; every renderer family → renderers;
    top-level tests/ → integration
  → cna_<group>_test_objects              [OBJECT libraries, EXCLUDE_FROM_ALL]
       ├── CnaTests                        [every group; the default target; the GoogleTest/CTest corpus]
       └── Cna<Group>Tests                [one group; EXCLUDE_FROM_ALL developer binary; links gtest_main]
  → gtest_discover_tests(CnaTests, DISCOVERY_MODE PRE_TEST, WORKING_DIRECTORY = repository root)
  → named CTest routes over CnaTests for platform, audio, input, glTF, X11 and Wayland behaviour
  → script-mode and policy gates (cmake/Tests, Harnesses, ModuleProbes) registered later

Object libraries are intentional: a static test archive can discard self-registering GoogleTest translation units when no symbol is directly referenced, so each group is an OBJECT library whose objects are added to CnaTests and to the group's focused executable. A focused executable links the owning module's dependency closure rather than the whole CNA umbrella wherever it can, so CnaMathTests and CnaCoreTests are genuinely narrower builds; the cross-module integration group, the renderers policy group and graphics-ext link the umbrella because they exercise the aggregate and need the selected renderer registry. A focused executable is not an extra CTest registration: it exists for fast local iteration, while the full CnaTests discovery remains the default corpus and is not run twice in CI. On the shared-library layout CnaTests names CNA first, so libcna.so precedes the static archives (sharp-runtime among them) that also contributed to it, or the process would carry two copies of their code and static state.

Conditional source filters matter. Implementation-specific platform tests (SDL3, SDL2, X11, Wayland, Win32, Terminal) are included only when that implementation exists in the link graph, and the code the X11 and Wayland backends share (Xkb, Freedesktop, Posix, Linux) has its tests only for those two selections. SDL2-native fixtures need separate binaries (cna_platform_sdl2_tests, cna_audio_sdl2_tests) because SDL2 and SDL3 targets advertise incompatible interface requirements. The audio suites that read SDL3_mixer's own tracks leave the corpus for every audio value except SDL3 (the XACT category suite also runs under ALSA, which has a mixer of CNA's own). Without FFmpeg the real-decoder fixtures leave but the no-backend contract tests remain; without networking the network and GamerServices suites leave; the inspector suites need CNA_BUILD_INSPECTOR; the tests of the optional sharp-runtime XML serialization component leave on Windows targets, as do suites that spawn a POSIX helper process (they and their helper executables are also excluded on Emscripten, Android and iOS). The experimental C API tests define their own main() and ABI conditions and live in the C API module, not in CnaTests. Therefore "all tests passed" means only "all tests configured and runnable in this build".

Test sources reach the code under test through cna_test_build_config: sharp-runtime and GoogleTest, compile-only include roots for the public modules, the test-support roots (modules/graphics/tests, modules/platform/tests, modules/input/tests, tools/common, tests/support, the vendored glTF header and, for one implementation test, modules/audio/src), SDL3 when SDL is configured and the tree is not SDL2-only, and, per compiled-in renderer, a CNA_RENDERER_PRESENT_<IDENTITY> definition plus that family's include roots and link-time switches. The PRESENT macros exist because only the default renderer's macro is defined project-wide: a family's suites guard their bodies on defined(CNA_RENDERER_<X>) || defined(CNA_RENDERER_PRESENT_<X>), so they compile to real tests whenever the family is compiled in and to nothing otherwise. Suites that include a family's headers are also filtered out of every tree that does not contain it (FNA3D, WebGPU, and the five EasyGL identities, which no longer includes OPENGL4, a family of its own with no easy-gl checkout).

Several tests drive a separate executable: helper processes for network loopback, the GamerServices dispatcher hang, audio without hardware, the SDL_Quit ordering hazard, mixer destruction with active voices, terminal restoration and resize, the X11 exit path, the CNB and glTF converters and cna-content. Their paths are baked into the object group as compile definitions and the group depends on the helper target, so a helper is always built before the test that spawns it. Those helpers are defined in cmake/Harnesses.cmake and the tool includes at the root.

Object groups and focused targets

The file counts are the C++ sources by owner, before per-configuration filtering (they sum with the C API, probe, Apple and fixture sources outside CnaTests to the 904 of the site method). A group with no sources in a configuration is not created.

A new module tests/ directory needs an entry for its group in CNA_TEST_FOCUSED_TARGET_<group>; the configure stops with "No focused test target name is defined for group" otherwise. CnaTests additionally links the build-time content libraries, the inspector, the network modules and cna_phone explicitly, because none of them is in the umbrella.

CTest names, labels and working directory

gtest_discover_tests registers individual CnaTests cases with DISCOVERY_MODE PRE_TEST, so enumeration happens when CTest reads the test files rather than at configure time: by that mode's design the cases are known only once CnaTests has been built (this was read from the GoogleTest module's behaviour and not observed here). The registration pins the repository root as the working directory because numerous fixture paths are relative to tests/assets. Under an X11 selection a TEST_FILTER removes the live suites (they need a private server, a window manager, uinput or a portal) from the discovered set, because dedicated entries run them once on the right server; under ALSA the discovered cases get the null playback and recording devices; under X11 and Wayland they get a nonexistent session bus so no test reaches the desktop portal of the machine it runs on; and where the test-display policy applies they carry the Wayland guard below.

Renderer-family examples and the named platform, audio, input and glTF routes register through cna_register_renderer_test (cmake/TestHelpers.cmake), which takes NAME, COMMAND and optional TIMEOUT, LABELS, ENVIRONMENT, WORKING_DIRECTORY and SKIP_REGULAR_EXPRESSION. It escapes list separators inside labels and environment values, and when no working directory is given it pins the top build directory: the historical layout in which content-loading examples resolve their assets next to the executable, kept explicit after registrations moved into their modules. The two policies differ (discovered cases run in the repository root, helper-registered entries in the build root unless they say otherwise; the glTF rungs and the Direct2D unit entry name the source root, CnaInputTests does not), and mixing them can create a test that passes by direct execution and fails only under CTest. An opt-in launcher variable, CNA_RENDERER_GPU_TEST_LAUNCHER, routes single-executable display tests through a wrapper script; only the WebGPU examples set it, so a headless run there skips rather than aborts.

The skip convention

CTest exit code 77 is the deliberate skip code. A one-shot at the end of cmake/UnitTests.cmake gives it to every test registered at root scope so far; each module-local registration file ends with cna_apply_skip_convention() for its own directory; and cmake/Tests/ModuleProbes.cmake, which registers later, reapplies it to the link-closure tests. A test skipped for an absent display, adapter or helper is not successful feature evidence, and it is worth reading a test's registered properties before assuming that a report with no failures means it rendered pixels. Note that the module link-closure gates read CMakeFiles/<target>.dir/link.txt, which only Makefile generators write, so in a Ninja tree they report skipped.

Output gates: Vulkan and OpenGL4

Some defects are reported after the last statement a test can execute (during device destruction, for instance), out of reach of an in-process assertion, so CNA gates on the process output as well. The Vulkan renderer's debug messenger prints [Vulkan Validation] <message> for every layer message at warning severity or above; any CTest whose output contains that prefix fails. The OpenGL4 renderer installs a KHR_debug callback (Debug builds, or CNA_OPENGL4_DEBUG_OUTPUT=1) that prints [OpenGL4 GL Error] <message> for every GL error, undefined-behaviour or high-severity message, and a stock program that fails to build is reported on the same line; the same shape of gate fails those tests. The gates are added to FAIL_REGULAR_EXPRESSION as list members in cna_register_renderer_test, and for the thousands of discovered cases (which are in no directory's TESTS property at configure time) through the one hook that reaches them, gtest_discover_tests(PROPERTIES …) in cmake/UnitTests.cmake, where the two patterns are joined into one alternation because a semicolon inside PROPERTIES would split the pair. Each gate is applied only where a device of that API can exist (the identity is the default or a member of the compiled set), and each has a per-test exemption list that is empty and must stay reasoned: an entry must name the exact message and why it is not CNA's. Registrations that replace FAIL_REGULAR_EXPRESSION after registering would drop the gate, so a directory that does so re-applies it last: the Vulkan examples call cna_apply_vulkan_validation_gate and the OpenGL4 examples cna_apply_opengl4_gl_error_gate.

Platform and input contracts

PlatformFactory::GetAvailable() (modules/platform/src/PlatformFactory.cpp) feeds a parameterised conformance suite: the selected implementation (SDL3, SDL2, Win32, X11 or Wayland) joins Headless in every build, and Terminal joins as well on POSIX targets, so several implementations can be held to one contract in one process. CnaPlatformTests runs the display-independent contract with a long list of suite filters, shuffled and repeated three times to catch global subsystem-refcount leakage. Window tests have a separate CnaPlatformWindowTests registration because SDL's dummy video driver must be initialised in an isolated process: a shared CnaTests process may already have selected another video driver, in which case they would skip and contribute nothing. That entry also holds the cross-module window tests (GraphicsDevicePlatformWindowTests, GameWindowPlatformTest), so CNA_PLATFORM_CTEST_BINARY, which can redirect every platform and X11 route to the focused CnaPlatformModuleTests binary for a narrow CI cell, proves a smaller contract: that binary does not contain the graphics and runtime suites. CnaPlatformXErrorHandlerTests needs a real X11 connection (its subject is what Xlib does with a failing request), so it forces the video driver and display.

The native backends split their suites by what each needs, because a bare Xvfb has no window manager (so maximise, minimise, EWMH fullscreen and focus do not happen there at all) and the translation tables need no server whatsoever:

No Wayland test reaches the desktop it runs on: the test environment points WAYLAND_DISPLAY and the session bus at nothing before any test starts. The sibling pages trace each backend's suites: X11, Wayland, Win32, Terminal and the backend overview. The Win32 backend is also validated through a separate standalone project, tools/platform/standalone_tests/CMakeLists.txt, which builds the platform module and its whole suite without the rest of CNA (cross-built with mingw-w64 and run under Wine, or natively with MSVC); it compiles the same sources under the same selection rules and does not replace the normal registrations.

The input label points to one canonical CnaInputTests filter maintained in cmake/UnitTests.cmake (CNA_INPUT_TEST_FILTER, a list of suite-name tokens), run with --gtest_shuffle --gtest_repeat=5 under a 1,200 second timeout. The comment above the run step in input-ci.yml still calls the entry “shuffled x3” and places the filter in CMakeLists.txt; both statements are stale, and the registration in cmake/UnitTests.cmake (five repeats, filter defined there) is what runs (read at 009d40f5, not executed). InputManager and related state are process-wide, so an order-dependent leak can escape a one-pass run; and if you add an input suite whose name matches none of the tokens you must extend that single filter, otherwise ctest -L input silently omits the new suite. Audio has analogous entries: CnaAudioPlatformTests (the contract, conformance and null-device suites, shuffled and repeated three times, with dummy SDL audio and null devices), the script-mode CnaAudioPlatformSelection_* cases, and for ALSA trees CnaAudioAlsaTests and a recording test that plays through ALSA's file device and measures what was captured.

Configuration and policy tests that need no GPU

A large share of the registered tests are not GoogleTest at all: they run CMake in script mode, small fixtures or Python gates. They are cheap, run anywhere and pin the rules the rest of the build depends on. Names that contain a variable in the table are generated per case.

Display isolation and meaningful GPU evidence

tools/platform/run_gpu_tests_private.sh <build-dir> -R '^Vulkan_' starts a private Weston compositor and a rootful Xwayland with DRI3, then runs CTest with DISPLAY and WAYLAND_DISPLAY pointing there; it cleans both up when done and never paints onto the developer's desktop (tools/platform/run_gpu_tests_private.sh). It refuses a build tree whose CNA_TEST_DISPLAY cache value forces a display, because CTest's configured environment would override isolation. That cache value's policy is in cmake/TestDisplayPolicy.cmake: empty by default (tests inherit the caller's display), the live desktop is honoured only with CNA_TEST_ALLOW_LIVE_DISPLAY=ON, any other value (an Xvfb, say) is honoured, and every test carries the Wayland guard WAYLAND_DISPLAY=string_append:, which turns "unset" into "empty" (connection refused) so libwayland cannot fall back to the owner's compositor while an exported value, such as the private runner's, is left alone. Xvfb often lacks DRI3 and cannot substitute for real Vulkan presentation. A launcher's code 77 means "environment unavailable", not "renderer correct". Some Wine-based differential tests have a documented private-runtime hang and need separate treatment; after a ctest run the wrapper names tests that died on a graphics-profile refusal before their assertions.

Renderer examples are selected by more than the compiled-in identity. Vulkan's example and oracle block, for instance, requires CNA_BUILD_EXAMPLES=ON, CNA_BUILD_TESTS=ON, a non-Emscripten, non-Windows target and a CNA_GRAPHICS_RENDERER STREQUAL "VULKAN" guard, and its tests are named Vulkan_*. That guard is re-pointed to each compiled identity in turn by the renderers loop in modules/renderers/CMakeLists.txt, so a multi-renderer build that includes Vulkan also enters the block even when another identity is the default; other families guard differently (some compare against the default identity), so read each family's example CMake file rather than assuming. There are also diagnostic corpus executables that emit raw pixels for a separate two-build comparison gate (cna_diag_compare diffs two dumps within a tolerance); they intentionally are not CTest entries, because CNA_GRAPHICS_RENDERER is a compile-time choice and two renderers need two builds. To claim parity, record both comparison inputs and the gate outcome. WebGPU's display tests go through a preflight and retry launcher that turns a missing display or adapter into a skip and propagates real failures unchanged (modules/renderers/webgpu/webgpu_test_launcher.sh).

What CI covers — and what it does not

Twenty workflow files define 28 jobs. The rows below say what each gates at this snapshot and where its boundary is. These are workflow roles, not a promise that every named job runs on every push: most have path filters, several ignore documentation-only changes, and the general suite was added specifically because older specialised jobs ran only filtered subsets. A green filtered job must not be read as an unfiltered regression run, and whether the current jobs are green was not checked for this page.

What no workflow runs: the whole XNA oracle corpus, the FNA differential harness, the WebGPU, SDL_gpu and OpenGL4 parity-fixture entries, and the Vulkan, Software and Direct3D 11 glTF golden sets. Renderer evidence remains per family and per configuration; see Renderer coverage is configuration-scoped.

The 32-bit arithmetic lane in detail

No workflow builds the CNA library or CnaTests for a 32-bit target. The one lane that runs code at a 32-bit width is 32bit-arithmetic-ci.yml. It checks something narrow: that size computations that multiply into std::size_t refuse to wrap when size_t is 32 bits wide. A 64-bit host cannot show this, because a uint32_t simulation there does not exercise the real code paths. The 64-bit unit tests of the audio-tag parser, for example, can only simulate a narrow size_t.

The lane builds a standalone CMake project, tools/media/arithmetic32bit, on ubuntu-latest with gcc-multilib and -m32. The project is not added to the main build, whose SDL3, SDL3_image, SDL3_mixer and FFmpeg dependencies are not provided for a 32-bit target in this toolchain. It compiles only the implementation files the checks need, into three executables registered as CTest tests:

• AudioTagParser32BitOverflowCheck: crafted ID3v2.3 frame-size and FLAC vendor-length fields that would wrap a 32-bit sum, checked against AudioTagParser.cpp (built with ContentLoadException.cpp).
• FramebufferAllocation32BitOverflowCheck: the software framebuffer planner, framebuffer_allocation_32bit_check.cpp.
• TextureAllocation32BitOverflowCheck: the software texture planner.

Each harness first asserts sizeof(std::size_t) == 4, so a mis-built 64-bit binary fails instead of passing vacuously. The framebuffer harness then checks four things:

• A 3840×2160 4× MSAA layout with stencil plans to exactly 174,182,400 bytes.
• A 16384² 4× request fails as ArithmeticOverflow before any multiplication wraps.
• An 11000² request, whose product is representable, fails as ByteBudgetExceeded.
• Generated mip storage counts against the budget.

The planners behind these checks (SoftwareFramebufferAllocation.hpp) contain no #ifdef; they are width-portable by construction. The byte budget of 512 MiB per surface (SoftwareFramebufferMaxBytes) is chosen to keep one surface practical in a 32-bit process while still admitting that 4K GDI back buffer.

The workflow triggers on pushes and pull requests to next, develop and main that touch the guarded sources, and on manual dispatch. It is evidence for these three computations only. It says nothing about a 32-bit build of CNA as a whole. The tag-parser, framebuffer and texture checks come from CNA's remediation items REMED-MEDIA-001, GDI-067 and GDI-076. Checked by reading the workflow, the standalone project and the harnesses at 009d40f5; not executed.

Change-oriented minimum and broader checks

These are representative recipes; the complete matrix is on What to test after changing X. Commands identify actual preset, label and target names but were read from CMake and the workflows, not executed against every dependency configuration.

Avoid running the entire GPU suite on a live desktop or interpreting skips as passes. For a hang, retain the CTest timeout and capture thread stacks; for a lifetime bug, reproduce both the explicit Dispose and the destructor or device-outliving paths under ASan where possible. The generated test index is a syntactic inventory of registrations; the owning CMake registration and a configured ctest -N are authoritative for a particular build.

Source tour for test ownership

1. cmake/UnitTests.cmake: follow the source filters, object groups, focused targets, GoogleTest discovery and the platform, audio, input and glTF routes.
2. cmake/TestHelpers.cmake: renderer-test registration (working directory, environment, timeout, skip pattern), the Vulkan and OpenGL4 output gates and the skip convention.
3. cmake/TestDisplayPolicy.cmake and cmake/TestDisplayPolicyRules.cmake: which display tests use and the Wayland guard.
4. cmake/Tests/ModuleProbes.cmake and cmake/Harnesses.cmake: the module probes, the policy gates and the helper executables.
5. CMakePresets.json: distinguish configure from build presets and use the exact binary tree they declare.
6. tools/platform/run_gpu_tests_private.sh: inspect the environment isolation and skip behaviour before relying on screenshots.
7. .github/workflows/: identify the job that builds and runs a test on a relevant host.