What to test after changing X

CNA snapshot c1c316b9  ·  Development › Human Takeover  ·  source links pinned to c1c316b9

Choose tests from the changed contract and host, not from the file extension. A CNA build excludes sources and tests according to its CMake options, so a green run in one configuration says nothing about the others. For every result record the configured platform, audio and renderer axes, the exact test name, the skip count, whether a real native device was used (and which kind), and which other implementations remained untested. Everything on this page was read from the source and CMake at snapshot c1c316b9; none of the commands was run for it.

A repeatable local baseline

# From a CNA checkout with prerequisites and submodules installed:
cmake --preset unit
cmake --build --preset unit
./cmake-build-unit/CnaTests
ctest --test-dir cmake-build-unit -N

The unit configure and build presets select Debug, the STUB renderer, tests on, and examples, C API, networking, FFmpeg video and Draco off (CMakePresets.json). The direct binary runs the aggregate GoogleTest corpus in one process; ctest -N shows what that configured tree registers. The presets define no test presets, so CTest is driven with ctest --test-dir. These commands are source-verified from the preset file, not executed across host types in this documentation pass. Because the STUB renderer draws nothing, a unit run cannot establish rendering behaviour; the oracle choices are in the graphics section below.

For the desktop tests preset, the preset's own description recommends running the CnaTests binary directly, with SDL_AUDIODRIVER=dummy, because ctest runs each discovered case as its own short-lived process, which reports every unbuilt display-dependent graphics smoke executable as failed and races tests that share hard-coded fixture paths. Use CTest for a deliberately selected registered test, label or renderer oracle, and inspect its registration first. A test that exits with code 77 is reported as skipped, not passed (see below); a GTEST_SKIP() inside a GoogleTest binary prints [ SKIPPED ]. Read the skip counts, not the exit colour.

The presets that matter for validation, with the full table in the Building CNA presets section:

How the test targets are assembled

Read UnitTests.cmake before assuming which tests a configuration contains.

• Object groups. Test sources are globbed from modules/*/tests (every renderer family's tests form one renderers group; anything else forms integration) and each group is compiled once into an object library cna_<group>_test_objects. The same objects feed both the aggregate and the group's focused executable.
• Aggregate and focused targets. CnaTests links every group; the focused executables (CnaCoreTests, CnaMathTests, CnaGraphicsTests, CnaContentTests, CnaRuntimeTests, CnaStorageTests, CnaDevicesTests, CnaDevicesExtTests, CnaAudioTests, CnaInputModuleTests, CnaPlatformModuleTests and the others) are EXCLUDE_FROM_ALL iteration targets. By the file's own comment they are not additional CTest registrations, so the aggregate stays the single default suite in CI.
• Two kinds of CTest entry. gtest_discover_tests(... DISCOVERY_MODE PRE_TEST) registers every case in CnaTests as its own test at test time, with the source directory as working directory so fixture assets resolve; the discovered cases exist only once CnaTests is built. Separate explicit entries run a filtered subset with their own environment, timeout and labels: CnaInputTests (label input, shuffled and repeated), CnaAudioPlatformTests and CnaAudioAlsaTests (audio, platform), the CnaPlatform* entries (among them CnaPlatformSdl3X11Tests and CnaPlatformSdl3WaylandTests), the CnaGltfConformance* ladder, the CnaXnb* conformance entries, the configuration-policy entries (label configuration), the CApi_* tests and each renderer family's example tests (Software_*, Vulkan_*, Headless_*, Stub_Smoke and so on, labelled by family).
• Source filters. Whole test files are dropped by the configuration: the Sdl3 platform tests exist only when SDL3 is the selected CNA_PLATFORM, the Terminal tests only off Windows, the Inspector tests only with the Inspector, the shutdown-ordering suite not on Windows, Emscripten, Android or iOS, and so on. Absence from ctest -N is information; confirm it before reporting a suite as run.

The deeper explanation, with the CI roles, is on Test architecture and change recipes; the generated Test target index lists the declarations.

Minimum loop and broader gate

CTest reports a registered test that exits with code 77 as skipped: UnitTests.cmake applies SKIP_RETURN_CODE 77 to every test registered in its scope and module-local files apply the same convention, so a launcher or fixture that finds no display, GPU or server can skip without failing.

Private displays, GPU hosts and native-host caveats

A test that opens a window must not open it on the developer's desktop, and the evidence it produces is only as good as the host it ran on.

• Display policy. TestDisplayPolicy.cmake sets CNA_TEST_DISPLAY empty by default, so a test inherits the caller's DISPLAY and where windows appear is decided by how the run is launched. A value naming the live display (:0) is honoured only with CNA_TEST_ALLOW_LIVE_DISPLAY=ON and is otherwise reset to empty; on non-Windows hosts every test also carries a guard that stops it connecting to the live Wayland compositor unless an explicit WAYLAND_DISPLAY is exported.
• Private GPU display. run_gpu_tests_private.sh runs a build's CTest suite (or, with --exec, one command) on a private headless Weston with a private rootful Xwayland that has DRI3 and therefore presents Vulkan on the real GPU, which Xvfb cannot. It refuses a build tree whose cache forces CNA_TEST_DISPLAY (exit 2), exits 77 when Xwayland or a compositor is missing, warns when the private display has no DRI3, and lists failures that died on a graphics-profile refusal before their first check, which are test defects and not renderer results. The Wine-based XNA interop tests hang in its private runtime directory and must be excluded or run on their own. It does not build the configuration or prove that the expected oracles were registered.
• SDL3 on real window systems. CnaPlatformSdl3X11Tests runs the SDL3 window suites with SDL_VIDEODRIVER=x11 on a private Xvfb started by tools/platform/x11_test_server.sh, and CnaPlatformSdl3WaylandTests with SDL_VIDEODRIVER=wayland on a private headless Weston started by tools/platform/wayland_test_server.sh; each exits 77 where its server is missing. A result must name the server it came from: an Xvfb or a headless Weston proves protocol behaviour only (no GPU, compositor effects or real input), Xwayland is not native Xorg, and a software renderer must be labelled as one.
• Windows. The SDL3 platform’s windows video driver is exercised by the native MSVC job in platform-ci.yml, which runs on manual dispatch only; per-monitor DPI, real keyboard and IME stacks and clipboard interoperability on a physical desktop remain things only a person on Windows hardware establishes.
• Other hosts. Metal, Apple, Android, browser and hardware-sensor behaviour each need their own host; report them as not run unless they were.

Device and host-service changes

For sensor registration, callback or shutdown work (Devices and sensor lifetime), build CnaDevicesTests or the aggregate CnaTests in a configuration with CNA_DEVICES=ON (the option defaults to OFF). Start with an exact Accelerometer/Gyroscope or Compass/Motion interleaving test and include, when lifetime changed, NoDispatchAfterStop, same-callback Dispose (DisposeFromWithinOwnCallbackDoesNotDeadlock), other-thread Dispose (ConcurrentDisposeFromMultipleThreadsNeverCorruptsInstanceCount) and the platform session-close barrier (SensorSessionContractTests.CallbackMayDestroyItsOwnSession, PlatformSensorMigrationTests.HandlerMayDisposeItsOwnSensorWithoutDeadlock). Use devices-tsan for races and devices-asan for use-after-free; the devices-tsan preset's description records one pre-existing race in Sharp Runtime's TimeSpan copy constructor, but that text describes an older sharp-runtime: the counters became atomic in sharp-runtime 9c2cb0ae (2026-07-07), which every current sharp-runtime branch contains, so with a current checkout classify any such report instead of suppressing it silently, and treat any new one as worth investigating. The separate shutdown-ordering harness (cna_devices_shutdown_ordering_harness, spawned by DevicesShutdownOrderingTest.*) is required for late-singleton and SDL teardown work; it calls the real SDL_Quit(), so it exists only where SDL3 is configured and is excluded on Windows, Emscripten, Android and iOS. A live Android sensor is needed to establish native queue and orientation behaviour: CNA's own event-contract document leaves the Android backend unverified beyond its fake-backend seam.

For camera, dialog or tray work (Device extensions and host services), the focused CnaDevicesExtTests target covers fake-service parameter translation, unavailable behaviour and owned session and icon destruction; run it only after confirming CNA_DEVICES=ON, then test the selected platform service and the native host. Camera frame upload (Camera::TryAcquireFrame fills a Texture2D) additionally needs a graphics texture path on a real renderer. File-dialog continuation changes need a case for the rejection path: when the platform has no dialog service, FileDialog now delivers an empty result to the callback (FileDialogWithoutAServiceTest.TheCallbackStillFiresSoACallerIsNotLeftWaiting), but when the platform service itself throws, the callback has already been moved and the caller sees a dialog that never opened (FileDialog.cpp).

Verified mechanisms, configuration caveats

cmake --build --preset unit-graphics builds CnaGraphicsTests; the unit preset's aggregate CnaTests must be built separately to run CTest-discovered cases. For a specific test, use the focused binary's --gtest_filter='Suite.Case' or ctest --test-dir cmake-build-unit -R '^Suite.Case$' --output-on-failure after the aggregate exists. Confirm CTest's exact names with -N. Use ctest --test-dir <build-dir> -L input for the intentionally shuffled and repeated input route: CNA_INPUT_TEST_FILTER in UnitTests.cmake is the single source of truth for that label (input-build-and-test.md), so when adding an input suite whose name matches none of its tokens, extend that filter or the label silently excludes it. The route is order-independence evidence for process-wide input state, and it leaves the video driver to the runner because the mouse-cursor tests need real cursors.

For native Vulkan tests use tools/platform/run_gpu_tests_private.sh <build-dir> -R '^Vulkan_' as described above. For sanitizers, devices-asan, devices-tsan and devices-ubsan are real presets with CNA_DEVICES=ON; they are not universal sanitizer matrices. If a different backend or subsystem is affected, configure CNA_SANITIZE (a comma-separated list such as address,undefined; AddressSanitizer and ThreadSanitizer cannot be combined; Emscripten and MSVC are refused) in its own build with CNA_SANITIZE_OPTIMIZATION as needed, and verify that instrumentation reached the target (BuildPerformance.cmake). The test architecture page explains object groups, fixture working directories, labels and CI roles; CMake option index locates the options.

Report evidence honestly

A useful validation note says: “CNA commit X; CNA_PLATFORM=…, CNA_AUDIO_PLATFORM=…, default and compiled renderers; focused case N passed; aggregate M passed; named GPU oracle skipped because no adapter; ASan run on build Y; native Windows not run.” A weak note says “tests pass.” Separate what was checked by reading (source verified), what was built, what was run (with the host kind: private Xvfb, Xwayland, real GPU, Wine, native), what was compared against an oracle and what was observed on hardware; never claim a runtime result that was not executed.

The generated Test target index reports the current number of test declarations; that inventory is a source listing of registrations, not a count of tests guaranteed to run in any one build. Find the target there, then follow its CMake registration and inspect the configured CTest and test-binary output. The site's own statement of what CNA's CI gates, and what it does not, is Verification and known issues: CI.

Storage and filesystem changes

Build the focused CnaStorageTests target and run StorageDeviceDeleteContainerTest.* plus the exception cases, but first verify what its fixture will delete: SetUp calls StorageDevice::SetAppNameEXT("CnaTestsContent002StorageDevice") and TearDown runs remove_all on the resolved storage root. That root comes from the same environment chain as production (XDG_DATA_HOME, LOCALAPPDATA, HOME, then the working directory), so point the relevant variable at a disposable directory, or prefer a disposable isolated environment (storage case study: the oracle). For PathContainment changes, run its core tests and every relevant caller, especially recursive delete, file creation and symlink cases, and keep an outside sentinel; the symlink case skips with GTEST_SKIP where directory symlinks cannot be created, so check the skip count. For app-identity changes, assert the Sharp Runtime .cna_isolated_storage override (AppNameAlsoScopesSharpRuntimeIsolatedStorage) and repeat on Windows with non-ASCII paths, since CNA's records report Windows-only containment failures. The existing tests do not prove a race-proof sandbox or concurrent root switching; record those as unverified rather than as a green test.

Choose the graphics oracle by the failure you could have caused

A fast unit run is not a general renderer sign-off. The snapshot contains several implementations with deliberately different observable outputs. Use the smallest one that can fail for the defect you fear, then add a second implementation or host to challenge the abstraction boundary.

Where a change could alter what a user sees compared with real XNA, the reference-image corpus is a further oracle with its own limits: see Using the XNA oracle as evidence. Neither it nor any row above is established by this page; the tests named here exist in the source and were not executed.

Worked decision. If you change a neutral render-target readback guard, STUB can exercise some argument validation and HEADLESS can prove that a non-rasterizer refuses fabricated data, but neither can prove target color. Run a Software target and readback pixel fixture (ctest -N -R '^Software_' lists the family's entries), then the relevant Vulkan, SDL_gpu or EasyGL native oracle, and inspect ctest -N and the fixture's renderer-specific branches so that “pass” does not secretly mean “pixel check skipped”. If you change a descriptor or no-window construction path, run both Headless_Smoke and Stub_Smoke where configured, then a real-window renderer. Their registration is conditional: the HEADLESS fixtures register only when HEADLESS is the default renderer identity, and Stub_Smoke only when the STUB family is in scope, so confirm with ctest -N. STUB's fixture links SDL3 unconditionally even though the STUB renderer does not, so an SDL-free build must not be reported as having run that test; the HEADLESS fixtures link SDL3 only when it is configured.