Tutorial 160: Running Focused Tests, Bounded Runs and Parity Fixtures

CNA Tutorials  ·  CNA snapshot 009d40f5

Before you start — Tutorial 99 covers GoogleTest for your own game logic and Tutorial 125 explains how CNA verifies rendering. This tutorial is about running CNA's own tests sensibly: one module at a time, in bounded memory, without windows appearing on your desktop, and including the shared cross-renderer parity fixtures. You need a CNA checkout on branch next, its sibling sharp-runtime checkout on its next branch, and a Linux machine (the scripts below are POSIX shell).

1. Configure one build tree per renderer

CNA selects its renderer at configure time, and several test groups only exist for the renderer you configured (the parity fixtures in step 5 are the clearest example). Keep one build directory per renderer you care about.

git clone -b next https://github.com/libcna/cna.git
git clone -b next https://github.com/libcna/sharp-runtime.git   # must be its 'next' branch
git clone https://github.com/libcna/easy-gl.git                 # the five GL-profile renderers
git clone https://github.com/libcna/meta-gl.git                 # needed by easy-gl

cd cna
git submodule update --init          # non-recursive is correct; fetches SDL, googletest, Draco

cmake -S . -B build -DCNA_GRAPHICS_RENDERER=OPENGLES3

CNA_BUILD_TESTS and CNA_BUILD_EXAMPLES both default to ON, so this tree already contains the tests. CNA also ships a tests preset (cmake --preset tests: OPENGLES3, Debug, its own cmake-build-tests directory). FFmpeg is optional and only matters for video tests.

At the end of configure, CMake prints one status line that is worth reading: CNA: focused unit-test targets: .... It lists exactly which focused targets this configuration defines.

2. Build and run one module’s tests

The aggregate CnaTests executable links every module’s tests into one binary. Since alpha.1 each module also has its own focused executable, built from the same object files, so you can iterate on one module without linking everything:

cmake --build build --target CnaMathTests

# run from the repository root
./build/CnaMathTests
./build/CnaMathTests --gtest_filter='CurveLoopTypeTest.*'
./build/CnaMathTests --gtest_list_tests | head

Things to know before you rely on them:

• They are developer iteration targets. CMake marks them EXCLUDE_FROM_ALL and does not register them with CTest, so a plain cmake --build build does not build them and ctest -N lists only what the aggregate CnaTests discovers. That is deliberate: CI must not run the same tests twice.
• Run from the repository root. CNA’s own CTest registration of CnaTests uses the source directory as the working directory, and tests that read fixtures resolve them relative to it. The binaries land in the build directory root on single-configuration generators such as Ninja and Makefiles.
• The preset’s own note says that running the CnaTests binary directly (with SDL_AUDIODRIVER=dummy for the audio suites) is the more reliable way to run the whole suite than ctest, which starts one short-lived process per test case. Use ctest -N to see what is registered and the binary to run it.
• A focused run says nothing about the other modules, and a static test count says nothing about how many of these pass. See what the test counts actually count.

3. Keep GPU tests off your desktop

Roughly a thousand renderer test registrations create a real window. Before the display policy existed, they defaulted to your live desktop (:0), and CNA’s own plans record a plain ctest opening hundreds of windows on the developer’s screen. CNA’s policy now, from cmake/TestDisplayPolicy.cmake:

• CNA_TEST_DISPLAY is empty by default. Tests then carry no DISPLAY of their own and inherit the caller’s, so how you launch decides where windows appear.
• Naming the live desktop is opt-in: -DCNA_TEST_DISPLAY=:0 is honoured only together with -DCNA_TEST_ALLOW_LIVE_DISPLAY=ON. Without it, CMake resets the value to empty and says so. Any other value (for example :99, an Xvfb) is honoured as before.
• No test falls back to your Wayland compositor either: with WAYLAND_DISPLAY unset the tests carry an empty value, so libwayland gets connection refused instead of connecting to your session. An exported value is left alone.
• The CTest CnaTestDisplayIsolation checks a configured tree’s registrations against this policy.
• On CMake older than 3.28 the per-test DISPLAY cannot be cleared: with an empty CNA_TEST_DISPLAY the tests would have no display, so set it explicitly there.

What the private runner builds around your command, per its own header: a headless Weston (GL renderer) in a private runtime directory with no session bus, plus a rootful Xwayland on a display number the X server picks itself, with DRI3 — so Vulkan can present on the real GPU, which Xvfb cannot. Nothing appears on your desktop, and everything it started is gone when it returns. It exits 77 (CTest’s skip code) if Weston or Xwayland is missing. Two documented limits: it refuses a build tree configured with a non-empty CNA_TEST_DISPLAY (reconfigure with -DCNA_TEST_DISPLAY=, because CTest would override the private display), and the Wine-based XNA interop tests hang inside its private runtime directory, so exclude them with -E 'XnaPipelineGenuineRuntime|XnaDifferentialBuildTest'. After a run, tools/platform/profile_dead_tests.py names failures that died on a graphics-profile refusal before their assertions — a test defect, not a renderer result.

4. Run a big binary in bounded memory

A CNA test builds a GraphicsDevice, and on some renderers a device is not cheap to have had: memory is not returned when it is destroyed, so it accumulates across the tests of one process. CNA’s runner header records the measurement that forced this — over the same 200 tests, peak memory of about 1,115 MB on WEBGPU against 167 MB on VULKAN — and that the whole CnaTests (10,564 tests on WEBGPU) could not be run there at all during the project’s WebGPU closeout, because every attempt at four and at twelve shards was killed by the machine’s low-memory reaper (481 tests in one WEBGPU process is killed on a 30 GB machine; 240 is not). The lever that works is how many tests share one process, not how many processes run at once.

# 200 tests per process by default, one process at a time
tools/tests/run_gtest_bounded.sh ./build/CnaGraphicsTests

# smaller shards, two at a time, logs kept in ./bounded-out
tools/tests/run_gtest_bounded.sh --tests-per-shard 100 --max-parallel 2 \
    --out bounded-out ./build/CnaGraphicsTests

# only some tests; then pick up after an interruption without redoing finished shards
tools/tests/run_gtest_bounded.sh --filter 'Vector2*' --out bounded-out ./build/CnaMathTests
tools/tests/run_gtest_bounded.sh --resume --out bounded-out ./build/CnaMathTests

# compose it with the private compositor so one compositor serves every shard
tools/platform/run_gpu_tests_private.sh --exec \
    tools/tests/run_gtest_bounded.sh --max-parallel 2 ./build/CnaTests

How it works: it lists the tests (--gtest_list_tests), derives the shard count, and starts the binary once per shard with GoogleTest’s own GTEST_TOTAL_SHARDS and GTEST_SHARD_INDEX variables and a private TMPDIR. Each shard writes shard-N.log, shard-N.xml and shard-N.status into the output directory. It reads no renderer setting, so the environment passes straight through: in a multi-renderer build, CNA_GRAPHICS_RENDERER=VULKAN tools/tests/run_gtest_bounded.sh ... shards the Vulkan run.

How to read the result, from its source:

• The last line is run_gtest_bounded: P passed / F failed / S skipped (N counted across K shard(s)), followed by one FAILED line per failing test.
• A shard killed by a signal — what the low-memory reaper does — is reported as KILLED, the counts are declared a lower bound, not a result, and the run fails. A runner that reported a partial pass as success would be the defect it exists to replace.
• Counts come from the XML and are cross-checked against GoogleTest’s text summary; if the two disagree the runner exits 3 and refuses to report a result.
• Exit 0 means every shard exited 0 and none was killed. Exit 2 means a usage error, an unusable binary, or a refusal: if DISPLAY is :0 or WAYLAND_DISPLAY is wayland-0 it declines to run (override with --allow-live-display only if a human is watching on purpose).

It creates no display of its own. Use it inside the private compositor, inside xvfb-run, or for suites that need no display.

5. Run the cross-renderer parity fixtures

A parity fixture is one renderer-neutral .cpp under modules/graphics/examples/parity/ that issues only public XNA calls and states its expected result in its own assertions. There are 32, listed once in ParityFixtures.cmake:

backbuffer_msaa, vertex_semantics, lit_untextured, lit_vertex_color, unlit_position_color, dual_texture_uv1, multi_stream_split, render_target_mip, hdr_render_target, compressed_cube, basic_effect_light_terms, basic_effect_vertex_color, basic_effect_alpha_scale, alpha_test_sweep, alpha_test_sources, dual_texture_terms, env_map_terms, skinned_terms, sprite_geometry, sprite_state, sprite_font, sampler_filters, blend_states, depth_states, stencil_states, stencil_compare, rasterizer_viewport, fill_mode_wireframe, sampler_max_mip_level, sampler_lod_bias, sprite_sampler_state and instanced_draw.

Four renderers call cna_register_parity_fixtures(), and each gets one executable and one CTest per fixture — but only in a tree configured for that renderer:

Every other renderer has no parity fixtures registered. Run them through CTest, or run one executable directly:

ctest --test-dir build -N -L Parity                       # what is registered in THIS tree
ctest --test-dir build -L Parity --output-on-failure
ctest --test-dir build -R '^EasyGL_Parity_alpha_test_sweep$' --output-on-failure

# inside the private compositor, so nothing opens on your desktop
tools/platform/run_gpu_tests_private.sh build -L Parity

# one fixture directly (the CTest registration sets SDL_VIDEODRIVER=x11 for it)
SDL_VIDEODRIVER=x11 ./build/cna_parity_alpha_test_sweep_easygl

A fixture prints one [PASS] or [FAIL] line per assertion and folds the result into its exit code. Its assertions are written to be renderer-independent: they sample interior regions rather than boundary pixels, compare region averages, prefer relational checks (this region is brighter than that one) to absolute colours, and use a small tolerance for shaded output and zero for a flat unshaded colour. The header states the rule that keeps the fixtures honest: a large difference is never papered over with a wide tolerance — an assertion that needs more than the shading tolerance is a finding, and belongs in the plan.

The SDL_GPU registrations use a 180-second timeout and treat validation-layer output (Validation Error, Validation Warning, VUID-) as failure.

6. Optionally compare two renderers’ frames

The fixtures’ own assertions are the primary oracle. As a second layer, an executable also accepts an output path and writes its whole back buffer as raw RGBA8, in the format the standalone comparator cna_diag_compare reads. Two scripts drive that; both need two separate builds, because the renderer is a compile-time choice:

cmake -S . -B build-gl     -DCNA_GRAPHICS_RENDERER=OPENGL33
cmake -S . -B build-webgpu -DCNA_GRAPHICS_RENDERER=WEBGPU
cmake --build build-gl --target cna_diag_compare cna_parity_sprite_state_easygl
cmake --build build-webgpu --target cna_parity_sprite_state_webgpu

# one fixture: both renderers' own assertions must pass AND the frames must match (tolerance 2)
CNA_PARITY_DISPLAY="$DISPLAY" scripts/run-parity-fixture.sh sprite_state build-gl build-webgpu

# the whole corpus, with a per-fixture frame policy; the comparison renderer is 'webgpu' or 'sdlgpu'
scripts/run-parity-corpus.sh build-gl build-webgpu webgpu

• run-parity-fixture.sh takes <fixture> [easygl-build] [webgpu-build] [tolerance]. Its defaults are build directories named cmake-build-debug (configured OPENGL33) and cmake-build-webgpu, and its display defaults to :131 — override it with CNA_PARITY_DISPLAY, as above. Tolerance 2 is rounding, not licence; raise it only for a fixture whose own header documents why.
• run-parity-corpus.sh gives every fixture exactly one policy, written in the script: strict (identical frames at tolerance 2, the default for a new fixture), allow:N:reason (at most N pixels may differ, with the reason) or internal:reason (frames are not compared, because the fixture’s own assertions are its whole contract). A fixture whose binary is missing is reported NOT BUILT and counts as a failure.

What this does and does not tell you

• The parity oracle is not real XNA. A passing fixture means that renderer satisfied that fixture’s own assertions. Comparison against the genuine XNA runtime is the separate oracle corpus in Tutorial 161.
• Four renderers, not 25. The fixtures are registered for EasyGL, OpenGL4, SDL_GPU and WebGPU only.
• Nothing here is in CI. The WebGPU, SDL_GPU and OpenGL4 fixture CTests run in no workflow. Only the EasyGL registrations can be reached through the general workflow’s unfiltered ctest on OPENGLES3, and whether that job is green was not verified.
• A registered test is not a passing test. This tutorial gives you the means to run and read them, not a result.

Where to go next

• Tutorial 161: Diffing a Renderer Against the XNA Oracle — the real-XNA comparison.
• Tutorial 125: Headless and Pixel Testing, and the XNA Oracle Corpus — how the pixel assertions themselves are written.
• Tutorial 99: Unit Testing CNA Game Logic — GoogleTest for your own game.
• Verification & Known Issues — the full inventory of verification infrastructure and its boundaries.