I need to add a regression test

CNA snapshot 009d40f5  ·  Development › Maintainer Handbook  ·  source links pinned to 009d40f5

A regression test in CNA has to do three jobs at once: fail for the reason the bug exists, join the right executable and CTest routes without a hand edit, and say only what it can honestly prove in the configurations that run it. This recipe is the procedure: choose the layer (GoogleTest object group, renderer example program or configuration gate), place the file so the CMake globs and filters pick it up, write a narrow failing case and its negative twin, run it through the focused target, and check that it is actually registered and not silently skipped. It ends with two worked, not executed, procedures in different modules. Everything was read from the CNA source at snapshot 009d40f5; no test was configured, built or run for this page.

Find the owner: which layer proves this?

Read first

1. UnitTests.cmake: the CNA_TEST_SOURCES glob and its filters, the group tables (CNA_TEST_GROUP_DEPENDENCY_<group>, CNA_TEST_FOCUSED_TARGET_<group>), the gtest_discover_tests call and the named routes.
2. TestHelpers.cmake: what cna_register_renderer_test sets by default and what its output gates do.
3. The neighbouring test file for the same subject; copy its fixture idiom rather than inventing one.
4. CMakePresets.json: which preset builds which focused target, so the loop you run is the loop CI's presets describe.

Place the file so it is picked up

• Location decides membership. Sources are globbed with CONFIGURE_DEPENDS from modules/*/tests/*.cpp, modules/renderers/*/tests/*.cpp, modules/renderers/common/*/tests/*.cpp and the top-level tests/*.cpp. A new .cpp under an existing module's tests/ joins that module's object group (cna_<group>_test_objects, with - in the module name becoming _) and therefore both the focused binary and CnaTests, with no CMake edit. Every renderer family's tests form the single renderers group; anything under top-level tests/ is integration.
• A brand-new module tests directory needs an entry in CNA_TEST_FOCUSED_TARGET_<group>; without it the configure stops with "No focused test target name is defined for group".
• Link closure. A focused executable links only its group's dependencies (for example CnaMathTests links cna_math). Compile-only include roots are visible everywhere, so a math test that calls graphics code compiles and then fails to link; add the implementation to the group's dependency list, as the content group does for cna_graphics_ext. The renderers, graphics_ext and integration groups link the whole umbrella.
• Files with their own main() or a helper process are not GoogleTest units. They are excluded from the glob (the C API tests, the module probes, the Apple smoke app) and registered as their own executables; a test that spawns a helper gets the helper's path through a compile definition and a dependency from the group, as the tool-spawning content tests do.
• Configuration filters. Whole files leave the corpus when the configuration cannot support them: platform-specific suites unless that backend is selected, the inspector suites without the inspector, network suites without networking, the XML-serialisation tests where the component is absent. A file that needs a family's headers must be guarded, and the guard should read defined(CNA_RENDERER_<X>) || defined(CNA_RENDERER_PRESENT_<X>): the plain macro names only the build's default renderer, so it would compile the body to nothing for the other families of a multi-renderer build. Absence from ctest -N is information.
• Naming. Files are <Subject>Tests.cpp; suites are usually <Subject>Test (RectangleTest, GraphicsAdapterTest, GraphicsProfileDrawLimitTest), with exceptions such as NpotTexture; a case name states the behaviour as a sentence (DifferentTypeSameNameThrowsContentLoadExceptionNotBadAnyCast). The discovered CTest name is Suite.Case, so a stable name also makes ctest -R '^Suite.Case$' and --gtest_filter reliable. A file header that names the plan task or defect it pins is the local convention.
• Named filters that must be extended. If your suite belongs to a route selected by a filter of name tokens, extend the filter: the input label runs one canonical CNA_INPUT_TEST_FILTER (ctest -L input, shuffled and repeated five times), the platform entries run token filters, and the glTF ladder (CnaGltfConformanceL0 to L6, Perf, Ledger, Tool) partitions every Gltf* suite into exactly one rung through CNA_GLTF_CONFORMANCE_RUNGS; a new glTF suite that fits no rung fails the run through GltfConformanceLadder.

Write the narrow failing case

1. Reproduce first, in the smallest form. Take the bug to a single call sequence with the fewest objects. A test that needs a window, a GPU or a second thread is a different test; make sure the bug really needs them.
2. Assert on the value that proves the cause, not on "did not throw". Where the expected value has a reference (XNA, an oracle file, a hand-derived table), quote its source in a comment. A golden produced by running CNA proves only that CNA did not change.
3. Make it fail for the right reason. Run it against the unfixed code and read the failure. Then check the reverse where you can: break the production code the test claims to cover and see it fail. NpotTextureTests.cpp records why: plain Texture2D::GetData is served from the framework's own CPU shadow, so those cases cannot reach a renderer's readback at all, and breaking one renderer's stride left every one of them green; only a render-target readback reaches the renderer. A test that cannot reach the code it names is worse than none.
4. Add the negative and boundary cases the bug lives beside: the wrong-type request that must throw the documented exception, the unsupported capability that must refuse, the exact limit, the empty and null inputs, disposal while bound, a second frame after a resize. Prefer several small cases to one long one, so a failure names the rule.
5. Keep fixtures self-contained. Use a unique scratch directory per test (the content tests use a ScratchContentRoot under the system temporary directory and remove it in the destructor); take shared read-only assets from tests/assets, resolved relative to the repository root because discovered cases run there; never point a destructive fixture at a shared or user location. The storage fixture, for instance, deletes the resolved storage root on teardown (storage changes).
6. Reset process-wide state you touch. Input, the renderer selection and the ambient platform are process-wide: use the reset hooks (InputManager::ResetAllForTests, GraphicsRendererSelection::ResetForTestingEXT) and leave no residue, or the test passes alone and fails in the shuffled, repeated routes.
7. Do not require a real desktop or device when a headless one is under test. Do not call SDL_Init from a unit case, sleep for wall-clock time, or depend on execution order.

Renderer example programs versus CTest

An example program is a standalone executable: it creates a real Game and device, draws, reads pixels back and returns an exit code. Single-frame pixel tests derive from CNA::Examples::PixelTestGame in PixelTestGame.hpp: override RunTest(), call ExpectPixel(label, rect, colour, tolerance) or CompareGoldenImage(label, rect, path, tolerance), and finish with return CNA::Examples::RunPixelTest<MyTest>();. Exit 0 is pass, 1 is failure, and 77 (kSkipExitCode) is "no display or GPU here", which CTest reports as skipped. A golden image is created or refreshed only by running once with CNA_UPDATE_GOLDEN=1, reviewing the PNG, and committing it; its path is relative to the process working directory, so its registration must name WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}" (the EasyGL golden entries do). Multi-frame state-machine tests hand-roll their own Game subclass.

The registration is a separate step in the family's examples/CMakeLists.txt: its own cna_<family>_test(target source) macro for linking, then cna_register_renderer_test(NAME <Family>_<Behaviour> COMMAND <target> TIMEOUT <s> LABELS "<Family>" ENVIRONMENT "SDL_VIDEODRIVER=x11;DISPLAY=${CNA_TEST_DISPLAY}"), and the file ends with cna_apply_skip_convention(). Three rules govern it.

• Working directory. Renderer-example registrations default to the top build directory; discovered GoogleTest cases and golden-image tests run in the repository root. A test that passes when run by hand and fails under CTest is usually this.
• Displays. Never hard-code DISPLAY=:0 or a wayland-0 socket; use DISPLAY=${CNA_TEST_DISPLAY} and let the launcher choose. CnaTestDisplayIsolation (check_test_display_isolation.py) fails a tree whose registrations could reach the live desktop, and a leftover empty DISPLAY= entry.
• Which tree runs it. A family's block registers only under its guard (see the EasyGL_*, Headless_* and Vulkan_* notes in Fix a renderer bug); several suites need an SDL3 target and return early without one. Output gates (Vulkan and OpenGL4) are added by the helper; a registration that overwrites FAIL_REGULAR_EXPRESSION must re-apply them last.

The unit route is fast and display-free but says nothing about pixels, and the example route is the opposite. Decide which you need for the failure, and add both only when the neutral rule and its native translation can fail independently.

Run it, and prove it ran

cmake --preset unit                                   # Debug, STUB, tests on; examples off
cmake --build --preset unit-math                      # or unit-core, unit-content, unit-graphics
./cmake-build-unit/CnaMathTests --gtest_list_tests | grep -A3 RectangleTest    # is it in the binary?
./cmake-build-unit/CnaMathTests --gtest_filter='RectangleTest.*'               # run from the repository root
cmake --build --preset unit                            # the aggregate CnaTests (a separate build)
ctest --test-dir cmake-build-unit -N -R '^RectangleTest\.'                     # discovered once CnaTests exists
ctest --test-dir cmake-build-unit -R '^RectangleTest\.' --output-on-failure

The unit configure preset writes cmake-build-unit; the four focused build presets (unit-core, unit-math, unit-content, unit-graphics) build one focused target each, and other groups are built by target name (cmake --build cmake-build-unit --target CnaStorageTests). The discovered CTest cases appear only after CnaTests itself is built (the discovery mode is PRE_TEST; read from the GoogleTest module, not observed here). Then check the result the way the evidence supports it: the case appears in --gtest_list_tests and ctest -N, it ran rather than skipped (read the skip count), it failed before the fix and passes after, and, for anything with process-wide state or ordering risk, it survives --gtest_shuffle --gtest_repeat=5. For a case that needs a rendering renderer, say which one and on which host it ran, and reach for the private display wrapper for anything that opens a window (private displays and host caveats).

Worked procedures (not executed)

A. A value-type regression in the math module

Suppose a report says Rectangle.Intersect returns a wrong origin for disjoint inputs. The steps, without claiming the bug exists:

1. Open RectangleTests.cpp, which already holds the RectangleTest suite with section comments and cases such as IntersectReturnsOverlapRegion and IntersectOfDisjointReturnsEmpty. Read what those assert first; a regression case pins what they do not, for example a disjoint pair with non-zero origins whose whole result must equal Rectangle::Empty, named as a sentence such as IntersectOfDisjointRectanglesWithNonZeroOriginsIsTheEmptyRectangle. Quote the expected value's source (XNA's documented result or an oracle value) in a comment. No CMake edit is needed: the file is already in the math group.
2. Build and run only that case: cmake --build --preset unit-math, then ./cmake-build-unit/CnaMathTests --gtest_filter='RectangleTest.Intersect*'. Confirm it fails against the unfixed code and read why.
3. Add the neighbours the bug lives beside: touching edges, one rectangle inside another, negative sizes, and the out-parameter overload separately from the value-returning one (CNA's rules require each overload to be covered).
4. Fix the implementation in modules/math/src/Rectangle.cpp, re-run the focused binary, then build CnaTests and run ctest -R '^RectangleTest\.' to prove the discovered entries agree.
5. Consider the blast radius before finishing: rectangles are used by graphics and input, so run cmake --build --preset unit-graphics and its filter for scissor and viewport cases, and check Change public XNA behavior if the fix moves a documented XNA result.

B. A cache-key regression in the content module

Suppose a report says loading the same logical name as two different types throws std::bad_any_cast instead of a ContentLoadException. That behaviour is already pinned by CnjAssetCacheTypeSafetyTests.cpp, which is the model to copy for a new neighbour.

1. Place the file under modules/content/tests/Microsoft/Xna/Framework/Content/, so the namespace path is mirrored and the file joins the content group (focused target CnaContentTests, preset unit-content). Reuse the file's idiom: a ScratchContentRoot with a unique directory, a WriteFile helper, and ContentManager cm(nullptr, root.path().string()).
2. Write the smallest failing sequence: register two loaders for two types under different .cnj type strings, load the name as the first type, then request it as the second and EXPECT_THROW(..., ContentLoadException). Add the control case that the same type and name still returns the cached instance, so a fix that simply disables caching fails.
3. Run cmake --build --preset unit-content and ./cmake-build-unit/CnaContentTests --gtest_filter='CnjAssetCacheTypeSafetyTest.*' from the repository root; several texture-reader suites in this group are renderer-gated and skip under STUB, so read the skip count rather than the exit colour.
4. If the change touches the tier order or the loose-file resolver, also run the resolver-order and cache tests named on Modify ContentManager, and the XNB and CNB conformance suites.

C. A pixel program in a renderer family (outline)

For a bug that only shows in pixels, put the shared logic in a PixelTestGame source under modules/graphics/examples/ if it is renderer-neutral, or beside the family if it is not; add a cna_<family>_test line and a cna_register_renderer_test entry with a 30 to 60 second timeout, the family label and the display environment above; ensure the file's guard matches the way the family's other tests are guarded; then run it under the private display wrapper with ctest -N -R '^<Family>_' first to confirm it registered. If the behaviour is neutral, add it to CNA_PARITY_FIXTURES instead and every parity-registering renderer gains it.

Keep the blast radius small

• Every new source in a group is compiled into both the focused binary and CnaTests; a compile or link failure in your file stops both. Keep includes narrow and avoid new heavy dependencies.
• A test that mutates process-wide state or writes outside its scratch directory can break unrelated suites in the same process (the aggregate runs one process; discovered cases run one process each, which hides such leaks until the shuffled routes run).
• A registration property that is too broad (a label, an environment variable, a working directory) changes how other tests are selected; prefer the helper's defaults and set only what the test needs.
• A filter edit (input tokens, glTF rungs, a platform token list) changes what a named route runs for everyone; extend it, and check that the route still contains what it contained.
• Do not turn a skip into a pass, and do not add a renderer-conditional assertion that silently weakens the test on the renderers that cannot satisfy it; skip explicitly with the gate macro so the coverage drop is visible.

Review checklist

• Does the test fail before the fix, for the reason in the report, and did anyone try breaking the production code to see it fail?
• Is it in the module and namespace path that mirrors the code, and is it visible in --gtest_list_tests and ctest -N of a tree that should run it?
• Are negative, boundary and lifetime cases present, not only the happy path?
• Does the fixture avoid shared locations, real desktops and process-wide residue?
• For an example program: exit codes, skip convention, working directory, display environment and family guard are right; a golden image was regenerated only deliberately.
• Does the description state the configuration, the renderer, the host, and the skip count, and say what was not run?