CNA-BUG-036: run-all-renderer-smoke-tests.sh ignores the build's exit status, and its '-- -k' build flag aborts every Ninja build, so smoke verdicts can come from stale binaries

CNA snapshot 009d40f5  ·  Known Issues › Current bugs  ·  source links pinned to 009d40f5

The orchestrator runs cmake --build <dir> -j4 -- -k and discards the result; -k without a count is a make option that Ninja rejects, so in a Ninja tree nothing is rebuilt and ctest runs whatever executables the tree already holds.

Identifier

CNA-BUG-036

Category

Bug

Subsystem

Testing & evidence

Status

Narrowed (partially fixed; describes only what survives)

Verified against

CNA 009d40f5 (009d40f5dd085c4e674d3479675fac84b12b3e0a)

Severity

Low (a triage suggestion, not a project priority)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

None

Affected contract

scripts/run-all-renderer-smoke-tests.sh (default, --tier and --multi modes): a renderer that configures and builds but fails its smoke test fails the run

Expected behaviour

The script's header says a renderer "that DOES configure/build but whose smoke test fails makes the overall run fail", so that it "composes cleanly as a CI step", and docs/runtime-renderer-selection.md recommends its --multi "HEADLESS;SOFTWARE;STUB" mode. A verdict only means something if the smoke tests run against binaries built from the current tree, and a build that did not happen must not be reported as a pass.

Actual behaviour at TARGET

Both modes build with cmake --build "${dir}" -j4 -- -k and never read the exit status (the comment explains -k as a way past an unrelated demo build error; "ctest is the real signal"). CMake appends the arguments after -- to the native tool unchanged. GNU make reads -k as keep-going; Ninja's -k takes a count, so ninja -j 4 -k stops with "option requires an argument -- 'k'" before building anything. The script configures a missing tree with CMake's default generator, but it reuses existing trees whatever their generator, and the --multi tree, cmake-build-multi, is exactly the binaryDir of the multi-renderer preset in CMakePresets.json, which inherits base-ninja and names the same HEADLESS;SOFTWARE;STUB set.

ctest then runs regardless. A smoke executable that was never built is reported Not Run, so the renderer shows FAIL with a misleading cause; one left from an earlier build runs the old code, so the renderer can show PASS for a tree that did not build. Under make the same happens to a smoke target whose own compile fails, since -k leaves the previous executable in place.

Source locations

• scripts/run-all-renderer-smoke-tests.sh — the build step of the --multi branch and of the per-renderer loop: cmake --build ... -j4 -- -k, exit status unused
• CMakePresets.json — multi-renderer preset: binaryDir cmake-build-multi, inherits base-ninja
• docs/runtime-renderer-selection.md — Smoke-testing several renderers from one build: the documented --multi command

Evidence

Read at 009d40f5. Executed: only Ninja's argument parsing, with Ninja 1.12.1 on the review host (ninja -k with no count prints "option requires an argument -- 'k'" and its usage text). The script was not run and no CNA tree was configured. What changed since this was first reported, and is not part of this entry: the renderer set is no longer a fixed four-identity list (--tier routine|full, where full covers every family that registers a GraphicsSmoke test), the multi-renderer mode was added, its label lookup refuses to count "no matching test" as a pass, and a configure failure is deliberately reported as SKIPPED ("not available on this machine"), which is the script's stated contract. No workflow calls the script; it is a manual maintainer tool.

Focused reproduction

Illustrative; not run for this entry.

cmake --preset multi-renderer        # Ninja tree in cmake-build-multi
scripts/run-all-renderer-smoke-tests.sh --multi "HEADLESS;SOFTWARE;STUB"
# build step: ninja: option requires an argument -- 'k'
# the script continues and reports per-renderer results from whatever the tree contains

Current tests

None: the script has no self-test and no workflow invokes it. multi-renderer-ci.yml builds and tests the same HEADLESS;SOFTWARE;STUB set with its own steps and does check its build.

Regression test

Treat a failed build as a result (record the renderer as FAIL (build) and skip its ctest, or exit non-zero), and pass keep-going in the generator's own spelling (-k for make, -k 0 for Ninja) or drop it. A small shell test can point the script at a tree whose build is made to fail and assert a non-zero exit.

Blast radius

Local renderer smoke runs by maintainers, in every mode (existing cmake-build-debug, cmake-build-vulkan, cmake-build-sdl, cmake-build-webgpu trees that happen to be Ninja trees, --tier, and --multi). CI and shipped code are unaffected.

Workaround

Build the tree first and check that the build succeeded (cmake --build cmake-build-multi), then run the script; or let the script create a fresh tree with a make generator.