CNA-BUG-193: general-tests-ci.yml still allowlists EasyGL_GraphicsDevice_ReferenceStencil as a known failure after REMED-GFX-236 fixed it, so a regression of that test cannot turn the job red

CNA snapshot 009d40f5  ·  Known Issues › Current bugs  ·  source links pinned to 009d40f5

The only unfiltered CI job classifies failures against a one-entry KNOWN_FAILURES list whose entry is the EasyGL ReferenceStencil test, which CNA's own record says now passes; if EasyGL's ReferenceStencil override breaks again, the job stays green.

Identifier

CNA-BUG-193

Category

Bug

Subsystem

Build & CI

Status

Open

Verified against

CNA 009d40f5 (009d40f5dd085c4e674d3479675fac84b12b3e0a)

Severity

Low (a triage suggestion, not a project priority)

Evidence basis

Source-verified: read at TARGET, not executed

Tests touching this area

Yes: see Current tests

Affected contract

.github/workflows/general-tests-ci.yml, step 'Classify failures (known pre-existing vs. genuinely new regressions)': KNOWN_FAILURES

Expected behaviour

The workflow header describes the classification step as a temporary stand-in for WILL_FAIL annotations whose job is to ensure "a real regression is never silently absorbed", with every allowlisted name a tracked, still-failing limitation. An entry must go when its failure is fixed.

Actual behaviour at TARGET

general-tests-ci.yml sets KNOWN_FAILURES=("EasyGL_GraphicsDevice_ReferenceStencil"), justified in its header as a "disclosed in-source limitation (Task 319/872)". REMED-GFX-236 (closed 2026-08-29, plans/plan_postaudit.md) implemented EasyGLRenderer::SetReferenceStencil (EasyGLRenderer.cpp), and the test's own header now says exit 0 is the expected result. The step counts any failed test whose name is on the list as known, so a future regression of EasyGL's ReferenceStencil override - the one behaviour this test pins - is absorbed. remediation/REMEDIATION_PROGRESS.md, which the header names as the tracking record, still calls the test "confirmed still-open".

Source locations

• .github/workflows/general-tests-ci.yml — header comment and the KNOWN_FAILURES array of the classification step
• modules/renderers/easygl/examples/easygl_graphicsdevice_reference_stencil_test.cpp — header: fix history and exit-code contract
• modules/renderers/easygl/src/EasyGLRenderer.cpp — EasyGLRenderer::SetReferenceStencil
• plans/plan_postaudit.md — REMED-GFX-236
• remediation/REMEDIATION_PROGRESS.md — the allowlist's tracking row
• modules/renderers/easygl/examples/CMakeLists.txt — Task 319 known-failure comment on the ReferenceStencil registration

Evidence

Checked by reading at 009d40f5; not executed. No CI run was inspected, so that the test passes on the hosted runner (Mesa under Xvfb) was not observed; CNA's record of the fix is a local run ("1 FAIL - 2 PASS", mutation-checked). The site's testing tutorial and verification page report the one-entry allowlist accurately but not that its entry is fixed.

Independently observed as a separate finding (merged): EasyGLRenderer::SetReferenceStencil now re-issues the stencil function, but the test's CMake registration still calls it a documented known failure and general-tests-ci.yml's KNOWN_FAILURES list absorbs it, so pass or fail is invisible.

Independent re-verification: Checked by reading at 009d40f5; not executed. No CI run was inspected, so that the test passes on the hosted runner (Mesa under Xvfb) was not observed; CNA's record of the fix is a local run (1 FAIL - 2 PASS, mutation-checked). general-tests-ci.yml is the only workflow that runs ctest unfiltered. Workaround correction: the classification step prints only counts for known failures ('Total failed this run: N', 'Known pre-existing ... allowed'), never their names; a regression of this test is visible only in the preceding ctest step's log ('The following tests FAILED'), whose failure is masked by continue-on-error, so the job stays green.

Focused reproduction

No focused reproduction is known. Nothing has been invented here; the evidence above is what exists.

Current tests

EasyGL_GraphicsDevice_ReferenceStencil itself. Nothing checks the allowlist against actual results: an allowlisted test that passes is silently accepted.

Regression test

Remove the entry, and make the classification step fail (or at least warn) when an allowlisted test passes, so an entry cannot outlive its failure; or move to the WILL_FAIL annotations the header plans (REMED-BUILD-003).

Blast radius

The CI signal for one EasyGL test on OPENGLES3. Code and every other test's classification are unaffected.

Workaround

Read the step's "Total failed this run" output, which still lists the test by name if it fails.