Verification tiers: evidence forms, oracle authority and CI reporting

CNA snapshot 009d40f5  ·  Deep Dives › Reference matrices  ·  source links pinned to 009d40f5

This reference is a decoding key for verification claims about CNA at snapshot 009d40f5. It does not rank tests on one line from weak to strong: a compile probe, a hostile-input test, a pixel comparison and a native-host run answer different questions. The useful record is an evidence vector naming the subject, the build and host, proof that the intended route engaged, the observation point, the oracle and the retained artifact. The tables below give the claim labels, what each evidence form does and does not establish (with a CNA instance of each), which authority decides which question, what CNA's automation actually runs, and how to report a CI result without overclaiming. The vocabulary itself is defined on Compatibility levels and the evidence vector; this page applies it.

Claim labels

The labels describe what was established; none is a synonym for "supported". The right-hand column gives the nearest term of the evidence scale these pages use (source exists, configures, builds, links, unit tested, integration tested, runtime observed, visual compared, real hardware observed, XNA oracle compared).

"Test exists", "test was registered", "test ran" and "oracle discriminated" are four separate statements. A skipped test is not a pass, a dry-run mutation is not a killed mutation, and an uploaded log is useful only when it identifies the binary, configuration, host and observation that produced it.

Evidence forms are not interchangeable

The strict probes, the pixel harness and the oracles named here are explained on When failing to compile is the pass and Oracles, tolerances and engagement gates.

Oracle authority depends on the question

For API shape, CNA's census takes Microsoft's XNA runtime XML documentation and DLL metadata as the target surface (audit_xna_runtime_surface.py); a converted, searchable copy of that documentation is a reading aid, not a gate and not a substitute for the original assemblies. For behaviour, CNA's contributor rules name FNA as the ordinary reference and measured XNA as the tie-break, while a format specification or a native API can outrank both for a narrower question. A project-authored expected value is an oracle only when its derivation is reviewable and independent of the code under test.

A renderer or platform claim needs a complete record

   subject + revisions               build + host identity
   (CNA sha, sibling shas,           (selector, family, options,
    local diff, test name)            OS, driver, display, prefix)
              \                            /
               v                          v
 engagement --> [        bounded claim        ] <-- observation point
 proof (token,                                      (return value, trace,
 browser verdict,                                    readback, frame, file)
 native handle)   ^                          ^
                 /                            \
   oracle + policy                     outcome + artifact
   (authority, tolerance,              (pass / fail / expected fail /
    normalisation)                      skip / unsupported / blocked /
                                        not attempted; log, image, digest)

The vector separates, for example, a MinGW binary running through Wine and DXVK from a native MSVC Direct3D run; a browser page served over loopback HTTP from one opened as a local file; an iOS simulator launch (what apple-ci.yml runs, beside a device final link) from a signed build on a physical device; and an Android source path (no workflow builds Android at this snapshot) from an APK that installs and presents a frame from its selected renderer. All are useful milestones, and none is a synonym for another.

Automation at the snapshot

The trigger facts below were read from the 20 files in .github/workflows. They are source-verified configuration, not a record that any job passed.

The native C ABI shows why a workflow file cannot promote a state. Its five workflows are build-free checks, the checked-in release gate reads "Not ready" with one unmet criterion (468 public symbols unmapped), and the generator behind the coverage criterion refuses to run while three public modules are unclassified in its scope table (see Known Issues). Neither a green subset of those checks nor the presence of the files makes the ABI release-ready; browser harnesses, sanitizers, mutation routes and native-platform jobs likewise establish only the configurations and artifacts they execute. The per-workflow scope table is on What CI covers, and what it does not.

What the specialised lanes add, and where they stop

• Portable Linux. The general job runs an unfiltered ctest of one EasyGL identity under Xvfb and classifies failures against a one-entry expected-failure list; the input and devices jobs run focused matrices with sanitizer selections; the 32-bit job builds three arithmetic checks, not a 32-bit CNA; the glTF jobs' oracles are narrower than a model-rendering campaign.
• Browsers and runtime selection. htmldom-ci.yml is the strongest automated browser route (served page, real Chromium, verdict object, compositor pixels). The Emscripten multi-renderer job is designed to prove a compatible bundle links (but it, htmldom-ci.yml, devices-tests.yml and the Apple and Metal workflows pin a sharp-runtime revision that lacks a component CNA's default build now requests, so by a static reading they stop at configure at this snapshot: CNA-BUG-199), and the native multi-renderer job proves descriptor generation and early runtime selection for a CPU and diagnostic set; neither proves every renderer's pixels.
• Platforms. platform-ci.yml treats platform, renderer and audio as independent axes: a contract matrix (SDL3, SDL2, Vulkan, Software, Headless, Terminal), a MinGW Win32 harness executed under Wine, a manual native-MSVC Win32 job, two SDL-free X11 lanes and an SDL-enable matrix. It has no Wayland job: the live Weston, Mutter, XKB, input-method, clipboard and portal suites are registered for the environments they need and run only where a developer provides them, and a missing compositor is a skip, not a passing behaviour test.
• Apple and Windows. The macOS workflows build and run portable suites and the native Metal contract tests with MTL_DEBUG_LAYER and MTL_SHADER_VALIDATION set; they do not imply iOS or tvOS devices or every GPU. The Direct3D and GDI workflows are manual. Native MSVC, MinGW under Wine with DXVK, WARP, and a physical interactive Windows machine remain different evidence tiers (see Direct3D: what each tier proves).
• Content Pipeline. content-pipeline-windows-ci.yml builds the CLI and the CPU pipeline tests with MSVC and exercises the CLI (explain, workers, clean, deterministic rebuild) from a fixture directory deliberately named with non-ASCII characters (CNA-MSVC-Zażółć-曲線) under the runner's temporary directory, which catches path, encoding and packaging assumptions a Linux build cannot. A green build-time tool is still not runtime content proof: XNB, CNB and runtime loading keep their own container and object-graph tests.

Diagnostics and Inspector in automation

Neither subsystem has a workflow, and no workflow or configure preset sets CNA_DIAGNOSTICS above OFF or turns on CNA_BUILD_INSPECTOR. A CI build therefore compiles only the two level-independent diagnostics tests and none of the STATS or FULL cases, and the 27 Inspector tests are never compiled there (the per-level table is on What the diagnostics corpus proves). The overhead figures for both are CNA's own benchmark records from one machine (an AMD Ryzen 7 PRO 7840U with GCC 14.2): evidence for regression on that machine, not a universal budget. CNA's notes record a Linux-plus-Chrome and a native-MSVC end-to-end Inspector run and list macOS runtime validation as pending; those are CNA-recorded, not reproduced here. The generated audits that do run (the XNA census, the renderer registry, platform containment, module closure, the content inventories) are structural proof, not behavioural parity.

Sanitizers, GPU validation, fuzzing, mutation and coverage

• Sanitizers. ASan and UBSan run in one row of input-ci.yml (with detect_leaks=0, so no leak check) and in gltf-sanitizers-ci.yml (leak detection on, but deliberately scoped to the glTF selection: its header, written on 2026-08-12, records that a full run of the same binary then reported 58 UBSan findings and 7 leaks elsewhere, among them signed-integer overflow in the math GetHashCode implementations (CNA-BUG-252), and it refuses an allowlist; a dated CNA note, not re-measured here); devices-tests.yml uses the devices-ubsan preset. Leak suppressions must name a leak reproduced without CNA code (the model is on ASan and UBSan). A sanitizer-clean CPU route says nothing about a native driver's state or pixels.
• GPU validation is family-specific and must itself prove engagement: the [Vulkan Validation] and [OpenGL4 GL Error] output gates, the Direct3D debug layers and the Direct2D debug-log gate in the native workflow, Metal's validation variables, and WebGPU's error scopes inside the renderer.
• Fuzzing. Fuzz-named GoogleTest suites are deterministic adversarial loops that run with the ordinary corpus. libFuzzer entry points exist (the compiled-effect harness with CNA_FX_FUZZER_ENTRY_POINT=ON, the model and intermediate-format harnesses, and two C API fuzz sources compiled as an object library), but no workflow runs a continuous fuzzing campaign.
• Mutation. The Direct2D mutation checker runs only as a dry run (anchors still apply) in a manual workflow; applying, rebuilding and observing each expected failure is a manual step.
• Coverage. No workflow configures gcov, lcov, gcovr or llvm-cov, so reports named "coverage" describe API or feature inventories unless they say otherwise.

How to report CI honestly

A useful CI statement names the workflow and the run, the trigger, the commit, the matrix tuple, the targets built, the tests selected, the skips, the implementation-engagement token and the retained artifacts. "Covered by CI" is too coarse: automatic browser pixels from htmldom-ci.yml, a manual native-Windows Direct3D run, a generated ABI baseline and a source-only registry comparison are all valid evidence, and they prove four different things.

# the facts a CI claim should carry, filled from the run itself
workflow=platform-ci.yml run=<run id> trigger=push commit=009d40f5
tuple="CNA_PLATFORM=HEADLESS renderer=OPENGLES3 audio=NULL"
selected="ctest -L platform" ran=R skipped=S failed=F
engagement="none needed (headless contract)" artifacts="<log locator>"

The block is an illustrative template, not output of a real run.

A minimum claim recipe

Before writing "works", write a narrower sentence that answers six questions:

1. What exact subject and revision were examined?
2. Which build, renderer, platform and host actually engaged?
3. Where was behaviour observed: a return value, a trace, a readback, a frame, a file or a network peer?
4. What independent authority supplied the expected result?
5. Which negative or perturbed case shows the check can fail?
6. What artifact lets the next reader reproduce or falsify the claim?

A claim that cannot yet answer all six can still be useful: label it with the strongest label it earns from the first table (source-proven, compile-proven, runtime-observed, renderer-engaged, oracle-compared, pixel-verified, manually observed, historically recorded, blocked, unsupported or not attempted) and do not silently promote it to behavioural parity. Worked sentences that follow these rules are on How to word a claim.

Where current defects are listed

Defects that exist at this snapshot, each with its evidence and the discriminating regression that would prove a fix, are published on Known Issues, not on this page. The rule that applies to them is the one above: "source-proven" there means the defective condition is visible in the pinned source, not that a reproducer was run, and a fix earns its status only with a test that fails before it and passes after.

The earlier conserved backlog of 31 findings was re-verified at this snapshot. Twenty-four keep their CNA-BUG-### number and have an entry (001, 002, 003, 004, 006, 008, 016, 017, 021, 022, 025, 026, 027, 028, 029, 036, 045, 047, 050, 052, 055, 058, 061, 062; 029, 036, 045, 050, 055 and 058 are narrowed to what survives). The other seven have no page under their old number, for the reasons below.

Evidence and limits

The trigger table, the lane descriptions and the "not covered" list were read from the workflow files at 009d40f5; the instances in the evidence-form and authority tables were read from the CMake registrations, tools and tests named. No workflow, test, benchmark or gate was run for this page, and whether any job is green at the snapshot was not checked. Benchmark, Inspector end-to-end and recorded pass figures are CNA's own records, attributed as such.