glTF conformance: corpus, oracle ladder and evidence

CNA snapshot 009d40f5  ·  Deep Dives › Models, glTF & 3D  ·  source links pinned to 009d40f5

CNA's glTF support is an evidence system as much as an importer: a pinned specification, a generated and digest-guarded corpus, numerical oracles from container structure up to the effect parameters a draw binds, renderer-owned pixel campaigns, a small comparison with the Khronos reference renderer, and an append-only defect ledger. This page describes each piece at this snapshot, what it proves, what it does not, and how to read the resulting claims. It is for anyone who needs to know how far "CNA imports glTF correctly" actually reaches, and for maintainers adding fixtures or chasing a regression.

A pinned authority

gltf-conformance.md pins the Khronos glTF 2.0 specification at commit 2b29723d025a995971726f2989697cdc49b1222a of KhronosGroup/glTF, path specification/2.0/Specification.adoc: a 149,830-byte AsciiDoc source with SHA-256 55986799907693d3f51b0a474497852c0d6318b85084811fdc05ff0db4b27967, pinned on 2026-08-11, with a curl and sha256sum recipe to prove any later reading is the same text. The AsciiDoc source is pinned rather than a registry HTML snapshot because it is the input the published HTML and PDF are generated from, it carries a stable digest, and it is retrievable at an immutable revision. The specification is not vendored and CNA has no build-time or run-time dependency on it; it is the research and test authority that fixture expectations cite. The same pin is recorded in the corpus manifest and in every fixture's expectation file.

A generated, deterministic corpus

The standard-library Python generator under tools/gltf_fixtures emits, from one source of truth per fixture, the asset, its GLB twin, its independently stated expectations and, where applicable, its byte goldens, so the asset and the expectation cannot drift apart. The outputs are committed under tests/assets/gltf and guarded by manifest.json, which records a SHA-256 per file; CnaTests re-verifies those digests at test time, so no Python interpreter is needed to run the suite. At this snapshot the tree holds 148 distinct assets — 148 .gltf files, 148 .glb twins and 148 .expected.json records — in 744 files, organised into fifteen owning groups (container, accessors, component types, topology, normals, transforms, materials, textures, skinning, animation, cameras, lights, scenes, Draco, robustness). A paired container audit therefore examines 296 containers, while the validator gate separately classifies the valid and deliberately invalid fixtures it feeds to the external validator. Those denominators answer different questions and must not be merged into one "number of glTF tests".

Each .expected.json keeps three things strictly apart: layer expectations L1–L4 that are derived from the fixture's authored values and the pinned specification and never change when CNA is fixed; an L5 section (the one layer that is not purely specification-derived, since it describes CNA's own packing); and a defects list recording, for any defect the fixture exposes, the first divergent layer, the owning tasks and CNA's measured current output as dated evidence. Every fixture is CNA-authored: the CI job runs scripts/check-gltf-asset-provenance.sh to prove no third-party glTF asset is committed, regenerates the corpus with --check to prove byte identity, and runs the generator twice to prove determinism across processes. The external Khronos glTF Validator is pinned (validator-pin.json) and fetched for that gate; it is not needed to build or run CNA. Khronos sample assets and the Khronos sample renderer are likewise fetched on demand for the bounded comparison campaigns below, never redistributed or made runtime dependencies.

The oracle ladder, including its L0 gate

The semantic ladder has seven observation layers, and a higher layer is meaningful only when the lower ones agree. Beneath them an L0 gate proves the corpus and oracle machinery are intact — generator determinism, manifest and asset agreement, .gltf/.glb equivalence, the ladder's own partition. The oracle helpers live in test scope only (namespace CnaTest::GltfOracle, compiled into CnaTests); nothing under modules/*/src may call them.

UnitTests.cmake registers the ladder as separately named CTest entries under the gltf-conformance label: CnaGltfConformanceL0 through L6, plus Perf, Ledger and Tool. Each entry is its own test so that a divergence has a layer — CTest's result names it without anyone reading a log — and they are registered lowest first, so the first failing entry is the first divergent layer, while higher layers still run. The rung list in that file is the single source of truth: GltfConformanceLadder parses it and asserts that every registered Gltf* suite falls into exactly one rung, so a new suite cannot silently sit outside the label. L7 is deliberately absent from CNA's own CTest list because it needs a renderer with a real 3D pipeline and a viewer; see pixel evidence.

L5: byte goldens and their absences

140 of the 148 fixtures carry committed .vb.bin/.ib.bin goldens (multi-primitive fixtures carry one pair per primitive). Eight have none, each for a stated reason: seven are deliberate import rejections (an unsupported required extension, three malformed-accessor cases, an out-of-range index, a joint index out of range and a 73-joint skin) and produce no buffers at all; the eighth, interleaved-pos-nrm-uv, is a positive case whose packed bytes contain a generated tangent basis, which the generator refuses to reimplement as its own oracle. The golden layout table is stated twice — in the generator and in the C++ comparator — and a test asserts the two agree; the comparator's own tests prove it reports a perturbed byte at the right offset, field and vertex, a truncation at the first missing byte, and an unknown stride rather than guessing at one.

L6: what reaches an effect

CaptureDrawParamsEXT records, per drawn part, the GpuDrawParams block a renderer would receive, after binding bone, world, view and projection exactly as Model::Draw does, by calling the same virtual Effect::FillGpuDrawParams() that GraphicsDevice::DrawIndexedPrimitives calls on the same effect instance. It stops before rasterization and deliberately leaves out the device's own buffer additions (stream bindings, start offsets), which L5 owns. Effect state with no draw-parameter field — alpha mode, cutoff and double-sidedness — is captured from the effect's CNAEXT properties beside the block, so "carried" and "bound" stay visibly different. L6 is what exposed D7: the factor-only gold material decoded perfectly at L3 and still rendered white, because nothing assigned its factors to an effect.

Pixel evidence is renderer-specific

CNA retains four whole-corpus L7 campaign reports, captured on 2026-08-18 against CNA commit b6c180fd (before the alpha.1 tag) with an external production viewer: OPENGLES3/EasyGL on Mesa llvmpipe with LIBGL_ALWAYS_SOFTWARE=1, VULKAN, SOFTWARE, and DIRECTX11 under Wine with DXVK 2.6.0 (gltf-l7-corpus-report.json, gltf-l7-vulkan-corpus-report.json, gltf-l7-software-corpus-report.json, gltf-l7-directx11-report.json under docs). Each gives all 148 canonical assets an explicit disposition: 140 deterministic 512×512 captures and eight safe rejections. Each renderer owns its goldens at an RGB and alpha tolerance of 0, and a golden is accepted only when two independent viewer processes produce byte-identical PNGs; no renderer borrows another's image or tolerance. A fifth report covers only the fourteen material fixtures on Vulkan.

These reports are CNA's recorded evidence: their images, hashes, commands and guards are in the tree at this snapshot, but these pages did not reproduce the campaigns, and same-renderer repeatability is not an independent implementation oracle. What runs automatically is narrower: the glTF renderer conformance workflow (gltf-renderer-stride-ci.yml) runs the whole CTest ladder and the stride-conformance cases on STUB, HEADLESS, OPENGLES3, VULKAN and SOFTWARE, and a separate job builds the external viewer (openeggbert/cna-gltf-viewer, pinned to a fixed commit) against the CNA commit under test and runs its whole-corpus L7 rung on OPENGLES3 only. The glTF sanitizers workflow (gltf-sanitizers-ci.yml) builds STUB with AddressSanitizer and UndefinedBehaviorSanitizer, with Draco both on and off, runs the validator gate, the provenance, regeneration and determinism checks, every *Gltf* test and the ladder.

An independent second perspective

Two bounded campaigns compare CNA with the pinned Khronos sample renderer (commit 863b981f). The reference comparison of 2026-08-15 (gltf-reference-comparison.json) covers 13 generated assets — JSON and GLB, ordinary and non-indexed triangles, interleaved and sparse accessors, converted strips, unlit and textured materials, normalized vertex colour, independent UV0/UV1 selection and the Draco path — gated on a non-clear-mask intersection-over-union of at least 0.99, foreground coverage within [0.99, 1.01] and an intersection RGB mean absolute error of at most 80 (colour is not required to match byte for byte because the two applications use different fixed light rigs and only the Khronos renderer tone-maps). All 13 passed, minimum IoU 0.999579. Its first run caught a real renderer defect: EasyGL's shared lit program evaluated specular maths for an unlit effect, normalize(0) produced NaNs and an RGB error of 189.33, and the shader now branches before lighting maths.

The final viewer retake matrix of 2026-08-14 (gltf-viewer-retake-report.json) runs 14 gate rows as 15 capture cases against pinned Khronos sample assets (Fox, DamagedHelmet and Sponza among them, with Sponza's resource closure of 52,686,624 bytes and 262,267 triangles), with IoU at least 0.99, coverage within [0.99, 1.01] and RGB error at most 100; all 15 passed, minimum IoU 0.998719. These are broader than a smoke test and much narrower than pixel-identical proof of all 148 fixtures.

Tests that guard the evidence itself

A ladder is only as good as its oracles. Several suites under the glTF import tests exist to stop the evidence from quietly going wrong. They check the oracles, the decoders that audits have already proved correct, and the parts of production that more than one piece of code computes.

• Locking a path that is already correct. GltfAccessorDecodeLockTests.cpp (plan task GLTF-041) pins the attribute decode route, UnpackAccessor over cgltf_accessor_unpack_floats. A forensic audit found this route correct for offsets, strides and interleaving, normalized integers and sparse attribute accessors, so later remediation is told to add checks around it and never to replace it. The lock has two halves. The L2 half checks decoded arrays against the generated manifests. The L3 half, ProductionExtractMeshAgreesWithTheL2DumpForEveryVerifiedFixture, checks that what production ExtractMesh packed agrees with the L2 dump component for component. A replacement decoder that rounds, strides or resolves sparse data differently fails the L3 half, even if it satisfies the L2 half.
• An oracle must not change what it measures. GltfOracleEXT.EvaluateWorldPositionsEXTDoesNotAlterProductionBehaviour (GltfOracleEXTTests.cpp) extracts the scene meshes of xf-shared-mesh before and after evaluating the world-position oracle and requires identical dumps. Agreement between CNA and the specification must come from the import path, not from an oracle feeding its own composition back into it.
• Two computations of one fact must agree. The scene-graph bone checks in GltfSceneGraphBonesTests.cpp load committed fixtures through the real ContentManager::Load<Model>. They compare every node's world matrix composed once by BuildSceneGraph at import with the matrix Model::CopyAbsoluteBoneTransformsTo recomposes from the bone tree at draw time, so neither computation can drift from the other unnoticed.
• A skip names the missing capability. Content suites that must build a live Model skip with GTEST_SKIP when SupportsCapability(GraphicsCapability::ThreeD) is false. RuntimeGltfModelTests.cpp and GltfToCnjToolTests.cpp do this, and so do the device-dependent cases of two import suites, GltfStrideAndBufferTests (uploads and draws of every stride) and GltfMaterialToPbrMaterialTests. A skip reads as the renderer's documented boundary rather than as a defect. The pure importer suites need no device. STUB answers false for every capability, so a STUB test build runs the importer suites and skips every Model-building case. HEADLESS reports ThreeD true, so those cases run there. A green STUB run is therefore no evidence for the Model routes.
• The converter is tested as a separate process. GltfToCnjToolTests.cpp spawns the real cna_tool_gltf_to_cnj binary. UnitTests.cmake makes the content test objects depend on the tool target and bakes its path in as CNA_GLTF_TO_CNJ_TOOL_PATH. The CNJ-pipeline and CNB model-equivalence suites reuse the same definition.

Checked by reading these test sources and the two renderers' SupportsCapability overrides at 009d40f5; nothing was built or run.

The append-only defect ledger

Normative expectations and measured behaviour are separate fields. A known-defect test asserts two things at once: that the specification expectation is still not met, and that the divergence is exactly the recorded one — CNA broken in the documented way, not a new way. A record moves through known-failing, partially-remediated and fixed; when fixed, its currentActual becomes today's output, the old measurement moves to priorActual, the layer is asserted in full and the record stays as a regression witness. Records are never deleted, and a bidirectional completeness test fails if an open defect lacks a test or a fixed one still has one. At this snapshot the ledger in the manifest holds nine records, all fixed:

D5 now carries point and line topology and converts loops, strips and fans explicitly; D6 stores rigid scene-node clips on unskinned models and reports the remaining mixed-carrier loss; D7 reaches the L6 material contract and the recorded renderer campaigns; D8 keeps skeleton-root ancestry and mesh-node cancellation in a separate prefix so animation cannot overwrite them (see D8). Fixed records are regression history; none of them is a current defect.

Current capability and loss reporting

GltfImportReportEXT makes a successful load distinguishable from a lossless one: direct glTF and converter-produced .cnj models expose scene, mesh, skin, camera, light, animation, primitive and clip counts plus ordered diagnostics with stable codes, and AnythingLost() with the dropped and approximated counts summarises them (details on the import core). The runtime model also exposes every imported skin mapping, cameras, material-variant names and selection, whole-model bounds and per-part topology, closing several old "parsed but unreachable" gaps. The deliberate limits remain:

• at most two distinct texture-coordinate sets per PBR primitive;
• four joint influences per vertex and a 72-matrix palette for the stock skinned effects;
• point and spot lights approximated through XNA's three directional slots, with range and cone reported as lost;
• EXT_mesh_gpu_instancing imports one placement per node, and EXT_meshopt_compression is refused;
• KTX2/BasisU and WebP images need a PNG/JPEG fallback;
• mirrored placements are reported but their winding is left to the application's cull state.

What the milestone names mean

CNA's plan records the declaration GLTF CORE 2.0 IMPORT/RUNTIME MODEL CORRECT (2026-08-18). It covers the scoped importer, the .cnj route, the model graph, effect and state transport, the generated corpus and the recorded renderer evidence, with renderer coverage stated beside it rather than inside it; an earlier, unqualified name was withdrawn because a correct import model cannot by itself prove every renderer's final pixels. The broader GLTF ROBUST milestone is assessed as 9 of 12 rows in the plan, with three open:

1. KHR_materials_specular texture sampling on every PBR renderer — among the current renderers only METAL still samples neither map (per the renderer inventory test);
2. native point and spot lights with falloff and cones, which needs a light block shared by every renderer's shader ABI that XNA's IEffectLights cannot express — the only open row that is a whole feature rather than coverage;
3. a fourth renderer passing the bounded Khronos retake matrix; three qualify (OPENGLES3, VULKAN and DIRECTX11 under Wine and DXVK), and the plan names SOFTWARE, which already has whole-corpus L7 evidence, as the cheapest fourth candidate, blocked only by the harness's external inputs (a built Khronos sample renderer and a sample-asset checkout) being absent on the recording host.

The open rows do not contradict the qualified core declaration; they define why the stronger milestone stays open. Both statuses are CNA's own plan records, restated here, not a new assessment.

How to read a glTF claim

A green import plus an empty loss report establishes the represented model for the exercised path; L5 establishes the bytes; L6 establishes what reaches an effect; only renderer-engaged pixel evidence establishes the final image, and only on the renderer and host it was recorded on. Keep "parsed", "represented", "reported" and "drawn by this renderer" as separate claims. The feature-by-feature version of this rule is the glTF evidence matrix; the site-wide evidence vocabulary is on Verification & Known Issues.

Evidence

Checked by reading the TARGET corpus manifest, conformance document, report JSON files, CMake registration, workflows and test suites at 009d40f5; nothing was built, captured or executed for this page. Counts were taken from the committed tree. Plan statuses and campaign results are CNA's own records and are attributed as such.