Review large AI-generated areas without trusting them

CNA snapshot 3b43bd86  ·  Development › Human Takeover  ·  source links pinned to 3b43bd86

CNA was built at scale with AI assistance, and the tree carries the paper trail: commit subjects with task IDs, 62 plan files, root-level handoff and campaign records, and comments that cite tasks. This page says how far each of those can be trusted, which of CNA's own mechanical guards are stronger evidence than any prose (and what each one really enforces), and how to review a very large diff without trusting its author, using one of the campaign merges in the snapshot's history as the worked example. Read it before you accept a change because a plan row, a status banner or a comment says it is done.

The rule

Current source and active CMake are authoritative. Plans, audits, handoff notes and comments explain intent and history, and can be stale. CNA's own files say the same about themselves. plans/README.md says the status recorded inside each plan is authoritative and refuses to copy task counts into the index, and several plans open with a banner that marks them superseded or historical (for example plans/plan_ascii.md and plans/plan_glbackends.md); finished or superseded plans are also deleted outright, with Git history keeping them (on 26 October 2026 alone, plan_graphics.md, plan_sdl3free.md, plan_vulkan_parity.md and four others). AGENTS.md and CLAUDE.md tell an assistant not to treat “old CNA code or AI-generated stubs” as authoritative when they conflict with the FNA reference API.

In practice: behaviour is answered by the implementation, the active CMake that compiles it and the tests registered for a given configuration; why, by git log and the plan; whether something is supported, by the registry (cmake/RendererIdentities.cmake), the CI workflows and Known uncertainty and history.

How the assistance shows in the tree

The instruction files address assistants directly (porting conventions, CNAEXT tagging, “always commit after finishing a task”), and handoff.md opens with an “Exact next task” written for the next session. Those are instructions to a past or future agent, not project policy and not authorization for you. Git cannot separate assistant-written from human-written work: all 10,679 reachable commits carry one author name, none of the 4,657 commits since the v0.1.0-alpha.1 tag has a Co-authored-by trailer, and 4,154 of those 4,657 carry a signature header (212 of them SSH rather than PGP signatures), which attests the committing key and not that anyone read the diff.

The artifacts and how far each can be trusted

Figures were counted from the TARGET tree and history at 3b43bd86; patterns are simple, so treat the percentages as approximate.

Plan rows are claims, and CNA's plans record their own corrections. SOFTWARE-375, 376 and 390 were closed as done, and SOFTWARE-457 (2026-09-12) records that Microsoft's assembler invalidated the evidence behind them: the fixtures used operand forms the compiler forbids, and both CNA compile paths had accepted seven forged violations (plans/plan_software.md). plans/plan_webgpu.md has 201 done rows and one partial one (counted from its own row tables with the logic of tools/count_webgpu_plan_status.sh) and says “read the row count as a count of rows, not as a parity claim”; a later plan, plans/plan_webgpu_failing_eight.md (no longer in the tree), recorded that ctest -L WebGPU had eight failures on rows marked done, three of them tests asserting behaviour the framework had deliberately changed: “a plan row being green says the work was done, not that the test still describes the framework it was written against.”

Drift found while verifying these pages

Each row was checked by reading the named source at 3b43bd86. It is a sample, not a census; the point is the shape of the mismatch.

CNA's mechanical guards: what each really enforces

A guard is stronger evidence than prose only for what it matches, in the configuration that runs it, and with the exit code it actually produces. The table gives, for each, when it runs, what it enforces and what it cannot see. Registrations were read from CMake and workflows; nothing was executed.

Sibling audits in the same style are scripts/check_renderer_target_discipline.py (RendererTargetDiscipline), scripts/check_renderer_combinations.py (RendererCombinationRegistry) and scripts/check_removed_renderer_api.py (RendererCurationApiDecisions). The CNAEXT guard, matrix, naming, nodiscard and Doxygen checks (scripts/check_cnaext_guards.sh and its neighbours) are still in the tree, but no CMake file or workflow runs them at this snapshot.

Three reading rules follow. Know when a guard runs: configure time, CTest, CI or by hand. Know what it matches: a regular expression, a compile, or a registration list. Know how it fails: skip or fail, and which condition sets the exit status. The last matters most for oracle scripts, where a printed difference does not always produce a non-zero exit; the oracle page gives the case.

A review method for a big diff

1. Read the campaign structure first. git log --first-parent --merges <base>..<head> lists the integration story. For one merge M: git rev-list --count M^1..M^2, git diff --shortstat M^1 M, and git diff-tree --cc -r --name-only M, which lists the files the merge itself edited relative to both parents. Then open the plan that the branch name or IDs point to and compare its stated scope with the commits.
2. Map changed paths to owners. Group git diff --numstat by module directory and look each owner up in Source ownership. Shared modules (modules/graphics tests and headers) mean every renderer is in scope even when the title names one.
3. Read the tests before the implementation. Which existing tests changed expectations, and does the message name a measurement (a real-XNA probe, a task ID) or merely a new result? Check where each test is registered: renderer gate lists such as CNA_RENDERER_IS(…), the source filters in cmake/UnitTests.cmake, and the skip convention.
4. Extract the invariants the change relies on (architectural invariants) and find a test for each: per-device GL contexts, compute bindings restored after a dispatch, the selection latch, ownership order (ownership map).
5. Check comment and behaviour drift in the touched files. Header doc against .cpp comment against code; prose counts against registrations and lists; task citations that resolve to nothing.
6. Look for hedged evidence. “Measured” against “reasoned”, “expected”, “should”, “not measured”; a claim with no command or test; one GPU, one driver, one date. tools/xnb/check_plan_status.py is a model of the rule: a plan's stated totals and its closed decisions must agree with its own rows.
7. Distrust “all tests pass” without a configuration. Ask for renderer, platform, audio option, generator, display, skips and known failures (reporting evidence honestly).
8. Record what you could not verify (investigation record), including hardware you lack and numbers you could only cross-check.

Worked example: reading a campaign merge, 0f690d7d5

The WebGPU per-draw binding campaign reached next as merge(WebGPUPerf), 2026-09-27, an ancestor of this snapshot. The commands are those of the method; the numbers were counted from Git objects.

Structure

The first parent is b1e4a2414, the tip of next at the time; the second is 196b14bd0 (WEBGPUPERF-0006). The merge base is 85c3baa09, the commit that had already put WEBGPUPERF-0001..0004 on next directly, and next had moved three commits past it, so the branch is a linear stack of five commits on that base. git diff-tree --cc lists no file: the merge commit edits nothing of its own. The message claims WEBGPUPERF-0005..0008; the five commits carry exactly those four IDs, one of them in a documentation commit that records what the ShaderEffect and compiled-effect routes still cost per draw, and the campaign's plan, plans/plan_webgpu_perf.md (in the tree at the merge, deleted before this snapshot), had rows 0001 to 0008, every one with a commit. Compare the fold merges of other campaigns, whose merge commits can carry edits that no branch commit shows.

Owners

git diff --shortstat b1e4a2414 0f690d7d5 reports 15 files, +1,505 and −548 lines. By directory: modules/renderers/webgpu/src 4 files (+719, −486), its include 2 (+166, −12), its examples 7 (+432, −41) and its tests 1 (+114), and the plan (+74, −7). A merge titled for one renderer that stays inside that renderer's directory is the narrow case: no shared test, header or module outside the family moved, so the blast radius is the WebGPU paths of the stock effects, SpriteBatch, ShaderEffect and compiled effects, which the message names. The first half of the campaign, 85c3baa09, had the same shape (7 files, all under modules/renderers/webgpu and the plan).

Checks that cost nothing

• Tests before implementation. The merge extends webgpu_binding_cost_test.cpp by 365 lines; it is registered as the CTest WebGPU_BindingCost (label WebGPU) in webgpu/examples/CMakeLists.txt. Its header lists checks A to O, and each pair pins both halves of the change: a draw still puts its own colour in its own cell, and a steady frame creates no bind group and makes a number of queue writes that does not grow with its draw count; a texture re-created every frame never meets a stale cached group; four render-target cycles before the back buffer are one submission, and a SetData between two of them lands in order. The cost checks read the renderer's own counters, so they prove the counters, not wall-clock time, and the program needs a display and a WebGPU adapter: a host without them says nothing. The merge also adds one GoogleTest case, WebGPUCompiledEffectDrawTest.SteadyDrawsBuildNoGroupsAndCopyNoGeometry, which skips unless the build selected the WebGPU renderer.
• Numbers in the message against the tree. “CnaGraphicsExtTests 73 + 3 HLSL skips” sums to 76, exactly the static test-definition count under modules/graphics-ext/tests/ at the merge, which a WebGPU-only diff should leave untouched. “CnaRendererTests 273” cannot be checked against a static count: the renderer test directories hold 1,029 definitions in 69 files at the merge, and the source filters in cmake/UnitTests.cmake decide which of them a WebGPU build compiles. “ctest -L WebGPU 147/147” was not recounted, and “the same 20 pre-existing failures” of CnaGraphicsTests are not named, so the message alone cannot show that no new failure replaced an old one.
• A claim about the parts not changed. The message says “next already carried 0001-0004 (85c3baa09)”; git merge-base returns exactly that commit, which confirms the split of the campaign across a direct commit and a merge.

What stays unverified

Every pass, fail and skip count, the street frame times (91–95 ms before, 26–29 ms after) and the render-target switch cost (about 57 µs before, about 16 µs after) are CNA's records from one Linux development host with wgpu-native and from one application, street-simulator (then cna-street); none was reproduced, and none says anything about the browser build, where WebGPU goes through Emscripten's port. The WebGPU family has no internals page here; its place among the families is on the renderer family map.

Short checklist

• Which file decides the behaviour: source, active CMake, a registered test? Not a plan, note or comment.
• Which merge or campaign brought it in, did a fold merge edit files on its own, and which owners and renderers does it touch beyond the one in the title?
• Did any test expectation change, and is the reason a measurement or merely a new result?
• Which invariant does the change rely on and which test pins it; do header docs, comments and behaviour agree; do stated counts match a registry?
• Is each evidence claim measured, dated, and tied to a host and a configuration?
• Would the guard that covers this area run in your configuration, and would it fail on this change?
• What did you not verify, and who can?

Limits of this page

Nothing was built or run; every statement was checked by reading files and Git objects at 3b43bd86. The drift table is a sample, percentages come from simple patterns, and the guard descriptions state registrations and script logic, not outcomes at this snapshot. Which files an assistant wrote is not recoverable from Git and is not claimed. The oracle-specific evidence rules are on Using the XNA oracle as evidence.