Review large AI-generated areas without trusting them

CNA snapshot 009d40f5  ·  Development › Human Takeover  ·  source links pinned to 009d40f5

CNA was built at scale with AI assistance, and the tree carries the paper trail: commit subjects with task IDs, 106 plan files, frozen audit and remediation archives, root-level handoff notes, and comments that cite tasks. This page says how far each of those can be trusted, which of CNA's own mechanical guards are stronger evidence than any prose (and what each one really enforces), and how to review a very large diff without trusting its author, using the snapshot's own head merge as the worked example. Read it before you accept a change because a plan row, a status banner or a comment says it is done.

The rule

Current source and active CMake are authoritative. Plans, audits, handoff notes and comments explain intent and history, and can be stale. CNA's own files say the same about themselves. The archives audit/AUDIT_FINAL_REPORT.md, remediation/README.md and modularization/RECONCILIATION.md each open with a banner that they predate the 2026-09-17 renderer curation and must be read “as a fact about that date, never as current CNA support”. plans/README.md says the status recorded inside each plan is authoritative and refuses to copy task counts into the index. 26 of the 106 plan files open with a retired-renderer, superseded or rejected-candidate banner (24 use the words “retired renderer — historical record”). AGENTS.md and CLAUDE.md tell an assistant not to treat “old CNA code or AI-generated stubs” as authoritative when they conflict with the FNA reference API.

In practice: behaviour is answered by the implementation, the active CMake that compiles it and the tests registered for a given configuration; why, by git log and the plan; whether something is supported, by the registry (cmake/RendererIdentities.cmake), the CI workflows and Known uncertainty and history.

How the assistance shows in the tree

The instruction files address assistants directly (porting conventions, CNAEXT tagging, “always commit after finishing a task”). Some archives are written as resume prompts: audit/AUDIT_PROGRESS.md says that if context is lost the session should resume from that file and continue “fully autonomously” with “no further questions”, and the last section of NEXT.md is a resume prompt for a session on the DIRECTX9 branch. Those are instructions to a past agent, not project policy and not authorization for you. Git cannot separate assistant-written from human-written work: all 8,899 reachable commits carry one author name, none of the 2,877 commits since the v0.1.0-alpha.1 tag has a Co-authored-by trailer, and 2,765 of those 2,877 carry a PGP signature header, which attests the committing key and not that anyone read the diff.

The artifacts and how far each can be trusted

Figures were counted from the TARGET tree and history at 009d40f5; patterns are simple, so treat the percentages as approximate.

Plan rows are claims, and CNA's plans record their own corrections. SOFTWARE-375, 376 and 390 were closed as done, and SOFTWARE-457 (2026-09-12) records that Microsoft's assembler invalidated the evidence behind them: the fixtures used operand forms the compiler forbids, and both CNA compile paths had accepted seven forged violations (plans/plan_software.md). plans/plan_webgpu.md counts 200 done rows and one open one (counted from its own row tables by tools/count_webgpu_plan_status.sh) and says “read the row count as a count of rows, not as a parity claim”; the plan plans/plan_webgpu_failing_eight.md then records that ctest -L WebGPU had eight failures on rows marked done, three of them tests asserting behaviour the framework had deliberately changed: “a plan row being green says the work was done, not that the test still describes the framework it was written against.”

Drift found while verifying these pages

Each row was checked by reading the named source at 009d40f5. It is a sample, not a census; the point is the shape of the mismatch.

CNA's mechanical guards: what each really enforces

A guard is stronger evidence than prose only for what it matches, in the configuration that runs it, and with the exit code it actually produces. The table gives, for each, when it runs, what it enforces and what it cannot see. Registrations were read from CMake and workflows; nothing was executed.

Sibling audits in the same style are scripts/check_renderer_target_discipline.py (RendererTargetDiscipline), scripts/check_renderer_combinations.py (RendererCombinationRegistry), scripts/check_removed_renderer_api.py (RendererCurationApiDecisions) and the CNAEXT guard, matrix, naming, nodiscard and Doxygen checks (scripts/check_cnaext_guards.sh and its neighbours).

Three reading rules follow. Know when a guard runs: configure time, CTest, CI or by hand. Know what it matches: a regular expression, a compile, or a registration list. Know how it fails: skip or fail, and which condition sets the exit status. The last matters most for oracle scripts, where a printed difference does not always produce a non-zero exit; the oracle page gives the case.

A review method for a big diff

1. Read the campaign structure first. git log --first-parent --merges <base>..<head> lists the integration story. For one merge M: git rev-list --count M^1..M^2, git diff --shortstat M^1 M, and git diff-tree --cc -r --name-only M, which lists the files the merge itself edited relative to both parents. Then open the plan that the branch name or IDs point to and compare its stated scope with the commits.
2. Map changed paths to owners. Group git diff --numstat by module directory and look each owner up in Source ownership. Shared modules (modules/graphics tests and headers) mean every renderer is in scope even when the title names one.
3. Read the tests before the implementation. Which existing tests changed expectations, and does the message name a measurement (a real-XNA probe, a task ID) or merely a new result? Check where each test is registered: renderer gate lists such as CNA_RENDERER_IS(…), the source filters in cmake/UnitTests.cmake, and the skip convention.
4. Extract the invariants the change relies on (architectural invariants) and find a test for each: per-device GL contexts, compute bindings restored after a dispatch, the selection latch, ownership order (ownership map).
5. Check comment and behaviour drift in the touched files. Header doc against .cpp comment against code; prose counts against registrations and lists; task citations that resolve to nothing.
6. Look for hedged evidence. “Measured” against “reasoned”, “expected”, “should”, “not measured”; a claim with no command or test; one GPU, one driver, one date. scripts/validate_direct2d_plan.py is a model of the rule: a done row must cite evidence that exists.
7. Distrust “all tests pass” without a configuration. Ask for renderer, platform, audio option, generator, display, skips and known failures (reporting evidence honestly).
8. Record what you could not verify (investigation record), including hardware you lack and numbers you could only cross-check.

Worked example: reading the head merge, 009d40f5

The snapshot is itself a merge: merge(OpenGL4ModernGraphics), 2026-09-24. The commands are those of the method; the numbers were counted from Git objects.

Structure

The first parent is b2a0a567, exactly the baseline that plans/plan_opengl4_modern_graphics.md records; the second is 725700d9 (GL4-0037), 24 commits later. The merge base equals the first parent, so the branch is a linear stack, and the merge tree is identical to the second parent: the merge commit edits nothing itself (git diff-tree --cc lists no files). Three earlier fold merges differ from both parents: 93ca4ffdf (Vulkan, 34 files), 0b046a8d0 (WebGPU, 13) and 1e8ec10e7 (software, 135); those merges are changes in their own right. The message claims “GL4-0001..GL4-0037”; the 24 commits carry 35 numbered IDs (0003 to 0037) in ranges plus two gate rows, while 0001 and 0002 exist only as plan rows.

Owners

git show --stat 009d40f5 and git diff --shortstat b2a0a567 009d40f5 report 180 files, +25,778 and −6,595 lines. By module: modules/graphics 87 files (30 tests, 54 examples and three new shared GL headers of 2,128 lines), modules/renderers/opengl4 42 (+18,412, −4,185), modules/graphics-ext 30 (+2,388), cmake 6, modules/renderers/easygl 5 (1,929 lines deleted), modules/content 4, plans 3, tools 2, docs 1. A merge titled for one renderer removes 1,929 lines from the family that serves five other identities, and edits the shared tests of every renderer: those two facts define its blast radius.

Checks that cost nothing

• Tests before implementation. Commit 80c158f36 (GL4-0018) changes shared test expectations to rules “measured from the XNA assemblies” (citing SOFTWARE-204, -246, -216 and GSC-0004) and adds OpenGL4 to the shared renderer gates: its message says 191 gates were extended, and the commit touches 56 files whose patch adds 290 lines that mention OpenGL4. The 191 was not recounted; sample the gate lists, because a list decides which renderers a test runs on or skips for, and a wrong entry silently changes what is tested.
• “Moved verbatim”. GlStockShaderSources.hpp states that its shader text is moved verbatim from EasyGL. Of the 1,869 non-blank lines the merge removes from EasyGL's renderer source and header, 1,835 (98.2 %) reappear, ignoring indentation, among the 2,055 lines added to the three new shared headers; the other 34 are declaration, qualifier and call-site edits (for example the EasyGLSurfaceState methods, CnaGlIblDecl and AdaptStockVertexShaderForOpenGL). That supports the text-level claim only; identical text can still compile differently in another namespace or include order.
• Numbers in the message against registrations. “CnaGraphicsExtTests 798/44/120 before, 960/0/7 after” sums to 962 and 967, which are exactly the static test-definition counts under modules/graphics-ext/tests/ at the two parents. “407” OpenGL4 CTests equals 26 direct registrations, 346 generated EasyGL-corpus entries (EasyGLParityCorpus.cmake), 32 parity fixtures and, with CNA_CNAEXT on, three modern tests, counted in opengl4/examples/CMakeLists.txt. CnaGraphicsTests is 2,833 + 57 = 2,890 against 2,906 static definitions under modules/graphics/tests; configuration filters can explain the 16, the message does not.

What stays unverified

Every pass, fail and skip count, the zero GL errors, the sanitizer runs and the 3,000-cycle stress are CNA's records from one Radeon 780M through the private runner; none was reproduced, and the merge message itself says findings for EasyGL, Vulkan, WebGPU and the Wayland platform are “recorded in the ledger, not changed”. The renderer-side reading is on OpenGL4 internals.

Short checklist

• Which file decides the behaviour: source, active CMake, a registered test? Not a plan, note or comment.
• Which merge or campaign brought it in, did a fold merge edit files on its own, and which owners and renderers does it touch beyond the one in the title?
• Did any test expectation change, and is the reason a measurement or merely a new result?
• Which invariant does the change rely on and which test pins it; do header docs, comments and behaviour agree; do stated counts match a registry?
• Is each evidence claim measured, dated, and tied to a host and a configuration?
• Would the guard that covers this area run in your configuration, and would it fail on this change?
• What did you not verify, and who can?

Limits of this page

Nothing was built or run; every statement was checked by reading files and Git objects at 009d40f5. The drift table is a sample, percentages come from simple patterns, and the guard descriptions state registrations and script logic, not outcomes at this snapshot. Which files an assistant wrote is not recoverable from Git and is not claimed. The oracle-specific evidence rules are on Using the XNA oracle as evidence.