I need to modify the Content Pipeline

CNA snapshot 009d40f5  ·  Development › Maintainer Handbook  ·  source links pinned to 009d40f5

The content pipeline is a typed build graph that turns source assets into .cnb or .xnb files: an importer chosen from the source, a processor chosen from the imported type, a writer chosen from the processed type and container, all driven by the cna-content command line with a fingerprinted manifest, atomic publication and safe cleanup. A change to any stage is a change to bytes that games load, to incremental-build correctness and to a dependency boundary. This recipe names the owner of each kind of change, the invariants that must survive it (including that the build must never need a device), how to keep output deterministic, and the tests that prove it. Everything was read from the CNA source at snapshot 009d40f5; nothing was built or run.

Find the owner

Four CMake targets share the work, and the split is a dependency boundary, not tidiness. Before editing, decide which of them your change belongs to.

Read first

1. ContentPipeline.hpp: the stable-type, request, component, registry and result contracts (ContentImporter, ContentProcessor, ContentTypeWriter, ContentPipelineRegistry) before any concrete codec. The extension API is declared experimental (ContentPipelineExtensionApiIsExperimental).
2. ContentPipeline.cpp: registration and freeze, route resolution, stage validation and contextual exceptions.
3. One concrete route end to end, Texture2DContentPipeline.cpp beside XnbOutputContentPipeline.cpp, to see shared processing with container-specific writing.
4. tools/content/content.cpp and ContentBuildManifest.cpp: discovery, graph, fingerprint, stage, publish and cleanup; this is where build correctness extends beyond one codec.
5. The content-pipeline CMakeLists.txt: which dependencies are private to the build-time module.

Invariants a change must keep

• The pipeline must not need a device. A text search of the pipeline sources, their headers, the build-time module and tools/content finds no occurrence of GraphicsDevice; a texture is decoded into plain CnbTextureData and the first GPU allocation happens in the runtime loader. This is a property of those sources, not of the link closure (cna_content still links the graphics core because its runtime half needs it). The Windows content lane relies on it, running the tool and tests with headless configuration and no GPU.
• The dependency boundary holds. FreeType, the build-time FFmpeg libraries, zlib for binary FBX and the effect-compiler defaults are private to cna_content_pipeline. Do not register a build-time codec in a runtime target to make a command-line case pass; CnaXnbDependencyBoundary checks both the CMake graph and, with nm, the built archives, because a translation unit moved across the boundary leaves the graph looking correct.
• Selection is by stable names and stable type strings, never by registration order or std::type_index. A component whose SelectedByNameOnly() is true does not compete for a default route (why .wav is a sound effect unless a project names the song importer); ambiguity fails with every candidate named. A writer is keyed by (container, processed type); use DocumentAbsentWriter to record why a container has no writer for a type rather than leaving a silent gap (ProcessedTypeCoverageTests.cpp fails a processed type with neither).
• The registry is frozen before use (Freeze(), called by ContentPipeline and by RunContentCompiler after adding the XNB writers); registration afterwards throws std::logic_error. A custom route must register everything before the freeze. Because one registered instance may serve concurrent build nodes when --workers is above 1, an importer, processor or writer must be immutable or reentrant; do not add a global scratch buffer.
• Every stage is checked. An importer must return a non-empty value whose stable type it declared; the processor must return its declared output type; the build checks the writer's declared asset, schema and codec identities before and after writing, forbids more than MaxContentBuildOutputs (256) outputs, and refuses two outputs under one logical name with different bytes. Do not bypass these checks to make one asset build.
• Write limits mirror read limits. XnbWriteLimits (file 64 MiB, payload 256 MiB, string 1 MiB, 4,096 type writers, 1,000,000 shared resources, 10,000,000 collection elements, nesting depth 256) are chosen so that a file CNA writes always satisfies the default XnbReadLimits; a non-positive limit is refused because it would widen to "unbounded". Move both sides together, and expect a very large model to fail at write time rather than at load.
• Strictness is narrow. Under ContentStrictness::XnaCompatible only an unknown parameter name and a wrong-typed value are downgraded to warnings; impossible operations still fail in both modes.
• Every failure carries its stage. Errors are rethrown as ContentPipelineError with the source, logical name, stage (Selection, Import, Process, Write, Graph, Publish) and component; keep that context.
• Sources are contained. The primary source must lie inside the source root; sibling files go through the context's dependency APIs (ResolveSourceDependency, AddContentBuildDependency, AddGeneratedDependency, AddRuntimeReference, AddDeploymentFile) so the manifest notices them and the rules (no absolute paths, no traversal, no symlink escape, external roots only by alias) stay intact.

Keep output deterministic

A non-deterministic build breaks two things quietly: every incremental rebuild becomes a coin toss, because the manifest fingerprints the output, and every recorded interoperability result stops meaning anything, because the file someone loaded in a genuine XNA runtime is not the file the next build produces. CNA's own tests state the rule: identical inputs produce identical files across worker counts and across process runs.

• Compression is deterministic (the same payload always produces the same bytes), including CNA's LZX encoder. Keep ordering explicit (sort by logical name, ordered dictionaries, stable iteration), and never let a hash-map order, a timestamp, a pointer value or a thread schedule reach an output byte or the manifest.
• XnbWorkerDeterminismTests.cpp runs the real cna-content in separate OS processes at several worker counts over a deliberately non-uniform asset mix (a Model, whose type table and shared-resource table are where an unordered container would first show, plus an effect built by an external compiler); XnaBuildDeterminismTests.cpp builds one corpus four times in four processes; ContentPipelineCliTest covers worker-count determinism of directory builds. Separate processes matter: one process can be deterministic by accident (a warmed cache, an allocator that returns the same addresses).
• The XNB writer identity is the codec version plus an options digest (platform, container version, profile, compression, reader-name spelling), so building for another target invalidates the previous artifacts instead of reusing them. When the encoding changes, bump the component or schema identity and check that old outputs rebuild; CustomWriterSchemaAndCodecEvolutionCannotSkipStaleOutput is the fixture. A test that only checks that a clean build works misses stale-cache and unsafe-cleanup regressions.
• The manifest is .cna-content-manifest.json at format version 9. A manifest that fails to parse is Corrupt, another version is Incompatible, and only a Current manifest may authorise deleting old outputs; cleanup refuses symlinks, escapes and files whose bytes no longer match the recorded SHA-256. It is written only when every item succeeded. Do not weaken any of this to make a clean-up test pass.

Reproduce narrowly

cmake --preset unit                                     # Debug, STUB, tests on; networking, video, Draco off
cmake --build --preset unit-content                     # CnaContentTests (canonical engine, CNB and XNB tests)
cmake --build cmake-build-unit --target CnaContentPipelineTests    # the XNA-facade and build-time suites (no preset)
cmake --build cmake-build-unit --target cna_content_tool           # the cna-content executable
./cmake-build-unit/CnaContentTests --gtest_filter='Texture2DContentPipelineTest.*:ContentPipelineCoreTest.*'   # from the repository root
cna-content build <sources> -o <output> --format cnb --workers 1 --explain    # one source directory, one worker, say why each node rebuilt

Two notes on that loop. The tests that spawn the tool receive its path through a compile definition and a dependency, and they are excluded on Windows, Emscripten, Android and iOS. And the canonical pipeline tests live in the content group (CnaContentTests) while the XNA-facade and effect-source tests live in the content_pipeline group (CnaContentPipelineTests); the unit-content preset builds only the first, so a facade change needs the second built by name. Reduce a failing build to one asset, one worker and --explain; the coordinator's own error carries the stage and component.

Make the smallest change

• New source type. Declare lowercase extensions and stable output types; add a processor for that type or require an explicit one; register CNB and XNB writers as appropriate, or document the absence with DocumentAbsentWriter; and test the full command-line-to-runtime round trip.
• New or changed writer or schema. Bump the component and schema identity, declare the asset, schema and codec identities the build will check, and prove old outputs rebuild. Container-specific limits belong to the writer: for textures, Reach and HiDef size limits apply only to .xnb output, and a block-compression request against .cnb is answered with a warning and Rgba8 pixels because CNB texture schema 1 stores Rgba8 only.
• Child assets. Give nested outputs stable logical names, schemas and dependency records; identical repeated nested outputs are shared, different ones under one name are a collision.
• The XNA facade. It is a view over the canonical engine, never a second engine or a second XNB dialect. A custom XNA route must register its importer, processor and writer before the freeze and be exercised through XnaCustomPipelineAcceptanceTests.cpp and the custom-pipeline example, not only an isolated class test.
• cna_add_content. It requires TARGET and OUTPUT_DIR and exactly one of SOURCE_DIR or CONTENT_PROJECT, forwards FORMAT, platform, profile and compression to the tool without re-validating them (the tool is the one authority on valid names), refuses FORMAT, XNB_PLATFORM, XNB_PROFILE, XNB_COMPRESS, CONFIG_FILE and XNA_COMPATIBLE together with a .contentproj, creates a custom target outside ALL, and needs a host CONTENT_EXECUTABLE when cross-compiling.
• Do not widen the runtime. A pipeline-only convenience must not add a dependency to cna_content's public closure; if a route needs a new library, it belongs in cna_content_pipeline and behind an option that defaults to something a headless build can satisfy.

Prove it

Nothing in the table was executed for this page. The unit-content preset builds only the canonical group, so state which group each result came from, and remember that tests needing the tool, an external effect compiler, FreeType or FFmpeg are present or absent by configuration: a green run means only what was configured. Windows behaviour comes from the content workflow's native lane (Windows CI), which runs the tool and content tests and checks a determinism hash; it is not built into CnaTests there.

Check the blast radius

Review checklist

• Does anything in the pipeline sources now name a device, a window or a runtime-only type?
• Are components immutable or reentrant, registered before the freeze, and selected by stable names?
• Is output deterministic across worker counts and processes, and was a component or schema identity bumped when bytes changed?
• Do old outputs rebuild, and does cleanup still refuse anything it cannot prove the pipeline owns?
• Is there a pipeline-to-runtime round trip, and, for the facade, are the XnaPipeline* gates and the frozen inventory respected?
• Were source, sibling-file and output containment rules preserved (no traversal, no symlink escape)?
• Does the note say which test group ran, which tests were absent for configuration reasons, and what was not run (Windows, genuine XNA runtime, external effect compiler)?