The XNB container and the ContentReader object graph

CNA snapshot 009d40f5  ·  Deep Dives › Content  ·  source links pinned to 009d40f5

An .xnb file is a small framing header, an optionally compressed body, a table naming the readers the body needs, and a serialized object graph that one ContentReader session turns into a C++ value. This page gives the byte-level and session-level rules CNA applies at this snapshot, where each resource limit is actually enforced, and the rules a custom reader must keep. The user-level view (what loads, the 61 built-in readers, writing XNB) is on XNB Loading & Interoperability and Tutorial 147; reader-by-reader wire contracts are on XNB type readers.

Three boundaries, checked in order

Passing one boundary proves nothing about the next: a valid header does not make the object graph safe, and a correct type reader cannot repair a truncated or allocation-hostile container.

 .xnb bytes (read whole; at most INT32_MAX bytes at run time)
    |
    v
 [1] CONTAINER  10-byte header -> length cross-check -> none | LZX | LZ4 body
    |           compressed payload <= 64 MiB, declared decoded size <= 256 MiB,
    |           decoded length must equal the declared size
    v
 [2] READER TABLE  7-bit count <= 4,096; per entry: name (<= 1 MiB, parsed,
    |              generic nesting <= 256) + Int32 version; every name must be
    |              registered and every version admitted, or the whole file fails
    v
 [3] OBJECT SESSION (one ContentReader)
        shared-resource count <= 1,000,000
        root object --1-based index--> table reader --> nested objects
        all shared objects read, then every queued fixup runs
        typed nesting <= 256; each collection <= 10,000,000 elements
    |
    v
 value returned to Load<T>    (external references re-enter ContentManager::Load)

The ten-byte header

ParseXnbHeader in XnbHeader.hpp validates as it reads and throws on the first problem.

The length cross-check

The declared length is a value the file asserts about itself, so LoadXnbAsset<T> checks it against the bytes it actually read before any offset arithmetic uses it: it must be at least 10 and no larger than the file, and a compressed file must be at least 14 bytes long so the four-byte decompressed-size field exists. The check exists because a file that claims more bytes than it holds would otherwise let the LZX branch compute a compressed size that reads past the end of the buffer. At run time the relationship is checked in one direction only: an uncompressed body is everything after byte 10 whatever the header says, and a compressed payload is totalLength - 14 bytes, so bytes beyond the declared length are tolerated. The build-time importer that reads an .xnb as a cna-content source (DecodeXnbCanonicalAsset in XnbCanonicalData.cpp) is stricter: the file must not exceed 64 MiB and the declared length must equal the file size exactly. The untyped external-reference decoder omits the 14-byte guard, as recorded on Content runtime internals.

Compression is four-valued

Two independent flag bits give four outcomes, kept as the enum XnbCompression rather than a Boolean so that every consumer branches on the actual scheme. Both bit values were taken from MonoGame's own ContentManager.cs.

LZX keeps its state across blocks

LzxDecoder.cpp is a line-by-line C++ port of FNA's LzxDecoder.cs (itself a port of libmspack's lzxd.c), keeping FNA's variable names, control flow and return codes so it can be compared against the original. DecompressXnbPayload in XnbDecompression.cpp constructs one decoder per file, so the sliding window, the repeated-offset LRU and the Huffman tables persist from block to block; a fresh decoder per block would pass trivial single-block payloads and corrupt real multi-block streams. The details:

• The window exponent is 16 (a 64 KiB window); the constructor accepts 15 to 21 and throws UnsupportedLzxWindowSizeRange otherwise.
• Each block starts with a two-byte big-endian compressed size and produces a 32 KiB frame; a leading 0xFF selects the five-byte form that also carries an explicit frame size. A zero block or frame size ends the loop.
• Before decoding, the compressed payload must be at most maxFileSize and the declared decompressed size at most maxDecompressedSize. After the last block the produced length must equal the declared size exactly. By reading, that comparison happens once, after the loop: the loop does not stop early when the output passes the declared size (not exercised).
• Intel E8 call translation is reproduced from FNA unfinished, so a stream that enables it fails to decode, as it would under FNA.
• A decoder error is reported as ContentLoadException ("Decompression of … failed").

The strongest evidence is cross-implementation. LzxDecoderDifferentialTests.cpp compares CNA's output byte for byte with reference files produced by running FNA's unmodified C# decoder under Mono on the payloads of two MonoGame fixtures, Explosion.xnb (a single block) and FontCalibri14.xnb (a multi-block stream of 44,032 bytes); the procedure is recorded in the fixture folder's README.md. A compressor and decoder that share one wrong assumption can agree with each other; an independent implementation cannot. LzxDecoderFuzzTest then mutates those two payloads and the size hint 2,000 times and requires every successful decode to produce exactly the declared length.

The port is line by line in control flow but not in memory safety. FNA's C# indexes its Huffman tables and window through CLR bounds-checked arrays, so a corrupt code-length table or match offset is a catchable exception there and undefined behaviour in a direct C++ port. LzxDecoder.cpp therefore rejects such input explicitly: MakeDecodeTable returns its error code when the long-code table-growth path would leave the table, and both match paths refuse an offset outside the window. The table-growth case was a real heap-buffer-overflow, found by the mutation fuzzer (not by the Mono comparison) and re-verified under AddressSanitizer and UBSan (plans/plan_xnb.md XNB-30 and XNB-30A). Keep those guards when comparing the port with the C# original.

LZ4 is one bounded raw block

MonoGame's LZ4 flag means one raw LZ4 block, decoded by CNA's own DecompressXnbLz4Payload with no external library. The compressed size must be positive and at most maxFileSize, the declared size at most maxDecompressedSize; the output buffer is allocated at the declared size up front, every literal run and match is checked against both the remaining input and the remaining output, a zero match offset or one reaching before the decoded history is refused, length extensions are overflow-checked, and at the end both the input and the output must be consumed exactly. The untyped external-reference path does not decode LZ4 and refuses such a file by name.

The type-reader table

The decoded stream begins with the table (XnbTypeReaderTable.hpp): a 7-bit-encoded count between 0 and maxTypeReaderCount (4,096), then per entry a length-prefixed name and an Int32 version. Each entry keeps both the raw name and its normalised canonical key. The 1 MiB name limit is applied after the string is read, because BinaryReader::ReadString takes no cap; sharp-runtime's reader refuses, before allocating, a declared length larger than what remains of a seekable stream (read at an unpinned sibling revision). A malformed name's std::invalid_argument is caught at this boundary and rethrown as ContentLoadException, so a caller needs one exception type for a malformed table.

Canonical names need a real parser

A reader name may carry nested generic arguments, assembly qualifiers, versions, cultures and public-key tokens, so cutting at the first comma breaks every generic name. XnbTypeName::ParseOne in XnbTypeName.hpp matches brackets by depth, recurses into each argument, drops the assembly metadata and rebuilds the key as Base[[arg1],[arg2]], so a file's

Microsoft.Xna.Framework.Content.ListReader`1[[Microsoft.Xna.Framework.Rectangle, Microsoft.Xna.Framework, Version=4.0.0.0, Culture=neutral, PublicKeyToken=842cf8be1de50553]]

becomes the registry key Microsoft.Xna.Framework.Content.ListReader`1[[Microsoft.Xna.Framework.Rectangle]]. Array rank specifiers ([], [,], jagged [][]) are told apart from argument lists by their contents and kept verbatim after the arguments, so List<int[]> parses. The recursion is bounded by maxObjectNestingDepth: a crafted name costs about four bytes per level, and before the bound a name under one megabyte exhausted the stack (XnbTypeNameTest.NestingDepthExceedingDefaultLimitThrowsInvalidArgumentNotCrash).

Reader versions are admitted strictly

Every entry's version must pass reader->SupportsVersion(version), whose default requires equality with getTypeVersionProperty(), which defaults to 0. FNA reads the integer and ignores it, so this is a CNA compatibility boundary separate from the container version. No built-in reader overrides either member, so every built-in admits version 0 only, and the canonical decoders that read nested objects inline (SpriteFont's atlas and lists, the Int32, Single and String fields of Song and Video) require version 0 of the nested reader as well. A custom ContentTypeReader<T> can override either member; the XNA-shaped pipeline façade lets a ContentTypeWriter declare a non-zero version (the custom-pipeline example writes 2), and the game-side reader must then report the same one. ReflectiveTypeReaderBuilder<T> offers no version setting, so its readers admit version 0 only.

One ContentReader session per file

Initialisation

The manager constructs the ContentReader positioned just after the container header, and ReadAsset<T>() runs three steps (ContentReader.cpp):

1. InitializeTypeReaders() parses the table at the current position, creates one fresh reader per entry through ContentTypeReaderManager::CreateReader and checks its version; an unregistered name fails the whole file even if no object ever uses that reader. A second pass then calls Initialize() on every reader, so a reader may rely on its peers existing, matching FNA's two-pass LoadAssetReaders. The ContentTypeReaderManager handed to Initialize() is a temporary local; a reader must not keep a reference to it. Finally the shared-resource count is read and bounded (0 to 1,000,000).
2. The root object is read through the one-based dispatch.
3. ReadSharedResources() reads every shared object and only then runs the queued fixups.

The constructor's comment still says the caller has already consumed the table; it is stale, since ReadAsset parses it. A second ReadAsset on the same reader parses again at the current position instead of rewinding, so one reader is one body. The reader instances, the parsed table and the fixup lists die with the session. The reader holds its stream and manager as raw pointers and, when the manager has a direct device pointer, a renderer-thread context lease for the session's lifetime.

Object indices and null

Every object reference is a 7-bit-encoded index into the file's table, one-based. Zero means null: the reader returns an existing instance if the caller supplied one, otherwise a value-initialised T{} when T is default-constructible, and otherwise throws ContentLoadException, because C#'s free default(T) has no C++ counterpart for a type such as SpriteFont. A negative index or one beyond the table is "an incorrect type reader index". A positive n selects reader n - 1, and its result is converted with std::any_cast<T>, so a reader whose result type differs from the requested one surfaces std::bad_any_cast.

Typed reads (ReadObject<T>) run under an RAII ObjectDepthGuard that increments a counter, refuses depths beyond maxObjectNestingDepth and decrements on every exit including exceptions, so one failed nested read cannot poison the session's remaining budget (ContentReaderTest.ObjectNestingDepthExceedingLimitThrowsContentLoadException). The untyped path, InnerReadObjectAny, used by ReadObject() without a type (Model tags, Dictionary<String, Object> values), by the shared-resource pass and by the untyped ReadAsset(), neither checks nor increments that counter. By reading, a chain of object-typed values that nest through the untyped path, such as a Dictionary<String, Object> whose values are further such dictionaries, is not bounded by the 256 limit; this was not exercised.

Shared resources and fixups

ReadSharedResource<T>(fixup) reads an index: a positive one queues the fixup against slot index - 1 (an out-of-range slot is a ContentLoadException), while zero queues nothing. The implementation tests only index > 0, so a negative index also queues nothing and is not reported. Because every shared object is read before any fixup runs, references may point forward and may form cycles, and identity is preserved: several mesh parts naming one effect receive the same object. A fixup whose value has the wrong type throws std::bad_any_cast, and fixups are not a transaction: side effects of fixups that ran before a failure remain.

External references

The typed ReadExternalReference<T>() reads a string: empty gives std::nullopt, a reader with no manager throws, and otherwise the path is resolved relative to the referencing asset's logical folder (absolute, drive and UNC spellings refused; a lexically normalised result above the root refused; a sibling ../textures/foo allowed) and loaded with ContentManager::Load<T>, so it walks the full ladder and shares the typed cache. It is defined in the .cpp and explicitly instantiated for Texture2D, TextureCube and shared_ptr<Effect> only; another T does not link. The untyped ReadExternalReference(), used by ExternalReferenceReader, loads through LoadUntypedXnbReference, which accepts only a compiled .xnb, caches under typeid(std::any) and refuses LZ4.

Disposal recording is narrow

RecordDisposable fires only for a statically typed shared_ptr<U> where U derives from IDisposable, and only when the caller passed a callback. Load<T> passes none, the untyped path never records, and the manager-side fallback is still marked deferred in ContentReader.hpp; the comment on ReadAsset<T> that an empty callback lets the manager record the disposables itself does not match that code. Only a derived manager that calls ReadAsset<T> with its own callback receives them (ContentRuntimeContractTest.ReadAssetHandsEveryDisposableToTheRecorder). The reader is not a general lifetime coordinator; asset lifetime is the shared ownership described on ContentManager resolution rules.

ContentTypeReader<T>, type erasure and existing instances

ContentTypeReader.hpp keeps XNA's name for the template a ported reader derives from and introduces ContentTypeReaderBase as the one forced rename, because C++ cannot declare a plain class and a template with the same bare name. The target type is a canonical name string, not a runtime Type; the eventual any_cast is the effective type check. A reader built with the parameterless constructor gets a name only when T is a default-constructible System::Object; otherwise it is unnamed and can be invoked explicitly but is never selected by a table. Every named reader records typeid(T) against its name in a second process-wide map, which is what lets ReadRawObject<T>() and GetTypeReader(Type) find a reader without reflection.

• Boxing. ReadUntyped stores a copyable result directly in the std::any and boxes a move-only one as shared_ptr<T>; an abstract T throws, because it has no value representation. Indexed dispatch unwraps both forms, which is how a bare SoundEffect root loads. The explicit-reader overloads ReadObject<T>(reader) and ReadRawObject<T>(reader) always cast to a bare T and therefore fail with std::bad_any_cast for move-only targets; use indexed dispatch for those.
• Existing instances. existingInstance is a std::optional<T> moved into the reader, and a value comes back; the caller's original C++ object is not updated in place. CanDeserializeIntoExistingObject is declared and overridden to true by ListReader and DictionaryReader, but ContentReader never consults it, so it is inert metadata where FNA uses it to decide whether an existing instance is passed down.
• Collections. ListReader appends to an existing list (as FNA's list.Add loop does), DictionaryReader clears first, ArrayReader resizes to the stored count instead of trusting the caller's size, and the Dictionary<String, Object> reader keeps an existing map's entries without clearing. Dictionaries insert with emplace, so a duplicate key silently keeps its first value where FNA's Dictionary.Add throws.
• Element readers. Without reflection, each closed collection is its own registration that names its element reader; the element reader is created fresh from the global registry on each read rather than taken from the file's initialised table. That is correct for CNA's stateless built-ins; a stateful or Initialize()-dependent custom element reader receives an uninitialised instance, so register and test each exact combination. Reference-type elements (String, shared_ptr values and nested lists or arrays) carry their own one-based index and go through indexed dispatch; value-type elements use the element reader directly.

Limits and where each is enforced

XnbReadLimits (XnbReadLimits.hpp) is consulted at named sites, not as one aggregate budget.

Declared-length blobs (pixels, samples, vertices, indices, bytecode) are read with ReadBytesExactOrThrow, because BinaryReader::ReadBytes silently trims at end of stream and callers used to pass the original count to pointer-and-count APIs, a confirmed heap-buffer-overflow. Dimension products go through CheckedMultiplyOrThrow in XnbArithmetic.hpp, which division-checks each step, because widening to 64 bits was not enough: two dimensions near INT32_MAX times four overflow int64_t as well. How these fit the wider hostile-input picture is on Content input boundaries.

The process-wide reader registry

ContentTypeReaderManager.cpp keeps a function-local static map from canonical name to factory. AddTypeCreator keeps the first factory for an exact key and silently ignores later ones; it does not reject an empty name or an empty factory (an empty factory fails only when a file names it, with std::bad_function_call). CreateReader returns a fresh instance per call. ClearTypeCreators() empties the registry for every manager in the process and exists for test isolation; RemoveTypeCreatorEXT erases one name. Nothing is locked, so registration must finish before any concurrent load. The registry is one of four registration systems, and the similar names are not interchangeable:

RegisterAllBuiltInXnbReaders() registers every built-in family and is idempotent. Game's constructor calls it; ContentManager's constructors deliberately do not, because many tests clear the registry first to exercise the unregistered-reader path. Its header comment still names the general EffectReader as a known-unsupported placeholder; the implementation registers the real reader, and RegisterKnownUnsupportedXnbReaders() now registers nothing.

Rules for a custom binary reader

Field-by-field decoding can look right while breaking identity, null semantics, lifetime or resource bounds. A reader that behaves like the built-ins:

1. registers its exact canonical name, generic arguments and all, before the first load;
2. treats index 0 as null and positive indices as one-based;
3. declares its serialized-version policy deliberately (getTypeVersionProperty() or SupportsVersion);
4. uses indexed dispatch for polymorphic and move-only members;
5. bounds every count and decoded size before allocating, with CheckCollectionElementCount, CheckDecodedByteSize and ReadBytesExactOrThrow;
6. queues shared-resource fixups and lets them run after all shared objects are read;
7. loads external files only through ReadExternalReference, so containment and the cache apply;
8. keeps no reference to the reader, the temporary manager or the stream beyond the session.

A complete test uses a real container and table, loads through ContentManager, includes nested and shared references where the type has them, adds a failure case that must be refused before allocating, and checks an observable result beyond successful construction. Tutorial 148 builds such a reader and its test file step by step.

What the in-tree tests prove about registration

Two focused suites show that the registry is the whole extension story. Neither needs a Game.

• A reader defined entirely outside CNA. CustomContentTypeReaderTests.cpp declares a stand-in game type GameLevelData (a room count and a name) and a GameLevelDataReader that uses nothing but the public ContentTypeReader<T> base. It registers the reader with one AddTypeCreator("MyGame.Content.GameLevelDataReader", …) call, writes a hand-built level1.xnb into a scratch root and reads it back through ContentManager(nullptr, root).Load<GameLevelData>("level1"). A second case leaves the reader unregistered and expects a ContentLoadException, not a crash or a wrong-type result. A third registers the custom reader next to a replacement for a built-in canonical name and shows that the two entries in the process-wide registry do not interfere.
• A standalone manager after explicit registration. XnbBuiltInReaderRegistrationTests.cpp clears the registry and calls RegisterAllBuiltInXnbReaders() in its fixture, and checks that a repeat call is idempotent. Six FreshContentManagerLoads…WithNoOtherSetup cases then load a real retained fixture through a manager built as ContentManager(nullptr, dir) plus setGraphicsDevice, one each for Texture2D, TextureCube, Model, SpriteFont, SoundEffect and Song. That is exactly the standalone pattern a tool or test needs. A Game subclass gets the same registration from its constructor.

Both suites use a hand-built or retained container and a real manager load, which is the complete-test shape described above. Checked by reading at 009d40f5; not executed.

Evidence and what is not proven

Read at this snapshot from the headers and sources linked above and from the XNB test folder; nothing was built or executed. The tests named on this page exist and were read, not run. Established by reading only: the late LZX length comparison, the missing depth guard on the untyped object path, and the ReadAsset<T> disposal comment. The sharp-runtime ReadString behaviour was read at a sibling revision the snapshot does not pin.