The CNJ model toolchain: gltf_to_cnj, the Model envelope and sidecars

CNA snapshot 009d40f5  ·  Deep Dives › Models, glTF & 3D  ·  source links pinned to 009d40f5

Direct runtime glTF loading keeps every mesh group in one Model but fixes the unit scale at 1. The offline converter cna_tool_gltf_to_cnj runs the same import core and material packing, then writes each mesh group as an explicit .cnj Model descriptor with binary sidecars — the reproducible path for a controlled unit scale, one asset per group, and reviewable, committed build artefacts. This page states what the tool emits, the version-2 Model envelope, what the .cnj reader validates and what it trusts, how the sidecars are read, what "parity" between the two routes means, and the one lossy material approximation both share. It is for build pipelines that convert glTF offline and for anyone inspecting or editing converter output.

A renderer-free build tool

ToolGltfToCnj.cmake builds cna_tool_gltf_to_cnj unconditionally rather than behind CNA_BUILD_TESTS, linking CNA and its sharp-runtime components; it never initialises a window, a graphics device or a renderer, because it only constructs math value types. It also compiles the test-scope oracle source GltfOracleEXT.cpp, which serves its diagnostic --dump-oracle mode (and, in a Draco build, links the decoder for that mode's independent position decode). The command line, from gltf_to_cnj.cpp:

cna_tool_gltf_to_cnj <input.gltf|input.glb> <outputDir> <baseName> [unitScale]
cna_tool_gltf_to_cnj --dump-oracle <input.gltf|input.glb> <emptyOutputDir>

# a centimetre-authored scene, scaled into metres
cna_tool_gltf_to_cnj art/city.glb Content city 0.01

unitScale must parse as a number and be positive; either failure, and any conversion error, prints error: … and exits with status 1. The output directory is created if needed. The same conversion is compiled into the content library as CNA::Tools::Gltf::ConvertGltfToCnj, which cna-content and cna_tool_gltf_to_cnb call before compiling the .cnj into a .cnb; the tool is one of three front ends of one implementation (see the import core).

What a conversion writes

A file with one group keeps the base name. With several, the unskinned group gets _static and each skinned group _<skinName> (or _skinN for an unnamed skin); names that collide after filename sanitising are refused rather than overwriting each other. Runtime and offline routes preserve the same group set but package it differently: one multi-skin runtime model versus separately named converter outputs. Every path the descriptor names is a bare file name resolved against the ContentManager root, not the descriptor's folder, which is why converting straight into the content root works and converting into a subfolder needs the names rewritten — the practical consequence is on Tutorial 111.

The Model envelope is version 2

Every .cnj document carries cnjVersion and type. The version ceiling is per type: ValidateCnjEnvelope (CnjEnvelope.hpp) receives the requested type and that type's maximum, Model passes 2 and every other type 1, so raising the Model ceiling did not bless version 2 for textures, effects or clips — a version one reader accepts can still be rejected by another. CnjEnvelopeTests.cpp pins that a raised ceiling does not leak between types. The Model reader also rejects sourceFile: a Model .cnj is always self-contained.

Version 2 adds a top-level bones array — each entry a name, a parent index and a 16-float local transform in XNA row-major order, parent before child, with an identity root at index 0 — and a parentBone index on each mesh entry. Those two fields preserve the glTF scene hierarchy and rigid instancing placement that a flat list of pre-transformed meshes could not. The version-2 document can additionally carry, as additive fields within the same ceiling: the structured import report, rigid (scene-node) clips on an unskinned model, material-variant names and per-variant part entries (variantOf/materialVariant), a per-part primitiveTopology, packed-UV selectors and per-map texture transforms, the newer PBR fields (normal scale, occlusion strength, IOR, specular, alpha mode and cutoff, double-sidedness, unlit), a partOfMesh index that groups consecutive primitive entries into one ModelMesh, and the flat-normal morph flag. It does not carry imported cameras — only the report's camera-node count — so cameras exist on direct glTF loads alone. The reader supplies empty or glTF-default values for every field an older document lacks.

Version-1 Model files remain loadable. With no general hierarchy (no bones, or only a root name), the reader builds the older compatibility shape: one Root plus one synthetic child bone per mesh, named after the mesh — which is what lets ported XNA samples find a named bone per rigid part through Model.Bones["PartName"]. Version 2 is an additive fidelity improvement, not a flag day.

Producer invariants the reader does not enforce

Three shape rules hold for every generated file but are not fully validated by the reader in ContentManager.cpp:

• Entry 0 is always built as the root with the entry's name (or Root); its serialized transform and parent are not applied.
• For later entries the parent index is bounds-checked against the whole array (an out-of-range parent, or a mesh parentBone out of range, is a ContentLoadException), but parent < child is not required. A forward parent builds a valid pointer tree whose CopyAbsoluteBoneTransformsTo order is wrong, because that composition is a single forward pass.
• A transform array with any count other than 16 silently becomes identity.

A hand-edited version-2 descriptor can therefore load into an internally inconsistent graph rather than fail at the schema boundary. The JSON is readable, but its numbers and the binary files form one asset contract: do not hand-edit counts, offsets or hierarchy without regenerating or validating the sidecars.

Binary sidecars and bounded reads

Vertex and index bytes use the packed ABI from vertex packing. The descriptor supplies a vertexStride (default 16 when absent) and file names; the reader derives vertex and index counts from the file sizes and uses 32-bit indices exactly when the vertex count exceeds 65,535. Its validation is partial, and precisely so:

• a stride of zero or less silently skips that mesh entry;
• a positive stride outside the canonical table is refused with a ContentLoadException naming the stride (it used to produce a vertex buffer that was never filled);
• vertex and index byte lengths that are not whole multiples of their element size are truncated by integer division rather than rejected.

Morph sidecars are read strictly: a target count outside 0–100,000, a per-target vertex count other than 0 or the mesh's vertex count, a normal or tangent flag other than 0 or 1, an unknown trailing block, a tangent trailer with an unsupported version or a different target count, and any bytes left over after the trailer are each a ContentLoadException. The tangent deltas live in that optional, magic-tagged trailer so that sidecars written before it existed stay readable and older readers ignore the new data. What remains trusted is the semantic agreement between the descriptor and its sidecars.

The skeleton sidecar's bone count must lie between 0 and 100,000 before anything is allocated. Historically it held two matrix blocks — bind poses and inverse binds — and version 2 may append a third, SkeletonRootPrefix (see the root prefix). The reader takes the third block only when at least boneCount × 64 bytes remain, and otherwise leaves the prefix empty, which the player reads as identity. That test cannot distinguish a genuinely old file from a partially written new one: a trailing fragment shorter than a whole block is ignored, not rejected, and bytes after a complete block are not required to be consumed. Appending an optional fixed-size block instead of changing the meaning of the first two is what keeps earlier sidecars valid; producers should still publish each generated sidecar atomically.

Clip references that name a .cnj go through the manager's ordinary cached Load<AnimationClipEXT>, so several models (or repeated names) can share one clip object — CnjModelSharedAnimationClipTests.cpp pins that two models share one clip, that a raw .clip.bin still works and that a missing shared clip throws. Mesh buffers, stock effects and most graph objects are still created per described part and retained by the model's private ownership bundle; a clip marked as a joint-palette clip on a model with no skeleton is skipped with a warning rather than attached where it would pose the wrong bones.

Offline and runtime parity

Sharing GltfImportCore reduces drift but does not prove parity by itself: after extraction one front end creates textures, effects, buffers and a live model, while the other serializes bytes and descriptors that a later .cnj reader recreates. The contract is not byte identity between a live object and JSON. It is that a group imported directly and the same group converted and loaded from .cnj produce the same scene-node placement, palette coordinate spaces, vertex and index bytes, material state and animation records. A fix made in only one front end is a regression even if the shared parser still builds.

GltfToCnjToolTests.cpp spawns the real converter binary as a separate process (so it is excluded on Windows, Emscripten, Android and iOS builds, where that process API is not used) and compares both routes: TheOfflineAndRuntimePathsProduceIdenticalSkinningDataForEverySkinFixture, TheOfflineAndRuntimePathsProduceIdenticalAnimationClipsForEveryAnimFixture, OfflineAndRuntimePathsHaveIdenticalL6MaterialStateForTheCorpus, MorphedFlatNormalsAreRecomputedIdenticallyOnBothLoadPaths and StructuredImportDiagnosticsSurviveBothLoadPaths, alongside round trips for PBR and skinned-PBR materials, morph targets and cubic weight tracks, punctual lights, material variants, multi-primitive meshes, unit scale on vertices and on the node hierarchy, and Draco.

Occlusion through DualTextureEffect

glTF occlusion uses 1.0 for full light and 0.0 for full occlusion. XNA's DualTextureEffect multiplies its second texture contribution by 2, so 0.5 is neutral in its equation; passing glTF's image unchanged would brighten, not preserve, an unoccluded surface. For the dual-texture approximation, RemapOcclusionImageForDualTextureEXT decodes the occlusion image, halves its RGB channels (alpha unchanged) and re-encodes it as PNG:

o_dual = o_gltf / 2, so 2 * o_dual = o_gltf.

The halving is integer division on 8-bit channels, so full white becomes 127 and the neutral value lands at 254/255 rather than exactly 1 — a sub-percent darkening. An image the codec cannot decode is refused by the remap function rather than guessed at, but neither front end turns that into an import failure: the converter warns that the occlusion texture "could not be decoded for the DualTextureEffect brightness fix" and writes the original bytes unmodified (so that _texoccN file keeps the source extension and the surface is again about twice as bright where unoccluded), and the runtime reader loads the original image without a diagnostic. The remapped image is cached separately from the original because the same source image may also be bound, unmodified, to a PbrEffect occlusion map, where glTF's convention is correct; the converter writes it as a distinct _texoccN.png. Both front ends use the one remap function, covered by GltfOcclusionRemapTests.cpp and by RuntimeGltfModelTest.RemapsOcclusionTextureBrightnessForDualTextureEffectFromGltf.

At this snapshot the route is narrow. An ordinary metallic-roughness material with base-colour and occlusion maps is imported through PbrEffect, which samples a real occlusion map (BaseColorAndOcclusionTexturesImportThroughPbrEffectWithARealOcclusionMap); only an unskinned, uncoloured primitive whose material declares KHR_materials_unlit and carries both maps reaches DualTextureEffect and the remap.

Choosing a route

Use direct glTF for rapid iteration on a file that is in metres, whether it holds one group or several. Use the converter when a unit conversion is required, when separately named assets per group are wanted, or when the asset bytes must be reviewed and committed; compile the result to .cnb with cna_tool_cnj_to_cnb, or go from glTF to .cnb in one step with cna_tool_gltf_to_cnb --unit-scale (see CNB Format). Use .xnb when compatibility with an existing XNA content pipeline governs. The legacy .skinnedmodel.json path is for SkinnedModelEXT, not an alternate .cnj spelling. What each route keeps is tabulated on Model Loading.

Evidence

Checked by reading the TARGET tool, reader and tests at 009d40f5; not built or executed. The converter's platform-gated process tests and the envelope suite are the main evidence; the command-line catalogue is on Command-Line Tools.