sharp-runtime streams, UTF-8 text and tasks

CNA snapshot 009d40f5  ·  Deep Dives › sharp-runtime  ·  source links pinned to 009d40f5

Three sharp-runtime families carry most of the behaviour a CNA game touches below the XNA surface: the stream classes every content load goes through, the UTF-8 encoding that decides what malformed text becomes, and the task and thread types a game may use for background work. This page gives their exact semantics, the places where they deliberately differ from .NET, and the evidence behind each rule. sharp-runtime statements were read at next @ 41b918c9, which CNA snapshot 009d40f5 does not pin; CNA statements at 009d40f5.

Stream: a lightweight subset, honest about it

System::IO::Stream calls itself a "lightweight subset of the .NET Stream API". A subclass must implement three members: Read(buffer, offset, count), Close() and getLengthProperty(). Everything else has a default that describes a read-only, non-seekable stream: Write and WriteByte throw NotSupportedException, the Position getter and setter throw NotSupportedException (and the default Seek is written in terms of Position and Length, so it fails the same way), getCanWriteProperty() and getCanSeekProperty() return false, and getCanReadProperty() returns true. Note that getCanSeekProperty() does not gate Seek(); it only reports. That is the shape of .NET's own read-only streams, and it spares a subclass from implementing a full read/write/seek contract it does not support.

The header documents the two ways the defaults can lie. A stream that overrides Write but not getCanWriteProperty() writes while claiming it cannot, and a caller that checks the property first never writes; a stream that cannot read must override getCanReadProperty(), because the default says yes. The rule is: every capability you implement, you also report. A minimal honest subclass looks like this:

class MemoryBlobStream final : public System::IO::Stream
{
public:
    explicit MemoryBlobStream(std::vector<SharpRuntime::bytecs> data) : data_(std::move(data)) {}

    SharpRuntime::intcs Read(SharpRuntime::bytecs buffer[], SharpRuntime::intcs offset,
                             SharpRuntime::intcs count) override
    {
        if (buffer == nullptr && count != 0) throw System::ArgumentNullException("buffer");
        if (offset < 0) throw System::ArgumentOutOfRangeException("offset");
        if (count < 0) throw System::ArgumentOutOfRangeException("count");
        const auto remaining = static_cast<SharpRuntime::intcs>(data_.size()) - position_;
        const SharpRuntime::intcs n = std::min(count, remaining);
        if (n > 0) std::memcpy(buffer + offset, data_.data() + position_, static_cast<std::size_t>(n));
        position_ += n;
        return n;                     // 0 only at the end, never for a bad argument
    }
    void Close() override {}
    [[nodiscard]] SharpRuntime::intcs getLengthProperty() const override
    {
        return static_cast<SharpRuntime::intcs>(data_.size());
    }
    // Write/WriteByte/Seek keep their NotSupportedException defaults, and the
    // inherited CanWrite == false and CanSeek == false describe them truthfully.
private:
    std::vector<SharpRuntime::bytecs> data_;
    SharpRuntime::intcs position_ = 0;
};

MemoryStream

System::IO::MemoryStream is the concrete stream CNA actually uses. Its default constructor creates an empty, writable buffer, as .NET's does. The buffer-copy constructor makes the same default explicit and adds a C++-only way to ask for a read-only copy:

MemoryStream(const bytecs* buffer, intcs size, bool writable = true);

The arguments are validated before the copy is formed. A null pointer with a non-zero size throws ArgumentNullException, and that check runs first, matching .NET's order; a negative size throws ArgumentOutOfRangeException. The one deliberate adaptation is (nullptr, 0), accepted as an empty range because C++ can express an empty pointer range where .NET's array overload simply rejects a null array (sharp-runtime's own BinaryData::ToStream() reaches the constructor that way for empty content). Passing false creates a readable, seekable, non-writable copy whose Write and WriteByte throw NotSupportedException.

CNA relies on exactly that read-only form. ContentManager::OpenStream in modules/content/src/Xna/ContentManager.cpp reads the whole asset first, from the file system or, for a relative path on Android, from the packaged assets through the platform file system, and returns std::make_unique<System::IO::MemoryStream>(bytes.data(), size, false); modules/content/src/Xna/ResourceContentManager.cpp does the same for embedded resources. Because the stream copies the range, it owns its bytes outright and nothing stays open on the manager's behalf. An asset larger than INT32_MAX bytes is refused with a ContentLoadException before the cast to intcs. The XNB decompressor uses two further memory streams for the compressed input and the decompressed output. How loading continues from that stream is on content runtime internals.

Closing closes operations, not the bytes

MemoryStream::Close() changes only its open flag; it keeps both the byte vector and the position. After it, Read, Write, WriteByte, Seek, Length and Position throw ObjectDisposedException ("Cannot access a closed Stream."), and CanRead and CanSeek return false. CanWrite deliberately keeps the construction-time writable flag, because .NET's property is CanWrite => _writable and does not fold in the disposed state. ToArray() and GetBuffer() remain usable, since .NET's own Dispose leaves the buffer in place ("allow TryGetBuffer, GetBuffer & ToArray to work") and the ensureNotClosed() guard follows the same rule. A producer can therefore finish an in-memory blob, close the stream to catch accidental later I/O, and still hand a copy on. The two extraction methods differ: ToArray() returns an independent vector, safe across later writes, while GetBuffer() returns a reference to the live vector that a later reallocation can invalidate.

System::IO::MemoryStream stream;
const SharpRuntime::bytecs payload[] = {1, 2, 3, 4};
stream.Write(payload, 0, 4);
stream.Close();

std::vector<SharpRuntime::bytecs> saved = stream.ToArray();   // still {1, 2, 3, 4}
// stream.getLengthProperty();   // throws ObjectDisposedException

System::IO::MemoryStream readOnly(payload, 4, false);
// readOnly.WriteByte(9);        // throws NotSupportedException

Argument and size limits

Two further properties keep a "memory only" stream from becoming an unchecked byte container. A null buffer throws ArgumentNullException and a negative offset or count throws ArgumentOutOfRangeException; neither is mistaken for an end-of-stream return of zero (before commit 0bf2faf5, Write silently did nothing on invalid arguments and a negative offset reached an out-of-bounds read). A caller may set Position beyond the current end, but the next write computes position + count in int64_t before resizing, and if the sum exceeds the intcs range it throws IO::IOException ("Stream was too long.") instead of wrapping negative and writing outside the vector. The narrower calculation it replaced had the same signed-overflow shape that a UBSan reproduction had already confirmed in Span<T>::Slice, and a wrapped sum would have compared as small enough to skip the resize, turning undefined behaviour into a heap overflow; StreamTests.cpp sets Position to 2147483647 and writes ten bytes to require the exception, for both MemoryStream and UnmanagedMemoryStream.

UTF8Encoding: validation and fallback

System::Text::UTF8Encoding does not pass bytes through as opaque char values. GetBytes() and GetString() validate that the input is well-formed UTF-8 and route each ill-formed byte through the configured fallback before resuming. The rule lives in one place: System::detail::TryDecodeUtf8Scalar() in System/detail/Utf8Scalar.hpp (owned by Core.Base). Five copies of the decode had accumulated across the text module; two refactors (tickets #2014 and #2354, the second on 2026-08-18) moved every caller onto the shared function, and UTF8Encoding.cpp's local wellFormedUtf8Length() is now a thin wrapper that returns the validated length or zero.

The decoder walks the lead byte against the real UTF-8 grammar and checks every continuation byte for the 10xxxxxx pattern. A structurally complete sequence is still rejected if it is overlong (a two-byte form decoding below U+0080, a three-byte form below U+0800, a four-byte form below U+10000), if it encodes a UTF-16 surrogate (U+D800 to U+DFFF, which exist only as a UTF-16 artefact), or if it lies above U+10FFFF. Overlong forms are the historically security-relevant case: a decoder that accepts C0 AF as / lets that byte past a filter that checks only the raw stream. TryDecodeUtf8Scalar is the reporting form: for a structurally ill-formed sequence (bad lead or continuation byte, truncation, overlong) it reports a length of one, but for a structurally valid encoding of a non-scalar (a surrogate, or a value above U+10FFFF) it reports the sequence's own length, so that Rune::TryGetRuneAt can skip it whole (RuneConsumesTheWholeSequenceForANonScalarButOneByteForABreak). Encodings use the substituting form, DecodeUtf8Scalar, which always advances one byte and yields U+FFFD, and wellFormedUtf8Length() returns zero for every failure, after which UTF8Encoding itself advances one byte, so replacement stays per byte for every ill-formed case, a directly encoded surrogate included.

The constructor installs U+FFFD (EF BF BD) as both encoder and decoder replacement, as .NET's UTF8Encoding.SetDefaultFallbacks() does, not the base Encoding class's "?", which is only right for single-byte code pages. Because the loop advances one byte per failure, replacement is per invalid byte, not per attempted sequence:

System::Text::UTF8Encoding utf8;

const SharpRuntime::bytecs euroSign[] = {0xE2, 0x82, 0xAC};        // U+20AC, well formed
std::string ok = utf8.GetString(euroSign, 0, 3);                    // unchanged

const SharpRuntime::bytecs overlongSlash[] = {0xC0, 0xAF};          // '/' in a 2-byte form
std::string rejected = utf8.GetString(overlongSlash, 0, 2);         // two U+FFFD

const SharpRuntime::bytecs loneContinuation[] = {0x80};
std::string alsoRejected = utf8.GetString(loneContinuation, 0, 1);  // one U+FFFD

Three test surfaces pin this at 41b918c9 (named here, not run): UTF8EncodingTests covers round trips, bad continuation bytes, overlong input, truncated sequences, directly encoded surrogates and the exception fallbacks (EncoderFallbackException, DecoderFallbackException); Utf8SharedScalarDecodeTests pins the shared decoder, including AnEncodingSubstitutesOneReplacementPerByteForBothKinds; and the Utf8Tests family validates the span-oriented validator for overlong sequences, surrogates, code points above U+10FFFF, truncation and lone continuation bytes. BinaryReader::ReadChar(), which CNA's XNB readers call, applies the same grammar but throws FormatException instead of substituting (how that method arrived in sharp-runtime). Remember that lengths and indices here are bytes (the UTF-8 unit).

Tasks: a real but scoped async model

System::Threading::Tasks::Task lives in the Threading.Tasks component, which is not in CNA's closure; CNA's modules include no task header, so a game that uses tasks names that component itself (how). The model is real but deliberately small.

• No pool, no scheduler. Task::Run and the task constructors start the work with std::async(std::launch::async, …), one thread per task. In a single-threaded Emscripten build they throw PlatformNotSupportedException ("std::async requires pthreads"); threaded WebAssembly needs SHARP_RUNTIME_ENABLE_EMSCRIPTEN_THREADS, which CNA sets from CNA_ENABLE_EMSCRIPTEN_THREADS.
• Wait. Wait() rethrows a faulted task's stored exception directly, not wrapped in AggregateException, and throws TaskCanceledException for a canceled one.
• Continuations run inline. A continuation runs synchronously on whichever thread completes the antecedent, or immediately on the calling thread if the antecedent has already finished: always as if TaskContinuationOptions::ExecuteSynchronously were set. NotOnFaulted, NotOnCanceled, NotOnRanToCompletion and their OnlyOn… compositions are honoured; PreferFairness, LongRunning, AttachedToParent, DenyChildAttach, HideScheduler and LazyCancellation are accepted for API parity and have no effect. The continuation captures the antecedent's state through a std::weak_ptr, so the antecedent's own continuation list cannot keep itself alive in a cycle. A continuation does not rethrow the antecedent's exception; it inspects it.
• Generic results. TaskT<TResult> has both an action overload of ContinueWith returning Task and a result-producing overload returning TaskT<TNewResult>. The lightweight antecedent handed to a callback shares the terminal status, result and exception, but not the original cancellation token: its token property reports CancellationToken::None().

using System::Threading::Tasks::Task;

Task loadTask = Task::Run([] { LoadLevelFromDisk(); });   // its own thread
Task continuation = loadTask.ContinueWith([](Task antecedent) {
    if (antecedent.getIsFaultedProperty()) {
        LogLevelLoadFailure();       // the exception is NOT rethrown here
        return;
    }
    FinishLevelSetup();              // runs inline on the thread that finished loadTask
});
continuation.Wait();

WhenAll, WhenAny and the missing generic combinators

Task::WhenAll(std::vector<Task>) returns an already completed task for an empty vector without starting a thread, throws ArgumentException for a moved-from task (this port's equivalent of a null Task), and otherwise waits for every input, never short-circuiting on the first fault. If any faulted, the first fault in input order is rethrown when the result is waited on, the same "rethrow, do not aggregate" simplification Wait() makes. If none faulted but one was canceled, waiting throws TaskCanceledException, yet the returned task's status reads Faulted, not Canceled, because it is built from an action that can only escape by throwing; that is a documented deviation. Since 2026-07-14 it tells a genuinely canceled input from one that faulted with a directly thrown TaskCanceledException by the input's own status rather than by the exception type.

Task::WhenAny(std::vector<Task>) returns a TaskT<Task> that always completes successfully with the first input to reach any terminal state; the caller inspects that task's own status. An empty vector throws ArgumentException. It was rebuilt on 2026-07-14 to follow .NET's zero-extra-thread design: it registers an inline callback on each input's shared state, the first callback to win an atomic compare-exchange completes the result, and later ones are no-ops, so no watcher thread is created, joined or detached. Neither combinator has an overload for std::vector<TaskT<TResult>>: a caller can continue one generic task but cannot express .NET's generic WhenAll/WhenAny that produce a result array or a typed winner.

TaskCompletionSource and Thread

TaskCompletionSource<TResult> bridges an external producer into a task. Its Try… methods claim completion with an atomic compare-exchange, so two racing producers cannot both settle the promise. TrySetCanceled() and a caller's SetException(TaskCanceledException) both settle the promise with the same exception type, so a separate producer-set atomic flag identifies only the genuine cancellation path; a caller's own exception stays a Faulted result. The same file carries a repaired race between two valid-looking steps: TrySetResult claimed completion before copying the result into the promise, so a throwing copy constructor left the source claimed but never ready, and every waiter hung. The fix stores the copy failure in the promise and still rethrows to the immediate producer:

bool TrySetResult(const TResult& result) {
    bool expected = false;
    if (!completed_.compare_exchange_strong(expected, true)) return false;
    try {
        promise_.set_value(result);
    } catch (...) {
        promise_.set_exception(std::current_exception());   // waiters become ready
        throw;                                              // the producer still sees it
    }
    return true;
}

System::Threading::Thread is a thin wrapper over std::thread that does not start at construction. A second Start() throws ThreadStateException ("Thread is running or terminated; it cannot restart."), .NET's one-shot contract, instead of silently restarting; a parameterised thread started with the parameterless Start() runs with a null argument and no exception, which is .NET's own asymmetry and is pinned by a test. Channels use notify_all() rather than notify_one() on their condition variables, because several writers or readers can be blocked at once and waking one could strand the others; that lost wake-up was found by a code review and reproduced before the fix (5770e476).

What these types prove and what they do not is part of the evidence model on parity and verification: the sanitizer and review findings behind several of the rules above are listed there.