Sound effects, streaming, capture and the audio evidence

CNA snapshot 009d40f5  ·  Deep Dives › Input, audio, media & services  ·  source links pinned to 009d40f5

This page collects the caller-visible semantics of CNA's sound-effect, streaming and microphone classes that sit between the user guide and the mixer internals: the global audio properties, the exact 3D formulas and their limits, how XNB sound content is decoded, how a dynamic stream counts its pending buffers, what the microphone does when a device cannot open, and, just as important, what the automated audio evidence proves and what it does not. It is for anyone porting XNA audio code or chasing a report of wrong pitch, missing sounds or distortion. The guide is Audio System; the playback path, mixers, devices and lifetimes are on Audio engine internals.

Where the audio facts are decided

Three independent choices decide what a program hears. CNA_AUDIO_PLATFORM selects the device and, through it, the mixer (SDL3 with SDL3_mixer, ALSA with CNA's own mixer, or SDL2 and NULL with no mixer at all), independently of CNA_PLATFORM and of the renderer; the window, event and input behaviour belong to the platform, and pixels to the renderer. The public SoundEffect, DynamicSoundEffectInstance, XACT, Microphone and MediaPlayer contracts are the same under every selection; only capability, device enumeration and audibility change. The selection table, the reserved spellings and the SOUND_ENABLED rule are on Audio: implementations and are not repeated here.

SoundEffect and its instances

Construction routes and conversions

SoundEffect is move-only and has four sources: a file path, FromStream(std::istream&) (which expects a complete container such as a WAV file and returns a heap pointer the caller owns), the two raw-buffer constructors that take headerless 16-bit little-endian PCM with an explicit rate and channel layout, and the content pipeline. Handing a whole RIFF/WAV file to a raw-buffer constructor is a caller error the constructor cannot reject, because a header read as samples is still syntactically valid PCM; it only prints an advisory warning and the header plays as a click. The static SoundEffect::GetSampleDuration(bytes, rate, channels) and GetSampleSizeInBytes(duration, rate, channels) convert between a 16-bit PCM buffer length and time; DynamicSoundEffectInstance and Microphone expose the same conversion for their own format.

Process-wide properties

Four static properties in SoundEffect.cpp affect every live instance: MasterVolume (default 1, applied once at the mixer's master gain, never multiplied into each track), DistanceScale (default 1; a value of 0 or less throws ArgumentOutOfRangeException), DopplerScale (default 1; a negative value throws) and SpeedOfSound (default 343.5, not validated). Without a mixer (SDL2 or NULL) the master volume is stored in a static and nothing else changes.

Pitch and 3D, exactly

Pitch is a frequency ratio of 2pitch, so Pitch = 1 is an octave up and -1 an octave down. Apply3D computes three values from the listener and emitter and latches the instance into its spatial path; call it again whenever either moves (SoundEffectInstance.cpp):

This is a defined stereo approximation, not XACT's DSP graph: there is no elevation, no HRTF, no multi-speaker diffusion, no orientation cone and no per-sample spatial filter, and with several listeners the nearest one decides all three values. Stereo pan is applied as a crossfeed matrix in the track's callback, so a hard pan moves the opposite channel across instead of simply muting it, and that callback slot is shared with the optional per-track filter. The auxiliary reverb send is a no-op because neither mixer has a send/return bus.

AudioListener listener;
listener.setPositionProperty(cameraPosition);
listener.setForwardProperty(cameraForward);
listener.setUpProperty(Vector3::Up);
listener.setVelocityProperty(cameraVelocity);

AudioEmitter emitter;
emitter.setPositionProperty(explosionPosition);
emitter.setVelocityProperty(Vector3::Zero);

SoundEffectInstance explosion = explosionSound.CreateInstance();
explosion.Apply3D(listener, emitter);   // before Play: fixes the instance in 3D mode
explosion.Play();

Syntax-checked against the TARGET headers (not executed). The mode rules (Apply3D after Play on a pan-mode instance throws, and the reverse) and the multi-listener overloads are on Audio: 3D positional audio; Tutorial 119 walks through the attenuation curve.

Sound content formats

The XNB SoundEffectReader (SoundEffectContentTypeReader.cpp) does not hand compressed data to the mixer. It decodes the payload to 16-bit PCM in the content library and constructs the effect through the raw-buffer constructor with the rate, channels and loop region. 16-bit PCM is taken directly; 8-bit PCM is widened; IEEE float 32-bit, MS-ADPCM 4-bit and IMA-ADPCM 4-bit are wrapped as WAV data and decoded by CNA's own WAV decoder, which is compiled into every build, so XNB sounds decode the same whatever audio platform is selected (XnbCanonicalData.cpp). XMA2 and unknown codecs are rejected with a ContentLoadException naming the format. XNA's own content pipeline writes MS-ADPCM with cbSize = 0, that is without the coefficient table a WAV decoder needs; when the format block carries no complete extension, CNA synthesizes the standard one with its seven coefficient pairs (256, 0), (512, -256), (0, 0), (192, 64), (240, 0), (460, -208), (392, -232).

Files loaded by path or stream go to the selected mixer. CNA's configure step switches off the SDL3_mixer back ends that need external codec libraries (GME, XMP tracker modules, mpg123, FluidSynth, Opus, libvorbisfile, Tremor, WavPack and libFLAC, in ThirdPartySDL.cmake), leaving SDL3_mixer's built-in decoders: WAV (including MS-ADPCM and IMA-ADPCM), FLAC through dr_flac, MP3 through dr_mp3 and Ogg Vorbis through stb_vorbis, plus whatever other built-in formats upstream enables by default (older audits of the same configuration list AIFF, VOC and AU). AAC, WMA and Opus are not decodable. The SDL3_mixer submodule is not part of the extracted TARGET tree, so the built-in list follows CNA's configure arguments and the upstream defaults rather than a reading of the pinned revision; in particular, whether that revision keeps a built-in MIDI synthesizer enabled was not established. The ALSA route's own decoder set is on Audio: decoders.

Dynamic streaming: what the pending count means

DynamicSoundEffectInstance accepts headerless 16-bit chunks through SubmitBuffer and, as a CNAEXT, normalized float chunks through SubmitFloatBufferEXT; switching between the two is allowed only while stopped, and a submission of the other kind while playing or paused throws InvalidOperationException (DynamicSoundEffectInstance.cpp). The instance owns its queued mixer stream; the mixer does not. A track that runs dry is not halted, so a later submission simply resumes the sound. As in FNA, Stop(false) throws instead of draining, because there is no authored loop to release into.

PendingBufferCount is the number of chunks queued but not yet handed over plus the chunks handed over and not yet consumed. The consumption rule was corrected once: an earlier version removed a whole chunk from the count as soon as any byte of it had been consumed. The current Update computes one consumed-byte budget (total submitted bytes minus the bytes the stream still reports queued) and removes a chunk only when the budget covers its full size, carrying the remainder to the next chunk. BufferNeeded is then raised once for every buffer the instance is short of MINIMUM_BUFFER_CHECK = 3, so it can fire more than once per update. Three limits remain:

• The count measures consumption by the stream, not playback at the speakers, so it is not a sample-accurate playback clock; the tests that pin the accounting use a source format equal to the mixer format, and whether the byte units stay aligned when the stream resamples a 22.05 or 48 kHz source was not established.
• Submissions are range-checked but not checked for frame alignment: a buffer whose length is not a whole number of sample frames is accepted, where XNA throws. Keep chunk sizes a multiple of 2 × channels bytes (16-bit) or 4 × channels bytes (float).
• Without a mixer (SDL2, NULL) the class compiles and plays nothing; Play() throws NoAudioHardwareException where a mixer exists but no device opens.

The task-level walkthrough, including the three-buffer rule and disposing from inside the handler, is Tutorial 118.

Microphone capture

Microphone (Microphone.cpp) enumerates the selected implementation's recording devices once and keeps that list for the process lifetime; getDefaultProperty() is its first entry, or nullptr under SDL2 and NULL, which have no recording provider. There is no refresh or hot-plug of the list.

• BufferDuration defaults to one second and accepts 100–1000 ms in 10 ms steps, judged on the total milliseconds as XNA's own setter does. FNA reads only the sub-second component, which made 1,100 ms pass and rejected 1,000 ms, the very value the property reports by default; CNA follows XNA, pinned by BufferDurationUsesTotalMillisecondsAsXnaDoes. Descriptions of the FNA behaviour as CNA's are out of date.
• Start() opens the device asking for mono signed 16-bit at 44,100 Hz, keeps it only if the device really delivers mono 16-bit (adopting the negotiated rate for SampleRate and the duration helpers), and otherwise closes it. Like FNA it sets the state to Started even when the device could not be opened, so a caller cannot tell a failed start from a healthy stream that has no data yet: both return 0 bytes from GetData.
• GetData drains what is available into the caller's buffer and returns the byte count; offsets and counts are range-checked without integer overflow, and a read with no data leaves the buffer untouched. It does not check frame alignment: XNA's Microphone.GetData throws ArgumentException for a buffer length, offset or count that is not a whole number of 2-byte mono frames, while CNA accepts an odd count and may return an odd byte count, so keep buffers and counts even. The class also takes no lock (XNA guards every member with one), so start, stop, read and the BufferReady pump all belong on the game-loop thread.
• BufferReady is raised from FrameworkDispatcher::Update when a subscriber exists and more than BufferDuration of audio is queued. Under ALSA the capture thread keeps up to eight seconds, so a game polling once a frame loses nothing between polls.

Microphone* mic = Microphone::getDefaultProperty();
if (mic == nullptr) return;                       // no recording provider or device

DynamicSoundEffectInstance monitor(mic->getSampleRateProperty(), AudioChannels::Mono);
auto token = mic->BufferReady.Add(
    [mic, &monitor](System::Object*, const System::EventArgs&) {
        std::vector<SharpRuntime::bytecs> bytes(4096);
        const int count = mic->GetData(bytes);
        if (count > 0)
        {
            bytes.resize(count);
            monitor.SubmitBuffer(bytes);
        }
    });

mic->Start();
monitor.Play();
// Before monitor leaves scope:
mic->Stop();
mic->BufferReady.Remove(token);

Syntax-checked against the TARGET headers and sharp-runtime next at 41b918c9 for EventHandler::Add and Remove (not pinned by TARGET; not executed). The rate read before Start() is the requested 44,100 Hz; a device that negotiates another rate changes it at Start(), so a monitor that must match exactly should be created after starting.

Which device is Microphone::Default

Microphone::getDefaultProperty() simply returns getAllProperty()[0] (Microphone.cpp). What "the default microphone" means is therefore decided by the order in which the recording provider lists its devices. Both providers follow one contract: only real devices are listed, each under the name the driver gives it, and the host's default recording device comes first, followed by the others in ascending id order. This is XNA's shape. XNA's enumeration puts the system default first and names it.

• SDL3 (Sdl3AudioRecordingDevice.cpp). The provider sorts the devices by id. It then moves to the front the lowest-id device whose name equals the name that SDL resolves for SDL_AUDIO_DEVICE_DEFAULT_RECORDING. If SDL reports no default name, or no device matches it, the list stays in id order and no entry claims to be the default.
• ALSA (AlsaAudioRecordingDevice.cpp). With CNA_AUDIO_RECORDING_DEVICE set, that device is the one and only entry, and it is the default. Otherwise the list begins with ALSA's default PCM, and only when the host configuration lists that PCM for input. The PCM is labelled with its description, which on a desktop is usually the PipeWire or PulseAudio plugin. Every card's capture devices follow as "card, device", ordered by id.

FNA instead prepends a synthetic entry named "Default Device", and CNA once copied it. The source comment records why it was dropped. A sample whose whole display is Microphone.Name showed "Default Device" where XNA showed the driver's name. Ordering matters as much as naming: dropping the synthetic entry without putting the default first briefly made All[0] a second, unconnected microphone, and the sample captured only silence. TheHostDefaultComesFirstAndTheRestStayInIdOrder and TheMachinesDevicesAreListedAsTheContractOrdersThem pin the order (the second asserts that no entry is named "Default Device"). Both enumerate whatever devices the test machine has. Checked by reading at 009d40f5; not executed.

How user reports changed the tests

Reports of high-pitched, distorted or missing sound led to three independent corrections: compressed XNB and XACT formats were not decoded, stereo pan discarded the opposite channel instead of crossfeeding it, and pitch used a linear approximation instead of 2pitch. The pitch error survived because the earlier tests happened to use pitch values -1, 0 and 1, where the linear and exponential formulas agree; a test at a midpoint value exposed it. The lesson generalizes: a property test at the formula's fixed points proves nothing about the formula.

The investigation also produced the offline renderer in OfflineAudioRenderer.hpp: it drives SDL3_mixer's real decode, resample, property, callback and mix stages into memory with no audio device and no wall clock, so the output can be measured deterministically. OfflineAudioRendererTests.cpp measures the dominant frequency within 0.1% for 22,050, 44,100, 48,000 and 96,000 Hz sources declared at their correct rates, in mono and stereo, checks pitch ratios, and reproduces the exact double-frequency signature of a 22,050 Hz buffer declared as 44,100 Hz. That rules out SDL3_mixer's resampler as the cause of the pitch report for correctly declared content, and points at mis-declared source rates as a class; it is not a perceptual-quality oracle and does not cover the ALSA mixer. The three user-reported release blockers in plans/plan_audio.md are still unchecked at this snapshot: the corrections narrow those reports, but the plan itself requires matching captures of the original game before they may be closed.

Memory-safety work in the same period fixed a use-after-free around mixer destruction (each queued stream now holds its own reference to SDL's audio subsystem, so tearing down the mixer cannot shut audio down under a live dynamic instance), a buffer over-read in the XACT name parser (names are now bounded by the real end of the data) and a race between wave-bank disposal and decoding (the wave cache is guarded by one mutex). Two narrower crash reports recorded against an earlier revision, a late failure in the audio-filter tests and a process-teardown failure that depended on the order of the cue and dynamic-instance suites, could not be confirmed or ruled out by reading the source for this page; both were reported as perturbed by sanitizers and are open findings, not evidence that normal runs crash.

What the audio evidence establishes

Source reading and focused tests establish the format routes, the scalar 3D formulas, state and lifetime rules, XACT parsing bounds and the offline waveform measurements. They do not establish perceptual equivalence to XACT or FAudio, hardware capture behaviour (the ALSA capture tests use null and file devices), resampled pending-count accuracy, or the release-blocker campaign. "Playback implemented" and "audio conformance complete" are different claims at this snapshot. Nothing on this page was executed.