FFmpeg video boundary

CNA snapshot 009d40f5  ·  Development › Module internals  ·  source links pinned to 009d40f5

modules/video-ffmpeg/ is a small physical target with a large stateful responsibility: two source files that implement the VideoDecoder and AudioDurationProbe interfaces declared in modules/media. The public Video and VideoPlayer types stay in cna_media whether or not this target exists, and FFmpeg is optional at this snapshot: a build without it links and runs, and playback throws NotSupportedException. A maintainer therefore has to decide which implementation the configured build actually linked before debugging any video behaviour. This page covers the configure-time switch, the decoder's ownership and packet state machine, seek and track switching, and what the tests can and cannot show; the user guide is FFmpeg setup in Video Playback.

Configure-time selection, not runtime codec discovery

modules/CMakeLists.txt interprets CNA_ENABLE_VIDEO, declared in the root CMakeLists.txt with the default AUTO (the value is upper-cased and anything other than OFF, AUTO or ON is a configure error). The value is one of CNA_ENABLE_VIDEO=OFF|AUTO|ON. The switch is a link-time implementation choice: it is not a renderer or platform selection and it is not a deferred run-time fallback.

The excluded platforms are deliberate: a cross-build must never resolve the build host's pkg-config files and poison its own include and library paths, so AUTO falls back there and ON fails instead of silently changing the request. The outcome is published three ways: the variable CNA_FFMPEG_AVAILABLE (exported to the parent scope because the root-level test registration keys its exclusions on it, and would silently drop the fixture suites on a capable host if it were not), the compile definitions CNA_VIDEO_AVAILABLE and CNA_FFMPEG_AVAILABLE on the shared build-config target, and a configure status line that reads either video enabled (…; FFmpeg backend cna_video_ffmpeg) or video disabled (…); Video playback reports NotSupportedException. That line is the first thing to read in a bug report.

When the backend is available, modules/CMakeLists.txt adds the subdirectory and links cna_video_ffmpeg privately to cna_media. modules/video-ffmpeg/CMakeLists.txt makes a static library (alias CNA::VideoFfmpeg), reaches the media headers through a private include path, links Sharp Runtime Core.Base privately and owns every FFmpeg imported target (PkgConfig::LIBAVCODEC, LIBAVFORMAT, LIBAVUTIL, LIBSWRESAMPLE). No public media header exposes an FFmpeg header or symbol; VideoDecoder.hpp only forward-declares the FFmpeg structs. The probe states no minimum FFmpeg version; the sources use swr_alloc_set_opts2 and the ch_layout channel-layout API, so an old FFmpeg that pkg-config can see would presumably fail at compile time rather than at configure time (not tried).

Otherwise VideoDecoderUnavailable.cpp and AudioDurationProbeUnavailable.cpp satisfy the same symbols. IsVideoDecoderAvailable returns false, RequireVideoDecoderAvailable throws NotSupportedException with a message that names -DCNA_ENABLE_VIDEO=ON, Open, SeekToStart, SetAudioStream, SetVideoStream, NextFrame and DrainAudio all throw it, Close does nothing, and the duration probe reports zero. In the enabled build the roles are reversed: IsVideoDecoderAvailable returns true and RequireVideoDecoderAvailable is empty.

Video / VideoPlayer                       [cna_media, public XNA layer, always built]
  -> CNA::Internal::Media::VideoDecoder     [contract in the media header]
       -> FFmpeg implementation             [cna_video_ffmpeg, optional]
       OR unavailable implementation        [cna_media]
  -> RGBA CPU buffer -> Texture2D::SetDataRGBA -> selected graphics renderer
  -> float audio buffer -> VideoPlayer audio-stream submission

MediaLibrary -> AudioDurationProbe contract
  -> FFmpeg container metadata  OR  unavailable, always zero (unknown)

Neighbouring switches and what actually runs

• Content pipeline. CNA_ENABLE_MEDIA_PIPELINE (MP3, WMA and WMV sources at content build time) reuses the same CNA_FFMPEG_AVAILABLE probe but links the four pkg-config targets into cna_content_pipeline itself, and only cna_content_compiler links that module. It is separate from the runtime backend described here, and with CNA_ENABLE_VIDEO=OFF it cannot find FFmpeg either.
• Packaging. The C API install deliberately does not ship the FFmpeg libraries: with the backend enabled they remain a system dependency, and an explicit CNA_ENABLE_VIDEO=OFF package has no FFmpeg edge at all (modules/c-api/CMakeLists.txt).
• Presets and workflows. In CMakePresets.json the dev, unit and release-modules presets (and the presets that inherit from them) set CNA_ENABLE_VIDEO=OFF; the remaining configure presets (the sanitizer, tests, multi-renderer, cnaext and platform presets) do not set it, so AUTO or the platform exclusion applies. The Linux workflows install the four FFmpeg development packages and the Apple workflows brew install ffmpeg, so AUTO would select the backend there; the only workflow that names the option explicitly, content-pipeline-windows-ci.yml, passes OFF. Which workflow exercises the disabled branch on a host where FFmpeg would otherwise have been found was not traced.

Decoder ownership and an actual frame path

Open and Close own the native state

VideoPlayer uniquely owns a VideoDecoder; the decoder owns raw FFmpeg pointers: the AVFormatContext, one video and one optional audio AVCodecContext, a SwrContext resampler, one reusable AVFrame and one reusable AVPacket. VideoDecoder.cpp Open starts by calling Close, opens the demuxer, reads stream information, and picks the best video and best audio stream. Video is mandatory: no video stream, no decoder for it, or a failure allocating or opening its context closes the format context and returns false. Width and height come from the opened video context, the frame rate from the stream's average frame rate (24 when the denominator is zero), and the duration from the container in seconds when it is known.

Audio is optional and degrades to video-only. The audio context is opened, and a resampler is built and initialised transactionally; if either step fails, the audio context is freed, the audio stream index reset and the decoder carries on without audio. That resampler converts sample format to interleaved 32-bit float and keeps the source sample rate and channel layout unchanged; nothing in the decoder changes the rate or remixes channels, so the player opens its mixer stream at the file's own rate and channel count. HasAudio is true only when both the audio context and the resampler exist, because a decoded audio frame is silently dropped without a working resampler. Finally the frame and packet are allocated; an allocation failure closes everything and returns false. Every codec context is created with strict error recognition (AV_EF_CRCCHECK | AV_EF_BITSTREAM | AV_EF_BUFFER | AV_EF_EXPLODE), so detected corruption becomes a hard decode error instead of a concealed glitch.

Close clears the retained-packet flag, then frees, in order, the packet, frame, resampler, audio codec, video codec and format context, resets the stream indices and metadata, and clears the queued decoded audio. The destructor calls it. A cleanup refactor must preserve the retry-packet reset and the pending-audio reset, not only the pointer frees: a reused decoder would otherwise resend a stale packet or return the previous file's samples from its first DrainAudio.

NextFrame is a packet state machine

NextFrame loops until it can return one converted frame, and its order matters:

1. It first tries to receive a decoded frame from the video codec. Success sets the presentation time to the frame's best-effort timestamp times the stream time base (0 when there is none), converts the frame and returns true. AVERROR_EOF returns false as a clean end of stream. Any other error except EAGAIN throws std::runtime_error.
2. If a packet is being retained (havePendingVideoPacket_), it is sent again now that the receive has had a chance to drain the codec. Success releases it; EAGAIN loops back to receive; any other error releases it, clears the flag and throws.
3. Otherwise it reads packets. An error other than end of file throws an I/O error. At end of file it flushes the video codec (sending a null packet, tolerating AVERROR_EOF) and flushes the audio codec and the resampler through ProcessAudioPacket(nullptr), then returns to step 1 to drain what the flush produced. An audio packet goes to ProcessAudioPacket and reading continues; a video packet is sent to the codec, where EAGAIN keeps the packet alive and sets the flag and any other failure throws, and then the loop returns to step 1; packets of other streams are released and reading continues.

The retained-packet flag is a class member, not a function local, on purpose. After a send returns EAGAIN the next receive almost always yields a buffered frame at once and returns to the caller before the retained packet is resent, so a local flag would lose the packet across calls and silently drop a frame. DecodesTheFullFileWithoutSilentlyDroppingAnyFrame pins the observable result (50 frames, strictly increasing timestamps) on one fixture. The decoder file contains no thread, mutex or atomic; decode, resample and conversion run on whichever thread calls NextFrame, and the header states that all methods of one decoder must be called from the same thread.

Conversion to RGBA

Converted video is always an 8-bit RGBA vector with opaque alpha. VideoDecoder.hpp declares the converters and the implementation chooses by the frame's pixel format:

The colour math is one BT.601 integer matrix for every planar format: it uses no range expansion and has no BT.709 branch, so the YUVJ and non-J formats take the same coefficients. The 10- and 12-bit route drops precision to 8 bits before converting; it is not an HDR-preserving pipeline. The planar converters size their output from the width and height captured at Open (or at a video-track switch), while the generic converter uses the frame's own dimensions. A new pixel format is a code change in this table, not something FFmpeg falls back to automatically.

Audio packets and the drain

ProcessAudioPacket sends the packet to the audio codec, retrying the same packet while the codec reports EAGAIN (the audio-side equivalent of the video retention), receives every available frame, converts each through the resampler, and appends the interleaved floats to pendingAudio_. A failing send, receive or convert throws; nothing is skipped silently. The end-of-file flush sends a null packet and additionally drains the resampler's internal delay, which is a separate buffer from the codec's. DrainAudio appends everything accumulated to the caller's vector and clears it. The player drains after every NextFrame call, including the one that returns false, because audio packets read while looking for the next video packet or the end of file would otherwise be stranded.

How the player consumes this is described on the media page: VideoPlayer::OpenDecoder verifies the file's metadata against the Video, applies track preferences before creating the output texture and audio stream, and decodes the first frame inside a catch that calls CloseDecoder; VideoPlayer::GetTexture then advances by play position, uploads through Texture2D::SetDataRGBA, and either loops via SeekToStart or waits for queued audio at end of file. A blank video can be decode, CPU conversion, texture upload, renderer sampling or timing; the first boundary that fails identifies the owner.

Seek, track switching and failure invariants

SeekToStart

SeekToStart returns at once when nothing is open. It seeks the demuxer to the start (backward flag); if that seek fails it changes nothing at all, and the next NextFrame simply carries on from wherever the stream was, because flushing on top of an unmoved position would build a wrong assumption. After a successful seek it flushes both codecs' buffers, clears pendingAudio_ (samples decoded before the seek must not be spliced onto post-seek audio), discards a retained pre-seek video packet and clears its flag, and replaces the resampler with a fresh one so its internal delay cannot surface after the seek. If that resampler cannot be rebuilt, audio becomes unavailable (HasAudio turns false) rather than the looping path gaining a new throw. Audio that the player already submitted to the mixer before the loop point is not discarded by the seek.

SetAudioStream and SetVideoStream

Both enumerate the streams of the requested kind by zero-based index among that kind. They return false for a negative or out-of-range index and for the already active stream, and otherwise return the result of opening the replacement, so the return value tells the caller whether a real switch happened; VideoPlayer uses it to skip needless output reconfiguration. An audio switch builds the replacement codec context and its resampler first and only then frees the old pair and clears pendingAudio_, so a failed switch leaves the previous working audio path intact. A video switch opens the replacement codec context before freeing the current one and then updates dimensions, frame rate and time base. Re-selecting the current video track is a deliberate no-op for a decode reason: a freshly created context has no reference frame, and the demuxer is already past the first keyframe, so recreating it mid-stream would make the next non-keyframe fail. Neither switch seeks the demuxer, so a genuine switch to a different video stream continues from the current position; how that decodes depends on the new stream's keyframes at that point, and no fixture exercises it (see below).

What throws, what returns false, what is silent

Changing these methods needs tests for the same-track no-op, a failed switch, a post-switch frame and the VideoPlayer output reconfiguration; the returned bool is the contract that tells the caller whether anything changed.

The duration probe

AudioDurationProbe::ProbeDurationMS in AudioDurationProbe.cpp opens the container, reads stream information and returns the container duration in whole milliseconds, or zero when the file cannot be opened, stream information fails, the duration is unset or it is not positive. It does not decode audio, so computing a duration never decodes an album, but every indexed song costs one open and one stream-information read at library construction. The library consumes it to give songs a duration; the unavailable implementation returns zero on purpose. The decoder and the probe have different consumers and different failure behaviour, so review them separately when changing FFmpeg version requirements or error handling.

What tests prove, and what they do not

cmake/UnitTests.cmake excludes the decoder suite (VideoDecoderTests.cpp), the real Video and VideoPlayer suites (VideoTests.cpp, VideoPlayerTests.cpp) and the content video XNB suite when CNA_FFMPEG_AVAILABLE is off, while VideoBackendAvailabilityTests.cpp and the reader and containment suites still build. That file compiles one of two exclusive tests. EnabledBuildReportsDecoderAvailable asserts the availability functions in an enabled build. DisabledBuildRetainsMetadataApiAndRejectsDecoding asserts, in a disabled build, that a missing file still throws FileNotFoundException first, that the raw-file constructor and FromUriEXT throw NotSupportedException, that a metadata-only Video keeps its values, that Play throws and leaves the player stopped with no video, that Pause, Resume and Stop are harmless, that every decoder operation except Close throws, and that the duration probe returns zero. A green disabled or fallback build is therefore not evidence that video decodes: inspect the configured ctest -N list and the test binary's filters rather than assuming AUTO selected FFmpeg.

In an enabled build, VideoDecoderTests.cpp (21 tests) covers dimensions and frame rate, 4:2:0, 4:2:2 and 4:4:4 conversion, 10- and 12-bit conversion, audio presence and draining, AV1 with its audio track, audio-track selection and the same-video-stream no-op, truncated and corrupted input, full-file frame count, seek reset, close reset and the pending-audio resets. VideoPlayerTests.cpp (29 tests) covers orchestration beyond the decoder: metadata mismatch, real-fixture texture size, audio stream resume and pause, end-of-stream behaviour with an audio tail, looping, track preferences and mid-playback track changes, first-frame failure leaving the player closed, the audio buffer not growing without a device, and the frame generation never restarting. The suites and counts are as registered in source, not executed.

Read the fixtures before generalising. Per the fixture manifest they are synthetic 160 by 90, 25 fps SMPTE colour-bar clips produced with ffmpeg: FFV1 (lossless) MKV files for 4:2:0, 4:2:2, 4:4:4, 10-bit and 12-bit and a FLAC audio-tail case, one AV1 plus FLAC file, a two-audio-track file whose audio tracks differ in sample rate, and one H.264 MP4 that is only ever used as a base for byte corruption. Colour checks compare seven bar samples on the first frame with a tolerance of 20 per channel against values sampled at authoring time, which shows the frame is neither clipped nor magenta and not that colours are accurate. The multi-track fixture has a single video stream, so no test switches to a second video stream. No fixture uses H.265, VP8, VP9, Theora, NV12 or RGB output, the YUVJ formats, or 10- and 12-bit 4:2:2 and 4:4:4, and the user guide's list of supported containers and codecs describes what FFmpeg can be asked to decode, not what these tests decode. Player tests that exercise audio depend on the mixer available in that configuration, and a CPU decoder fixture does not prove texture upload or audible playback on every graphics and audio backend.

For a bug fix, record the selected CNA_ENABLE_VIDEO and the configure status line first. Reproduce at the narrowest boundary: the decoder for packet, colour or seek problems, VideoPlayer for timing, texture or audio hand-off, MediaLibrary for metadata. Preserve the Open/Close and EAGAIN invariants, add a fixture for the exact troublesome stream, and run enabled and disabled configurations so the unavailable implementation is not removed by accident. A new FFmpeg dependency or a public-header include is a build and packaging change, not merely a decoder edit; see I need to change build configuration.

Source-reading order

1. modules/CMakeLists.txt, media/CMakeLists.txt and video-ffmpeg/CMakeLists.txt: prove which implementation and which native libraries entered this build.
2. VideoDecoder.hpp and VideoDecoderUnavailable.cpp: establish the media-owned contract before reading the optional native implementation.
3. video-ffmpeg/VideoDecoder.cpp: follow Open and Close, the NextFrame packet state, conversion, audio drain, seek and the track-switch transaction, in that order.
4. VideoPlayer.cpp: connect decoder results to texture and audio output, error cleanup and timing.
5. AudioDurationProbe.cpp and the tests named above: separate metadata indexing from video playback, and confirm which fixture exercises each claim.

Related pages: Media internals for the player and library, the CMake option index, CMake architecture, the ownership map and the thread and callback map. The module's place among its siblings is in the module index.