Tutorial 63: Stencil Buffer Effects

CNA Tutorials · Advanced Rendering

The stencil buffer is an 8-bit integer channel attached alongside the depth buffer. It acts as a per-pixel mask that you write in one draw call and read back in subsequent draw calls to include or exclude pixels from rendering. Combined with CNA's DepthStencilState API it enables a wide range of effects that are impossible or expensive to achieve any other way.

DepthStencilState Stencil Settings

DepthStencilState is a pipeline state object that controls both the depth test and stencil test. The stencil-relevant fields are:

Field	Type	Description
`StencilEnable`	`bool`	Master switch. Must be `true` to use stencil at all.
`StencilFunction`	`CompareFunction`	How the stencil test compares the buffer value to `ReferenceStencil`.
`StencilPass`	`StencilOperation`	Operation on the stencil buffer when both stencil and depth tests pass.
`StencilFail`	`StencilOperation`	Operation when the stencil test fails (depth test not evaluated).
`StencilDepthBufferFail`	`StencilOperation`	Operation when stencil passes but depth test fails.
`ReferenceStencil`	`int`	Reference value compared against the buffer. Range 0–255.
`StencilMask`	`int`	AND mask applied to both reference and buffer before comparison.
`StencilWriteMask`	`int`	AND mask controlling which bits are written to the stencil buffer.
`DepthBufferEnable`	`bool`	Enable depth testing.
`DepthBufferWriteEnable`	`bool`	Enable depth writes.

CNA provides several built-in presets: DepthStencilState::Default (depth on, stencil off), DepthStencilState::DepthRead (depth test without write, stencil off), and DepthStencilState::None (both off). For stencil effects you always construct a custom state.

StencilFunction (CompareFunction)

The StencilFunction field uses the same CompareFunction enum as depth testing. The comparison is: (buffer_value & StencilMask) OP (ReferenceStencil & StencilMask).

Value	Passes when
`Always`	Always passes (used when writing to stencil without testing it)
`Never`	Never passes
`Equal`	`buffer == reference`
`NotEqual`	`buffer != reference`
`Less`	`buffer < reference`
`LessEqual`	`buffer <= reference`
`Greater`	`buffer > reference`
`GreaterEqual`	`buffer >= reference`

StencilOperation Enum

StencilOperation controls what happens to the stencil buffer value for each pixel, independently for three outcomes (pass, stencil-fail, depth-fail):

Value	Effect on buffer
`Keep`	Do not change the existing value.
`Zero`	Set buffer to 0.
`Replace`	Set buffer to `ReferenceStencil`.
`IncrementSaturation`	Increment, clamped at 255.
`DecrementSaturation`	Decrement, clamped at 0.
`Invert`	Bitwise NOT of the current value.
`Increment`	Increment with wrap-around (255 + 1 = 0).
`Decrement`	Decrement with wrap-around (0 - 1 = 255).

Writing to the Stencil Buffer

The typical pattern for writing a stencil mask is:

Set StencilEnable = true, StencilFunction = Always (always write, never test), StencilPass = Replace, ReferenceStencil = 1.
Disable colour writes so the mask geometry does not appear in the image (BlendState with ColorWriteChannels::None).
Draw the mask geometry (e.g., a sphere silhouette, a portal quad, a mirror plane). Pixels covered by this geometry will have stencil value 1 after this pass.

DepthStencilState writeStencil;
writeStencil.StencilEnable          = true;
writeStencil.StencilFunction        = CompareFunction::Always;
writeStencil.StencilPass            = StencilOperation::Replace;
writeStencil.ReferenceStencil       = 1;
writeStencil.DepthBufferEnable      = true;
writeStencil.DepthBufferWriteEnable = true;

BlendState noColorWrite = BlendState::Opaque;
noColorWrite.ColorWriteChannels = ColorWriteChannels::None;

gd.setDepthStencilState(writeStencil);
gd.setBlendState(noColorWrite);
drawMaskGeometry(gd);
gd.setBlendState(BlendState::Opaque);

Masking with the Stencil Buffer

Once the stencil buffer contains the mask, draw the content that should only appear inside (or outside) the mask:

// Draw content only where stencil == 1
DepthStencilState testStencil;
testStencil.StencilEnable          = true;
testStencil.StencilFunction        = CompareFunction::Equal;
testStencil.StencilPass            = StencilOperation::Keep;
testStencil.ReferenceStencil       = 1;
testStencil.DepthBufferEnable      = true;
testStencil.DepthBufferWriteEnable = true;

gd.setDepthStencilState(testStencil);
drawContent(gd); // only pixels where buffer == 1 survive

Object Outline / Silhouette Effect

A two-pass technique produces a coloured outline around any object. It is widely used for selection highlighting in strategy games, interactable object indicators in adventure games, and enemy highlighting in shooters.

Pass 1 — draw the object normally. Set StencilPass = Replace and ReferenceStencil = 1 so every pixel covered by the object writes stencil=1. This also draws the object's normal appearance to the colour buffer.

Pass 2 — draw the same object again but scaled up by a few percent (e.g. 1.05x). Set StencilFunction = NotEqual, ReferenceStencil = 1. Only pixels where the stencil is not 1 pass, i.e., only the thin ring of pixels that the scaled-up version covers but the original did not. Disable depth testing so the outline appears in front of all geometry.

class OutlineGame final : public Game {
    DepthStencilState writeStencil_;   // pass 1: write stencil
    DepthStencilState outlineStencil_; // pass 2: draw outline ring

    std::unique_ptr<Effect> solidEffect_;
    std::unique_ptr<Effect> outlineEffect_;

    void LoadContent() override {
        // Pass 1: draw object, write stencil=1 everywhere it covers
        writeStencil_.StencilEnable          = true;
        writeStencil_.StencilFunction        = CompareFunction::Always;
        writeStencil_.StencilPass            = StencilOperation::Replace;
        writeStencil_.ReferenceStencil       = 1;
        writeStencil_.DepthBufferEnable      = true;
        writeStencil_.DepthBufferWriteEnable = true;

        // Pass 2: draw scaled-up object only where stencil != 1
        outlineStencil_.StencilEnable          = true;
        outlineStencil_.StencilFunction        = CompareFunction::NotEqual;
        outlineStencil_.StencilPass            = StencilOperation::Keep;
        outlineStencil_.ReferenceStencil       = 1;
        outlineStencil_.DepthBufferEnable      = false; // always in front
        outlineStencil_.DepthBufferWriteEnable = false;

        solidEffect_   = Content.Load<Effect>("effects/solid");
        outlineEffect_ = Content.Load<Effect>("effects/flat_color");
    }

    void Draw(const GameTime&) override {
        auto& gd = getGraphicsDeviceProperty();
        gd.Clear(Color::DarkSlateGray);

        // --- Pass 1: draw object + write stencil ---
        gd.setDepthStencilState(writeStencil_);
        solidEffect_->Parameters["u_world"].SetValue(objectWorld_);
        solidEffect_->Parameters["u_view"].SetValue(camera_.View());
        solidEffect_->Parameters["u_projection"].SetValue(camera_.Projection());
        drawObjectMesh(gd, *solidEffect_);

        // --- Pass 2: draw scaled-up outline where stencil != 1 ---
        gd.setDepthStencilState(outlineStencil_);
        float scale = 1.05f;
        Matrix outlineWorld = Matrix::CreateScale(scale, scale, scale) * objectWorld_;
        outlineEffect_->Parameters["u_world"].SetValue(outlineWorld);
        outlineEffect_->Parameters["u_view"].SetValue(camera_.View());
        outlineEffect_->Parameters["u_projection"].SetValue(camera_.Projection());
        outlineEffect_->Parameters["u_color"].SetValue(Vector4(1.0f, 0.5f, 0.0f, 1.0f));
        drawObjectMesh(gd, *outlineEffect_);

        // Restore default state before drawing anything else
        gd.setDepthStencilState(DepthStencilState::Default);
        gd.Present();
    }

    Matrix objectWorld_ = Matrix::CreateTranslation(0.0f, 0.0f, 0.0f);
};

Portal Rendering

The stencil buffer is ideal for portals: rectangular openings in the world that display a different scene or location.

Clear the stencil buffer to 0.
Draw the portal quad with StencilPass = Replace, ReferenceStencil = 1, and colour writes disabled. Pixels inside the portal opening now have stencil=1.
Set StencilFunction = Equal, ReferenceStencil = 1. Render the "other side" scene (using a different camera that looks through the portal). Only the portal pixels receive the other-scene rendering.
Render the current scene normally (with stencil test disabled or with a NotEqual mask to skip the portal opening).

This technique requires depth buffer management too: after drawing the portal destination scene, reset the depth buffer at the portal pixels to the portal quad's depth value before drawing the main scene, so main-scene geometry in front of the portal still occludes it correctly.

Shadow Volumes (Advanced)

The classic Carmack's Reverse (depth-fail) shadow volume algorithm uses stencil increment and decrement operations to count how many shadow volume surfaces surround a pixel:

Render the scene without shadows.
Disable colour and depth writes. Enable stencil write only.
For front-facing shadow volume surfaces: StencilDepthBufferFail = Increment.
For back-facing shadow volume surfaces: StencilDepthBufferFail = Decrement.
After both passes, pixels with stencil > 0 are inside a shadow volume and should be darkened.
Re-render the scene with StencilFunction = Equal, ReferenceStencil = 0 (lit pixels) and then with StencilFunction = Greater, ReferenceStencil = 0 (shadowed pixels) using a dark blending pass.

Shadow volumes produce pixel-perfect hard shadows and require no shadow map resolution compromises, but they are expensive when the shadow caster has complex silhouettes. Shadow mapping (Tutorial 59) is usually preferred in modern engines.