Tutorial 63: Stencil Buffer Effects

CNA Tutorials  ·  Advanced Rendering

The stencil buffer is an 8-bit integer channel attached alongside the depth buffer. It acts as a per-pixel mask that you write in one draw call and read back in subsequent draw calls to include or exclude pixels from rendering. Combined with CNA's DepthStencilState API it enables a wide range of effects that are impossible or expensive to achieve any other way.

DepthStencilState Stencil Settings

DepthStencilState is a pipeline state object that controls both the depth test and stencil test. The stencil-relevant fields are:

CNA provides several built-in presets: DepthStencilState::Default (depth on, stencil off), DepthStencilState::DepthRead (depth test without write, stencil off), and DepthStencilState::None (both off). For stencil effects you always construct a custom state.

StencilFunction (CompareFunction)

The StencilFunction field uses the same CompareFunction enum as depth testing. The comparison is: (buffer_value & StencilMask) OP (ReferenceStencil & StencilMask).

StencilOperation Enum

StencilOperation controls what happens to the stencil buffer value for each pixel, independently for three outcomes (pass, stencil-fail, depth-fail):

Writing to the Stencil Buffer

The typical pattern for writing a stencil mask is:

1. Set StencilEnable = true, StencilFunction = Always (always write, never test), StencilPass = Replace, ReferenceStencil = 1.
2. Disable colour writes so the mask geometry does not appear in the image (BlendState with ColorWriteChannels::None).
3. Draw the mask geometry (e.g., a sphere silhouette, a portal quad, a mirror plane). Pixels covered by this geometry will have stencil value 1 after this pass.

DepthStencilState writeStencil;
writeStencil.setStencilEnableProperty(true);
writeStencil.setStencilFunctionProperty(CompareFunction::Always);
writeStencil.setStencilPassProperty(StencilOperation::Replace);
writeStencil.setReferenceStencilProperty(1);
writeStencil.setDepthBufferEnableProperty(true);
writeStencil.setDepthBufferWriteEnableProperty(true);

BlendState noColorWrite = BlendState::Opaque;
noColorWrite.setColorWriteChannelsProperty(ColorWriteChannels::None);

gd.setDepthStencilStateProperty(writeStencil);
gd.setBlendStateProperty(noColorWrite);
drawMaskGeometry(gd);
gd.setBlendStateProperty(BlendState::Opaque);

Masking with the Stencil Buffer

Once the stencil buffer contains the mask, draw the content that should only appear inside (or outside) the mask:

// Draw content only where stencil == 1
DepthStencilState testStencil;
testStencil.setStencilEnableProperty(true);
testStencil.setStencilFunctionProperty(CompareFunction::Equal);
testStencil.setStencilPassProperty(StencilOperation::Keep);
testStencil.setReferenceStencilProperty(1);
testStencil.setDepthBufferEnableProperty(true);
testStencil.setDepthBufferWriteEnableProperty(true);

gd.setDepthStencilStateProperty(testStencil);
drawContent(gd); // only pixels where buffer == 1 survive

Object Outline / Silhouette Effect

A two-pass technique produces a coloured outline around any object. It is widely used for selection highlighting in strategy games, interactable object indicators in adventure games, and enemy highlighting in shooters.

Pass 1 — draw the object normally. Set StencilPass = Replace and ReferenceStencil = 1 so every pixel covered by the object writes stencil=1. This also draws the object's normal appearance to the colour buffer.

Pass 2 — draw the same object again but scaled up by a few percent (e.g. 1.05x). Set StencilFunction = NotEqual, ReferenceStencil = 1. Only pixels where the stencil is not 1 pass, i.e., only the thin ring of pixels that the scaled-up version covers but the original did not. Disable depth testing so the outline appears in front of all geometry.

#include "Microsoft/Xna/Framework/Graphics/ShaderEffect.hpp"
#include "System/IO/File.hpp"

class OutlineGame final : public Game {
    DepthStencilState writeStencil_;   // pass 1: write stencil
    DepthStencilState outlineStencil_; // pass 2: draw outline ring

    std::unique_ptr<ShaderEffect> solidEffect_;
    std::unique_ptr<ShaderEffect> outlineEffect_;

    // ShaderEffect's matrix setter takes raw column-major floats.
    static void SetMat4(ShaderEffect& fx, const char* name, const Matrix& m) {
        float cm[16];
        m.ToColumnMajor(cm);
        fx.SetUniformMat4(name, cm);
    }

    void LoadContent() override {
        // Pass 1: draw object, write stencil=1 everywhere it covers
        writeStencil_.setStencilEnableProperty(true);
        writeStencil_.setStencilFunctionProperty(CompareFunction::Always);
        writeStencil_.setStencilPassProperty(StencilOperation::Replace);
        writeStencil_.setReferenceStencilProperty(1);
        writeStencil_.setDepthBufferEnableProperty(true);
        writeStencil_.setDepthBufferWriteEnableProperty(true);

        // Pass 2: draw scaled-up object only where stencil != 1
        outlineStencil_.setStencilEnableProperty(true);
        outlineStencil_.setStencilFunctionProperty(CompareFunction::NotEqual);
        outlineStencil_.setStencilPassProperty(StencilOperation::Keep);
        outlineStencil_.setReferenceStencilProperty(1);
        outlineStencil_.setDepthBufferEnableProperty(false); // always in front
        outlineStencil_.setDepthBufferWriteEnableProperty(false);

        // Three arguments: device, vertex source, fragment source. Not file paths.
        auto& gd = getGraphicsDeviceProperty();
        solidEffect_ = std::make_unique<ShaderEffect>(
            gd,
            System::IO::File::ReadAllText("Content/effects/solid.vert.glsl"),
            System::IO::File::ReadAllText("Content/effects/solid.frag.glsl"));
        outlineEffect_ = std::make_unique<ShaderEffect>(
            gd,
            System::IO::File::ReadAllText("Content/effects/flat_color.vert.glsl"),
            System::IO::File::ReadAllText("Content/effects/flat_color.frag.glsl"));

        // Neither constructor throws on a compile failure -- check both.
        if (!solidEffect_->IsEffectValid() || !outlineEffect_->IsEffectValid()) {
            // The GLSL did not compile. Do not draw with it.
        }
    }

    void Draw(const GameTime&) override {
        auto& gd = getGraphicsDeviceProperty();
        gd.Clear(Color::DarkSlateGray);

        // --- Pass 1: draw object + write stencil ---
        gd.setDepthStencilStateProperty(writeStencil_);
        // Apply() binds the program; the uniform setters write into it.
        solidEffect_->Apply();
        SetMat4(*solidEffect_, "u_world",      objectWorld_);
        SetMat4(*solidEffect_, "u_view",       camera_.View());
        SetMat4(*solidEffect_, "u_projection", camera_.Projection());
        drawObjectMesh(gd, *solidEffect_);

        // --- Pass 2: draw scaled-up outline where stencil != 1 ---
        gd.setDepthStencilStateProperty(outlineStencil_);
        float scale = 1.05f;
        Matrix outlineWorld = Matrix::CreateScale(scale, scale, scale) * objectWorld_;
        outlineEffect_->Apply();
        SetMat4(*outlineEffect_, "u_world",      outlineWorld);
        SetMat4(*outlineEffect_, "u_view",       camera_.View());
        SetMat4(*outlineEffect_, "u_projection", camera_.Projection());
        outlineEffect_->SetUniformVec4("u_color", 1.0f, 0.5f, 0.0f, 1.0f);
        drawObjectMesh(gd, *outlineEffect_);

        // Restore default state before drawing anything else
        gd.setDepthStencilStateProperty(DepthStencilState::Default);
        gd.Present();
    }

    Matrix objectWorld_ = Matrix::CreateTranslation(0.0f, 0.0f, 0.0f);
};

Portal Rendering

The stencil buffer is ideal for portals: rectangular openings in the world that display a different scene or location.

1. Clear the stencil buffer to 0.
2. Draw the portal quad with StencilPass = Replace, ReferenceStencil = 1, and colour writes disabled. Pixels inside the portal opening now have stencil=1.
3. Set StencilFunction = Equal, ReferenceStencil = 1. Render the "other side" scene (using a different camera that looks through the portal). Only the portal pixels receive the other-scene rendering.
4. Render the current scene normally (with stencil test disabled or with a NotEqual mask to skip the portal opening).

This technique requires depth buffer management too: after drawing the portal destination scene, reset the depth buffer at the portal pixels to the portal quad's depth value before drawing the main scene, so main-scene geometry in front of the portal still occludes it correctly.

Shadow Volumes (Advanced)

The classic Carmack's Reverse (depth-fail) shadow volume algorithm uses stencil increment and decrement operations to count how many shadow volume surfaces surround a pixel:

1. Render the scene without shadows.
2. Disable colour and depth writes. Enable stencil write only.
3. For front-facing shadow volume surfaces: StencilDepthBufferFail = Increment.
4. For back-facing shadow volume surfaces: StencilDepthBufferFail = Decrement.
5. After both passes, pixels with stencil > 0 are inside a shadow volume and should be darkened.
6. Re-render the scene with StencilFunction = Equal, ReferenceStencil = 0 (lit pixels) and then with StencilFunction = Greater, ReferenceStencil = 0 (shadowed pixels) using a dark blending pass.

Shadow volumes produce pixel-perfect hard shadows and require no shadow map resolution compromises, but they are expensive when the shadow caster has complex silhouettes. Shadow mapping (Tutorial 59) is usually preferred in modern engines.