Tutorial 54: AlphaTestEffect and Alpha Masking

CNA Tutorials  ·  Built-in Effects

AlphaTestEffect is one of CNA's built-in effects that replicates the XNA 4.0 alpha-test pipeline. Instead of using the fixed-function alpha test that was available in older GPU APIs, CNA implements it as a shader-level discard instruction in the fragment shader. A pixel is either fully kept or fully discarded based on whether its texture alpha passes a programmable comparison against a reference threshold. This technique — commonly called alpha masking or clip-alpha — is essential for rendering foliage, fences, decals, and other cutout geometry efficiently without the sorting overhead required by true alpha blending.

AlphaTestEffect Properties

AlphaTestEffect exposes a focused set of properties that mirror the XNA 4.0 API exactly. Below is a summary of each property and its purpose:

  • Texture — setTexture(Texture2D*). The diffuse texture whose alpha channel drives the masking decision. An RGBA texture is required; RGB-only textures will always produce an alpha of 1.0, making all pixels pass the test.
  • Alpha — setAlpha(float). A global alpha multiplier in the range [0, 1] applied to the entire surface before the per-pixel test. Combined with vertex alpha and texture alpha before comparison.
  • DiffuseColor — setDiffuseColor(Vector3). A linear RGB tint multiplied into the final output colour. Does not affect the alpha test comparison itself.
  • FogEnabled / FogColor / FogStart / FogEnd — Optional depth-based linear fog. setFogEnabled(bool) toggles it. setFogColor(Vector3), setFogStart(float), setFogEnd(float) control the gradient. The fog factor is computed per-vertex and interpolated to the fragment.
  • VertexColorEnabled — setVertexColorEnabled(bool). When true, expects VertexPositionColorTexture vertices and multiplies the per-vertex colour (including its alpha) into the test.
  • World / View / Projection — setWorld(Matrix), setView(Matrix), setProjection(Matrix). Standard transform matrices uploaded to the vertex shader.

AlphaFunction Enum

The CompareFunction enum (the same enum used by depth and stencil tests) controls how the fragment alpha is compared against the reference value. Pass it to setAlphaFunction(CompareFunction). The fragment is discarded (clipped) when the comparison fails:

CompareFunction value Pass condition Typical use case
Always Every fragment passes. Effectively disables alpha masking; use for opaque surfaces.
Never No fragment passes. Invisible surface (can be useful for shadow receivers that should not colour the buffer).
Less alpha < reference Keep only nearly-transparent pixels — unusual but useful for inverse masks.
LessEqual alpha ≤ reference Keep pixels up to and including the threshold.
Equal alpha == reference Rare; requires exact alpha values (palette textures).
GreaterEqualalpha ≥ reference Keep pixels at or above the threshold — the most common setting for cutout foliage.
Greater alpha > reference Keep pixels strictly above the threshold; reference = 127 keeps alpha 128–255.
NotEqual alpha != reference Discard pixels at exactly one alpha value; useful for palette-keyed transparency.

The most practical combination for foliage is CompareFunction::Greater with ReferenceAlpha = 128. This keeps all fragments whose alpha (after texture sample and per-vertex multiplication) is greater than 128 out of 255, cleanly discarding semi-transparent fringe pixels caused by texture filtering at leaf edges.

ReferenceAlpha

The ReferenceAlpha property accepts an integer in the range 0–255, consistent with the XNA 4.0 API. Internally CNA normalises this to a float [0, 1] before uploading it to the GLSL uniform that feeds the fragment-level if (alpha < threshold) discard; instruction.

Choosing the right reference value depends on your texture. Textures exported from Photoshop or GIMP with "straight alpha" typically have clean binary alpha (0 or 255) — in that case any reference between 1 and 254 gives the same visual result. Textures with pre-multiplied alpha or anti-aliased edges benefit from a higher reference (around 180–220) to avoid a faded halo at the silhouette.

// Keep pixels with alpha > 50% (out of 255)
alphaEffect_->setAlphaFunction(CompareFunction::Greater);
alphaEffect_->setReferenceAlpha(128);

// Keep only fully-opaque pixels (reference = 254 avoids fp rounding issues)
alphaEffect_->setAlphaFunction(CompareFunction::GreaterEqual);
alphaEffect_->setReferenceAlpha(254);

Vertex Color Support

When setVertexColorEnabled(true) is called, the effect expects vertex data conforming to the VertexPositionColorTexture layout, which adds a Color field (RGBA bytes) to each vertex. The per-vertex colour is multiplied component-wise into the texture colour and the per-vertex alpha is multiplied into the texture alpha before the threshold comparison. This allows you to fade out individual billboard sprites by reducing their vertex alpha, causing more of them to fall below the reference threshold and be discarded:

// Fade a billboard by reducing vertex alpha
VertexPositionColorTexture v;
v.Position = billboardPos;
v.Color    = Color(255, 255, 255, fadeAlpha);   // fadeAlpha in [0, 255]
v.TextureCoordinate = Vector2(u, v_coord);

When VertexColorEnabled is false the effect uses VertexPositionTexture (no colour field), saving 4 bytes per vertex.

Use Cases

Alpha masking with AlphaTestEffect is the standard technique for several categories of geometry:

  • Foliage: Billboard trees and grass use a leaf texture atlas with alpha cutouts. Thousands of billboards can be rendered in a single draw call without sorting, because discarded pixels write nothing to the colour buffer and do not participate in blending.
  • Chain-link fences and grilles: A tiled texture with an alpha mask produces the appearance of a complex wireframe structure using just two triangles per fence panel.
  • Cutout decals: Bullet-hole or crack decals on walls use alpha masking to avoid rectangular borders appearing on the surface.
  • Stylised characters: 2.5D games (e.g., Paper Mario style) render flat character sprites as world-space quads. Alpha masking removes the rectangular quad border.
  • UI elements in 3D scenes: Health bars or markers placed in world space often use alpha-masked textures.

The critical advantage of alpha masking over alpha blending is that masked geometry does not need to be sorted. The depth buffer works correctly for masked surfaces — a passing fragment writes depth normally, a discarded fragment writes nothing. This makes alpha masking O(1) with respect to object count, whereas alpha blending requires a back-to-front sort that is O(n log n).

Comparison with Alpha Blending

Alpha blending (BlendState::AlphaBlend or BlendState::NonPremultiplied) composites the source fragment over the destination colour using the source alpha as a mixing weight. It can produce smooth, anti-aliased silhouettes and partially-transparent surfaces. However it comes with two significant restrictions:

  1. Depth-write must be disabled (or carefully managed) — blended fragments behind already-drawn opaque geometry are lost if they don't pass the depth test, but if depth-write is on, they may incorrectly occlude geometry drawn later.
  2. Back-to-front sorting is required — blended geometry must be drawn from furthest to nearest so that the compositing equation accumulates colour correctly. Sorting breaks batching and can be expensive at runtime.

Alpha masking has a hard silhouette edge (stairstepping is visible at low texture resolution), but it requires zero sorting and works correctly with depth-write enabled. A common real-world approach is to use alpha masking for dense foliage (grass, leaf canopy) where the hard edge is less noticeable, and reserve alpha blending for hero translucent objects (glass, particles, smoke) where softness matters and the sort count is small.

Complete Example: Foliage Rendering

class FoliageGame final : public Game {
    std::unique_ptr<AlphaTestEffect> alphaEffect_;
    std::unique_ptr<Texture2D>       leafTex_;
    std::unique_ptr<VertexBuffer>    treeBillboard_;
    int                              triCount_ = 0;
    Camera                           camera_;

    void LoadContent() override {
        leafTex_ = Content.Load<Texture2D>("textures/leaf_atlas");

        alphaEffect_ = std::make_unique<AlphaTestEffect>(getGraphicsDeviceProperty());
        alphaEffect_->setTexture(leafTex_.get());
        alphaEffect_->setAlphaFunction(CompareFunction::Greater);
        alphaEffect_->setReferenceAlpha(128);
        alphaEffect_->setVertexColorEnabled(false);
        alphaEffect_->setDiffuseColor(Vector3(0.85f, 1.0f, 0.75f));  // slight green tint
        alphaEffect_->setAlpha(1.0f);

        // Optional fog for distance fade
        alphaEffect_->setFogEnabled(true);
        alphaEffect_->setFogColor(Vector3(0.6f, 0.7f, 0.8f));
        alphaEffect_->setFogStart(40.0f);
        alphaEffect_->setFogEnd(120.0f);

        // Build a grid of billboard quads (cross-shaped: 2 quads per tree)
        BuildTreeBillboards(200 /* tree count */);
    }

    void Draw(const GameTime&) override {
        auto& gd = getGraphicsDeviceProperty();
        gd.Clear(Color::CornflowerBlue);

        // Alpha-tested geometry writes depth correctly — no sort needed
        gd.setRasterizerState(RasterizerState::CullNone);  // both sides of billboards

        alphaEffect_->setWorld(Matrix::Identity);
        alphaEffect_->setView(camera_.View());
        alphaEffect_->setProjection(camera_.Projection());

        gd.setVertexBuffer(*treeBillboard_);
        for (auto& pass : alphaEffect_->getCurrentTechnique().Passes) {
            pass.Apply();
            gd.DrawPrimitives(PrimitiveType::TriangleList, 0, triCount_);
        }

        gd.Present();
    }

    void BuildTreeBillboards(int count) {
        std::vector<VertexPositionTexture> verts;
        // For each tree, create two crossed quads (X shape from above)
        for (int i = 0; i < count; ++i) {
            float x = (float)(rand() % 200) - 100.0f;
            float z = (float)(rand() % 200) - 100.0f;
            AddBillboardQuad(verts, Vector3(x, 1.5f, z), 3.0f, 4.0f, 0.0f);
            AddBillboardQuad(verts, Vector3(x, 1.5f, z), 3.0f, 4.0f, 90.0f);
        }
        triCount_ = (int)(verts.size() / 3);
        treeBillboard_ = std::make_unique<VertexBuffer>(
            getGraphicsDeviceProperty(),
            VertexPositionTexture::VertexDeclaration,
            (int)verts.size(),
            BufferUsage::WriteOnly
        );
        treeBillboard_->SetData(verts.data(), (int)verts.size());
    }
};