How does this simple FxAA work?

Question

I came across this FxAA shader that does anti-aliasing and seems to be working quite well. But, Somehow could not understand the logic. Can someone explain?

[[FX]]

// Samplers
sampler2D buf0 = sampler_state {
    Address = Clamp;
    Filter = None;
};

context FXAA {
    VertexShader = compile GLSL VS_FSQUAD;
    PixelShader = compile GLSL FS_FXAA;
}



[[VS_FSQUAD]]

uniform mat4 projMat;
attribute vec3 vertPos;
varying vec2 texCoords;

void main(void) {
    texCoords = vertPos.xy; 
    gl_Position = projMat * vec4( vertPos, 1 );
}


[[FS_FXAA]]

uniform sampler2D buf0;
uniform vec2 frameBufSize;
varying vec2 texCoords;

void main( void ) {
    //gl_FragColor.xyz = texture2D(buf0,texCoords).xyz;
    //return;

    float FXAA_SPAN_MAX = 8.0;
    float FXAA_REDUCE_MUL = 1.0/8.0;
    float FXAA_REDUCE_MIN = 1.0/128.0;

    vec3 rgbNW=texture2D(buf0,texCoords+(vec2(-1.0,-1.0)/frameBufSize)).xyz;
    vec3 rgbNE=texture2D(buf0,texCoords+(vec2(1.0,-1.0)/frameBufSize)).xyz;
    vec3 rgbSW=texture2D(buf0,texCoords+(vec2(-1.0,1.0)/frameBufSize)).xyz;
    vec3 rgbSE=texture2D(buf0,texCoords+(vec2(1.0,1.0)/frameBufSize)).xyz;
    vec3 rgbM=texture2D(buf0,texCoords).xyz;

    vec3 luma=vec3(0.299, 0.587, 0.114);
    float lumaNW = dot(rgbNW, luma);
    float lumaNE = dot(rgbNE, luma);
    float lumaSW = dot(rgbSW, luma);
    float lumaSE = dot(rgbSE, luma);
    float lumaM  = dot(rgbM,  luma);

    float lumaMin = min(lumaM, min(min(lumaNW, lumaNE), min(lumaSW, lumaSE)));
    float lumaMax = max(lumaM, max(max(lumaNW, lumaNE), max(lumaSW, lumaSE)));

    vec2 dir;
    dir.x = -((lumaNW + lumaNE) - (lumaSW + lumaSE));
    dir.y =  ((lumaNW + lumaSW) - (lumaNE + lumaSE));

    float dirReduce = max(
        (lumaNW + lumaNE + lumaSW + lumaSE) * (0.25 * FXAA_REDUCE_MUL),
        FXAA_REDUCE_MIN);

    float rcpDirMin = 1.0/(min(abs(dir.x), abs(dir.y)) + dirReduce);

    dir = min(vec2( FXAA_SPAN_MAX,  FXAA_SPAN_MAX),
          max(vec2(-FXAA_SPAN_MAX, -FXAA_SPAN_MAX),
          dir * rcpDirMin)) / frameBufSize;

    vec3 rgbA = (1.0/2.0) * (
        texture2D(buf0, texCoords.xy + dir * (1.0/3.0 - 0.5)).xyz +
        texture2D(buf0, texCoords.xy + dir * (2.0/3.0 - 0.5)).xyz);
    vec3 rgbB = rgbA * (1.0/2.0) + (1.0/4.0) * (
        texture2D(buf0, texCoords.xy + dir * (0.0/3.0 - 0.5)).xyz +
        texture2D(buf0, texCoords.xy + dir * (3.0/3.0 - 0.5)).xyz);
    float lumaB = dot(rgbB, luma);

    if((lumaB < lumaMin) || (lumaB > lumaMax)){
        gl_FragColor.xyz=rgbA;
    }else{
        gl_FragColor.xyz=rgbB;
    }
}

Urtication

whoa that is far far FAAAAR from simple – Beetlebrowed

Answer 1

FxAA is a filter algorithm that performs antialiasing on images. In contrary to other AA techniques it is applied on the pixels of an image, not while drawing it's primitives. In 3D applications like games it is applied as a post processing step on top of the rendered scene.

The basic idea is: Look for vertical and horizontal edges. Blur in orthogonal direction if at the end of the edge.

Here's a good description and the original paper on the topic.

Transpose

I guess there are multiple versions of FxAA, the version that I have posted here is different from what the paper describes. Want to know here what does rgbA and rgbB signify here. Any what does the following logic mean, if((lumaB < lumaMin) || (lumaB > lumaMax)){ gl_FragColor.xyz=rgbA; }else{ gl_FragColor.xyz=rgbB; } – Urtication

Answer 2

The idea of standard FxAA is: For each pixel, sample the 8 neighboring pixels to determine if it's part of a high contrast edge. If it is, smooth it out in the direction of the edge.

The problem is, you can't detect the direction of the edge accurately when only looking at the 3x3 square around the pixel. Imagine if before anti-aliasing, you have a staircase line like this:

XXXXXXX   
       XXXXXXX
              XXXXXXX
                     XXXXXXX

Pixels on that line can't see from direct neighbors that the tangent of the slope is 1/7. The original algorithm solves this with these steps:

• Determine from the 3x3 neighborhood if the slope is closer to horizontal or closer to vertical
• Step both left/right (or up/down for vertical) in a loop, sampling surrounding pixels again
• Stop when you reach the end of the local edge, or when you reach a maximum FXAA_SEARCH_STEPS
• Based on how far you traveled, shift the uv of the original pixel so that using texture filtering, it gets smoothed

The search loop is expensive, so the FxAA code you posted uses an alternative approach. It works like this:

First of all, instead of sampling all 8 neighbors, it only samples 4 diagonal neighbors. Then, it approximates the direction of the edge by computing its gradient vec2 dir. This vector is scaled by rcpDirMin, and dir.xy is clamped to (-8, 8) pixels. I don't understand the exact computation of the length scaling, but the closer the gradient is to perfectly horizontal or vertical, the closer to the maximum length it gets. The smoothing happens by sampling the pixels along this gradient, in two amounts:

• rgbA is short blur, it samples at -1/6*dir and +1/6*dir (I don't know why those constants are written in the code as 1/3-0.5 and 2/3-0.5, it makes it more confusing to me)

• rgbB is longer blur, sampling at -3/6*dir and +3/6*dir and blending it with rgbA.

Example: when the pixel to smooth is on a horizontal line and dir has length 8, it will sample 1.33 and 4.0 pixels to the left and right.

The final check ((lumaB < lumaMin) || (lumaB > lumaMax)) selects rgbA or rgbB as final pixel color. It prefers the longer blur rgbB, unless that results in the luminance getting out of the original luminance range of the 3x3 neighborhood, which indicates it sampled pixels not part of the edge.

Differences

• The original uses 8 neighbours to compute the gradient, the alternative uses only 4 neighbours
• The original FxAA rounds the gradient to 'horizontal' or 'vertical'. The alternative doesn't, but scales the gradient so it's longer in those cardinal directions
• The original FxAA samples in steps along the (rounded) gradient in a loop to find how to shift the UV of the pixel to smooth. The alternative samples a fixed 4 steps along the gradient, which is the smoothing

The advantage of the alternative FxAA shader is that it does less texture lookups, and has less branches, resulting in better and more consistent performance, at the cost of lower accuracy.

Notably absent are the quick exits the original FxAA has, which skips dark or low-contrast areas, while the alternative FxAA still blends 4 texture lookups. When your image is full of dark / flat color, the performance difference may be smaller.

I wonder who authored this FxAA version. A GitHub search for rcpDirMin shows it's copy-pasted a lot, but I haven't found a version with attribution. At most there's a reference to the original Timothy Lottes paper, which doesn't mention this variant.

Doings