Preventing pixelshader overdraw for a single ERG

Question

Background

Using gluTess to build a triangle list in Direct3D9 from a GDI+ DrawString(..) path:

A pixel shader (v3.0) is then used to fill in the shape. When painting with opaque values, everything looks fine:

The problem

At certain font sizes, if the color has an alpha component (ie Argb #55FFFFFF) we begin to see these nasty tessellation artifacts where triangles may overlap ever so slightly:

At larger font sizes the problem is sometimes not present:

Using Intel's excellent GPA Frame Analyzer Pixel History tool, we can see in areas where the artifacts occur, the pixel has been "touched" 3 times from the single Erg.

I'm trying to figure out how I can stop my pixel shader from touching the same pixel more than once.

Other solutions relating to overdraw prevention seem to be all about zbuffer strategies, however this problem is more to do with painting of a single 2D triangle list within a single pixel shader pass.

I'm at a bit of a loss trying to come up with a solution on this one. I was hoping that HLSL might have some sort of "touch each pixel only once" flag, but I've been unable to find anything like that. The closest I've found was to set the BLENDOP to MAX instead of ADD. But the output is not correct when blending over other colors in the scene.

I also have SRCBLEND = ONE, DSTBLEND = INVSRCALPHA. The only combination of flags which produce correct output (albeit with overdraw artifacts.)

I have played with SEPARATEALPHABLENDENABLE in the GPA frame analyzer, which sounded like almost exactly what I need here -- set blending to MAX but only on the "alpha" channel, however from what I can determine, that setting (and corresponding BLENDOPALPHA) affects nothing at all.

One final thing I thought of was to bake text as opaque onto a texture, and then repaint that texture into the scene with the appropriate alpha value applied, however this doesn't actually work in this project because I also support gradient brushes, where stop values may contain alpha, meaning either the artifacts would still be seen, or the final output just plain wrong if we stripped the alpha away from the stop values prior to baking to a texture. Also the whole endeavor would be hideously expensive.

Any hints or pointers would be appreciated. Thanks for reading.

Vast

Answer 1

The problem you're seeing shouldn't happen.

If two of your triangles are overlapping it's because you've placed the vertices in such a way that when the adjacent triangles are drawn, they overlap. What's probably happening is that these two adjacent triangles share two vertices, but each triangle has its own copy of each vertex that's been calculated to be in a very, very slightly different position.

The solution to the problem isn't to try and make the pixel shader touch the pixel only once it's to use an index buffer (if you aren't already) and have the shared vertices between each triangle actually share the same vertex and not use one that's ever-so-slightly not in the same place as the one used by the adjacent triangle.

If you aren't in control of the tessellation algorithm being used you may have to run a pass over the vertex buffer after its been generated to detect and merge vertices that are within some very small tolerance of one another. Even without an index buffer, a naive solution would be this:

1. For each vertex in the vertex buffer, compare its position to every other vertex in the rest of the vertex buffer.

2. If two vertices are within some small tolerance of another, replace the second vertex's position with the position of the one you are comparing it against.

This should have the effect of pairing up the positions of two vertices if they are close enough that you deem them to be the same.

You now shouldn't have any problem with overlapping triangles. In everyday rendering two triangles share edges with each other all the time and you won't ever get the effect where they appear to every-so-slightly overlap. The hardware guarantees that a sample point is either on one side of the line or the other, but never both at the same time, no matter how close the point is to the line (even if it's mathematically on the line, it still fails on one side or the other).

Exemplificative

Thanks Adam, yeah the overlapping is actually from AA extruding/intruding points I had just discovered a moment ago -- if I reduce the offset from .25 to .05, the artifacts are gone but the outside edges are of course much more jaggared. I think you're absolutely correct in that the problem needs to be solved in the source of the data rather than trying to have the pixel shader clean up the sloppy points, so to speak. Thanks again man. – Vast

After playing with this (merging close-by vertices) I've found that all it does is undo anti-aliasing as the intruded/extruded edges become collapsed. Also finding that edge intrude/intrude for path anti-aliasing is always going to result in overlaps when it gets down to really thin 1-2px wide font glyphs (think Segoe UI Light.) I can eliminate artifacts by only ever extruding and never intruding, but the result is noticeably "fatter" text. Reckon maybe a pixel shader solution is in order after all, or perhaps I should rethink my AA strategy such that no path manipulation takes place. – Vast

What tolerance are you using to decide on the merge? I can imagine how an extruded edge/border around the character might have vertices quite close together (but deliberately far enough apart that you do actually see a fade off), but not so close that they should get merged. The ones inside the character itself are probably within 0.0001f of one another, whereas I'd expect a measurable gap on the edge. Is there a value of tolerance small enough that it eliminates the artifact inside the font but large enough that it doesn't remove the AA? – Exemplificative

It didn't seem to matter what tolerance was used, as much as a full 2px (enough to corrupt the shape) and the artifacts would still actually be there. Since I was extruding/intruding by .25px either side of the boundary, anything less than .50px tolerance did very little (maybe 6 points collapsed on a figure 8, none the cause of overdraw.) Higher and it collapsed the AA. An arbitrary balance I've found (will probably just roll with) is to extrude by .19999f and intrude by .09999f. Everything from fat 300px fonts down to 12px skinny fonts look alright with only the odd gimpy pixel popping up. – Vast

Those pixels are going to drive me crazy. I'm going to keep toying around see if I can't identify "edges" which meet vs outer edges which should be the only candidates for extrusion anyhow. – Vast