Anti-Aliasing issue with MSAA, drawing CSG with depth and FBO

Question

I have reimplemented OpenCSG for modern OpenGL version.

PixelFormatAttributes:

NSOpenGLPFAColorSize    , 24 ,
NSOpenGLPFAAlphaSize    , 8  ,
NSOpenGLPFADepthSize    , 32 ,
NSOpenGLPFAStencilSize  , 8  ,
NSOpenGLPFAAccelerated  ,
NSOpenGLPFADoubleBuffer ,
NSOpenGLPFASupersample  ,
NSOpenGLPFASampleBuffers, 1  ,
NSOpenGLPFASamples      , 4  ,

FBO specs: (tried render to FBO with multisample, but lines getting more strong and visible, look on screenshot at bottom)
- created texture with power of 2, GL_RGBA (tried GL_RGBA8 and GL_RGBA32F)
- GL_DEPTH24_STENCIL8 (tried GL_DEPTH32_STENCIL8, no result)

Simply algorithm Goldfeather:

while (i < depth complexity) {
    take channel for render
       merge layers if no free channel 
    render each layer with stencil func, mask and depth params to channel (FBO)
}
merge layers (taking texture from FBO and render objects again with applying shader below)

In shader I have code for merging (result texture from FBO overlaps on top of rendering for testing, in OpenCSG it's setupProjectiveTexture):

     vec2 ndcPos = gl_FragCoord.xy / sizetexture.xy;
     vec4 maskColor = texture2D(maskTexture, ndcPos.xy);
     if (maskColor[channel] < 0.5) {
        discard;
     }

Looks like after FBO getting not enough clear texture or not right size.

EDIT:
Those lines appears only in overlapping places of subtracting mesh.

EDIT 2:
Fixed with rendering to non MSAA FBO and applying FXAA on result.

Risotto

32-bit color buffer + 8-bit alpha strikes me as particularly unusual. Usually pixel formats are setup like: 32-bit (8-bits unused) RGB + 0 A or 24-bit RGB + 8-bit A, almost never 32-bit RGB + 8-bit A (which produces a 40-bit color buffer). It should not make a difference since CGL will match the closest format, but you are asking it for something fundamentally strange. – Machuca

@AndonM.Coleman actually you right, will change! But it did not resolve my issue. I found few articles about how to resolve similar issue: 1. alphanew.net/index.php?section=articles&site=multisampling 2. mtnphil.wordpress.com/2013/06/26/know-your-ssao-artifacts But for me hard to understand fully, because I'm not native speaker. Maybe will understand later. – Nelly

By the way, when you specify Supersample in your pixel format you are asking GL for SSAA instead of MSAA. This is OS X; I have never encountered a hardware accelerated pixel format on OS X that supports supersampling, only the software renderer. I would suggest you remove that from your pixel format too. At this point this is getting pretty nit-picky, but since the only information in the question right now is related to your pixel format... :) – Machuca

In the branch of your fragment shader, does anything change if, instead of discard you write something like this: gl_FragDepth = 1.0f; gl_FragColor = vec4 (0.0f, 0.0f, 0.0f, 0.0f); return;? I have a wild hunch, but without the code to try it myself the best I can do is ask you if it changes anything. – Machuca

@AndonM.Coleman with that code, model getting blue and lines still visible, picture. Supersample more optimal for Mac OS X then Multisample. I will try change it to MSAA and look on GL_SAMPLE_ALPHA_TO_COVERAGE. – Nelly

Actually, my point about multisampling was that unless you are using the software renderer on OS X, you actually will not get supersampling. The Supersample part of the pixel format is a hint, if you are expecting it to give you supersampling you are in trouble. You might want to try supersampling + software rendering just to see if that is in fact the issue; if it is you may have no choice but to use single-sampled rendering. – Machuca

Must be some way to use multi-sampling for this implementation. Single-sampled left for last. And as well I have this code runs on iOS with single-sampled rendering. It has similar and very strange bug, at the corner in 512x512 square rendering is good, but in another part of screen everything has jagged edges and black lines, looks like MSAA not applied, but I did not use MSAA on iOS (can't show you screenshot here). On iPad 3/4 gen issue is appearing, on iPad Air everything fine. – Nelly

@AndonM.Coleman I've changed to Single-sampled render and applied FXAA, not so perfect as MSAA, but ok. FXAA made a bit blurry in some place. – Nelly

Way late to this, but you should read up on "centroid sampling" wrt MSAA. That's probably your problem and could be fixed by setting your texture sampling up as such. – Spoonbill

Answer 1

Not sure I got your explanations right. What kind of texture are you sampling from in the shader?

I don't know the specifics of the algorithm you're implementing but if that texture contains the results of some previous calculations with multisample gathering already done, I'm really not sure you can use that result for a whole fragment (and all of its samples). The mask is going to fade at edges, which seems to be the problem here.

Maybe you should do sample gathering "by hand", e.g. use a sampler2DMS and read individual samples with texelFetch in the shader.

Another solution might be to set gl_SampleMask to all 1's in the first pass to prevent the fading at borders.

Whew