GLSL/OpenGL shader tessellation flickering and failure

Question

I just started with OpenGL tessellation and have run into a bit a trouble. I am tessellating series of patches formed by one vertex each. These vertices/patches are structured in a gridlike fashion to later form a terrain generated by Perlin Noise.

The problem I have run into is that starting from the second patch, and every 5^th patch after that, sometimes have a lot of tessellation (not the way i configured) but most of the time it doesn't get tessellated at all.

Like so:

  

The two white circles mark the highly/over tessellated patches. Also note the pattern of untessellated patches.

The strange thing is that it works on my Surface Pro 2 (Intel HD4400 graphics) but bugs on my main desktop computer (AMD HD6950 graphics). Is it possible the hardware is bad?

The patches are generated with the code:

    vec4* patches = new vec4[m_patchesWidth * m_patchesDepth];
    int c = 0;
    for (unsigned int z = 0; z < m_patchesDepth; ++z) {
        for (unsigned int x = 0; x < m_patchesWidth; ++x) {
            patches[c] = vec4(x * 1.5f, 0, z * 1.5f, 1.0f);
            c++;
        }
    }
    m_fxTerrain->Apply();
    glGenBuffers(1, &m_planePatches);
    glBindBuffer(GL_ARRAY_BUFFER, m_planePatches);
    glBufferData(GL_ARRAY_BUFFER, m_patchesWidth * m_patchesDepth * sizeof(vec4), patches, GL_STATIC_DRAW);
    GLuint loc = m_fxTerrain->GetAttrib("posIn");
    glEnableVertexAttribArray(loc);
    glVertexAttribPointer(0, 4, GL_FLOAT, GL_FALSE, sizeof(vec4), nullptr);
    delete(patches);

And drawn with:

    glPatchParameteri(GL_PATCH_VERTICES, 1);
    glBindVertexArray(patches);

    glPolygonMode(GL_FRONT_AND_BACK, GL_LINE);
    glDrawArrays(GL_PATCHES, 0, nrOfPatches);

Vertex Shader:

#version 430 core
in vec4 posIn;

out gl_PerVertex {
    vec4 gl_Position;
};

void main() {
    gl_Position = posIn;
}

Control shader:

#version 430
#extension GL_ARB_tessellation_shader : enable
layout (vertices = 1) out;

uniform float OuterTessFactor;
uniform float InnerTessFactor;

out gl_PerVertex {
    vec4 gl_Position;
} gl_out[];

void main() {

    if (gl_InvocationID == 0) {
        gl_TessLevelOuter[0] = OuterTessFactor;
        gl_TessLevelOuter[1] = OuterTessFactor;
        gl_TessLevelOuter[2] = OuterTessFactor;
        gl_TessLevelOuter[3] = OuterTessFactor;

        gl_TessLevelInner[0] = InnerTessFactor;
        gl_TessLevelInner[1] = InnerTessFactor;
    }
    gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;
}

Evaluation shader:

#version 430 
#extension GL_ARB_tessellation_shader : enable
layout (quads, equal_spacing, ccw) in;

uniform mat4 ProjView;
uniform sampler2D PerlinNoise;

out vec3 PosW;
out vec3 Normal;
out vec4 ColorFrag;
out gl_PerVertex {
    vec4 gl_Position;
};

void main() {
    vec4 pos = gl_in[0].gl_Position;
    pos.xz += gl_TessCoord.xy;
    pos.y = texture2D(PerlinNoise, pos.xz / vec2(8, 8)).x * 10.0f - 10.0f;
    Normal = vec3(0, 1, 0);
    gl_Position = ProjView * pos;
    PosW = pos.xyz;
    ColorFrag = vec4(pos.x / 64.0f, 0.0f, pos.z / 64.0f, 1.0f);
}

Fragment shader:

#version 430 core
in vec3 PosW;
in vec3 Normal;
in vec4 ColorFrag;
in vec4 PosH;

out vec3 FragColor;
out vec3 FragNormal;

void main() {
    FragNormal = Normal;
    FragColor = ColorFrag.xyz;
}

I have tried to hardcode the different tessellation levels but that did not help. I recently started out with OpenGL so please let me know if i am doing something stupid.

So does anyone have any idea what could be causing this "flickering" of certain patches?

Update: I had a friend run the project and he got the same pattern of flickering tessellation but the failing patches were not drawn at all except when being overly tessellated. He has the same graphics card as I do (AMD HD6950).

Cullender

How do you set your tesselation levels? – Astrodome

@Astrodome I set them like: GLuint loc = m_fxTerrain->GetUniform("OuterTessFactor"); glUniform1f(loc, 8.0f); But i have also tried hardcoding them in the shader, but still same bug. – Cullender

Answer 1

You should use triangle/quad tessellation, in which each patch has 3 or 4 vertices. As I can see, you use quads (I use them too). In that case, you can set it like this:

glPatchParameteri(GL_PATCH_VERTICES,4);
glBindVertexArray(VertexArray);

(TIP: use drawelements for your terrain, much better performance for 2D-displacement based mesh.)
In the control shader, use

layout (vertices = 4) out;

since your patch has 4 control points. The ordering is still important (CCW/CW).
Personally I don't like to use built-in variables, so for the vertex shader you can send your vertex data to the tesscontrol like this:

layout (location = 0) out vec3 outPos;
....
outPos.xz = grid.xy;
outPos.y = noise(outPos.xz);

Tess control:

layout (location = 0) in vec3 inPos[]; //outPos (location = 0) from vertex shader
//'collects' the 4 control points to an array in the order they're sended
layout (location = 0) out vec3 outPos[];  //send the c.points to the ev. shader
...
gl_TessLevelOuter[0] = outt[0];
gl_TessLevelOuter[1] = outt[1];
gl_TessLevelOuter[2] = outt[2];
gl_TessLevelOuter[3] = outt[3];

gl_TessLevelInner[0] = inn[0];
gl_TessLevelInner[1] = inn[1];

outPos[ID] = inPos[ID];//gl_invocationID = ID

Note that both in and out vertex data is an array.

The tessev is simple:

layout (location = 0) in vec3 inPos[]; //the 4 control points
layout (location = 0) out vec3 outPos; //this is no longer array, next is the fragment shader
...
//edit: do not forgot to add the next line
layout (quads) in;

vec3 interpolate3D(vec3 v0, vec3 v1, vec3 v2, vec3 v3) //linear interpolation for x,y,z coords on the quad
{
    return mix(mix(v0,v1,gl_TessCoord.x),mix(v3,v2,gl_TessCoord.x),gl_TessCoord.y);
};
...main{...
outPos = interpolate3D(inPos[0],inPos[1],inPos[2],inPos[3]); //the four control points of the quad. Every other point is linearly interpolated between them according to the TessCoord.
gl_Position = mvp * vec4(outPos,1.0f);

A good representation of the quad domain: http://ogldev.atspace.co.uk/www/tutorial30/tutorial30.html.

I think the problem is with your one-vertex patch. I cannot imagine how a one vertex path can be divided into triangles, I don't know how it works on another hardware. The tessellation is for divide primitives into other simple primitives, to triangles in case of OGL, since it can be handled by a GPU easily (3 points always lie in a plane). So, the minimum number of patch vertices should be 3, for a triangle. I like quads, because it simplier to index, and the memory cost is less. It will be divided into triangles too during tessellation. http://www.informit.com/articles/article.aspx?p=2120983 Also, there is another type, the isoline tessellation. (check out the links, the second is pretty good.)

All in all, try it with quads or triangles, and set the control vertices to 4 (or 3). My (pretty complex) terrain shader is here with frustum culling, tessellation shader culling for a geoclipmap based terrain. Also, without tessellation it works with vertex morph in vertex shader. Maybe some part of this code will be useful. http://speedy.sh/TAvPR/gshader.txt

A scene with tessellation at about 4 pixels/triangle runs at 75 FPS (with fraps) with runtime normal calculation and bicubic smoothing and other things. I'm using AMD HD 5750. It still could be much faster with better code and pre-baked normals:D. (runs at max 120 w/o normal calc.)

Oh, and you can only send the x and z coords if you displace the vertex in the shader. It will be faster too.

Lots of vertices.

Aharon