Vertex shader attribute mapping in GLSL

Question

I'm coding a small rendering engine with GLSL shaders:

Each Mesh (well, submesh) has a number of vertex streams (eg. position,normal,texture,tangent,etc) into one big VBO and a MaterialID.

Each Material has a set of textures and properties (eg. specular-color, diffuse-color, color-texture, normal-map,etc)

Then I have a GLSL shader, with it's uniforms and attributes. Let's say:

uniform vec3 DiffuseColor;
uniform sampler2D NormalMapTexture;
attribute vec3 Position;
attribute vec2 TexCoord;

I'm a little bit stuck in trying to design a way for the GLSL shader to define the stream mappings (semantics) for the attributes and uniforms, and then bind the vertex streams to the appropriate attributes.

Something in the lines of saying to the mesh :"put your position stream in attribute "Position" and your tex coordinates in "TexCoord". Also put your material's diffuse color in "DiffuseColor" and your material's second texture in "NormalMapTexture"

At the moment I am using hard-coded names for the attributes (ie. vertex pos is always "Position" ,etc) and checking each uniform and attribute name to understand what the shader is using it for.

I guess I'm looking for some way of creating a "vertex declaration", but including uniforms and textures too.

So I'm just wondering how people do this in large-scale rendering engines.

Edit:

Recap of suggested methods:

1. Attribute/Uniform semantic is given by the name of the variable (what I'm doing now) Using pre-defined names for each possible attribute.The GLSL binder will query the name for each attribute and link the vertex array based on the name of the variable:

//global static variable

semantics (name,normalize,offset) = {"Position",false,0} {"Normal",true,1},{"TextureUV,false,2}

 ...when linking
for (int index=0;index<allAttribs;index++)
{
   glGetActiveAttrib(program,index,bufSize,length,size[index],type[index],name);      
   semantics[index]= GetSemanticsFromGlobalHardCodedList(name);
} 
... when binding vertex arrays for render
 for (int index=0;index<allAttribs;index++)
{
    glVertexAttribPointer(index,size[index],type[index],semantics[index]->normalized,bufferStride,semantics[index]->offset);

}  

2. Predefined locations for each semantic

GLSL binder will always bind the vertex arrays to the same locations.It is up to the shader to use the the appropriate names to match. (This seems awfully similar to method 1, but unless I misunderstood, this implies binding ALL available vertex data, even if the shader does not consume it)

.. when linking the program...
glBindAttribLocation(prog, 0, "mg_Position");
glBindAttribLocation(prog, 1, "mg_Color");
glBindAttribLocation(prog, 2, "mg_Normal");

3. Dictionary of available attributes from Material, Engine globals, Renderer and Mesh

Maintain list of availlable attributes published by the active Material, the Engine globals, the current Renderer and the current Scene Node.

eg:

 Material has (uniformName,value) =  {"ambientColor", (1.0,1.0,1.0)}, {"diffuseColor",(0.2,0.2,0.2)}
 Mesh has (attributeName,offset) = {"Position",0,},{"Normals",1},{"BumpBlendUV",2}

then in shader:

 uniform vec3 ambientColor,diffuseColo;
 attribute vec3 Position;

When binding the vertex data to the shader, the GLSL binder will loop over the attribs and bind to the one found (or not? ) in the dictionary:

 for (int index=0;index<allAttribs;index++)
    {
       glGetActiveAttrib(program,index,bufSize,length,size[index],type[index],name);      
      semantics[index] = Mesh->GetAttributeSemantics(name);
}

and the same with uniforms, only query active Material and globals aswell.

Kesterson

I believe that your 3 points are somewhat the same, depending on your needs of configuration - ranging from 'keeping it naively simple' (2) to 'data-driven' (3). Talking of vertex attributes, I actually started with (2) a few years ago, thinking I'd go to (1) or (3) depending on my needs. I never had the need to change yet. I'm not saying that the other options are bad, it all depends on your needs. As I'm actually in the process of deleting over-engineered code in our engine, I'm perhaps biased ;) – Candycandyce

I tend to preffer (3) atm, because it seems the easiest to extend in the future. I would like to start simple with (2);what I don't understand in (2): if you have a VBO with 4-5 attribute streams, how do you know what streams to bind depending on the shader? do you simply bind all of them? – Kesterson

Answer 1

Attributes:

Your mesh has a number of data streams. For each stream you can keep the following info: (name, type, data).

Upon linking, you can query the GLSL program for active attributes and form an attribute dictionary for this program. Each element here is just (name, type).

When you draw a mesh with a specified GLSL program, you go through programs attribute dictionary and bind the corresponding mesh streams (or reporting an error in case of inconsistency).

Uniforms:

Let the shader parameter dictionary be the set of (name, type, data link). Typically, you can have the following dictionaries:

• Material (diffuse,specular,shininess,etc) - taken from the material
• Engine (camera, model, lights, timers, etc) - taken from engine singleton (global)
• Render (custom parameters related to the shader creator: SSAO radius, blur amount, etc) - provided exclusively by the shader creator class (render)

After linking, the GLSL program is given a set of parameter dictionaries in order to populate it's own dictionary with the following element format: (location, type, data link). This population is done by querying the list of active uniforms and matching (name, type) pair with the one in dictionaries.

Conclusion: This method allows for any custom vertex attributes and shader uniforms to be passed, without hard-coded names/semantics in the engine. Basically only the loader and render know about particular semantics:

• Loader fills out the mesh data streams declarations and materials dictionaries.
• Render uses a shader that is aware of the names, provides additional parameters and selects proper meshes to be drawn with.

Burkholder

pretty good summary, but you have to define the relationship between your mesh attributes and your shader inputs at some point. Of course, hardcoding is somewhat blunt, but somewhere you got to define this relationship in an arbitrary way (in you loader, in your exporter, in your shader, or in your modeling package). Moreover, often, with multiple meshes/multiple shaders you can't afford to have too many semantics floating around. Homogeneity can sometimes be simpler that genericity. – Candycandyce

So rather than hard-coded semantics, we keep several dictionaries with custom semantics. I like the ideea. Do you think the shader's dictionary should be a separate file alongside the vertex&fragment code files or comments inside the code? – Kesterson

@rotoglup. nobody forces you to use 'too many semantics', but if you do - the "genericity" approach fits that usage pattern better than homogeneity one. The problem with the latter that you'll use more vertex attribute slots (as you pre-define each slot to a semantic, but not use all semantics in each mesh). So in a large-scale project you'll have to do some tricks in order to fit those 16 slots GL provides. At the same time, Peer-to-peer approach I described doesn't have this problem. – Burkholder

@Radu094: you may have understood the idea in your own way :) In the original idea the dictionaries are created in code (material - in loading procedure, engine - in global init, render - in render constructor). Shaders, on the other side, are not supposed to be parsed in a special way. Their active uniforms are queried in a standard GL manner. – Burkholder

@kvark. I understand your proposal, but my point is that, for vertex attributes, I can't see what is the concrete use case of such a flexible system in an application. Do you ever happen to have two meshes with positions named in a different way in the same application? or vastly different vertex attributes from one mesh to another? For me, it may quickly become an authoring nightmare to know which shader is meant to work on which mesh. In which context are you using such a system? – Candycandyce

@rotoglup. The point of a system is not to use the same semantics under different names (semantics is checked by name+type, but no one stops you from declaring the same mesh stream under different names in this scheme). The point is to use different mesh formats independently and without any global hard-coded names. For example, your particle sub-system (one of them) may use attributes like speed and size, which have nothing to do with your main rendering pipeline. The proposed system allows introducing such extensions without touching the core. – Burkholder

@kvark. Ok, I hope I understand what you mean by now. The materials and engine are the ones 'publishing' any available data, and the shaders are just passive 'consumers' of whatever they are interested, based on name+type for a type of weak-contract. Is that correct? – Kesterson

Answer 2

From my experience, OpenGL does not define the concept of attributes or uniforms semantics.

All you can do is define your own way of mapping your semantics to OpenGL variables, using the only parameter you can control about these variables: their location.

If you're not constrained by platform issues, you could try to use the 'new' GL_ARB_explicit_attrib_location (core in OpenGL 3.3 if I'm not mistaken) that allows shaders to explicitly express which location is intended for which attribute. This way, you can hardcode (or configure) which data you want to bind on which attribute location, and query the shaders' locations after it's compiled. It seems that this feature is not yet mature, and perhaps subject to bugs in various drivers.

The other way around is to bind the locations of your attributes using glBindAttribLocation. For this, you have to know the names of the attributes that you want to bind, and the locations you want to assign them.

To find out the names used in a shader, you can:

• query the shader for active attributes
• parse the shader source code to find them yourself

I would not recommend using the GLSL parsing way (although it may suit your needs if you're in simple enough contexts): the parser can easily be defeated by the preprocessor. Supposing that your shader code becomes somewhat complex, you may want to start using #includes, #defines, #ifdef, etc. Robust parsing supposes that you have a robust preprocessor, which can become quite a heavy lift to set up.

Anyway, with your active attributes names, you have to assign them locations (and/or semantics), for this, you're alone with your use case.

In our engine, we happily hardcode locations of predefined names to specific values, such as:

glBindAttribLocation(prog, 0, "mg_Position");
glBindAttribLocation(prog, 1, "mg_Color");
glBindAttribLocation(prog, 2, "mg_Normal");
...

After that, it's up to the shader writer to conform to the predefined semantics of the attributes.

AFAIK it's the most common way of doing things, OGRE uses it for example. It's not rocket science but works well in practice.

If you want to add some control, you could provide an API to define the semantics on a shader basis, perhaps even having this description in an additional file, easily parsable, living near the shader source code.

I don't get into uniforms where the situation is almost the same, except that the 'newer' extensions allow you to force GLSL uniform blocks to a memory layout that is compatible with your application.

I'm not satisfied by all this myself, so I'll be happy to have some contradictory information :)

Candycandyce

ARB_explicit_attrib_location seems to be such a step in the right direction! I am worried though, to rely the entire engine/shaders onto a new extension that might or might not be generally supported. I guess it's hard-coded names for attributes and uniforms, unless some other tehnique comes up. – Kesterson

@Radu094: I have mixed feelings with explicit_attrib_location, I feel like it's only usable with proper #defining the locations values to name them. And #include extensions for GLSL is quite clumsy, so... Predefined attributes names (hardcoded or configurable) can be more easy to configure. I find it hard to find proper references of 'state of the art' renderers done in GLSL that would allow to establish a knowledge base of these issues. – Candycandyce

Answer 3

You may want to consider actually parsing the GLSL itself.

The uniform/attribute declaration syntax is pretty simple. You can come up with a small manual parser that looks for lines that start with uniform or attribute, get the type and name and then expose some C++ API using strings. This will save you the trouble of hard coded names. If you don't want to get your hands dirty with manual parsing a couple of likes of Spirit would do the trick.
You probably won't want to fully parse GLSL so you'll need to make sure you don't do anything funny in the decelerations that might alter the actual meaning. One complication that comes to mind is conditional compilation using macros in the GLSL.

Distributee

Hi! Well, even with parsing the GLSL, if I'm not using hard-coded names then how can I induce the semantics of the attribute? (eg. "attribute vec3 lPos2;") Unless I write some sort of (hard-coded) comment after the attribute (" //LIGHT_POS "). I can't find a way to attach semantics data to attributes. – Kesterson

Why would you want to attach semantic data to attributes? It's difficult from your question to see the big picture of what you're trying to do. – Distributee

Because I need the graphics engine to know what data stream the shader wants binded at that attribute. Is is vertex position, second set of UV, reflectivity, temperature, some-future-funky-per-vertex-param? – Kesterson