How can I hunt down these OpenGL calls that are distorting objects in my scene?

Question

I'm mixing two libraries that use OpenGL: Qt and OpenSceneGraph. I'm targeting OpenGL ES 2, so everything is done with shaders and ES 2 compatible calls.

I'm specifically using OSG with QtDeclarative by trying to paint OSG onto a QDeclarativeItem. I do this the way suggested in Qt documentation: wrap all OpenGL calls between beginNativePainting()/endNativePainting().

This works fine until I use textures in my OpenSceneGraph scene. When I do this, my QML window gets "messed up" for lack of a better word. To keep it as simple as possible, my OSG scene consists of a plane with a texture applied to it. I recreated the scene using basic OpenGL calls and the problem no longer occurs. Here's the problem summarized as a bunch of pictures:

• The QtDeclarative engine uses OpenGL to paint stuff. I set up a simple QML page:

• I create a simple scene using OpenGL directly. It's a plane with a texture painted onto it.

• Now I try to set up the same scene in OSG... identical shaders, etc.

You can see something odd is going on with the last screenshot. Don't worry about the black background where the original OpenGL scene was transparent, that's just OSG using a black clear color. The problem is that the other items set up with QML (the rectangles) get messed up.

Edit: To clarify what happens: The rectangles I draw with QML are all stretched out to the right edge of the screen. I also noticed if I draw rectangles after the OpenSceneGraph item in QML, they don't show up (I didn't notice this before). I draw the purpley black rectangle after the OSG item in the following screenshots... note that it disappears. There might be more weird stuff happening, but this is all I've observed playing with rectangles.

Before

After

I'm fairly new to OpenGL so I don't know what kind of call/state setting would cause something like this to happen. I think that OpenSceneGraph makes some OpenGL state change that's messing up Qt's paint engine. I also know that this only occurs when OSG uses textures... if I don't apply textures in my OSG scene, this doesn't happen. This is where I'm stuck.

Also, I tried to use BuGLe to get an OpenGL call trace with and without textures enabled in OSG to see if I could figure out the problematic state change(s). I found a few differences, and even some global state that OSG changed (such as glPixelStorei()) between the two, but resetting the changes I found made no difference. It would help a lot if I knew what to look for. If anyone's feeling insane, I also have the stack traces:

• OSG with texturing: http://pastie.org/4223182 (osg texture stuff is lines 637~650)
• OSG without texturing: http://pastie.org/4223197

Edit 2: Here's a diff that might be helpful. You'll need to scroll way down before the relevant lines are apparent. http://www.mergely.com/nUEePufa/

Edit 3: Woah! Okay, that diff helped me out quite a bit. OSG enables VertexAttribArray 3 but doesn't disable it. Calling glDisableVertexAttribArray(3) after OSG renders its frame seems to partially solve the problem; there's no more stretching of the QML rectangles. However, rectangles drawn after the OSG item still don't show up.

Schaaf

So just to be clear, the first two images are perfectly fine, and your problem in the third image, is that the gray rectangle in the center of the screen is extending further to the right than in the first two images? Or if that's not right, can you edit one of the images to circle exactly where is the problem? – Glycogen

I edited the question to address this. – Schaaf

Actually it is pretty easy to diff them, if you black out the raw memory addresses s/0x[a-z0-9]+//. I did diff them and look at it for a few minutes but I'm stumped. OSG looks like it does clean up after itself pretty well, and the number of differences between the logs is pretty minimal. Looking at the fact that it works fine without textures, and looking at the diffs in the log, I can't think of any explanation. Just out of curiousity, is that OSG or Qt making the calls after BeginNativePainting start and BeginNativePainting end? – Glycogen

Thanks for the reply and for looking into the issue. It's Qt making the calls after begin/endNativePainting. I realized that I could use regex to erase the memory addresses a few minutes ago as well. I updated my post with a link to the diff. – Schaaf

Answer 1

After obsessing over the trace logs, I think I've found two OpenGL things that need to be reset before passing control back to Qt to cause the issues above to go away. I mentioned one in an edit... I'll summarize both in this answer.

Rectangle/QML Item distortion

QPainter uses Vertex Attributes 3, 4, and 5 directly for something that looks like its related to the geometry of those rectangles. This can be seen in the trace:

[INFO] trace.call: glVertexAttrib3fv(3, 0x2d94a14 -> { 0.00195312, 0, 0 })
[INFO] trace.call: glVertexAttrib3fv(4, 0x2d94a20 -> { 0, -0.00333333, 0 })
[INFO] trace.call: glVertexAttrib3fv(5, 0x2d94a2c -> { 0.2, 0.4, 1 })

Disabling the corresponding vertex attribute arrays fixes the stretchy rectangles issue:

 glDisableVertexAttribArray(3);
 glDisableVertexAttribArray(4);
 glDisableVertexAttribArray(5);

Items drawn after the OSG Item don't render

In retrospect, this was one was easy and didn't have anything to do with texturing. I hadn't noticed this before trying to add textures to my scene though, so mixing the two issues was my fault. I also screwed up with the traces and diff I posted; I never updated them to account for the ordering problem after I discovered it (sorry!)

Anyways, QPainter expects depth testing to be turned off. Qt will turn depth testing off when you call beginNativePainting(), and also when it starts to paint its items... but you're expected to turn it back off whenever handing control back:

1. QPainter paints stuff (DEPTH_TEST = off)
2. OSG draws stuff (DEPTH_TEST = on)
3. QPainter paints more stuff [expects DEPTH_TEST = off]

The right trace logs showed that I wasn't doing this... So the fix is

glDisable(GL_DEPTH_TEST)

Schaaf

Answer 2

Maybe you just need to reenable GL_TEXTURE_2D? I notice in your example with textures that OSG enables, and subsequently disables GL_TEXTURE_2D. Thus the difference between your two cases (with texture vs without), is that the one that uses textures finishes with texturing disabled, while the one without texturing leaves GL_TEXTURE_2D in it's initial state.

If Qt needs/expects texturing enabled to draw quads it could cause nothing to show up.

Glycogen

glEnable/glDisable(GL_TEXTURE_2D) aren't used in OpenGL ES 2, and shouldn't be called at all! In fact, calling glEnable(GL_TEXTURE_2D) should cause an error with ES 2. The trace calls are off my desktop; the code and settings I'm using with Qt and OSG should be ES 2 compatible, but in this case OSG has a bug in it (Qt also calls the deprecated function if I'm not mistaken). It works fine though since I'm on desktop (I should really be running an ES/2 emulator though). I figured out what was causing the issue though, see my answer (and thanks for the help/replies!) – Schaaf