Automated testing for OpenGL application

Question

I have a Java Application that uses JOGL to provide a large part of the GUI.

Is there any tool which you know of, or have used which can automate the testing of OpenGL applications (or more specificly those using JOGL)

Just to update: The tool can run on either linux or windows.

Harlamert

pardon the noob question, but what makes an OpenGL application testing different from any other app testing ? – Crosley

What I really meant is to test the OpenGL view itself. Clicking, draging, rotating zooming etc .... There are tools to allow you to record your GUI interaction with say a Swing App and replay them. You can do this for all your core interactions and replay it as regression testing. I'd like a similar solution for OpenGL but I don't know of one that exists. – Harlamert

Answer 1

I have written unit-tests for C++ (Qt on Linux) & OpenGL before. I don't know any reason it shouldn't work for Java too.

The things which worked for me are:

• Abstract your OpenGL context provider so the rest of your code is independent of it. In my case the main app used Qt's QGLWidget, but the unittests used a pbuffer-based one which I could create with no windowing infrastructure at all (other than a designated X11 DISPLAY). Later I added an "offscreen Mesa" (pure software OpenGL implementation) so they'd even work on a headless build machine with no GPU at all.

• Keep your OpenGL code independent of your GUI code. In my case the OpenGL "rendering engine" didn't know anything about Qt classes (e.g mouse events). Define your own testable API which isn't tied to any specific GUI concepts, and write tests for it.

• In the unittests, read the contents back from the framebuffer using glReadPixels and either hit them with some assertions about which pixels should be particular values, or go down the regression-testing route and compare the framebuffer capture with a stored image you know is good (either from manual verification, or because it's produced from some other reference model).

• Allow a bit of fuzziness in any image regression testing; most OpenGL implementations produce slightly different output.

• (I didn't do this, but...) Ideally you want to be able to test the GUI layer by asserting that it makes the expected sequence of calls to your rendering engine in response to GUI activity. If it does that, and you're confident of your render layer because of the above tests... well, no need to actually do the rendering. So create a suitable mock object of your rendering layer for use when GUI testing (I'd imagine tests like "mouse drag from here to there results in a call to the rendering layer to set a particular transform matrix"... stuff like that).

Tomkin

+1 for all good advice. For the framebuffer comparison we wrote a little app that does a fuzzy difference on the images that looks at the perceptual deltas, rather than absolute pixel values. – Farly

The problem is the GUI is a custom built framework built using JOGL. Thats why number 2 can't work. For example we use to create graphical elements on the screen which the user can interact with all using JOGL. I need a way to automate the user interaction.... – Harlamert

Using any particular library to implement the GUI shouldn't matter. Create a GraphicsContext class that abstracts the OpenGL Canvas/frame, and a GUI class that abstracts whatever GUI features you need. Then, the GUI class implementation can render to the display context implementation, but you app won't need to care how they're working together. – Panada

Do you have webstarts/demos of your JOGL based app ? I would be interested in a JOGL testing framework as well. – Gavelkind

Answer 2

You can test your JOGL based application using Sikuli which performs UI automation via image recognition techonolgy on screen-shots.

I am currently using Sikuli to functional-test a Java app that is predominantly based on the NASA Worldwind Java SDK (which is based on JOGL). Using the Sikuli Java API my test suite can recognise icons within the OpenGL canvas, click on them and also drag them around. Sikuli can also recognise and extract text from the canvas via OCR, however the performance of this seems to be a little hit-and-miss (depending on language, font, size and background colours behind the text).

I have done a lot of automated UI-Testing using other tools that work by introspecting the windowing toolkit (e.g. Swing, SWT, native Windows) and found that Sikuli runs much slower than those, however that is understandable given the amount of image-processing it needs to do behind the scenes. Also note that Sikuli currently requires your application to run in a Window (not in full screen mode).

Sikuli runs on both Windows and Linux. I'd recommend you try it out. I couldn't find any other tool capable of doing that level of functional-testing of an OpenGL based application.

Vern

Answer 3

I've been thinking of using a picture-diff tool such as PDiff to test OpenGL code, by taking snapshots, saving them to disk and comparing with previous regression output. That way, really bad stuff (missing textures) pop up but humanly unnoticable things (such as the above mentioned small diffs between implementation) go through fine.

Also, for automating user interaction, either the GUI classes should be sufficiently open for you to send events or call 'clicked' on a button, or you have to inject OS events manually to your apps. This is possible, but much more troublesome. Might be easier to open up the GUI layer, if it's Open Source.

Ramirez

SSIM ("Structural Similarity index") - en.wikipedia.org/wiki/Structural_similarity - is another image comparator based on perceptual considerations. – Tomkin