What's the recommended way to unittest Python GUI applications?

Question

I'm currently foolish enough to try to maintaintain two parallel code bases for a Python desktop application, one using PyGObject introspection for GTK 3 and one using PyGTK for GTK 2. I work mainly on the PyGObject branch and then I port changes over to the PyGTK branch. Due to all the minor differences between these implementations, I often overlook things and cause breakage that I miss and accidentally release, only to be caught by the users.

I'm trying to figure out a good way to design some unittests that would, preferably, be suitable to run on both code bases. It's not an overly complicated program, (it's essentially a library management tool, imagine like iTunes):

- Main Window
  |- Toolbar with some buttons (add/edit/remove items, configure the program)
  |
  |- VPaned
  |--- Top HPaned
  |------ ListView (listing values by which a library of items can be filtered)
  |------ ListView (listing the contents of the library
  |--- Bottom HPaned
  |------ Image (displaying cover art for the currently selected item in the library)
  |------ TextView (displaying formatted text describing the currently selected item)
 - Edit dialog
 - Configuration dialog
 - About dialog 

I've tried to separate views from models as much as possible. Each of those items is implemented in its own class (well, in classes which inherit from the GTK classes listed). The ListViews are coupled with other classes which inherit from ListStores. The library itself is handled by a different class. Nonetheless, there are interactions between the widgets that need to be tested. For example, if the user selects a particular item in the filter view, filtering the library, and then selects an item from the filtered results, the text view must display the information for the correct library entry, which is semi-complicated due to translating iters between TreeModelFilter and the original ListStore, etc etc.

So, I ask, what is the recommended way for writing robust unit tests for such a GUI application? I've seen that there are some libraries for this but the main ones for pygtk haven't been updated in years and so they will almost certainly fail with PyGObject introspection. Perhaps I'm not creative enough to figure out a good way to do it using Python's unittest module, so I'm open to suggestions.

Thyrse

And before anyone asks: pygtk doesn't support GTK 3, but I found that support for pygobject introspection of GTK 2 was too incomplete to depend on. – Thyrse

For me, the best way to avoid regression when using GUI is to do manuals tests. Unittest is really good for testing work functions (in case of MVC as example) but i don't know how you can deal with gui... – Harvard

This is what I have been doing but I'm apparently not very good at it, since every release I miss something. – Thyrse

have you checked out @ unpythonic.blogspot.com/2007/03/unit-testing-pygtk.html? – Tarmac

You should have put that as an answer. I haven't encountered that method. I think I could piece together some complicated test scripts using that method. – Thyrse

Answer 1

Are you sure you want to unit test the GUI? Your example of complexity involves more than 1 unit and is therefore an integration test.

If you really want to unit test, you should be able to instantiate a single class providing mocks or stubs for its dependencies, then call methods on it like the GUI framework would for e.g. a user click. This can be tedious and you have to know exactly how the GUI framework dispatches user input to your classes.

My advice is to put even more stuff in models. For your given example you could create a FilterManager which abstracts all the filter/select/display stuff behind a single method. Then unit test it.

Kristine

Ah, perhaps I have my terms wrong then (it has been almost 10 years since I learned/used any of this). If I may naively ask, then, is integration testing as robustly defined or designable as unit testing? Nonetheless, I do think that a lot of it can be unit tested the way I've written it, using the method mentioned in a comment on my question. – Thyrse

Well the core concept of unit testing is to test a single unit at a time, in order to keep tests complete, simple, fast and be able to pinpoint failures exactly. Sometimes this is hard, especially for GUIs. You may be able to use some of Python's unittest modules for non-unit tests too, but you also lose some of the benefits, and I wouldn't call it unit testing any more. – Belda

This is not a real answer... Obviously he wants integration testing with PyGTK, so point him in the right direction instead of correcting terminology. – Assail

@Assail I'd prefer if you would provide your answer instead of hinting at a different take and downvoting. I have no integration testing experience myself. – Belda

Maybe you should check out this link: obeythetestinggoat.com/… – Assail

Answer 2

There is a great way to test PyGTK functions and widgets directly, without going through acceptance/functional/integration testing frameworks that clikety their way into oblivion. I learned about this in this post which is fairly self explanatory. But the basic idea is that you treat your widgets as functions/classes, and you can test them directly. If you need to process callbacks and so on, there's a neat trick that I will reproduce here:

import time
import gtk

# Stolen from Kiwi
def refresh_gui(delay=0):
  while gtk.events_pending():
      gtk.main_iteration_do(block=False)
  time.sleep(delay)

As mentionned in the blog post, this code is LGPL. Otherwise, when you think about it, as long as you don't show() windows or widgets, you can test them all you want and they should behave as if they were real because, in a way, they are. They are just not displayed.

Of course, you need to simulate interaction on buttons and interactive widgets yourself by calling clicked() on a button for example. See again Ali Afshar's excellent post about unit testing in PyGTK.

Quintanilla

Answer 3

In sticking with the theme that Jürgen was correct in that I'm not interested in unit testing, but actually interested in integration testing, I also found this framework from freedesktop.org: http://ldtp.freedesktop.org/wiki/

It allows automating a variety of tests for Accessibility-enabled GUI applications (including GTK, Qt, Swing, etc).

Thyrse

Answer 4

You can use a X11 framebuffer:

Xvfb :119 -screen 0 1024x768x16 &
export DISPLAY=:119
... run your tests

Be sure, to not enter gtk.main(), since this would wait for mouse or keyboard input. You can use this pattern to let gtk handle all events:

def refresh_gui():
  while gtk.events_pending():
      gtk.main_iteration_do(block=False)

AFAIK you can't see your application, but you can test your callbacks.

Bobker

xvfb-run -a testname is better – Two