How can I simulate mouse clicks by posting events to the Qt event system?

Question

I would like to do a rudimentary automation test of my Qt application. It records mouse events and writes them to a file (f.e. mousepress(300, 400)). When starting the automation, it reads the coordinates from the file, sends the appropriate mouse events and would do a pixel comparison with a previously saved screenshot.

Currently, I have an overlay widget that spans the app and has transparent mouse events. All it does is track the coordinates. When reading the data back in, that overlay paints a rectangle on the mouse press location. I need help when sending mousePressEvents to Qt's event system. It draws the points on the correct location but never actually does the physical click. Is there a way to do this with Qt or would I have to use Window's SendInput()?

Is there a way to pause and wait until the mouse event has finished? I would need to know when the event is complete in order to start the pixel by pixel comparison.

Widget::Widget( QWidget *parent )
: QFrame( parent )
, mPoint(QPoint(0,0))
{
   setWindowFlags(Qt::WindowStaysOnTopHint);
   setStyleSheet("background-color: rgba(0, 0,255, 2%);");
   setAttribute(Qt::WA_TransparentForMouseEvents, true);
   setGeometry(parent->geometry());
   ...
}

void Widget::run()
{
   QFile file( "coordinates.txt", NULL );
   if(!file.open( QIODevice::ReadOnly ))
       return;

   QTextStream in(&file);
   int i = 0;
   while (!in.atEnd())
   {
       QString line = in.readLine();
       if(line.startsWith("mousepress"))
       {
          int startIndex = line.indexOf('(');
          int endIndex = line.indexOf(')');

          QString coord = line.mid(startIndex+1, line.size() - startIndex - 2);
          QStringList nbr = coord.split(',');
          mPoint = QPoint(nbr[0].toInt(), nbr[1].toInt());
          QWidget *receiver  = QApplication::widgetAt(mPoint);
          QMouseEvent *event = new QMouseEvent(QEvent::MouseButtonPress, mPoint, Qt::LeftButton, Qt::LeftButton,  Qt::NoModifier);
          QCoreApplication::postEvent(receiver, event); // same result with sendEvent() 
          QCoreApplication::processEvents();
          update();
          // wait till the event finished, then continue with next point
      }
   }
}


void Widget::paintEvent(QPaintEvent *event)
{
  QPainter p( this );
  QPen pen;
  pen.setBrush(Qt::NoBrush);

  if(!mPoint.isNull())
  {
    pen.setColor(Qt::red);
    pen.setWidth( 2 );
    p.setPen(pen);

    p.drawRoundRect(mPoint.x(), mPoint.y(), 10, 10, 25, 25);
    p.drawText(mPoint, QString::number(mPoint.x()) + ", " + QString::number(mPoint.y()));
  }
}

[Edited]

I followed ddriver's suggestion and it works after a few changes: I save global and local positions in the file, to send to the QMouseEvent.

How could I be sure that the mouse click is complete before doing a screenshot and comparing it to a saved image?

void Widget::DoStep()
{
  if(!mInStream.atEnd())
  {
      QString line = mInStream.readLine();
      if(line.startsWith("MouseButtonPress"))
      {
          QPoint pos = parseGlobalPos();
          QPoint localPos = parseLocalPos();
          QWidget *receiver  = QApplication::widgetAt(pos);

          QMouseEvent *event = new QMouseEvent(QEvent::MouseButtonPress,localPos, pos, Qt::LeftButton, Qt::LeftButton,  Qt::NoModifier);
          QApplication::postEvent(receiver, event);
          QMouseEvent *eventRelease = new QMouseEvent(QEvent::MouseButtonRelease, localPos, pos, Qt::LeftButton, Qt::LeftButton,  Qt::NoModifier);
          QApplication::postEvent(receiver, eventRelease);

          // after successful click, take screenshot and compare them
      } 
   }

  if (!mInStream.atEnd())
      QMetaObject::invokeMethod(this, "DoStep", Qt::QueuedConnection);
  else
      file.close();
}

Lindner

It may not completely solve this, however, you submit an event of MouseButtonPress, but you're missing a corresponding MouseButtonRelease. In addition, I suggest not looping and waiting for the event to finish, but instead, use a single shot QTimer for each event and let the main even loop continue to run, without resorting to calling processEvents. – Milden

thanks, I tried posting an additional MouseButtonRelease event but that didn't work. – Lindner

I'm not quite sure I understand your QTimer idea. QTimer::singleShot(0, receiver, SlotSendEvent()); It's not possible to pass in the coordinates to the slot. What did you mean by that? – Lindner

It is if you create a QTimer object and connect to a lambda function. Even better, if you're using Qt 5.4, you can do this with the static call: QTimer::singleShot(1000, [=]() { PostEvent(evt); } ); – Milden

I got the same result and no actual mouse press in the application. Any other ideas? – Lindner

Answer 1

If I understand the problem correctly, its source is the blocking while loop, which blocks the thread and doesn't allow the event loop to spin and process events. There is no way to "pause" as that would block the event loop as well, and it wouldn't be able to do the work either, but there is a way to split the work to be done one step at a time, one event loop cycle at a time.

The solution is to not use a blocking while loop but implement an event system driven loop. Then you process one line on every event loop cycle until you run out of work.

Move the file, stream and all that stuff outside of the function, make them class members so they persist. Then in run() you simply setup the input for reading, and move all the event posting stuff to a new function, for example doStep(), and in run() after you setup you have a:

QMetaObject::invokeMethod(this, "doStep", Qt::QueuedConnection);

In doStep(), aside from the event stuff, at the end you also do scheduling:

if (!in.atEnd()) QMetaObject::invokeMethod(this, "doStep", Qt::QueuedConnection);
else // we are done, close file, cleanup, whatever

This way only one step will be executed per event loop cycle, allowing the event loop to spin and process the events. While there is work, a step will be scheduled for the next event loop cycle. You can also use a single shot timer to do this, or even a queued connection, which you can trigger by emitting overloading the event to emit a signal on completed. You do not, and should not force events to be processed, using this approach there will be no need to.

The concept in general is about a non-blocking async event driven self scheduling worker, I have posted a fully implemented one here. The upsides are you can track progress, pause, cancel and all that good stuff.

Wrestle

that worked: but how could I be sure that the mouse click is complete before doing a screenshot and comparing it to a saved image? see my edited question – Lindner

@Lindner - you can do another queued invocation - doStep() will schedule takeScreenshot(), and takeScreenshot() will schedule the next doStep(). This way all events will be processed before each scheduled invocation. – Wrestle

Another alternative is to schedule the screenshot from the click event handler of the receiver. It is not clear what you exactly mean by "the click is completed". I assume it is the drawing of the click position. So you can put it after the drawing code as well. – Wrestle

I don't quite understand the event sequence: can I be sure that the receiver object processes the event, that was sent through: QApplication::postEvent(receiver, eventRelease) first, before QMetaObject::invokeMethod(this, "screenshot", Qt::QueuedConnection); is processed? If they are both sent to the same queue and processed in order, then yes: that works! – Lindner

@Lindner - point is that when you use queued connections, the invocation is queued for the next event loop cycle, whereas input events are sent immediately. So the release event will be processed immediately, while the invocation event will be queued for the next event loop cycle, so it could not possibly be processed before the release event. And if the queuing goes step->screenshot->step then they will be executed in the proper order as well, as they will always be scheduled for the next event loop cycle, while their work is done in a blocking manner, so it will be completed before that. – Wrestle

your explanation makes sense. The problem is that some clicks result in a UI freeze for a second or so. f.e. click on toolbutton-> processing -> toolbar slides open. In that case the screenshot has already been taken before the toolbar slides down. I could fix it by adding QTimer::singleShot(2000, this, SLOT(SlotTakeScreenshot())); but I would like to understand why it's happening. – Lindner

@Lindner - without any code and no information what happens on a click I can't understand why that's happening either. I am not a psychic you know ;) If a timer works for you go for it. – Wrestle

Answer 2

Try sending event to QGraphicsView's viewport:

qApp->sendEvent(view->viewport(), &mousePressEvent);

Slantwise

This doesn't answer or even attempt to answer the question. – Wrestle