Noticable lag in a simple OpenGL program with mouse input through GLFW

Question

Here's a simple program that draws a triangle following the mouse cursor's position.

What I (and hopefully you) can notice, is that the triangle lags behind the cursor, it's not as tight as when dragging around even a whole window.

So my question is: What am I doing wrong? What leads to this lag?

One thing I realize is that it would suffice to shift the actual pixel values of the triangle, and not rasterize it again and again. But is rasterizing this one triangle really that expensive? I also tried using glTranslate instead of drawing at varying coordinates, but no improvement on the lag resulted. So I hope you can enlighten me on how to draw this efficiently.

#include <GLFW/glfw3.h>
#include <stdlib.h>
#include <stdio.h>
#include <math.h>

float x = 0.0f;
float y = 0.0f;

static void error_callback(int error, const char* description)
{
    fputs(description, stderr);
}

static void key_callback(GLFWwindow* window, int key, int scancode, int action, int mods)
{
    if (key == GLFW_KEY_ESCAPE && action == GLFW_PRESS)
        glfwSetWindowShouldClose(window, GL_TRUE);
}

static void cursor_callback(GLFWwindow *window, double xpos, double ypos)
{
    int width, height;
    glfwGetFramebufferSize(window, &width, &height);
    float ratio = width / (float) height;
    x = ratio*(2*xpos/(float)width - 1);
    y = 2*-ypos/(float)height + 1;
}

int main(void)
{
    GLFWwindow* window;
    glfwSetErrorCallback(error_callback);
    if (!glfwInit())
        exit(EXIT_FAILURE);
    window = glfwCreateWindow(640, 480, "Following Triangle", NULL, NULL);
    if (!window)
    {
        glfwTerminate();
        exit(EXIT_FAILURE);
    }
    glfwMakeContextCurrent(window);
    // Callbacks
    glfwSetKeyCallback(window, key_callback);
    glfwSetCursorPosCallback(window, cursor_callback);
    // geometry for the equal sided triangle
    float r = 0.1f; // outer circle radius
    float u = r * sin(M_PI_2/3.0f);
    float l = 2.0f * r * cos(M_PI_2/3.0f);

    while (!glfwWindowShouldClose(window))
    {
        int width, height;
        glfwGetFramebufferSize(window, &width, &height);
        float ratio = width / (float) height;
        glViewport(0, 0, width, height);
        glClear(GL_COLOR_BUFFER_BIT);
        glMatrixMode(GL_PROJECTION);
        glLoadIdentity();
        glOrtho(-ratio, ratio, -1.0f, 1.0f, 1.f, -1.f);
        glMatrixMode(GL_MODELVIEW);
        glLoadIdentity();

        glBegin(GL_TRIANGLES);
            glColor3f(1.f, 0.f, 0.f);
            glVertex3f(x+0, y+r, 0.f);
            glColor3f(0.f, 1.f, 0.f);
            glVertex3f(x-l/2.0f, y-u, 0.f);
            glColor3f(0.f, 0.f, 1.f);
            glVertex3f(x+l/2.0f, y-u, 0.f);
        glEnd();

        glfwSwapBuffers(window);
        glfwPollEvents();
    }
    glfwDestroyWindow(window);
    glfwTerminate();
    exit(EXIT_SUCCESS);
}

Michamichael

Do you have a compositor and/or vsync enabled? Does your main while loop always take ~16-17ms to execute? – Harbinger

To both questions I don't know the answer. I'll try measuring the main loop's single runtime. – Michamichael

Do you have a reasonably decent graphics card with current drivers? – Wilhite

It's good enough to play 3 year old games at medium settings, so yes. And like I said, I'm dragging whole windows around without noticable lag. – Michamichael

OK, I can confirm, my main loop takes 16.9ms (averaged across 100 runs) – Michamichael

@Phantrast: You would generally know if you have VSYNC enabled, it can be enabled in the application through requesting a non-zero swap interval (0 is usually the default) or forced on/off in a driver. Input latency is especially apparent when you draw a cursor in your software instead of using the operating system's built-in cursor. One thing you can do to help while VSYNC'd is to defer the swap buffer operation until as late as possible, this would mean moving CPU-based operations like glfwPollEvents (...) to come before glfwSwapBuffers (...). – Aviate

Thanks, Andon (and genpfault)! So shifting glfwSwapBuffers to the end of the loop didn't change much. Disabling VSYNC with glfwSwapInterval(0) however eliminated the lag at the cost of some tearing. "Input latency" seems to be the keyword here. So... I still have doubts that without VSYNC it's inevitable to have this much lag. Or why is it easily possible for the OS to move a window (even with video playing in it) without any noticable lag, and this very simple program exhibiting this so clearly? – Michamichael

@AndonM.Coleman: Why did you remove your answer? I was just about to reply: This sounds complicated but also interesting. I'll try this tomorrow and hope to understand the issue afterwards. Right now, I still fail to believe that this much lag is "normal". I'd be very happy to understand why it's then possible to drag desktop icons and windows around quickly. Is it because it's deeper in the OS or is it because they're also applying these delta tricks? – Michamichael

@Phantrast: Because about half way through proof-reading it, I realized I answered the wrong question. You are trying to position the cursor at an absolute position, I had it in my mind that you were trying to use the mouse input to rotate something. Minimizing input latency is a lot more complicated, though if your latency is as bad as you suggest, calling glFinish (...) after glutSwapBuffers (...) may help. If it does, I can re-open my answer and explain why. – Aviate

Answer 1

Your updates are purely event driven. Try replacing glfwPollEvents with glfwWaitEvents. I would then reimplement glfwSwapInterval(1). You gain nothing by updating more frequently than the monitor refresh rate - just tearing and burning through cycles.

Sharitasharity

That yields quite an improvement! However, it's still not quite as tight as I'd like it and know it should be possible (examples given above in other comments). Using this as a cursor replacement in a game still wouldn't seem right. One thing I also notice is that when quickly sliding across the touchpad, it's possible to halt the cursor at a position far from the triangle. – Michamichael

Hmm. I would use glViewport in a callback set by glfwSetFramebufferSizeCallback, rather than doing this on each redraw. Still... I wouldn't expect this to be a performance killer. I've long since moved to shaders I'm afraid, so I'm not real clear on what could be hurting performance. As you say though, this should be a negligible amount of work for the GPU! – Sharitasharity

Retrieving the framebuffer size in each run also seemed unnecessary to me, but removing it changed nothing, so it's cheap. Shaders... yes, I presume the above isn't "modern OpenGL", but I find it really hard to learn from the available material anyhow. Most of the new guides seem agnostic on the traditional pipeline. I'd very much prefer a guide that deliberately explains both and the differences but since I know none, I decided to first learn the old way and then the new way, simply so that the (often narrowly minded) recent texts start to make more sense. – Michamichael

@Michamichael - yeah. Legacy GL always kept the user in mind - but this just couldn't keep up with paradigms necessary for modern graphics hardware. Modern GL requires a very solid understanding of coordinate spaces (world space vs (homogeneous) clip space vs 3D window space), resources-as-objects, and more technical 'under-the-hood' details. The emphasis is on performance, and it's up to the user to track a relatively steep learning curve. That said, I've never bought a book, and there are some good tutorials to get started with. – Sharitasharity

Had the same problem as OP, replacing glfwPollEvents did not help me as it freezes my application unless I'm giving some input. However, glfwSwapInterval(1) made all the difference, I feel saved! Thank you! – Arnold

Answer 2

So yeah the usual approach to synchronising rendering to frame rate is with SwapInterval(1) and you weren't doing that, but that's not where the lag is coming from. In fact, typically with SwapInterval set to 0, you get less lag than with it set to 1, so I suspect it was actually set to 1 the whole time. So, I'm writing the rest of this supposing SwapInterval was set to 1.

You are seeing this lag for two independent reasons.

reason 0: your code is impatient.

Glossing over some details, the naïve method of rendering in OpenGL is with a loop like this:

while (!exitCondition()) {
    pollEvents();
    render();
    swap();
}

This loop is run once per frame. If render() is fast, most of the time is spent at swap(). swap() doesn't actually send out the new frame, though, until the moment it returns. New mouse and keyboard events may happen that entire time, but they will have no effect until next frame. This means the mouse and keyboard information is already one to two frames old by the time it reaches the screen. For better latency, you should not be polling events and rendering immediately after swap() returns. Wait for as many new events as possible, render, then send the frame just in time for it to be displayed. As little time as possible should be spent waiting for swap(). This can be achieved by adding a delay to the loop. Suppose tFrame is the amount of time between frames (1s/60 .= 16.67ms for a 60Hz screen) and tRender is an amount of time that is usually greater than the amount of time render() takes to run. The loop with the latency delay would look like this:

while(!exitCondition()) {
    sleep(tFrame - tRender);
    pollEvents();
    render();
    swap();
}

Patience is a virtue in computing too, it turns out.

reason 1: glfwSwapBuffers() does not behave like you would expect.

A newcomer would expect glfwSwapBuffers() to wait until the vsync, then send the newly-rendered frame to the screen and return, something like the swap() function I used in reason 0. What it actually does, effectively, is send the previously-rendered frame to the screen and return, leaving you with a whole frame of delay. In order to fix this, you have to obtain a separate means of synchronising your rendering, because OpenGL's mechanism isn't good enough. Such a mechanism is platform-specific. Wayland has such a mechanism, and it's called presentation-time. GLFW doesn't currently support this, but I was bothered so much by this synchronisation problem that I added it. Here's the result:

As you can see, it really is possible to synchronise rendering to the system cursor. It's just really hard.

Massorete

I'm trying to achieve similar result using Vulkan. Do you think this is applicable with Vulkan too? – Meliorism

The first problem applies everywhere that render pattern exists. The second one, well I'm really hoping vulkan is going to fix that and add support for something resembling wayland's presentation-time, but portable. I don't know. Something to do with WSI, though. – Massorete

I'm looking at your patch from the link and I don't see how you get around the glfwSwapBuffers() issue (reason # in your answer1). Is there more to the implementation than what your patch shows? I'm trying to implement something like this myself for mac. – Amphisbaena

On top of using the function I implemented, I also set swapInterval() to 0, causing glfwSwapBuffers() to act instantly. With swapInterval() set to 0, the GL implementation assumes either you don't care about synchronisation and the associated tearing, or you have your own way of synchronising. In our case, it's the latter. – Massorete

Answer 3

While the solution from enigmaticPhysicist is the best we can achieve from glfw with Wayland.

Since glfw 3.2 i was able to get smooth mouse experience by disabling vsync and manually tuning the correct timeout for

// Waits with timeout until events are queued and processes them.
GLFWAPI void glfwWaitEventsTimeout(double timeout);

Replace the usage of greedy glfwPollEvents with something like glfwWaitEventsTimeout(0.007)...

Wrath

didn't make any difference for me – Etheridge

Answer 4

I found that placing glFinish() call right after swapping buffer eliminates the lag. It is strange, because AFAIK swapping buffers must wait for OpenGL commands to complete, but nevertheless, this trick helps for whatever reason.

Isomerism

Answer 5

I think this is due to double buffering: the OpenGL is drawn is a second buffer while the first one is displayed in the window. And the buffers are swapped using the command

glfwSwapBuffers()

Thus there will always be a lag of at least one frame, i.e. 33 ms at 30 FPS or 17 ms at 60 FPS. This is inherent to the method and cannot be fixed. However, if the system mouse is hidden using the following command:

glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_HIDDEN);

Then without the reference mouse, the brain somehow gets used to the lag and it the lag disappears.

Caboose

You're right! I could confirm your claim in a small experiment. I used a metronome and I disabled the mouse acceleration and then moved the mouse steadily. The 33ms (resp. 17ms) input lag even at moderate mouse speeds indeed amounts to as much as one centimeter easily. Thanks for pointing that out! – Michamichael

I am unsatisfied with this. Windows can be dragged perfectly synchronous to the system mouse's movement. All of the rules that apply to the client program also apply to the window manager. How does the window manager get better responsiveness? The fact that the system cursor leads the opengl cursor proves better responsiveness is possible. How else would the system know where to put the cursor? It violates causality. Is the system violating the laws of physics? You see why there is still a problem, hopefully. – Massorete