How to detect when a WPF control has been redrawn?

Question

I am using D3DImage to display a sequence of frames that are rendered unto the same Direct3D Surface one after the other. My current logic is thus:

• Display last rendered frame (i.e.D3DImage.Lock()/AddDirtyRect()/Unlock())
• Start rendering next frame
• Wait for next frame to be ready and that it's time to display it
• Display last rendered frame
• ...

The problem with this approach is that when we are done calling Unlock() on D3DImage, the image isn't actually copied, it's only scheduled to be copied on the next WPF render. It's therefore possible that we render a new frame on the Direct3D surface before WPF has had the chance to display it. The net result is that we see missed frames on the display.

Right now I'm experimenting with using a separate Direct3D texture for rendering and performing a copy to a "display texture" just before display, which is giving better results but incurs substantial overhead. It would be preferrable to just be able to know when D3DImage is done refreshing and start rendering the next frame immediately after. Is this possible, if so how? Or do you have a better idea altogether?

Thanks.

Fourierism

Does CompositionTarget.Rendering help? – Secundas

My understanding is that this event is called just before a render, not after. If we have to wait for the next Rendering event to know the previous has completed, then it's too late to start rendering the frame. Ideally we'd like to get called on WPF's render thread right after it's done redrawing the D3DImage or its containing UIElement. – Fourierism

Can you do all your rendering work ahead of time (eagerly) but then use CompositionTarget.Rendering as a trigger to call Unlock() (if your render is ready) on your D3DImage and schedule another render? This means you would only ever be rendering and displaying images at most (but possibly less than) at WPF's frame rate. – Secundas

That reduces the likeliness but doesn't eliminate the possible race condition of scheduling another render right after Unlock(): the frame could finish rendering before the control is actually redrawn. – Fourierism

I don't follow. Once you call Unlock() you can immediately start another render into your D3DImage. Calling Lock() will block until the WPF render thread has copied the back buffer to the front buffer. Once you've finished your render, you would wait until the next CompositionTarget.Rendering before calling Unlock() and repeating the process again. It simply doesn't matter if you finish rendering before the control is redrawn, because it's redrawing from the front buffer, not the back buffer (which is what you're modifying). – Secundas

I'm not sure if I get what you're saying, but when you call Unlock() the buffer hasn't actually been copied, it's just scheduled to be copied on the next render pass. Even if you do this in the Rendering event, there's still some time between the Unlock() and the actual buffer copy, during which you risk overwriting the frame if you schedule another one to render. – Fourierism

Are you calling SetBackBuffer() before AddDirtyRect()? Also, the IsFrontBufferAvailableChanged event might be helpful. – Signally

Yes, and IsFrontBufferAvailableChanged only triggers on device lost/recovered which has nothing to do with the issue. – Fourierism

Answer 1

It looks like the clean way of doing this, in order to render in parallel with the UI, is to render into a separate D3D surface, and copy it to the display surface (i.e. the one passed to SetBackBuffer) between the calls to Lock() and Unlock(). So the algorithm becomes:

1. Copy and display the last rendered frame, i.e.
   • Lock()
   • Copy from render to display surface
   • SetBackBuffer(displaySurface)
   • AddDirtyRect()
   • Unlock()
2. Schedule a new render to the render surface
3. Wait for it to complete and that the timing is ok to display it
4. Goto 1

The documentation for D3DImage explicitely states:

Do not update the Direct3D surface while the D3DImage is unlocked.

The sore point here is the copy, which is potentially costly (i.e. >2ms if the hardware is busy). In order to use the display surface while the D3DImage is unlocked (avoiding a potentially costly operation at render time), one would have to resort to disassembly and reflection to hook into D3DImage's own rendering...

Fourierism

Answer 2

The CompositionTarget.Rendering event is called when WPF is going to render, so that's when you should do your Lock() and Unlock(). After the Unlock(), you can kick off the next render.

You should also check the RenderingTime because the event may fire multiple times per frame. Try something like this:

private void HandleWpfCompositionTargetRendering(object sender, EventArgs e)
{
    RenderingEventArgs rea = e as RenderingEventArgs;

    // It's possible for Rendering to call back twice in the same frame
    // so only render when we haven't already rendered in this frame.
    if (this.lastRenderTime == rea.RenderingTime)
        return;

    if (this.renderIsFinished)
    {
        // Lock();
        // SetBackBuffer(...);
        // AddDirtyRect(...);
        // Unlock();

        this.renderIsFinished = false;
        // Fire event to start new render
        // the event needs to set this.renderIsFinished = true when the render is done

        // Remember last render time
        this.lastRenderTime = rea.RenderingTime;
    }
}

Update to address comments

Are you sure that there's a race condition? This page says that the back buffer gets copied when you call Unlock().

And if there really is a race condition, how about putting Lock/Unlock around the render code? This page says that Lock() will block until the copy is finished.

Signally

As I was saying in the comments, this doesn't eliminate the race condition. The Rendering event is called on the UI thread, but the actual refresh takes place on WPF's render thread some (small) time after the event. Therefore scheduling a render right after Unlock() always risks overwriting the frame before it's been rendered. – Fourierism

Yes there is a race condition as the documentation suggests (it is "marked for rendering") and I've looked at the code in reflector. I've also validated in about 4 different ways that every time I call Lock()/Unlock(), the contents of my d3d frame are a separate new frame, so the issue here is really getting D3DImage to copy the buffer before I render the next. Additional calls to Lock()/Unlock() don't help; Lock() only blocks when the control is currently rendering. – Fourierism

@Fourierism If Lock() blocks while the control is rendering, isn't that exactly what you need? After Lock() returns, you know you can start rendering because the control is done. – Signally

I don't know if the control is rendering or not when I call Lock. This happens on WPF's render thread at time I have no grasp on. Therefore I don't know whether Lock() will wait for the buffer to be copied or not. – Fourierism

@Fourierism The Lock() link in my answer says that "calls to the Lock method block until the render thread has copied the contents of the back buffer to the front buffer". – Signally

"When the changes are committed and rendering occurs". This is exactly what I'm trying to determine in the first place. – Fourierism

@Fourierism I'm definitely not sure about this, but I think that "commiting changes" happens when you Unlock(). – Signally

Yes, but rendering happens at in indeterminate time on WPF's rendering thread. Calling Lock() on the UI thread, even in the Rendering event or whenever else, isn't guaranteed to happen when rendering occurs. Therefore this isn't a solution. – Fourierism

@Fourierism "calls to the Lock method block until the render thread has copied..." – Signally

... if the render thread was currently rendering the control at the time you called Lock. That's clear from the documentation, the code I see in reflector and my own experience, what's there to argue about? – Fourierism

@Fourierism I was pointing out that Lock() pays attention to the render thread, which is what you were concerned about in your previous comment. I contend that after you get the lock, the copy from back to front buffer is done and you can do anything you want to the back buffer regardless of WPF rendering. If you've tested and that's not the case, then I'm sorry but I can't be of any further help. – Signally