Synchronization between command buffers in Vulkan

Question

There are several ways to handle synchronization in Vulkan. This is how I understand it:

• Fences are GPU to CPU syncs.
• Semaphores are GPU to GPU syncs, they are used to sync queue submissions (on the same or different queues).
• Events are more general, reset and checked on both CPU and GPU.
• Barriers are used for synchronization inside a command buffer.

In my case I have two command buffers. And I want the second command buffer to execute after the first one.

submitInfo.pCommandBuffers = &firstCommandBuffer;
vkQueueSubmit(queue, 1, &submitInfo, VK_NULL_HANDLE);

// wait for first command buffer to finish
submitInfo.pCommandBuffers = &secondCommandBuffer;
vkQueueSubmit(queue, 1, &submitInfo, VK_NULL_HANDLE);

What kind of synchronization is best for this? If I use vkQueueWaitIdle(queue)), is that the same thing as using a fence or should I use event or semaphores for this?

If I send multiple commandbuffer to the queue at the same time:

std::vector<VkCommandBuffer> submitCmdBuffers = {
        firstCommandBuffer,
        secondCommandBuffer
    };
    submitInfo.commandBufferCount = submitCmdBuffers.size();
    submitInfo.pCommandBuffers = submitCmdBuffers.data();

Is there still a way to synchronize between the first and the second one?

Inexplicit

"And I want the second command buffer to execute after the first one." How much do you "want" that to happen? That is, what is the second command buffer doing that it's so imperative that the first one has completed execution before the second one starts? – Clarisaclarise

The first command buffer is rendering object with the depth test turned on. The second command buffer is rendering outlines of meshes with the depth test turned off. Because it has to be on top of the other objects. – Inexplicit

If you do one submission. then use vkCmdSetEvent in your first command buffer and vkCmdWaitEvents in your second and keep src and dst stage masks as narrow as possible. – Constructive

Answer 1

The first command buffer is rendering object with the depth test turned on. The second command buffer is rendering outlines of meshes with the depth test turned off. Because it has to be on top of the other objects.

For this case, what you need rather depends on what those command buffers are.

If those are secondary command buffers executed within the same render pass instance, then you don't need any synchornization. Not unless you are manually reading from the depth texture in the secondary command buffer. Why?

Because section 2.2.1's API Ordering protects you. Depth testing and depth writing within a render-pass instance will always proceed in API order. So later commands, whether in the same CB or a different one, will be ordered with regard to depth testing/writing.

However, if you need to read that depth buffer from the shader or your command buffers are in different render pass instances, then you need explicit synchronization via an event.

In this case, the stage mask for the vkCmdSetEvent command should be the stage that writes the depth value. This could be EARLY_FRAGMENT_TESTS_BIT or LATE_FRAGMENT_TESTS_BIT. To be safe, use both. However, since you're probably updating the same color buffer, you also need the COLOR_ATTACHMENT_OUTPUT_BIT stage. Insert this command at the end of the first command buffer (or after all the depth writing is done).

For the vkCmdWaitEvent, you want to wait on the pipeline stages that need it. In your case, this is again the fragment tests and color attachment. But if a shader stage is going to read the depth, you also need that stage in the wait command.

Since memory is involved, your vkCmdWaitEvent will also need to use a memory dependency on the depth and color buffers.

Really though, all of this complexity is why you should try to put these command buffers in the same render pass instance if at all possible. The only reason you would be unable to do so is if you needed to read from the depth buffer in a shader.

Clarisaclarise

your knowledge is impressive:), "If those are secondary command buffers", why does it matter if they are primary or secondary command buffers if the they are in the same render pass? – Inexplicit

@hidayat: Because you cannot have two primary command buffers in the same render pass instance. vkCmdEndCommandBuffer cannot be called while a renderpass instance is active. Only with secondary command buffers is it possible for them to both issue commands within the same renderpass instance. They do this by inheriting the renderpass instance of the primary command buffer that they execute within. – Clarisaclarise

Answer 2

For your scenario you should use events. These should be the most light-weight synchronization objects to sync execution of two command buffers in a given order, even if you submit them at once. But note that events do not work across different queues. If you only use one, use events and try to keep src and dst pipeline stage masks as narrow as possible.

Semaphores are another way of synchronizing command buffer execution, but these only work on the queue submission, so they're more heavy-weight than events.

Constructive

Couldn't a barrier work? I keep reading here that barriers do work across command buffer boundaries: https://mcmap.net/q/502512/-vulkan-ordering-image-memory-barriers-in-multiple-command-buffers Couldn't that apply here? Or is it just that events are more lightweight than barriers are? – Squid

@Squid I wonder the same thing. As far as I understand yes, but it would be really nice if someone more experienced could provide an answer. – Maddening