Barriers in OpenCL

Question

In OpenCL, my understanding is that you can use the barrier() function to synchronize threads in a work group. I do (generally) understand what they are for and when to use them. I'm also aware that all threads in a work group must hit the barrier, otherwise there are problems. However, every time I've tried to use barriers so far, it seems to result in either my video driver crashing, or an error message about accessing invalid memory of some sort. I've seen this on 2 different video cards so far (1 ATI, 1 NVIDIA).

So, my questions are:

1. Any idea why this would happen?
2. What is the difference between barrier(CLK_LOCAL_MEM_FENCE) and barrier(CLK_GLOBAL_MEM_FENCE)? I read the documentation, but it wasn't clear to me.
3. Is there general rule about when to use barrier(CLK_LOCAL_MEM_FENCE) vs. barrier(CLK_GLOBAL_MEM_FENCE)?
4. Is there ever a time that calling barrier() with the wrong parameter type could cause an error?

Tyrelltyrian

Answer 1

As you have stated, barriers may only synchronize threads in the same workgroup. There is no way to synchronize different workgroups in a kernel.

Now to answer your question, the specification was not clear to me either, but it seems to me that section 6.11.9 contains the answer:

CLK_LOCAL_MEM_FENCE – The barrier function will either flush any variables stored in local memory or queue a memory fence to ensure correct ordering of memory operations to local memory.

CLK_GLOBAL_MEM_FENCE – The barrier function will queue a memory fence to ensure correct ordering of memory operations to global memory. This can be useful when work-items, for example, write to buffer or image memory objects and then want to read the updated data.

So, to my understanding, you should use CLK_LOCAL_MEM_FENCE when writing and reading to the __local memory space, and CLK_GLOBAL_MEM_FENCE when writing and readin to the __global memory space.

I have not tested whether this is any slower, but most of the time, when I need a barrier and I have a doubt about which memory space is impacted, I simply use a combination of the two, ie:

barrier(CLK_LOCAL_MEM_FENCE | CLK_GLOBAL_MEM_FENCE);

This way you should not have any memory reading\writing ordering problem (as long as you are sure that every thread in the group goes through the barrier, but you are aware of that).

Hope it helps.

Neogene

Obviously CLK_GLOBAL_MEM_FENCE is often slower than CLK_LOCAL_MEM_FENCE. The reason is all the threads inside a block will have to wait for memory accesses to finish. And waiting for global memory accesses to finish is much more expensive than local memory accesses. Of course, this is not always true, it depends on the access patterns (Fermi now has caches, meaning global accesses can be cached on L1, which has similar latency with shared memory), the number of global/local memory accesses in the kernel, the occupancy, bank conflicts, coalescences, etc. – Pejoration

Answer 2

Reviving an old-ish thread here. I have had a little bit of trouble with barrier() myself.

Regarding your crash problem, one potential cause could be if your barrier is inside a condition. I read that when you use barrier, ALL work items in the group must be able to reach that instruction, or it will hang your kernel - usually resulting in a crash.

if(someCondition){
  //do stuff
  barrier(CLK_LOCAL_MEM_FENCE);
  //more stuff
}else{
  //other stuff
}

My understanding is that if one or more work items satisfies someCondition, ALL work items must satisfy that condition, or there will be some that will skip the barrier. Barriers wait until ALL work items reach that point. To fix the above code, I need to restructure it a bit:

if(someCondition){
  //do stuff
}
barrier(CLK_LOCAL_MEM_FENCE);
if(someCondition){
  //more stuff
}else{
  //other stuff
}

Now all work items will reach the barrier.

I don't know to what extent this applies to loops; if a work item breaks from a for loop, does it hit barriers? I am unsure.

UPDATE: I have successfully crashed a few ocl programs with a barrier in a for-loop. Make sure all work items exit the for loop at the same time - or better yet, put the barrier outside the loop.

(source: Heterogeneous Computing with OpenCL Chapter 5, p90-91)

Tyler

Thank you for your comment. I'm not sure either, but I suspect that all threads in a work group must hit a barrier if any thread from that work group does. So, the the loop case, if all of the threads in a work group break before hitting the barrier, no problem. But, if at least one thread from a work group hits the barrier on a particular iteration of the loop, then all the threads (from that work group) must as well. – Tyrelltyrian

Another case the documentation seems to skip is also the case of having an if/else and calling a barrier in both cases. This doesn't seem to crash the program, but it doesn't seem to have any effect either. I've found, that AMD's example code keeps all threads running in for-loops, even though some have no purpose but to hit the barrier. – Indecency