Real-life use cases of barriers (DSB, DMB, ISB) in ARM

Question

I understand that DSB, DMB, and ISB are barriers for prevent reordering of instructions. I also can find lots of very good explanations for each of them, but it is pretty hard to imagine the case that I have to use them.

Also, from the open source codes, I see those barriers from time to time, but it is quite hard to understand why they are used. Just for an example, in Linux kernel 3.7 tcp_rcv_synsent_state_process function, there is a line as follows:

    if (unlikely(po->origdev))
            sll->sll_ifindex = orig_dev->ifindex;
    else
            sll->sll_ifindex = dev->ifindex;

    smp_mb();

    if (po->tp_version <= TPACKET_V2)
            __packet_set_status(po, h.raw, status);

where smp_mb() is basically DMB. Could you give me some of your real-life examples? It would help understand more about barriers.

Pollypollyanna

This question was a recent example: https://mcmap.net/q/390584/-does-mutex_unlock-function-as-a-memory-fence/1163019 – Oswald

Cortex programmer's guide has a section for barriers (11.2) as well. infocenter.arm.com/help/topic/com.arm.doc.den0013c/index.html – Oswald

understand that the folks that work on the linux kernel very often use overkill or completely botch up the understanding an implementation of an errata, etc. it might work on their chip but may have to be repaired to work on yours. folks will throw these in even if not needed. Basically the linux source (or any large project like that, closed or open) is not an authority no the right way to do something. – Dorolice

Answer 1

Sorry, not going to give you a straight-out example like you're asking, because as you are already looking through the Linux source code, you have plenty of those to go around, and they don't appear to help. No shame in that - every sane person is at least initially confused by memory access ordering issues :)

If you are mainly an application developer, then there is every chance you won't need to worry too much about it - whatever concurrency frameworks you use will resolve it for you.

If you are mainly a device driver developer, then examples are fairly straightforward to find - whenever there is a dependency in your code on a previous access having had an effect (cleared an interrupt source, written a DMA descriptor) before some other access is performed (re-enabling interrupts, initiating the DMA transaction).

If you are in the process of developing a concurrency framework (, or debugging one), you probably need to read up on the topic a bit more - but your question suggests a superficial curiosity rather than an immediate need? If you are developing your own method for passing data between threads, not based on primitives provided by a concurrency framework, that is for all intents and purposes a concurrency framework.

Paul McKenney wrote an excellent paper on the need for memory barriers, and what effects they actually have in the processor: Memory Barriers: a Hardware View for Software Hackers

If that's a bit too hardcore, I wrote a 3-part blog series that's a bit more lightweight, and finishes off with an ARM-specific view. First part is Memory access ordering - an introduction.

But if it is specifically lists of examples you are after, especially for the ARM architecture, you could do a lot worse than Barrier Litmus Tests and Cookbook.

The extra-extra light programmer's view and not entirely architecturally correct version is:

• DMB - whenever a memory access requires ordering with regards to another memory access.
• DSB - whenever a memory access needs to have completed before program execution progresses.
• ISB - whenever instruction fetches need to explicitly take place after a certain point in the program, for example after memory map updates or after writing code to be executed. (In practice, this means "throw away any prefetched instructions at this point".)

Cassondracassoulet

Nice explanation. I can't help wonder a bit about the 3 instructions though. Basically, what you write is: DMB: make sure that above code is completed. DSB: make sure that above code is completed. ISB: make sure that above code is completed. (I prolly ought to read the links you supplied.) Still, wouldn't "volatile" tell the compiler to guarantee instruction order? (Eventually by inserting one of the above commands etc.) – Jewel

@Illishar: volatile affects the order in which the instructions are generated by the compiler, yes. These instructions affect when the results of those instructions are guaranteed to be architecturally consistent in an out-of-order and/or multi-cpu system. Paul McKenney's paper is the most complete one of the listed resources, and well worth spending some time with. As for the instructions, that's not what the descriptions say. When entering the world of concurrency, get used to having to pay a lot of attention to semantics. – Cassondracassoulet

@Jewel volatile actually does not affect the order in which instructions are generated by the compiler. Instead, it's telling the compiler to ensure a memory access will take place. It doesn't mean instruction ordering can't be optimized. Also, there's a lot of stuff that effect instruction ordering independent of the compiler. Barrier instructions can ensure that side execution waits until side effects (which may not be code dependent at all) are complete. – Sciatic

To clarify - I meant it does not affect order in the way implied. Compilers can still reorder volatile accesses and optimize code with volatile objects in ways that may not be intuitive. This link may give more context: gcc.gnu.org/onlinedocs/gcc/Volatiles.html – Sciatic

@Sciatic volatile affects the ordering (and number) of accesses to volatile locations. I did not mean to imply it prevents optimizations beyond that. – Cassondracassoulet

I wanted to make it clear that volatile does not guarantee a particular ordering of instructions and while it makes some guarantees about memory accesses, they are not intuitive and differ from what barrier instructions guarantee. Pretty much volatile is not an alternative to barriers and don't even guarantee a particular ordering of memory accesses. – Sciatic

Answer 2

Usually you need to use a memory barrier in cases where you have to make SURE that memory access occurs in a specific order. This might be required for a number of reasons, usually it's required when two or more processes/threads or a hardware component access the same memory structure, which has to be kept consistent.

It's used very often in DMA-transfers. A simple DMA control structures might look like this:

struct dma_control {
  u32 owner;
  void * data;
  u32 len;
};

The owner will usually be set to something like OWNER_CPU or OWNER_HARDWARE, to indicate who of the two participants is allowed to work with the structure.

Code which changes this will usually look like this

dma->data = data;
dma->len  = length;
smp_mb();
dma->owner = OWNER_HARDWARE;

So, data and len are always set before the ownership gets transferred to the DMA hardware. Otherwise the engine might get stale data, like a pointer or length which was not updated, because the CPU reordered the memory access for performance reasons.

The same is true for processes or threads running on different cores. They could communicate in a similar manner.

Gills

Answer 3

One simple example of a barrier requirement is a spinlock. If you implement a spinlock using compare-and-swap(or LDREX/STREX on ARM) and without a barrier, the processor is allowed to speculatively load values from memory and lazily store computed values to memory, and neither of those are required to happen in the order of the loads/stores in the instruction stream.

The DMB in particular prevents memory access reordering around the DMB. Without DMB, the processor could reorder a store to memory protected by the spinlock after the spinlock is released. Or the processor could read memory protected by the spinlock before the spinlock was actually locked, or while it was locked by a different context.

unixsmurf already pointed it out, but I'll also point you toward Barrier Litmus Tests and Cookbook. It has some pretty good examples of where and why you should use barriers.

Suzetta