spin_lock_irqsave vs spin_lock_irq

Question

On an SMP machine we must use spin_lock_irqsave and not spin_lock_irq from interrupt context.

Why would we want to save the flags (which contain the IF)?

Is there another interrupt routine that could interrupt us?

Grubb

Answer 1

If interrupts are already disabled before your code starts locking, when you call spin_unlock_irq you will forcibly re-enable interrupts in a potentially unwanted manner. If instead you also save the current interrupt enable state in flags through spin_lock_irqsave, attempting to re-enable interrupts with the same flags after releasing the lock, the function will just restore the previous state (thus not necessarily enabling interrupts).

Example with spin_lock_irqsave:

spinlock_t mLock = SPIN_LOCK_UNLOCK;
unsigned long flags;

spin_lock_irqsave(&mLock, flags); // Save the state of interrupt enable in flags and then disable interrupts
// Critical section
spin_unlock_irqrestore(&mLock, flags); // Return to the previous state saved in flags

Example with spin_lock_irq( without irqsave ):

spinlock_t mLock = SPIN_LOCK_UNLOCK;
unsigned long flags;

spin_lock_irq(&mLock); // Does not know if interrupts are already disabled
// Critical section
spin_unlock_irq(&mLock); // Could result in an unwanted interrupt re-enable...

Pleasance

This answer is incorrect. spin_lock_irq will uncondtinoally disable interrupts, while the irqsave variants will save the interrupt state in cases that you cannot know what state you are currently in. – Kizzie

@NoahWatkins That is the point of the last portion. It say Could result in an error unlock in a comment. The spin_lock_irq is to show that the IRQ is enabled when it should not be. The answer is not incorrect; just not very clear. – Matchboard

@artlessnoise When we say irq's are disabled, are all the irq's in the system disabled? This is not very clear to me. Can you explain? – Eiffel

Tip for total kernel noobs like me: spin_lock_irqsave is a macro, not a function, which is why it can change the "passed" flags variable. This baffled me until I actually looked it up in the kernel code. – Marcille

@mk.. Just interrupts on that logical core. The idea is that the thread will not be interrupted while holding the lock. – Flacon

@Eiffel When we say irq's are disabled, it means that all irq's on the local CPU are disabled, whereas other CPUs have their irq mask intact. – Nantucket

Answer 2

spin_lock_irqsave is basically used to save the interrupt state before taking the spin lock, this is because spin lock disables the interrupt, when the lock is taken in interrupt context, and re-enables it when while unlocking. The interrupt state is saved so that it should reinstate the interrupts again.

Example:

1. Lets say interrupt x was disabled before spin lock was acquired
2. spin_lock_irq will disable the interrupt x and take the the lock
3. spin_unlock_irq will enable the interrupt x.

So in the 3rd step above after releasing the lock we will have interrupt x enabled which was earlier disabled before the lock was acquired.

So only when you are sure that interrupts are not disabled only then you should spin_lock_irq otherwise you should always use spin_lock_irqsave.

Momently

Answer 3

The need for spin_lock_irqsave besides spin_lock_irq is quite similar to the reason local_irq_save(flags) is needed besides local_irq_disable. Here is a good explanation of this requirement taken from Linux Kernel Development Second Edition by Robert Love.

The local_irq_disable() routine is dangerous if interrupts were already disabled prior to its invocation. The corresponding call to local_irq_enable() unconditionally enables interrupts, despite the fact that they were off to begin with. Instead, a mechanism is needed to restore interrupts to a previous state. This is a common concern because a given code path in the kernel can be reached both with and without interrupts enabled, depending on the call chain. For example, imagine the previous code snippet is part of a larger function. Imagine that this function is called by two other functions, one which disables interrupts and one which does not. Because it is becoming harder as the kernel grows in size and complexity to know all the code paths leading up to a function, it is much safer to save the state of the interrupt system before disabling it. Then, when you are ready to reenable interrupts, you simply restore them to their original state:

unsigned long flags;

local_irq_save(flags);    /* interrupts are now disabled */ /* ... */
local_irq_restore(flags); /* interrupts are restored to their previous
state */

Note that these methods are implemented at least in part as macros, so the flags parameter (which must be defined as an unsigned long) is seemingly passed by value. This parameter contains architecture-specific data containing the state of the interrupt systems. Because at least one supported architecture incorporates stack information into the value (ahem, SPARC), flags cannot be passed to another function (specifically, it must remain on the same stack frame). For this reason, the call to save and the call to restore interrupts must occur in the same function.

All the previous functions can be called from both interrupt and process context.

Forayer

I think this is the best explanation because otherwise I don't see a case where you can use nothing else but _irqsave – Nutrient

Answer 4

Reading Why kernel code/thread executing in interrupt context cannot sleep? which links to Robert Loves article, I read this :

some interrupt handlers (known in Linux as fast interrupt handlers) run with all interrupts on the local processor disabled. This is done to ensure that the interrupt handler runs without interruption, as quickly as possible. More so, all interrupt handlers run with their current interrupt line disabled on all processors. This ensures that two interrupt handlers for the same interrupt line do not run concurrently. It also prevents device driver writers from having to handle recursive interrupts, which complicate programming.

Positive

This doesn't answer question. – Leonardoleoncavallo

Answer 5

Below is part of code in linux kernel 4.15.18, which shows that spiin_lock_irq() will call __raw_spin_lock_irq(). However, it will not save any flags as you can see below part of the code but disable the interrupt.

  static inline void __raw_spin_lock_irq(raw_spinlock_t *lock)
    {
        local_irq_disable();
        preempt_disable();
        spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
        LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
    }

Below code shows spin_lock_irqsave() which saves the current stage of flag and then preempt disable.

static inline unsigned long __raw_spin_lock_irqsave(raw_spinlock_t *lock)
{
    unsigned long flags;

    local_irq_save(flags);
    preempt_disable();
    spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
    /*
     * On lockdep we dont want the hand-coded irq-enable of
     * do_raw_spin_lock_flags() code, because lockdep assumes
     * that interrupts are not re-enabled during lock-acquire:
     */
#ifdef CONFIG_LOCKDEP
    LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
#else
    do_raw_spin_lock_flags(lock, &flags);
#endif
    return flags;
}

Watereddown

Answer 6

On an SMP machine we must use  spin_lock_irqsave and not spin_lock_irq from interrupt context.

I don't think this statement is true at all. For sharing between hardirq and softirq, it's fine to just use spin_lock_irq in the softirq.

you use _irq or _irqsave whenever you want to prevent preemption by EITHER hard OR softirq while holding the lock. the _irqsave one also saves EFLAGS for later restoration.

spin_lock_irq will always be wrong to use in an irq handler since unlocking re-enables IRQ which should be done by the caller. And according to kernel/irq/handle.c:__handle_irq_event_percpu, This will generate a warning if your handler returns with IRQ enabled

spin_lock_irqsave is universal but I struggle to find a specific use case where you can use nothing else. Maybe it's useful when you want to put un/locking into a function that might be called from both hardirq AND non-hardirq because in this case, you cannot use spin_lock_irq or spin_lock(_bh)

Nutrient

Answer 7

This question starts from the false assertion:

On an SMP machine we must use spin_lock_irqsave and not spin_lock_irq from interrupt context.

Neither of these should be used from interrupt context, on SMP or on UP. That said, spin_lock_irqsave() may be used from interrupt context, as being more universal (it can be used in both interrupt and normal contexts), but you are supposed to use spin_lock() from interrupt context, and spin_lock_irq() or spin_lock_irqsave() from normal context. The use of spin_lock_irq() is almost always the wrong thing to do in interrupt context, being this SMP or UP. It may work because most interrupt handlers run with IRQs locally enabled, but you shouldn't try that.

UPDATE: as some people misread this answer, let me clarify that it only explains what is for and what is not for an interrupt context locking. There is no claim here that spin_lock() should only be used in interrupt context. It can be used in a process context too, for example if there is no need to lock in interrupt context.

Anking

This is COMPLETELY FALSE answer. Just about everything said here is wrong. Please down vote this or completely remove it. – Urbani

The spin-lock methods are not tied to interrupt OR process context. The answer says spin_lock() is to be used from interrupt context, and spin_lock_irq()/spin_lock_irqsave() are to be used in process context. That is FALSE. Its all about access paths. If you know for sure that your data can be accessed from process-context and interrupt-context, then use spin_lock_irq()/spin_lock_irqsave(). If you know for sure that your code is only accessed from process context, then use spin_lock(), you don't need to be bothered about interrupts here. – Urbani

Now regarding spin_lock_irq() vs spin_lock_irqsave() .. it has already been detailed above in the accepted answers. And yes spin_lock_irqsave() is much safer. – Urbani

These primitives are different in only their handling of flags, and especially the IF flag. The reason is to make sure the interrupt handler never tries to acquire the same lock as already held in process context, which on UP would lead to a dead-lock. So in a process context, the locking function should clear the IF flag, which is not needed in an interrupt context. More importantly, in an interrupt context you should NOT set the IF flag at unlock (as it was probably cleared initially), so the _irq version does not suit. But universal _irqsave is good only when don't know what context you in. – Anking

@Urbani you misread me if you say "The answer says spin_lock() is to be used from interrupt context". I was explaining the case where locking in interrupt context is needed at all, as this was what the original question is about. If you don't need the lock in interrupt context, then of course you can use spin_lock() in a process context, but I was not covering that case, as the question was explicitly about interrupt context. My answer was only about what is and what is not to use in interrupt context. – Anking