Futexes and blocking in the kernel

Question

I was reading some documentation and trying out a few samples of code that issue the futex system call in Linux. I read that if a mutex is acquired by thread_a using FUTEX_LOCK_PI, and say if another thread thread_b tries to acquire the same, the latter(thread_b) is blocked in the kernel.

What exactly is meant by "blocked in the kernel"? Is it that the execution of thread_b in userspace does not resume? In the following example, the second thread seems to issue the printf after the syscall as well. Would that not suggest that it is not blocked in the kernel? What am I missing?

/*
 * This example demonstrates that when futex_lock_pi is called twice, the
 * second call is blocked inside the kernel.
 */

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

int mutex = 0;

#define  __NR_futex          240
#define FUTEX_WAIT 0
#define FUTEX_WAKE 1
#define FUTEX_FD 2
#define FUTEX_REQUEUE 3
#define FUTEX_CMP_REQUEUE 4
#define FUTEX_WAKE_OP 5
#define FUTEX_LOCK_PI 6
#define FUTEX_UNLOCK_PI 7
#define FUTEX_TRYLOCK_PI 8
#define FUTEX_WAIT_REQUEUE_PI 11
#define FUTEX_CMP_REQUEUE_PI  12


void *thread(void *arg) {
  int ret = 0;
  pid_t tid = gettid();
  printf("Entering thread[%d]\n", tid);

  ret = syscall(__NR_futex, &mutex, FUTEX_LOCK_PI, 1, NULL, NULL, 0);
  printf("Thread %d returned from kernel\n", tid);
  printf("Value of mutex=%d\n", mutex);
  printf("Return value from syscall=%d\n", ret);
}

int main(int argc, const char *argv[]) {
  pthread_t t1, t2;

  if (pthread_create(&t1, NULL, thread, NULL)) {
    printf("Could not create thread 1\n");
    return -1;
  }

  if (pthread_create(&t2, NULL, thread, NULL)) {
    printf("Could not create thread 2\n");
    return -1;
  }

  // Loop infinitely
  while ( 1 ) { }

  return 0;
}

The output can be found below :-

Entering thread[952]
Thread 952 returned from kernel
Entering thread[951]
Value of mutex=-2147482696
Return value from syscall=0
Thread 951 returned from kernel
Value of mutex=-1073740873
Return value from syscall=0

Dahle

You use futexes in priority-inversion mode, which actually works with mutexes (that's why LOCK substring in the futex operation name). In that mode futex values transition scheme is fully defined, and doesn't depends from val and val2 arguments of futex system call. Value of futex in unlocked state must be 0. Check that. – Reardon

Also, in PI mode kernel checks that process(thread) corresponded to tid value, stored in futex, exists and actually access it for perform priority-inversion. As I understand, if owner of the futex is exited, mutex treated as unlocked. Try to add sleep() at the end of your thread function and check, whether the second thread returns from futex() call while the first one(futex owner) sleeps. (This shouldn't be happen). – Reardon

What do you initialize the mutex to? Can you give us complete, compilable code? – Russom

@DavidSchwartz I have updated the code to be complete and compilable, thanks. – Dahle

Your code and output don't match though. Could you show the output from this code, and ideally add the syscall return value as well? – Iggy

You got rid of the sleep as well. What happens if the thread just spins & sleeps for ever after getting the futex? I suspect it's being released when the thread exits (this is the "robust" futex behaviour mentioned in the PI futex doc linked by @Tsyvarev) – Iggy

Add sleep(5); printf("Thread %d is exited\n", tid); to the end of your thread function. And see order of messages in the output. – Reardon

Answer 1

blocked in the kernel means that the thread is put to sleep state until an events wakes it up, in your case the event is the mutex to get available.

I changed a little bit your code to illustrate:

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/syscall.h>

static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

void *thread(void *unused) {
  printf("Entering thread %d\n", syscall(SYS_gettid));

  pthread_mutex_lock(&mutex);

  sleep(2);

  pthread_mutex_unlock(&mutex);

  return NULL;
}

int main(int argc, const char *argv[]) {
  pthread_t t1, t2;

  printf("using mutex @%p\n", &mutex);

  if (pthread_create(&t1, NULL, thread, &t1)) {
    printf("Could not create thread 1\n");
    return -1;
  }

  if (pthread_create(&t2, NULL, thread, &t2)) {
    printf("Could not create thread 2\n");
    return -1;
  }

  pthread_join(t1, NULL);
  pthread_join(t2, NULL);

  return 0;
}

The code is basically the same but I use explicitly a mutex of the pthread lib.

If I run this code with strace, I get:

using mutex @0x6010a0
Process 19688 attached
Entering thread 19688
Process 19689 attached
Entering thread 19689
[pid 19689] futex(0x6010a0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 19688] futex(0x6010a0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 19689] <... futex resumed> )       = 0
[pid 19688] +++ exited with 0 +++
[pid 19689] futex(0x6010a0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 19689] +++ exited with 0 +++
+++ exited with 0 +++

You may notice that we do not see the first thread taking the mutex, BUT we can see the next one caling futex for waiting (FUTEX_WAIT_PRIVATE). This is due to the fact that futex doesn't get called when the mutex is being taken.

But then you can see that the first thread (id 19688 here) then eventually calls futex(FUTEX_WAKE_PRIVATE) which tells the kernel to wake the other thread now the utex is free.

You may have noticed that the first call [pid 19689] futex(0x6010a0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...> is unfinished, which means that the process was hanged waiting for the kernel to finish the job and give back the hand. Then [pid 19689] <... futex resumed> ) = 0 sketches that the call is eventually finished (obviously because the mutex was released)

Boorer