SCHED_FIFO thread is preempted by SCHED_OTHER thread in Linux

Question

I have written a test program to test out SCHED_FIFO. I have learnt that SCHED_FIFO cannot be preempted by SCHED_OTHER threads. But I couldn't explain the results obtained when same program is run multiple times.

/* Includes */
#include <unistd.h>     /* Symbolic Constants */
#include <sys/types.h>  /* Primitive System Data Types */ 
#include <errno.h>      /* Errors */
#include <stdio.h>      /* Input/Output */
#include <stdlib.h>     /* General Utilities */
#include <pthread.h>    /* POSIX Threads */
#include <string.h>     /* String handling */
#include <sched.h>

/* prototype for thread routine */
void print_message_function ( void *ptr );
void print_message_function1 ( void *ptr );

/* struct to hold data to be passed to a thread
 * this shows how multiple data items can be passed to a thread
 */
typedef struct str_thdata
{
    int thread_no;
    int thread_value;
    char message[100];
} thdata;

int main()
{
    pthread_t thread1, thread2;  /* thread variables */
    thdata data1, data2;         /* structs to be passed to threads */

    /* initialize data to pass to thread 1 */
    data1.thread_no = 1;
    data1.thread_value = 0;
    strcpy(data1.message, "Hello!");

    /* initialize data to pass to thread 2 */
    data2.thread_no = 2;
    data2.thread_value = 10000;
    strcpy(data2.message, "Hi!");

    /* create threads 1 and 2 */    
    pthread_create (&thread1, NULL, (void *) &print_message_function, (void *) &data1);
    pthread_create (&thread2, NULL, (void *) &print_message_function1, (void *) &data2);

    /* Main block now waits for both threads to terminate, before it exits
     * If main block exits, both threads exit, even if the threads 
     * have not finished their work 
     */ 
    pthread_join(thread1, NULL);
    pthread_join(thread2, NULL);
          
    /* exit */  
    exit(0);
} /* main() */

/** 
* print_message_function is used as the start routine for the threads used
* it accepts a void pointer 
**/
void print_message_function ( void *ptr )
{
    thdata *data;            
    data = (thdata *) ptr;  /* type cast to a pointer to thdata */

    struct sched_param param;
    //int priority=10;
    /* sched_priority will be the priority of the thread */
    //param.sched_priority = priority;
    /* only supported policy, others will result in ENOTSUP */

    int policy = SCHED_OTHER;
    /* scheduling parameters of target thread */
    pthread_setschedparam(pthread_self(), policy, &param);
    printf("Thread %d says sched policy  %d \n", data->thread_no, SCHED_OTHER);
    pthread_getschedparam(pthread_self(),&policy,&param);

    printf("Thread %d says %s  %d \n", data->thread_no, data->message,policy);

    int i=0;
    /* do the work */
    printf("Thread %d says %s %d \n", data->thread_no, data->message,(int)pthread_self());
    for(i=0;i<100;i++) 
        printf("Thread %d says %d \n", data->thread_no,data->thread_value++);
    pthread_exit(0); /* exit */
} /* print_message_function ( void *ptr ) */

void print_message_function1 ( void *ptr )
{
    thdata *data;            
    data = (thdata *) ptr;  /* type cast to a pointer to thdata */

    struct sched_param param;
    int priority=10;
    /* sched_priority will be the priority of the thread */
    param.sched_priority = priority;
    /* only supported policy, others will result in ENOTSUP */

    int policy = SCHED_FIFO;
    /* scheduling parameters of target thread */
    pthread_setschedparam(pthread_self(), policy, &param);
    printf("Thread %d says sched policy %d \n", data->thread_no, SCHED_FIFO);

    pthread_getschedparam(pthread_self(),&policy,&param);

    printf("Thread %d says %s  %d \n", data->thread_no, data->message,policy);

    int i=0;
    /* do the work */
    printf("Thread %d says %s  %d \n", data->thread_no, data->message,(int)pthread_self());
    for(i=0;i<100;i++)
        printf("Thread %d says %d \n", data->thread_no,data->thread_value++);
    pthread_exit(0); /* exit */
} /* print_message_function ( void *ptr ) */

I have got unexpected results in multiple runs where I have seen SCHED_FIFO is preempted by SCHED_OTHER thread, i.e. as per program, thread 2 is in FIFO mode, while thread 1 is SCHED_OTHER mode. I have seen multiple times where thread2 is preempted by thread1.

Can someone help me out in finding the issue?

Alcoholize

Output of sysctl -a | grep _rt ? – Spermaceti

@martinjames kernel.sched_rt_period_us = 1000000 kernel.sched_rt_runtime_us = 950000 – Alcoholize

I am using ubuntu 11.10 with g++ 4.6.1 – Alcoholize

Answer 1

You probably have these sysctl settings in effect, which are default values:

kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000

This means that real time threads are allowed to hog only 95% of every 1 second period.

Also see: Can't provoke Priority Inversion in C++

Gymno

yes,these are my default values.But,i tried to put -1 to kernel.sched_rt_runtime_us to disable realtime.This also didnt work out – Alcoholize

How many cores do you have? How are you ensuring that the threads cannot simply execute on different cores? Also note that you have to set the priority. POSIX defines a pair of functions sched_get_priority_max and sched_get_priority_min which take a policy as an argument and give you the range of valid priorities for that policy. – Gymno

I disabled using all cores except one using command sudo sh -c "echo 0 > /sys/devices/system/cpu/cpu3/online".. Things are fine.THANKS :-) – Alcoholize

I was able to get a similar scenario working not by using -1; but rather by setting them both to the same value. That way it works out to be 100% as Kaz suggested: sudo sysctl -w kernel.sched_rt_runtime_us=1000000 . Making this change I was successful a locking out everything else (including the kernel) when I executed my program. – Rumpus

Answer 2

Also whenever your thread blocks on IO, perhaps from the printf statements, another thread may be scheduled.

Acherman