PLINQ AsParallel() with lower priority?

Question

is it possible to run some of my PLINQ AsParallel() - Queries with a lower priority than others? (Or some with a higher priority than others) Is this possible with PLinq or will I have to avoid PLINQ and do all the stuff on my own?

EDIT/UPDATE:

Would it be possible to call

Thread.Sleep(0)

inside the parallel executed method when I want to archive a lower priority? Or is that a very bad practice/hack?

Pinetum

Re the update: avoid Sleep(0), it has problems. You could give up time with Sleep(1) but it's really crude, not advised. – Horsemint

Maybe state why you want different priorities. That doesn't look very logical for PLINQ. – Horsemint

Answer 1

Unfortunately, this is not directly possible in PLINQ.

You can do it in most of the rest of the Task Parallel Library via creation of a custom TaskScheduler. This would allow you to have custom "priorities" when using Parallel.For or ForEach.

However, the ability to customize the TaskScheduler was not provided with PLINQ, since PLINQ requires very strict guarantees from the TaskScheduler, and the fear was that exposing this would be very problematic.

---

EDIT/UPDATE:

Would it be possible to call

Thread.Sleep(0)

This would "lower" the priority, but unfortunately, has it's own issues, especially when combined with PLINQ. This will potentially cause thread starvation in the ThreadPool, since you'll be "sleeping" on ThreadPool threads.

In addition, there's a fundamental problem with this - PLINQ is designed and intended to handle queries, and is not designed for processing. Introducing logical code to control the flow structure is really against the theory behind PLINQ, and will likely cause unintended performance repercussions you aren't expecting, especially if you're using the default partitioning.

Dorton

how would you set custom priorities in case of Parallel.For? – Freer

@Andrey: You can do this by providing a custom TaskScheduler to the ParallelOptions argument (msdn.microsoft.com/en-us/library/…) This would allow you to handle this any way you chose, whether it's "thread" priority, or even just using a priority queue for scheduling on the default thread pool. – Dorton

do you really think that it is easier then to create Threads? I am sure that it would need much more code. – Freer

@Andrey: It depends on the scenario - it's very easy to "steal" from the ParallelExtensionsExtra library, grab one of their schedulers, and just change the thread priority, if that's your goal. The advantage here, though, is that getting the scheduling correct yourself for manual threading is non-trivial. Using ParallelOptions, once you have the scheduler, allows you to write the code once and reuse it everywhere in a very clean manner. – Dorton

@Reed Copsey : The funny thing is, if you look inside ParallelEnumerable class using ILSpy, there is a method called WithTaskScheduler() that allow to use a custom TaskScheduler in PLINQ. However, this is marked internal (but seems to be used no where). Like it was cancelled just before final release (probably because it was not safe enough, as you explained). – Collude

Answer 2

AsParallel is very high level API. You should really use Threads if you want fine grained control over what is happening using Priority

Freer

Answer 3

The PLINQ library does not allow configuring the TaskScheduler, for good reasons:

The reason we didn't make this public initially and then haven't each time the question has come up is due to fear of deadlocks. Some PLINQ operators use barriers, such that all tasks/partitions involved in processing the query need to join before any of them can make further progress. If the underlying scheduler can't guarantee that all of the tasks queued as part of PLINQ's processing will be able to run at the same time, e.g. if it's a scheduler created with a fixed number of threads, then we risk deadlock.

If you feel adventurous you could consider doing this configuration with reflection. Here is a PLINQ operator that does exactly this:

/// <summary>
/// Sets the scheduler that should execute the query.
/// </summary>
/// <remarks>Has been tested on .NET 6 only.</remarks>
public static ParallelQuery<T> WithScheduler<T>(this ParallelQuery<T> source,
    TaskScheduler scheduler)
{
    if (source == null) throw new ArgumentNullException(nameof(source));
    if (scheduler == null) throw new ArgumentNullException(nameof(scheduler));
    const string name1 = "_specifiedSettings";
    FieldInfo fi1 = typeof(ParallelQuery).GetField(name1,
        BindingFlags.NonPublic | BindingFlags.Instance);
    if (fi1 == null) throw new InvalidOperationException($"Field {name1} not found.");
    object querySettings = fi1.GetValue(source);
    if (querySettings == null) throw new InvalidOperationException($"{name1} is null.");
    const string name2 = "TaskScheduler";
    PropertyInfo pi2 = querySettings.GetType().GetProperty(name2,
        BindingFlags.NonPublic | BindingFlags.Instance);
    if (pi2 == null) throw new InvalidOperationException($"Property {name2} not found.");
    pi2.SetValue(querySettings, scheduler);
    fi1.SetValue(source, querySettings); // The QuerySettings is a struct
    return source;
}

Usage example:

TaskScheduler lowPriorityScheduler = new ConcurrentExclusiveSchedulerPair(
    TaskScheduler.Default, Environment.ProcessorCount).ConcurrentScheduler;

var query = source
    .AsParallel()
    .WithScheduler(lowPriorityScheduler)
    //...

You could use the same lowPriorityScheduler to schedule other operations that should run with low priority, in parallel with this and other queries. All these operations combined will use at most Environment.ProcessorCount threads from the ThreadPool. All other threads that are available beyond this limit, will be used exclusively by non-low-priority operations.

You could also take a look at this somewhat related question.

Leveller