Yarn - How does yarn.scheduler.capacity.root.queue-name.maximum-capacity works?

Question

I have 4 queues under the root queue with the following configuration.

|-------------|-----------------|---------------------|-------------------|
| Queue Name  | Capacity (in %) | Max Capacity (in %) | User Limit Factor |
|-------------|-----------------|---------------------|-------------------|
| default     | 10              | 30                  | 10                |
|-------------|-----------------|---------------------|-------------------|
| thriftsvr   | 5               | 30                  | 10                |
|-------------|-----------------|---------------------|-------------------|
| stream      | 70              | 70                  | 10                |
|-------------|-----------------|---------------------|-------------------|
| batch       | 15              | 30                  | 10                |
|-------------|-----------------|---------------------|-------------------|

I have set up capacity by yarn.scheduler.capacity.root.<queue-name>.capacity and max capacity by yarn.scheduler.capacity.root.<queue-name>.maximum-capacity property.

My understanding is, above 2 properties set ABSOLUTE capacity and ABSOLUTE maximum capacity respectively. That means queue stream's 100% is equal to the 70% of cluster's total capacity and it can fill up to 100% of queue's capacity that is also the 70% of cluster's total capacity.

Now, the problem is when queue 'stream' is filled with 66.4% (i.e. when Used Capacity: 66.4% & Absolute Used Capacity: 46.5%) then new jobs are getting in the pending state which is submitted in queue 'stream' by saying "waiting for AM container to be allocated, launched and register with RM".

When I checked queue configuration on yarn UI it shows Configured Max Capacity: 70.0% & Absolute Configured Max Capacity: 70.0% but according to the configuration, queue 'stream' can be filled till Used Capacity: 100% & Absolute Used Capacity: 70%

Any idea, why new jobs are unable to utilize the queue stream's capacity till 100%?

Amalbergas

Answer 1

I suspect the confusing thing here is that .capacity and .maximum-capacity properties can both be defined as either

• relative to parent queue root's percentage (as float, e.g. 12.5)
• absolute resource value (using resource value syntax e.g. [memory=204800,vcores=122])

If you have any further questions, please do ask.

For full reference, just read the doc: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Queue_Properties

Inflection

Answer 2

I'll take an example from this book to explain how yarn.scheduler.capacity.root.queue-name.maximum-capacity works.

A sample Capacity Scheduler configuration file, called capacity-scheduler.xml. It defines two queues under the root queue, prod and dev, which have 40% and 60% of the capacity, respectively. Notice that a particular queue is configured by setting configuration properties of the form yarn.scheduler.capacity.., where is the hierarchical (dotted) path of the queue, such as root.prod.

    <?xml version="1.0"?>
<configuration>
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>prod,dev</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.queues</name>
    <value>eng,science</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.prod.capacity</name>
    <value>40</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.capacity</name>
    <value>60</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.maximum-capacity</name>
    <value>75</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.eng.capacity</name>
    <value>50</value>
  </property>
  <property>
    <name>yarn.scheduler.capacity.root.dev.science.capacity</name>
    <value>50</value>
  </property>
</configuration>

As you can see, the dev queue is further divided into eng and science queues of equal capacity. So that the dev queue does not use up all the cluster resources when the prod queue is idle, it has its maximum capacity set to 75%. In other words, the prod queue always has 25% of the cluster available for immediate use. Since no maximum capacities have been set for other queues, it’s possible for jobs in the eng or science queues to use all of the dev queue’s capacity (up to 75% of the cluster), or indeed for the prod queue to use the entire cluster.

See YARN doc to understand more on Queue configs.

Amboina

Answer 3

https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances-guidelines.html

As per the above link AWS has removed those labels starting from EMR version 6 and above.

So this works for EMR greater than 6, but for lower versions removing core label might be helpful.

Andra