What's the difference between Priority Class and QoS in Kubernetes?

Question

In Kubernetes we can set the priority of a pod to Guaranteed, Burstable or Best-Effort base on requests and limits. Another method to assign priorities in Kubernetes is to define a priorityClass object and assign a priorityClassName to a pod. How are these methods different and when we have to choose one method over another? According to https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#interactions-of-pod-priority-and-qos:

The scheduler’s preemption logic does not consider QoS when choosing preemption targets. Preemption considers Pod priority and attempts to choose a set of targets with the lowest priority.

So if the Kubernetes has to choose between a pod with Guaranteed QoS which has a lower "priorityClass" Value than a Burstable pod, does it put the Guaranteed pod in Preempting state?

Bunde

Answer 1

Quality of Service determines scheduling and eviction priority of pods. When a pod is not given any resources requests/ limits it is considered as low-priority pod (best effort). If node is out of resources it is the first pod to be evicted.

Medium priority (burstable) when a pod has any resource/limits specified (does not meet guaranteed class requirements).

Highest priority (guaranteed) when a pod has requests and limits set to the same value (both CPU and memory).

PriorityClass is used to determine pod priority when it comes to eviction. Sometimes you may want one pod to be evicted before another pod. Priority is described by integer in it's value field and the higher the value, the higher the priority.

When Pod priority is enabled, the scheduler orders pending Pods by their priority and a pending Pod is placed ahead of other pending Pods with lower priority in the scheduling queue. If there are no nodes with resources to satisfy high-priority node it will evict pods with lower priority. The highest priority is system-node-critical.

• QoS is used to control and manage resources on the node among the pods. QoS eviction happens when there are no available resources on the node.
• The scheduler considers the PriorityClass of the Pod before the QoS. It does not attempt to evict Pods unless higher-priority Pods need to be scheduled and the node does not have enough room for them.
• PriorityClass- when pods are preempted, PriorityClass respects graceful termination period but does not guarantee pod disruption budget to be honored.

Gallery

So correct me if I'm wrong in the following scenarios: (a) We have 3 nodes each running a Guaranteed pod, and each node have 200MiB free memory, when I apply a new deployment with HIGHER priority pods which requests 400MiB memory (but have limit of 500MiB i.e. Burstable ) scheduler preempt previous guaranteed pods to schedule these new Burstable pods. (b) a node running a HIGHER priority Burstable pod and a LOWER priority guranteed pod. Then the node has no availabe resource and has to evict one, it will evict the HIGHER priority Burstable pod. – Bunde

(c) after scenario b , the evicted Burstable pod has to be scheduled again (it's a deployment) but if there is no room for this on any node, the scheduler evicts the guaranteed pod to schedule the Burstable pod again. – Bunde

> "Quality of Service determines scheduling and eviction priority of pods." I don't understand how do QoS classes have anything to do with scheduling? They determine eviction in case of node going out of resources. Scheduling uses Priorities to set the order. So where is the link between QoS and scheduling, mentioned in the answer above, but also in the official page: kubernetes.io/docs/tasks/configure-pod-container/… – Theomorphic

Answer 2

The scheduler would simply ignore the QoS as mentioned. Preemption only takes into consideration the priorityClass, QoS has not effect here.

Chromolithograph