Whitelist "kube-system" namespace using NetworkPolicy

Question

I have a multi-tenant cluster, where multi-tenancy is achieved via namespaces. Every tenant has their own namespace. Pods from a tenant cannot talk to pods of other tenants. However, some pods in every tenant have to expose a service to the internet, using an Ingress.

This I how far I got (I am using Calico):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: tenant1-isolate-namespace
  namespace: tenant1
spec:
  policyTypes:
  - Ingress
  podSelector: {} # Select all pods in this namespace
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: tenant1 # white list current namespace

Deployed for each namespace (tenant1, tenant2, ... ), this limits communication between pods within their namespace. However, this prevents pods from the kube-system namespace to talk to pods in this namespace.

However, the kube-system namespace does not have any labels by default so I can not specifically white list this namespace.

I found a (dirty) workaround for this issue by manually giving it a label:

kubectl label namespace/kube-system permission=talk-to-all

And adding the whitelist rule to the networkpolicy:

...
  - from:
    - namespaceSelector:
        matchLabels:
          permission: talk-to-all # allow namespaces that have the "talk-to-all privilege"

Is there a better solution, without manually giving kube-system a label?

Edit: I tried to additionally add an "OR" rule to specifically allow communication from pods that have the label "app=nginx-ingress", but without luck:

  - from
    ...
    - podSelector:
        matchLabels:
          app: nginx-ingress # Allow pods that have the app=nginx-ingress label

Hendecagon

did you manage to expose a dedicated to pod in each ns to internet but still preventing any inter cluster communication between namespaces ? I tried your dirty workaround but my dedicated pods are not accessible from internet (AWS ELB) ... In my network policy for ingress I have - podSelector: {} - namespaceSelector: matchLabels:permission: kube-system The fact i can access my protected namespace from pods in kube-system namespace does not allow me to access ELB exposed pod in a protected namespace - seems like external traffic from ELB does not go via kube-system namespace – Gothurd

Answer 1

apiVersion: networking.k8s.io/v1

The namespaceSelector is designed to match namespaces by labels only. There is no way to select namespace by name.

The podSelector can only select pods in the same namespace with NetworkPolicy object. For objects located in different namespaces, only selection of the whole namespace is possible.

Here is an example of Kubernetes Network Policy implementation:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

Follow this link to read a good explanation of the whole concept of Network policy, or this link to watch the lecture.

apiVersion: projectcalico.org/v3

Calico API gives you more options for writing NetworkPolicy rules, so, at some point, you can achieve your goal with less efforts and mind-breaking.

For example, using Calico implementation of Network Policy you can:

• set action for the rule (Allow, Deny, Log, Pass),
• use negative matching (protocol, notProtocol, selector, notSelector),
• apply more complex label selectors(has(k), k not in { ‘v1’, ‘v2’ }),
• combine selectors with operator &&,
• use port range (ports: [8080, "1234:5678", "named-port"]),
• match pods in other namespaces.

But still, you can match namespaces only by labels.

Consider reading Calico documentation for the details.

Here is an example of Calico Network Policy implementation:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-tcp-6379
  namespace: production
spec:
  selector: role == 'database'
  types:
  - Ingress
  - Egress
  ingress:
  - action: Allow
    protocol: TCP
    source:
      selector: role == 'frontend'
    destination:
      ports:
      - 6379
  egress:
  - action: Allow

Thorny

In calico you can now also match namespaces by name: "Select a specific namespace by name using the projectcalico.org/name label." Check the documentation of namespaceSelector – Favouritism

So, it is label, that equals namespace name, but selector still can use labels only. Right? – Thorny

selector selects resources by their label (i.e. pods, services). namespaceSelector selects resources by the label of their namespace. They can also be used together. And yes afaik projectcalico.org/name is a special identifier that points to the namespace name – Favouritism

Answer 2

Indeed, tenant1 pods will need access to kube-dns in the kube-system namespace specifically.

One approach without requiring kube-system namespace to be labelled is the following policy. Although kube-dns could be in any namespace with this approach so it may not be suitable for you.

---                                                                                                                                                                                                           
# Default deny all ingress & egress policy, except allow kube-dns                                                                                                                                             
# All traffic except this must be explicity allowed.                                                                                                                                                          
kind: NetworkPolicy                                                                                                                                                                                           
apiVersion: networking.k8s.io/v1                                                                                                                                                                              
metadata:                                                                                                                                                                                                     
  name: default-deny-all-except-kube-dns                                                                                                                                                                      
  namespace: tenant1                                                                                                                                                                                        
spec:                                                                                                                                                                                                         
  podSelector: {}                                                                                                                                                                                             
  egress:                                                                                                                                                                                                     
  - to:                                                                                                                                                                                                       
    - podSelector:                                                                                                                                                                                            
        matchLabels:                                                                                                                                                                                          
          k8s-app: kube-dns                                                                                                                                                                                   
 - ports:             
   - protocol: TCP                                                                                                                                                                                         
     port: 53                                                                                                                                                                                                                                                                                     
   - protocol: UDP                                                                                                                                                                                           
     port: 53                                                                                                                                                                                                
 policyTypes:                                                                                                                                                                                                
 - Ingress                                                                                                                                                                                                   
 - Egress   

Then, you would need also need an 'allow all within namespace policy' as follows:

---                                                                                                                                                                                                           
# Allow intra namespace traffic for development purposes only.                                                                                                                                                
kind: NetworkPolicy                                                                                                                                                                                           
apiVersion: networking.k8s.io/v1                                                                                                                                                                              
metadata:                                                                                                                                                                                                     
  name: allow-intra-namespace                                                                                                                                                                                 
  namespace: tenant1                                                                                                                                                                                        
spec:                                                                                                                                                                                                         
  podSelector:                                                                                                                                                                                                
    matchLabels:                                                                                                                                                                                              
  ingress:                                                                                                                                                                                                    
  - from:                                                                                                                                                                                                     
    - podSelector: {}                                                                                                                                                                                         
  egress:                                                                                                                                                                                                     
  - to:                                                                                                                                                                                                       
    - podSelector: {}                                                                                                                                                                                         
  policyTypes:                                                                                                                                                                                                
  - Ingress                                                                                                                                                                                                   
  - Egress

Lastly, you will want to add specific policies such as an ingress rule. It would be better to replace the allow-intra-namespace policy with specific rules to suit individual pods, which your tenant1 could do.

These have been adapted from this website: https://github.com/ahmetb/kubernetes-network-policy-recipes

Laughry

I've been fighting with this for a while now - your kube-dns rule did the trick. Much appreciated! – Saveloy

Answer 3

If i understood correctly you are using calico. Just use their example on how to implement default-deny without breaking kube-dns communication. Found here

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: deny-app-policy
spec:
  namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name not in {"kube-system", "calico-system"}
  types:
  - Ingress
  - Egress
  egress:
  # allow all namespaces to communicate to DNS pods
  - action: Allow
    protocol: UDP
    destination:
      selector: 'k8s-app == "kube-dns"'
      ports:
      - 53

Favouritism

Answer 4

I'm on k3os with default flannel CNI. It has default label on kube-system namespace:

$ kubectl describe ns kube-system
Name:         kube-system
Labels:       kubernetes.io/metadata.name=kube-system
Annotations:  <none>
Status:       Active

This works for me:

- namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: kube-system

Here is my full yaml which allows all external traffic and kube-dns in kube-system namespace for egress:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
      - port: 53
        protocol: UDP
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16

Deliver