How to install nginx-ingress with hostNetwork on bare-metal?

Question

I created a single-node kubeadm cluster on bare-metal and after some research I would go for a host network approach (https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network), since NodePort is not an option due to network restrictions.

I tried installing nginx-ingress with helm chart through the command:

   helm install stable/nginx-ingress \
     --set controller.hostNetwork=true

The problem is that it is creating a LoadBalancer service which is Pending forever and my ingress objects are not being routed:

NAME                                                                 READY   STATUS    RESTARTS   AGE
pod/whopping-kitten-nginx-ingress-controller-5db858b48c-dp2j8        1/1     Running   0          5m34s
pod/whopping-kitten-nginx-ingress-default-backend-5c574f4449-dr4xm   1/1     Running   0          5m34s

NAME                                                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/kubernetes                                      ClusterIP      10.96.0.1       <none>        443/TCP                      6m43s
service/whopping-kitten-nginx-ingress-controller        LoadBalancer   10.97.143.40    <pending>     80:30068/TCP,443:30663/TCP   5m34s
service/whopping-kitten-nginx-ingress-default-backend   ClusterIP      10.106.217.96   <none>        80/TCP                       5m34s

NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/whopping-kitten-nginx-ingress-controller        1/1     1            1           5m34s
deployment.apps/whopping-kitten-nginx-ingress-default-backend   1/1     1            1           5m34s

NAME                                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/whopping-kitten-nginx-ingress-controller-5db858b48c        1         1         1       5m34s
replicaset.apps/whopping-kitten-nginx-ingress-default-backend-5c574f4449   1         1         1       5m34s

Is there any other configuration that needs to be done to succeed in this approach?

UPDATE: here are the logs for the ingress-controller pod

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.24.1
  Build:      git-ce418168f
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

I0707 19:02:50.552631       6 flags.go:185] Watching for Ingress class: nginx
W0707 19:02:50.552882       6 flags.go:214] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
nginx version: nginx/1.15.10
W0707 19:02:50.556215       6 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0707 19:02:50.556368       6 main.go:205] Creating API client for https://10.96.0.1:443
I0707 19:02:50.562296       6 main.go:249] Running in Kubernetes cluster version v1.15 (v1.15.0) - git (clean) commit e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529 - platform linux/amd64
I0707 19:02:51.357524       6 main.go:102] Validated default/precise-bunny-nginx-ingress-default-backend as the default backend.
I0707 19:02:51.832384       6 main.go:124] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
W0707 19:02:53.516654       6 store.go:613] Unexpected error reading configuration configmap: configmaps "precise-bunny-nginx-ingress-controller" not found
I0707 19:02:53.527297       6 nginx.go:265] Starting NGINX Ingress controller
I0707 19:02:54.630002       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"801", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/staging-ingress
I0707 19:02:54.727989       6 nginx.go:311] Starting NGINX process
I0707 19:02:54.728249       6 leaderelection.go:217] attempting to acquire leader lease  default/ingress-controller-leader-nginx...
W0707 19:02:54.729235       6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:54.729334       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:54.729442       6 controller.go:797] Service "default/test" does not have any active Endpoint.
I0707 19:02:54.729535       6 controller.go:170] Configuration changes detected, backend reload required.
I0707 19:02:54.891620       6 controller.go:188] Backend successfully reloaded.
I0707 19:02:54.891654       6 controller.go:202] Initial sync, sleeping for 1 second.
I0707 19:02:54.948639       6 leaderelection.go:227] successfully acquired lease default/ingress-controller-leader-nginx
I0707 19:02:54.949148       6 status.go:86] new leader elected: precise-bunny-nginx-ingress-controller-679b9557ff-n57mc
[07/Jul/2019:19:02:55 +0000]TCP200000.000
W0707 19:02:58.062645       6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:02:58.062676       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:02:58.062686       6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:02.406151       6 controller.go:373] Service "default/precise-bunny-nginx-ingress-default-backend" does not have any active Endpoint
W0707 19:03:02.406188       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:02.406357       6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:02 +0000]TCP200000.000
W0707 19:03:05.739438       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:05.739467       6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:05 +0000]TCP200000.001
W0707 19:03:09.072793       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:09.072820       6 controller.go:797] Service "default/test" does not have any active Endpoint.
W0707 19:03:12.406121       6 controller.go:797] Service "default/face" does not have any active Endpoint.
W0707 19:03:12.406143       6 controller.go:797] Service "default/test" does not have any active Endpoint.
[07/Jul/2019:19:03:15 +0000]TCP200000.000
I0707 19:03:54.959607       6 status.go:295] updating Ingress default/staging-ingress status from [] to [{ }]
I0707 19:03:54.961925       6 event.go:209] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"staging-ingress", UID:"9852d27b-d8ad-4410-9fa0-57b92fdd6f90", APIVersion:"extensions/v1beta1", ResourceVersion:"1033", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/staging-ingress

Monoicous

Answer 1

@ijaz-ahmad-khan @vkr gave good ideas for solving the problem but the complete steps for setup are:

1) Install nginx-ingress with:

helm install stable/nginx-ingress --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet

2) In your deployments put:

spec:
  template:
    spec:
      hostNetwork: true

3) In all your Ingress objects put:

metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"  

Monoicous

How did you manage to install the helm chart? Specifying an empty value for controller.service.type causes Error: release nginx-ingress failed: Service "nginx-ingress-controller" is invalid: spec.type: Unsupported value: "''": supported values: "ClusterIP", "ExternalName", "LoadBalancer", "NodePort". – Dictionary

I guesst step 2 is not required as with controller.hostNetwork you are already setting it github.com/helm/charts/blob/master/stable/nginx-ingress/… – Gilreath

@Gilreath at the time I run into that problem I am sure I tried without it. Maybe now with latests versions of kubernetes and nginx-ingress it is not necessary anymore. Can you try that? I can also confirm and update the answer. – Monoicous

Probably, just want to mention it ;-) – Gilreath

@KarlRichter Probably depends on the exact version of Kubernetes you are running. The bare metal installation docs recommend just deleting the Ingress Controller service since it is no longer used with the DaemonSet/hostNetwork set up. So you could use any legal value to get the installation to work and delete the service afterwards. – Brochette

Answer 2

In some tutorials you might be advised to simply do a kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml

But if you're working with a baremetal implementation on a private LAN without a load balancer, you'll need to modify this step a bit.

instead of applying that deploy.yaml directly, do a wget (e.g. wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.4/deploy/static/provider/baremetal/deploy.yaml) then edit the deploy.yaml - scroll down to the Deployment resource and add the hostNetwork: true key:value pair. So for example:

      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      hostNetwork: true
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission

etc.

So what you're doing is modifying the spec of the template for the controller deployment.

Deploy the modified version with something like:

kubectl apply -f deploy.yaml

The Ingress resource that you create to use this deployment/controller should refer to it like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-api-ingress
  annotations:
    kubernetes.io/ingress.class: nginx

You can then do a describe like this:

kubectl describe service -n ingress-nginx ingress-nginx-controller

And find out what lucky node has been designated as your ingress.

Paintbox

Answer 3

I would say you need sort of configuration close to that one that is provided in the Ingress on Custom Kubernetes article. And agree with @Ijaz Ahmad Khan - it should be kind: DaemonSet

Example:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
            hostPort: 80
          - name: https
            containerPort: 443
            hostPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          securityContext:
            runAsNonRoot: false

Rook

Given your answer and @IjazAhmadKhan I found some attributes in values.yaml that seems to be the proposed solution: helm install stable/nginx-ingress --set controller.hostNetwork=true,controller.service.type="",controller.kind=DaemonSet I didn't validade that it works as expected though. – Monoicous

This configuration behaves exactly the same as Deployment kind of controller, and does not work. Any url that should be responding gives me a timeout. – Monoicous