Kubernetes: how to access service if nodePort is random?

Question

I'm new to K8s and am currently using Minikube to play around with the platform. How do I configure a public (i.e. outside the cluster) port for the service? I followed the nginx example, and K8s service tutorials. In my case, I created the service like so:

kubectl expose deployment/mysrv --type=NodePort --port=1234

The service's port is 1234 for anyone trying to access it from INSIDE the cluster. The minikube tutorials say I need to access the service directly through it's random nodePort, which works for manual testing purposes:

kubectl describe service mysrv | grep NodePort
...
NodePort:                 <unset>  32387/TCP
# curl "http://`minikube ip`:32387/"

But I don't understand how, in a real cluster, the service could have a fixed world-accessible port. The nginx examples describe something about using the LoadBalancer service kind, but they don't even specify ports there...

Any ideas how to fix the external port for the entire service?

Methylal

If you set the type field to NodePort, the Kubernetes control plane allocates a port from a range specified by --service-node-port-range flag (default: 30000-32767).If you want a specific port number everytime, you can specify a value in the nodePort field at service creation time on your service manifest file. – Indivertible

Setting the nodePort as explained in the docs is not a good solution because it only controls the port at the node level. When accessing the service from outside the entire cluster without minikube (on GKE for example), how is the nodePort even relevant? It has to be something at the load balancer level but I don't know where. BTW Ingress is not an option since my traffic is not http-based (it's TCP though) – Methylal

Hi, if its non-http traffic, then i would suggest you to use exteranlIP in k8s service object. then you can access non-http application with externalIP:svcPort. kubernetes.io/docs/tutorials/stateless-application/… – Merkley

Suresh Vishnoi, thanks, it works! The problem was that I had to run 'minikube tunnel' in a separate shell, to get an external IP (otherwise it stays 'pending' forever). The 'port' config is indeed accessible outside the cluster as you mentioned – Methylal

Answer 1

The minikube tutorials say I need to access the service directly through it's random nodePort, which works for manual testing purposes:

When you create service object of type NodePort with a $ kubectl expose command you cannot choose your NodePort port. To choose a NodePort port you will need to create a YAML definition of it.

You can manually specify the port in service object of type Nodeport with below example:

apiVersion: v1
kind: Service
metadata:
  name: example-nodeport
spec:
  type: NodePort
  selector:
    app: hello # selector for deployment
  ports:
  - name: example-port
    protocol: TCP
    port: 1234 # CLUSTERIP PORT
    targetPort: 50001 # POD PORT WHICH APPLICATION IS RUNNING ON 
    nodePort: 32222 # HERE!

You can apply above YAML definition by invoking command: $ kubectl apply -f FILE_NAME.yaml

Above service object will be created only if nodePort port is available to use.

But I don't understand how, in a real cluster, the service could not have a fixed world-accessible port.

In clusters managed by cloud providers (for example GKE) you can use a service object of type LoadBalancer which will have a fixed external IP and fixed port.

Clusters that have nodes with public IP's can use service object of type NodePort to direct traffic into the cluster.

In minikube environment you can use a service object of type LoadBalancer but it will have some caveats described in last paragraph.

A little bit of explanation:

NodePort

Nodeport is exposing the service on each node IP at a static port. It allows external traffic to enter with the NodePort port. This port will be automatically assigned from range of 30000 to 32767.

You can change the default NodePort port range by following this manual.

You can check what is exactly happening when creating a service object of type NodePort by looking on this answer.

Imagine that:

• Your nodes have IP's:
  • 192.168.0.100
  • 192.168.0.101
  • 192.168.0.102
• Your pods respond on port 50001 with hello and they have IP's:
  • 10.244.1.10
  • 10.244.1.11
  • 10.244.1.12
• Your Services are:
  • NodePort (port 32222) with:
    • ClusterIP:
      • IP: 10.96.0.100
      • port:7654
      • targetPort:50001

A word about targetPort. It's a definition for port on the pod that is for example a web server.

According to above example you will get hello response with:

• NodeIP:NodePort (all the pods could respond with hello):
  • 192.168.0.100:32222
  • 192.168.0.101:32222
  • 192.168.0.102:32222
• ClusterIP:port (all the pods could respond with hello):
  • 10.0.96.100:7654
• PodIP:targetPort (only the pod that request is sent to can respond with hello)
  • 10.244.1.10:50001
  • 10.244.1.11:50001
  • 10.244.1.12:50001

You can check access with curl command as below:

$ curl http://NODE_IP:NODEPORT

---

In the example you mentioned:

$ kubectl expose deployment/mysrv --type=NodePort --port=1234

What will happen:

• It will assign a random port from range of 30000 to 32767 on your minikube instance directing traffic entering this port to pods.
• Additionally it will create a ClusterIP with port of 1234

In the example above there was no parameter targetPort. If targetPort is not provided it will be the same as port in the command.

Traffic entering a NodePort will be routed directly to pods and will not go to the ClusterIP.

From the minikube perspective a NodePort will be a port on your minikube instance. It's IP address will be dependent on the hypervisor used. Exposing it outside your local machine will be heavily dependent on operating system.

---

LoadBalancer

There is a difference between a service object of type LoadBalancer(1) and an external LoadBalancer(2):

• Service object of type LoadBalancer(1) allows to expose a service externally using a cloud provider’s LoadBalancer(2). It's a service within Kubernetes environment that through service controller can schedule a creation of external LoadBalancer(2).
• External LoadBalancer(2) is a load balancer provided by cloud provider. It will operate at Layer 4.

Example definition of service of type LoadBalancer(1):

apiVersion: v1
kind: Service
metadata:
  name: example-loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: hello
  ports:
    - port: 1234 # LOADBALANCER PORT 
      targetPort: 50001  # POD PORT WHICH APPLICATION IS RUNNING ON 
      nodePort: 32222 # PORT ON THE NODE 

Applying above YAML will create a service of type LoadBalancer(1)

Take a specific look at:

  ports:
    - port: 1234 # LOADBALANCER PORT 

This definition will simultaneously:

• specify external LoadBalancer(2) port as 1234
• specify ClusterIP port as 1234

Imagine that:

• Your external LoadBalancer(2) have:
  • ExternalIP: 34.88.255.5
  • port:7654
• Your nodes have IP's:
  • 192.168.0.100
  • 192.168.0.101
  • 192.168.0.102
• Your pods respond on port 50001 with hello and they have IP's:
  • 10.244.1.10
  • 10.244.1.11
  • 10.244.1.12
• Your Services are:
  • NodePort (port 32222) with:
    • ClusterIP:
      • IP: 10.96.0.100
      • port:7654
      • targetPort:50001

According to above example you will get hello response with:

• ExternalIP:port (all the pods could respond with hello):
  • 34.88.255.5:7654
• NodeIP:NodePort (all the pods could respond with hello):
  • 192.168.0.100:32222
  • 192.168.0.101:32222
  • 192.168.0.102:32222
• ClusterIP:port (all the pods could respond with hello):
  • 10.0.96.100:7654
• PodIP:targetPort (only the pod that request is sent to can respond with hello)
  • 10.244.1.10:50001
  • 10.244.1.11:50001
  • 10.244.1.12:50001

ExternalIP can be checked with command: $ kubectl get services

Flow of the traffic: Client -> LoadBalancer:port(2) -> NodeIP:NodePort -> Pod:targetPort

Minikube: LoadBalancer

Note: This feature is only available for cloud providers or environments which support external load balancers.

-- Kubernetes.io: Create external LoadBalancer

On cloud providers that support load balancers, an external IP address would be provisioned to access the Service. On Minikube, the LoadBalancer type makes the Service accessible through the minikube service command.

-- Kubernetes.io: Hello minikube

Minikube can create service object of type LoadBalancer(1) but it will not create an external LoadBalancer(2).

The ExternalIP in command $ kubectl get services will have pending status.

To address that there is no external LoadBalancer(2) you can invoke $ minikube tunnel which will create a route from host to minikube environment to access the CIDR of ClusterIP directly.

Muskrat

In an on-prem environment it's easy to understand the use case of a fixed NodePort. And in a cloud environment it's easy to understand the use case of a LoadBalancer. But what's the use case of a randomly assigned NodePort? Who is that useful for and how is it meant to be used? – Roop

Answer 2

There is a small mistake in Dawid Kruk’s answer,

Traffic entering a NodePort will be routed directly to pods and will not go to the ClusterIP.

But as k8s documented here:

NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting :.

Traffic entering a NodePort did go to ClusterIP.

Magi