How to deploy TURN server(coturn) inside Kubernetes

Question

I am trying to deploy coturn server in the Kubernetes cluster.

According to the startup manual, it seems to each server has to have own external IP address. But I can't find a way to bind external IP addresses to each coturn pods.

How can I solve this problem? Or should I place server outside of Kubernetes cluster?

Pull

Hey @Jaemin Park, were you able to deploy it in the end? How do you modify the external ip in the turnserver config based on the kubernetes service? – Coverup

@AlbertoElias As each turn server must have at least one unique physical IP address, I gave up using Kubernetes. Instead of using k8s I choose Instance Groups to manage turn servers. This configuration is not tested on production yet but I think it is doable. – Pull

oh, ok, thank you! – Coverup

Answer 1

Although you can't assign a static IP directly to a pod, you can create a service that exposes the pods and allows you to route traffic to them via an external IP address.

For example, you could expose the deployment by running the following command which would create a service (this command presumes your application is listening on port 8080):

kubectl expose deployment DEPLOYMENT_NAME --type=LoadBalancer --port 80 --target-port 8080

To retrieve the resulting external IP address run:

kubectl get services

There is some more information on this here

You could also generate an external IP by creating an ingress resource as detailed here.

Dowery

Thanks for answer. But from my knowledge TURN spec is designed with scale out capability. So there may be number of coturn servers in same network, not single instance. This situation introduces to the question: can I assign different external IP address to each pods without creating many services? – Pull

No, services are the method for assigning external IP addresses to pods. You would need to add a service to each pod in order to assign an external IP address to each. – Dowery

Hi, I have a service created by exactly the same means, but we cannot reach the service. coturn neither show any logs nor connects. Any pointers that might be missing. – Suannesuarez

While you can assign external IP and port through a service (type LoadBalancer or NodePort). I believe the problem is rather to expose multiple UDP/TCP ports in K8S. A TURN server will open udp/tcp ports, and expect traffic on them. However it is not feasible to define a service for each of those relay ports, right? – Policeman

This approach will not work correctly to expose a TURN server externally. The exposed port will be the TURN control port, but none of the relay ports (negotiated over that control port) will be exposed. Even if you added mappings for all possible relay ports (by default 49152-65535), you'd still have problems with return UDP traffic from the TURN server, which won't pass back through the load balancer, so will be mapped to a different IP that won't be accepted by the client. hostNetwork, multus, or similar approaches are the way to expose a TURN server. – Soutor

Answer 2

The simplest way for this would be to use a DaemonSet with hostNetwork: true. That way you will have a coturn server for each of the node servers with direct access to the external interface.

Grishilda

For those who want to deploy DaemonSet on specific set of nodes, you can use node pools while deploying coTURN as described in the offical doc here cloud.google.com/architecture/… – Cower

You would need to use header based routing, something like Istio, if you want to scale it. – Choose

@OliverDixon could you explain a bit more please, I am in the situation where I want to scale STUN/TURN server to become multiple replicas. – Intact

Answer 3

To do that, you should use a proper CNI plugin, like multus: read this.

In a Nutshell: Multus creates a bridge on top of the k8s external interface, than it will attach this bridge into your container, moreover, it will attach a second interface for Inter-Cluster communication using ClusterIP, and you could also use the default CoreDNS services for DNS queries between pods.

You create a NetworkAttachmentDefinition to configure multus, and than you provide a config like so:

  config: '{
  "cniVersion": "0.3.0",
  "type": "macvlan",
  "master": "eth0",
  "mode": "bridge",
  "ipam": {
    "type": "host-local",
    "subnet": "192.168.1.0/24",
    "rangeStart": "192.168.1.200",
    "rangeEnd": "192.168.1.216",
    "routes": [
      { "dst": "0.0.0.0/0" }
    ],
    "gateway": "192.168.1.1"
  }
}'

Multus will assign a IP Address dynamically to your coturn pod out of a pool based on a pre-configured range. if for some reason the pod containing the "coturn server" container will get killed(assuming you have configured a proper liveliness Probes), it will immediately be rescheduled to a different node using the SAME bridged external interface, with the same network settings.

Using "hostNetwork: true" is just a bad idea, it will expose the ports on every k8s node, and it makes networking very much complicated, also you will not be able to communicate with the coturn service using ClusterIP if any other service needs it or use k8s DNS services.

Ephemeris

hostNetwork is not necessarily a bad idea if used wisely when it is really needed, and I find this use case one of them. – Intact