UnknownHostException in kubernetes

Question

Im using microservice in kubernete and docker and I got an UnknownHostException when Zuul (gateway) forward request data to service I can't ping to service container by pod name (but when i use docker swarm instead of Kubernetes, i can ping by host name normally)

This is my service yaml file

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: merchantservice
  labels:
    run: merchantservice
spec:
  template:
    metadata:
      labels:
        name: merchantservice
    spec:
      containers:
      - name: merchantservice
        image: merchantservice:latest
        ports:
        - containerPort: 8001
        env:
          - name: EUREKA_SERVER
            value: "eureka1"
          - name: EUREKA_SERVER2
            value: "eureka2"
          - name: CONFIG_SERVER
            value: "configserver"
---
apiVersion: v1
kind: Service
metadata:
  name: merchantservice
spec:
  selector:
    name: merchantservice
  ports:
    - port: 8001
      targetPort: 8001
  type: LoadBalancer

And this is error output

2019-05-28 04:29:53.443  WARN 1 --- [nio-8444-exec-6] o.s.c.n.z.filters.post.SendErrorFilter   : Error during filtering

com.netflix.zuul.exception.ZuulException: Forwarding error
...
Caused by: com.netflix.client.ClientException: null
...
Caused by: java.lang.RuntimeException: java.net.UnknownHostException: merchantservice-79cc77d9cc-224mf: Try again
    at rx.exceptions.Exceptions.propagate(Exceptions.java:57) ~[rxjava-1.3.8.jar!/:1.3.8]
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463) ~[rxjava-1.3.8.jar!/:1.3.8]
    at rx.observables.BlockingObservable.single(BlockingObservable.java:340) ~[rxjava-1.3.8.jar!/:1.3.8]
    at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112) ~[ribbon-loadbalancer-2.3.0.jar!/:2.3.0]
    ... 158 common frames omitted
Caused by: java.net.UnknownHostException: merchantservice-79cc77d9cc-224mf: Try again
...

Philo

kube-dns does not create DNS records for pods, since they are ephemeral; if you want to address pods, use their IP or use a Service – Agony

@MatthewLDaniel but why my eureka server (service discovery) still discovery service normally? In config server, im using this server.port=8001 eureka.client.serviceUrl.defaultZone=http://${EUREKA_SERVER:localhost}:9091/eureka,http://${EUREKA_SERVER2:localhost}:9092/eureka – Philo

I don't understand your question; what does posting eureka.client.serviceUrl.defaultZone= have to do with your question about resolving the DNS name of a Pod? – Agony

@MatthewLDaniel the ${EUREKA_SERVER:localhost} return eureka server hostname => it will become to eureka.client.serviceUrl.defaultZone=http://eureka:9091/eureka <= why this hostname eureka work but zuul occur exception? – Philo

@KaijinSabrac merchantservice-79cc77d9cc-224mft is the pod name not the service name. your zuul is integrated with eureka and ribbon is doing client side discovery by querying the eureka server with service name. It is getting list of host names(pod names) and making a call using the pod name instead of the pod ip, its getting failed. Kindly follow Matthew suggestion of using ip address instead of pod name. – Azalea

Answer 1

Make sure that you have added eureka.instance.preferIpAddress = true in your application.properties file of the microservice which has to be routed via zuul.

Reference: https://cloud.spring.io/spring-cloud-netflix/multi/multi_spring-cloud-eureka-server.html#spring-cloud-eureka-server-prefer-ip-address

Kaleidoscope

This actually worked for me !! @kaijin-sabrac Please mark this as answer if you also feel the same !! – Annmaria

what happens if Ip changes? In k8s, the Ip of the pod will change if I created new pod – Twoedged

IP is read via Service discovery, assuming you are using DNS based service discovery, it'll map IP from pod's service host. – Kaleidoscope

Answer 2

in openshift you cannot hit a pod by pod-id but you can hit it by local DNS and service. ..svc.cluster.local: I think the same will work in your case. please try below

merchantservice.<bob-id/project-id>.svc.cluster.local:8081 

Bribery

Answer 3

It seems that container id or (pod id ) in case of kubernetes is not valid DNS name however, it is valid if you use docker-compose. I think the preferred way is to add this setting to all services that will be registered in Eureka eureka.instance.prefer-ip-address=true

Ferricyanide

Answer 4

I am posting @Matthew L Daniel answer from the comments for better visibility:

kube-dns does not create DNS records for pods, since they are ephemeral; if you want to address pods, use their IP or use a Service – Matthew L Daniel

Drayton

I just fixed this by restart docker. This is iptables rule problem (i think) – Philo