Istio distributed tracing with Jaeger not working

Question

I'm trying to set up a local k8s cluster and on minikube with installed istio and I have an issue with enabling distributed tracing with Jaeger. I have 3 microservices A -> B -> C. I am propagating the all the headers that are needed:

{"x-request-id", "x-b3-traceid", "x-b3-spanid", "x-b3-parentspanid", "x-b3-sampled", "x-b3-flags", "x-ot-span-context"}

But on Jaeger interface, I can only see the request to the service A and I cannot see the request going to service B.

I have logged the headers that are sent in the request. Headers from service A:

Header - x-request-id: c2804368-2ff0-9d90-a2aa-972537968924
Header - x-b3-traceid: 3a2400b40bbe5ed8
Header - x-b3-spanid: 3a2400b40bbe5ed8
Header - x-b3-parentspanid: 
Header - x-b3-sampled: 1
Header - x-b3-flags: 
Header - x-ot-span-context: 

Headers from service B:

Header - x-request-id: c2804368-2ff0-9d90-a2aa-972537968924
Header - x-b3-traceid: 3a2400b40bbe5ed8
Header - x-b3-spanid: 3a2400b40bbe5ed8
Header - x-b3-parentspanid:
Header - x-b3-sampled: 1
Header - x-b3-flags:
Header - x-ot-span-context:

So the x-request-id, x-b3-traceid, x-b3-sampled, and x-b3-spanid mathces. There are some headers that aren't set. Also, I'm accessing service A via k8s Service IP of type LoadBalancer, not via ingress. Don't know if this could be the issue.

UPD: I have setup istio gateway so now I'm accessing service A via istio gateway. However the result is the same, I can see the trace for gateway->A but no any further tracing

Resolvable

I've found that some frameworks give you empty string when you look for a specific header that does not exist so you may end up propogating a non-existent header to the next request with empty value. I solved a similar issue by only propogating tracing headers with non-empty values. – Plurality

@UtkuTurunç Wow, looks like this is the issue! Please make it as an answer so I can accept it! :) – Resolvable

glad it worked :) answer is done – Plurality

I'm facing the same issue. How did you solve it? In my case, B receives the same value of spanid as A sends it. I think the spanid should be different at B, as by that time a new span should have been created at the sidecar (envoy proxy), right? – Leonialeonid

@Nawaz please see the answer by UtkuTurunç . In short: filter the tracing headers that have empty values in your application – Resolvable

@SergiiBishyr: I'm not sending empty headers. I'm propagating only x-request-id, x-b3-traceid , x-b3-spanid and x-b3-sampled because those are what A receives in its headers. And all of these are non-empty. – Leonialeonid

@Nawaz are you sure that the values of these headers aren’t empty? – Resolvable

Yes. the service A and B return them as responses (in their corresponding body). So I can see them in the output. – Leonialeonid

I'm using Minikube BTW. – Leonialeonid

@Nawaz there are no additional requirements for minikube. Maybe you can share the code for forwarding the headers? – Resolvable

@SergiiBishyr: Please see the gists here: gist.github.com/snawaz/0fc8ff3f5e964f146d8abf9cf344aea6 – Leonialeonid

My issue is similar to this and this posted in different group – Leonialeonid

@Nawaz why don't you froward headers x-b3-parentspanid , x-b3-parentspanid and x-b3-flags? – Resolvable

@SergiiBishyr: they're empty .. actually they don't even exist in the headers. – Leonialeonid

Answer 1

Some web frameworks return empty string if a non-existent header is queried. I have seen this in Spring Boot and KoaJS.

If any of the tracing headers is not sent by Istio, this header logic causes us to send empty string for those non-existent headers which breaks tracing.

My suggestion is after getting the values for headers filter out the ones with empty string as their values and propogate the remaining ones.

Fezzan

Thank you for this answer. Worked perfectly for Istio 1.0.6 and Jaeger. I wasted two days trying to solve this issue in Go. – Crellen