How istio send tracing spans to jaeger?

Question

I want to use istio with existing jaeger tracing system in K8S, I began with installing jaeger system following the official link with cassandra as backend storage. Then installed istio by the helm way, but with only some selected components enabled:

helm upgrade istio -i install/kubernetes/helm/istio --namespace istio-system \   
--set security.enabled=true \   
--set ingress.enabled=false \   
--set gateways.istio-ingressgateway.enabled=true \   
--set gateways.istio-egressgateway.enabled=false \   
--set galley.enabled=false \   
--set sidecarInjectorWebhook.enabled=true \  
--set mixer.enabled=false \   
--set prometheus.enabled=false \
--set global.proxy.envoyStatsd.enabled=false \
--set pilot.sidecar=true \ 
--set tracing.enabled=false

Jaeger and istio are installed inside the same namespace istio-sytem, after all done, all pods inside it looks like this:

kubectl -n istio-system get pods                           
NAME            READY     STATUS    RESTARTS   AGE 
istio-citadel-5c9544c886-gr4db      1/1       Running   0          46m 
istio-ingressgateway-8488676c6b-zq2dz     1/1       Running   0        51m 
istio-pilot-987746df9-gwzxw               2/2       Running   1    51m 
istio-sidecar-injector-6bd4d9487c-q9zvk   1/1       Running   0    45m 
jaeger-collector-5cb88d449f-rrd7b         1/1       Running   0    59m 
jaeger-query-5b5948f586-gxtk7             1/1       Running   0    59m

Then I followed the link to deploy the bookinfo sample into another namespace istio-play, which has label istio-injection=enabled, but no matter how I flush the productpage page, there's no tracing data be filled into jaeger.

I guess maybe tracing spans are sent to jaeger by mixer, like the way istio do all other telementry stuff, so I -set mixer.enabled=true, but unfortunately only some services like istio-mixer or istio-telementry are displayed. Finally I cleaned up all the above installation and followed this task step by step, but the tracing data of bookinfo app are still not there.

My questions is: How indeed istio send tracing data to jaeger? Does sidecar proxy send it directly to jaeger-collector(zipkin.istio-system:9411) like how envoy does, or the data flows like this: sidecar-proxy -> mixer -> jaeger-collector? And how could I debug how the data flow between all kinds of components inside the istio mesh?

Thanks for any help and info :-)

---

Update: I tried again by installing istio without helm: kubectl -n istio-system apply -f install/kubernetes/istio-demo.yaml, this time everything works just fine, there must be something different between kubectl way and helm way.

Overwinter

I think --set tracing.enabled=false disables the tracing. You need --set tracing.enabled=true. That is why you're seeing the difference between kubectl way and helm way. – Keynote

Answer 1

Based on my experience and reading online, I found this interesting line in Istio mixer faq

Mixer trace generation is controlled by command-line flags: trace_zipkin_url, trace_jaeger_url, and trace_log_spans. If any of those flag values are set, trace data will be written directly to those locations. If no tracing options are provided, Mixer will not generate any application-level trace information.

Also, if you go deep into mixer helm chart, you will find traces of Zipkin and Jaeger signifying that it’s mixer that is passing trace info to Jaeger.

I also got confused which reading this line in one of the articles

Istio injects a sidecar proxy (Envoy) in the pod in which your application container is running. This sidecar proxy transparently intercepts (iptables magic) all network traffic going in and out of your application. Because of this interception, the sidecar proxy is in a unique position to automatically trace all network requests (HTTP/1.1, HTTP/2.0 & gRPC).

On Istio mixer documentation, The Envoy sidecar logically calls Mixer before each request to perform precondition checks, and after each request to report telemetry. The sidecar has local caching such that a large percentage of precondition checks can be performed from cache. Additionally, the sidecar buffers outgoing telemetry such that it only calls Mixer infrequently.

Update: You can enable tracing to understand what happens to a request in Istio and also the role of mixer and envoy. Read more information here

Clout

Thanks for your response:-) So you mean the sidecar envoy proxy inside each pod has nothing to do with jaeger/zipkin tracing system, but when sidecar generates the static config for envoy(/etc/istio/proxy/envoy-rev0.json), from the config template file /var/lib/istio/envoy/envoy_bootstrap_tmpl.json you'll see it'll generate envoy's tracing block and a static zipkin cluster, if envoy won't send tracing info by itself, what's this config info for? – Overwinter

To understand the role of mixer and envoy,You should check this answer here - #48640160 – Clout