Istio Distributed Tracing shows just 1 span

Question

I'm following this guide, with Zipkin. I have 3 microservices involed, A -> B -> C, I'm propagating headers from A to B and from B to C. But in the Zipkin dashboard I only see entries for A -> B and B -> C, not A -> B -> C.

Those are the headers:

[
    "x-request-id",
    "x-b3-traceid",
    "x-b3-spanid",
    "x-b3-parentspanid",
    "x-b3-sampled",
    "x-b3-flags",
    "x-ot-span-context"
]

I can see that in B x-b3-parentspanid is null and I guess that's wrong, but the other are working I think...how is it possible?

EDIT: added code snippets to show headers propagation

A -> B propagation:

app.post("/job", (req, res) => postJob(req.body, req.headers).then((response) => res.send(response)))

...

const postJob = (job, headers) => rp({
    method: "POST",
    uri: `${API_ENDPOINT}/api/job`,
    json: true,
    body: job,
    headers: Object.keys(headers).filter((key) => TRACING_HEADERS.includes(key)).map((key) => headers[key])
})

B -> C propagation:

@PostMapping("/api/job")
@ResponseBody
fun publish(
    @RequestBody job: Job,
    @RequestHeader("x-request-id") xreq: String?,
    @RequestHeader("x-b3-traceid") xtraceid: String?,
    @RequestHeader("x-b3-spanid") xspanid: String?,
    @RequestHeader("x-b3-parentspanid") xparentspanid: String?,
    @RequestHeader("x-b3-sampled") xsampled: String?,
    @RequestHeader("x-b3-flags") xflags: String?,
    @RequestHeader("x-ot-span-context") xotspan: String?
): JobResponse = jobsService.publishJob(
    job, mapOf(
        "x-request-id" to xreq,
        "x-b3-traceid" to xtraceid,
        "x-b3-spanid" to xspanid,
        "x-b3-parentspanid" to xparentspanid,
        "x-b3-sampled" to xsampled,
        "x-b3-flags" to xflags,
        "x-ot-span-context" to xotspan
    )
)

...

fun publishJob(job: Job, headers: Map<String, String?>): JobResponse {
        val enabled = restTemplate.exchange(
            "${gatekeeperConfiguration.endpoint}/",
            HttpMethod.GET,
            HttpEntity(headers),
            EnabledResponse::class.java
        ).body
        if (!enabled!!.isEnabled) // TODO we intentionally want this to crash if body is null
            return JobResponse(JobRequestStatus.REJECTED)

        return if (this.queue.publish(job)) JobResponse(JobRequestStatus.OK)
        else throw RuntimeException("I don't know what to do, yet")
    }

Gayegayel

Are you sure that your services propagate all headers? – Lewes

@LukasEichler updated the answer with some code – Gayegayel

You will probably be better off giving this a kotlin label (that's kotlin, right?) I don't know kotlin but a number of things look suspicious. Most suspicious is the HttpEntity(), apparently to propagate the headers. The term "entity" has a specific meaning in HTTP land. From that meaning I would infer that the header map was actually getting written into the body of the B->C request, not rendered as individual request headers. But you should ask this of people who know kotlin. – Apthorp

I had a similar issue, not sure that this is the same, but take a look a answer here: #52038525 – Homophonous

Possible duplicate of Istio distributed tracing with Jaeger not working – Homophonous

Have you confirmed that the headers are on the wire? – Noaccount

Try to use spring-cloud-sleuth, github.com/spring-cloud/spring-cloud-sleuth. I saw you are using spring already. – Radack

I had a similar issue at one point. I found that the trace had not been flushed due to a configuration to prevent chatty back-and-forth from the tracing client. – Interdiction

Answer 1

Object.keys(headers).filter((key) => TRACING_HEADERS.includes(key)).map((key) => headers[key]) returns an array.

What you want is:

Object.keys(headers)                                                                                                                                                                              
  .filter(key => TRACING_HEADERS.includes(key))                                                                                                                                                                    
  .reduce((obj, key) => {                                                                                                                                                                                          
    obj[key] = headers[key];                                                                                                                                                                                       
    return obj;                                                                                                                                                                                                    
  }, {})

I'm pretty sure this isn't an istio / distributed tracing issue ;-)

Tva

Answer 2

b3-propagation of x-b3-parentspanid (https://github.com/openzipkin/b3-propagation) can be configured in your application.yml by adding:

opentracing:
  jaeger:
    enable-b3-propagation: true

Dele