Unable to export traces to OpenTelemetry Collector on Kubernetes

Question

I am using the opentelemetry-ruby otlp exporter for auto instrumentation: https://github.com/open-telemetry/opentelemetry-ruby/tree/main/exporter/otlp

The otel collector was installed as a daemonset: https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-collector

I am trying to get the OpenTelemetry collector to collect traces from the Rails application. Both are running in the same cluster, but in different namespaces.

We have enabled auto-instrumentation in the app, but the rails logs are currently showing these errors:

E, [2022-04-05T22:37:47.838197 #6] ERROR -- : OpenTelemetry error: Unable to export 499 spans

I set the following env variables within the app:

OTEL_LOG_LEVEL=debug
OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318

I can't confirm that the application can communicate with the collector pods on this port. Curling this address from the rails/ruby app returns "Connection Refused". However I am able to curl http://<OTEL_POD_IP>:4318 which returns 404 page not found.

From inside a pod:

# curl http://localhost:4318/
curl: (7) Failed to connect to localhost port 4318: Connection refused

# curl http://10.1.0.66:4318/
404 page not found

This helm chart created a daemonset but there is no service running. Is there some setting I need to enable to get this to work?

I confirmed that otel-collector is running on every node in the cluster and the daemonset has HostPort set to 4318.

Fluoroscope

Answer 1

The problem is with this setting:

OTEL_EXPORTER_OTLP_ENDPOINT=http://0.0.0.0:4318

Imagine your pod as a stripped out host itself. Localhost or 0.0.0.0 of your pod, and you don't have a collector deployed in your pod.

You need to use the address from your collector. I've checked the examples available at the shared repo and for agent-and-standalone and standalone-only you also have a k8s resource of type Service.

With that you can use the full service name (with namespace) to configure your environment variable.
Also, the Environment variable now is called OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, so you will need something like this:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<service-name>.<namespace>.svc.cluster.local:<service-port>

Jocko

Thank you that worked. The OpenTelemetry errors have stopped. However I had to use this endpoint, the service name itself wasn't resolvable: <service-name>.<namespace>.svc.cluster.local:<service-port> In addition, the env variable I changed was: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. The non-specific OTEL_EXPORTER_OTLP_ENDPOINT didn't seem to make any difference. This helm chart should have clearer documentation that both the standalone collector AND daemonset agent need to enabled. – Fluoroscope

Thx @alig227, I'll update the answer with your insights. – Jocko

Answer 2

The correct solution is to use the Kubernetes Downward API to fetch the node IP address, which will allow you to export the traces directly to the daemonset pod within the same node:

  containers:
  - name: my-app
    image: my-image
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://$(HOST_IP):4318

Note that using the deployment's service as the endpoint (<service-name>.<namespace>.svc.cluster.local) is incorrect, as it effectively bypasses the daemonset and sends the traces directly to the deployment, which makes the daemonset useless.

Huerta

Thanks for adding this answer. I was wondering exactly why you'd use a Service with a Daemonset. It seems like you'd rather have the otel collector as a Deployment in that case. – Densify

If you're not using the logs and metrics features of the collector then it is better to use a deployment rather than a daemonset, see github.com/open-telemetry/opentelemetry-collector/issues/5405 – Huerta

This isn't the correct solution... – Jointly