datadog agent not reachable from inside docker container

Question

I installed dd-agent on Amazon linux ec2. If I run my python script directly on the host machine (I used the SDK named "dogstatsd-python"), all the metrics can be sent to datadog (I logged in to datadoghq.com and saw the metrics there). the script is something like:

from statsd import statsd
statsd.connect('localhost', 8125)
statsd.increment('mymetrics')

However, I launched a docker container and run the same script from inside the container:

from statsd import statsd
statsd.connect('172.14.0.1', 8125)
statsd.increment('my metrics')

'172.14.0.1' is the IP of the host, which was extracted with command

netstat -nr | grep '^0\.0\.0\.0' | awk '{print $2}'

No metrics were sent to datadog at all.....

I'm guessing that maybe it's due to some configuration issue like "address binding". Maybe the dd-agent I installed on the host can only receive metrics from 'localhost'.

Hope someone could help me. Thank you in advance.

Cline

Did you figure out a fix?. I set non_local_traffic: yes but it doesn't seem to be working. – Kutzenco

Answer 1

You will need to set non_local_traffic: yes in your /etc/dd-agent/datadog.conf file. Otherwise the agent will reject metrics from containers.

After setting, you will need to restart the agent for the change to take effect: sudo /etc/init.d/datadog-agent restart or sudo service datadog-agent restart

The docker-dd-agent image enables non_local_traffic: yes by default.

Divestiture

In the latest version, the file is /etc/datadog-agent/datadog.conf, not dd-agent. – Iatrics

This is exactly what was needed. My setup: OS Windows server 2016 datadog agent on host. windows containers. leaving the details for others who may need. – Sarad

I ran this in Docker and for me the easiest way to get this to work was with the environment variable DD_DOGSTATSD_NON_LOCAL_TRAFFIC=True. See also docs.datadoghq.com/agent/docker/…. – Zoroastrianism

Answer 2

If dd-agent listens on localhost it can receive data only from localhost (127.0.0.1). Try to change the dd-agent host to 0.0.0.0 instead of localhost.

We are using docker-dd-agent and it works OOTB.

Magda

Thank you! I changed "bind_host" from "localhost" to "0.0.0.0" in /etc/dd-agent/datadog.conf. and restarted dd-agent. But the problem still exists..... is there anything else I need to change? – Cline

Yes, I know the dd-agent docker image. But for some reason, we try to avoid using it. – Cline

Answer 3

You don't actually want to use the IP of the host in this case. If you're running the docker dd-agent, there are two environment variables you can tap into:

statsd.connect(DOGSTATSD_PORT_8125_UDP_ADDR, DOGSTATSD_PORT_8125_UDP_PORT)

That should do the trick. If not, you should be able to find the relevant info to your problem in this section of the Datadog docs.

Also, I should point out that the only Python library that Datadog shows in their docs is datadogpy.

Kyoko

Answer 4

Here are instructions for Datadog version and Ubuntu Linux. The config vars have changed a bit:

Agent 7.35.2 - Commit: edd49be - Serialization version: v5.0.16 - Go version: go1.17.6

Check nano /etc/datadog-agent/datadog.yaml:

# Remember to enable this section
apt_config:
    apm_non_local_traffic: true
    dogstatsd_non_local_traffic: true

Then:

sudo systemctl restart datadog-agent

and:

sudo datadog-agent status

Napkin