Spring + Prometheus + Grafana: Err reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused

Question

Hello I have an app in Spring Boot and I am exposing some metrics on Prometheus. My next goal is to provide these metrics on Grafana in order to obtain some beautiful dashboards. I am using docker on WSL Ubuntu and typed the next commands for Prometheus and Grafana:

docker run -d --name=prometheus -p 9090:9090 -v /mnt/d/Projects/Msc-Thesis-Project/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

docker run -d --name=grafana -p 3000:3000 grafana/grafana

Below I am giving you the Prometheus dashboard in my browser and as you can see, everything is up and running. My problem is in Grafana configuration where I have to configure Prometheus as Data Source.

In the field URL I am providing the http://localhost:9090 but I am getting the following error:

Error reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused

I've searched everywhere and saw some workarounds that don't apply to me. To be specific I used the following: http://host.docker.internal:9090, http://server-ip:9090 and of course my system's IP address via the ipconfig command http://<ip_address>:9090. Nothing works!!!

I am not using docker-compose but just a prometheus.yml file which is as follows.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'Spring Boot Application input'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 2s
    scheme: http
    static_configs:
      - targets: ['192.168.1.233:8080']
        labels:
          application: "MSc Project Thesis"

Can you advise me something?

Scag

Answer 1

You can use the docker inspect command to find the IP address of the Prometheus container and then replace the localhost word with it.

Winglet

Wow, I didn't exepected this. Seems like it worked with localhost earlier, but now your answer helped me. Thanks – Enneagon

Answer 2

You can simply replace localhost with prometheus in the URL field of Grafana.

Tested it on my device just now and using localhost I received the same error as you did. When I replaced the URL with http://prometheus:9090 and clicked on save it did work as intended.

Abnegate

Glad to hear that it works for you. – Abnegate

Thx. You saved my day. ^_^ – Vicinage

Answer 3

Looks like you're using the same host for Prometheus and Grafana docker containers;
In this case, setting up http://host.docker.internal:9090 as the Prometheus datasource will do the trick:

Bootle

Answer 4

• It works for https://mcmap.net/q/1859138/-spring-prometheus-grafana-err-reading-prometheus-post-quot-http-localhost-9090-api-v1-query-quot-dial-tcp-127-0-0-1-9090-connect-connection-refused

• Also, if you deploy the stack by docker compose and all dockers are in same network, you can do that:

  URL: http://prometheus:9090

In above, prometheus is the domain name of the prometheus docker, which can be resolved by all dockers within same network.

Doit

This does not provide an answer to the question. Once you have sufficient reputation you will be able to comment on any post; instead, provide answers that don't require clarification from the asker. - From Review – Disturb

Answer 5

neither of those worked for me, running on ubuntu with current latest versions of prometheus and grafana. what did the trick was to run grafana with this host: sudo docker run --add-host=host.docker.internal:host-gateway -p 3000:3000 ${imagename} and then use "http://host.docker.internal:9090" as target

Lark

Answer 6

Set net flag to use host network and access Prometheus using localhost or 127.0.0.1:

docker run -d --network host --name=prometheus ...

Hoffarth

Answer 7

I'll suggest you to use docker-compose, which better supports in DNS resolving and your issues of localhost will get resolved.

Palliate