Grafana showing "too many outstanding requests" error while querying loki dashboard

Question

I am trying to use loki and grafana for generating and visualizing log-based metrics. I have created a Grafana dashboard using the loki filters. While clicking the refresh button, all dashboards fail with the error "too many outstanding requests" and display no data. Refer to the screenshot attached. I deployed grafana and loki to the EKS cluster.

Is there a parameter I can adjust to resolve the issue? I examined the pod/deployment configurations but found nothing pertinent.

Please assist.

Encyclical

I also faced this issue, and updated my Loki configuration according to some of the examples at github.com/grafana/loki/issues/5123 and it seemed to help. – Validate

Thanks, the issue was fixed after adjusting the "max_outstanding_per_tenant" parameter in Loki configuration. – Encyclical

Answer 1

You can change default limits for some Loki internal variables. Here is an example of a tested config that works fine on 4 core virtual machine with simple file storage.

query_scheduler:
  max_outstanding_requests_per_tenant: 4096
frontend:
  max_outstanding_per_tenant: 4096
query_range:
  parallelise_shardable_queries: true
limits_config:
  split_queries_by_interval: 15m
  max_query_parallelism: 32

More details on this issue: https://github.com/grafana/loki/issues/5123

Downs

Answer 2

What worked for me was adding a max_outstanding_requests_per_tenant value to the local-config.yaml file, which is the default Loki config file. The file is stored in the official Docker image at /etc/loki/local-config.yaml. So I mounted a local-config.yaml file to that location:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

query_scheduler:
  max_outstanding_requests_per_tenant: 2048

Chadwick

Thanks. This settings solves 500 error for long time queries: max_outstanding_requests_per_tenant: 2048 – Rubel