How to edit a helm chart (loki) config files for Grafana Cloud

Question

I have installed loki and grafana via helm charts to my cloud hosted k8s cluster. How do I send the loki metrics to grafana cloud? I know I should edit the promtail config file but how do I locate and view/edit helm chart files?

My installation procedure for loki+grafana are:

helm upgrade --install loki --namespace=monitoring grafana/loki-stack
helm upgrade --install grafana --namespace=monitoring grafana/grafana
kubectl port-forward service/grafana 3000:80 -n monitoring

This installation of loki only exposes loki to grafana locally. I wish to enter the configuration of loki (or more specifically promtail) so that I can send the loki logs to my grafana cloud account and monitor the logs from grafana cloud as well. I have done this in a non-k8s setting by modifying the promtail-config.yaml which is referenced to send the logs to grafana cloud. Adding the grafana cloud url to the promtail config such as:

clients:
  - url: http://loki:3100/loki/api/v1/push
  - url: https://123456:[email protected]/api/prom/push

When I explore grafana cloud > intergrations and connections > loki hosted logs > k8s cluster. I receive the following instructions

Your configuration, complete with your API key, has been generated below. Copy and paste this code to promtail/config.yaml to send data using promtail.

The code snippet:

curl -fsS https://raw.githubusercontent.com/grafana/loki/master/tools/promtail.sh | sh -s 123456 eB9... logs-prod3.grafana.net default | kubectl apply --namespace=default -f  -

I am new to helm charts, so I do not know how to view the helm config files and where they are stored. Or how to obtain the promtail/config.yaml file from helm or the k8s cluster.

Roby

Answer 1

I'm not an expert in Loki but I downloaded the default values.yaml that Chart uses and there seems an option to mention the Promtail configuration which should help you (shown below) and looking at the commands above, it seems Helm would have used default values to install the chart, you can still modify them in 2 ways.

1st: Run the below to get the values.yaml that was used to install the chart by default, modify the Promtail configuration and then upgrade the installation using the modified values.yaml

helm get values -n monitoring grafana/loki-stack loki > values.installed.loki.yaml

*Modify the values.installed.loki.yaml, on line number 5 there are a few *promtail configuration you may need to set
 promtail:
   enabled: true
   config:
     lokiAddress: http://{{ .Release.Name }}:3100/loki/api/v1/push


Once done, upgrade the chart with the modified values with -f flag
helm upgrade --install loki --namespace=monitoring grafana/loki-stack -f values.installed.loki.yaml

2nd: Get the default values.yaml as shown below which is used for Loki installation, modify the promtail configuration and upgrade the installation as shown below

helm show values grafana/loki-stack > values.loki.yaml

Modify values.loki.yaml for the promtail configuration and upgrade the chart
 promtail:
   enabled: true
   config:
     lokiAddress: http://{{ .Release.Name }}:3100/loki/api/v1/push

Run an upgrade on the existing installation with the new values.yaml

helm upgrade --install loki --namespace=monitoring grafana/loki-stack -f values.loki.yaml

Coulombe

Thanks for point this out! Confirmed works as long as you change the value of lokiAddress! I could not get the values saved to a file with access denied errors, but I ended up just displaying them within the CLI and copy/pasting into a new file and upgrading that way. – Roby