How to change fluentd config for GKE-managed logging agent?

Question

I have a container cluster in Google Container Engine with Stackdriver logging agent enabled. It is correctly pulling stdout logs from my containers. Now I would like to change the fluentd config to specify a log parser so that the logs shown in the GCP Logging view will have the correct severity and component.

Following this Stackdriver logging guide from kubernetes.io, I have attempted to:

1. Get the fluentd ConfigMap as a yml file
2. Added a new <filter> according to my log4js log format
3. Created a new ConfigMap named fluentd-cm-2 in kube-system namespace
4. Edited the DaemonSet for fluentd and set its ConfigMap to fluentd-cm-2. I did this using kubectl edit ds instead of kubectl replace -f because the latter failed with an error message: "the object has been modified", even after getting a fresh copy of the DaemonSet yaml.

Unexpected result: The DaemonSet is restarted, but its configuration is reverted back to the original ConfigMap, so my changes did not take effect.

I have also tried editing the ConfigMap directly (kubectl edit cm fluentd-gcp-config-v1.1 --namespace kube-system) and saved it, but it was also reverted.

I noticed that the DaemonSet and ConfigMap for fluentd are tagged with addonmanager.kubernetes.io/mode: Reconcile. I would conclude that GKE has overwritten my settings because of this "reconcile" mode.

So, my question is: how can I change the fluentd configuration in a Google Container Engine cluster, when the logging agent was installed by GKE on cluster provisioning?

Flogging

Answer 1

Please take a look at the Prerequisites section on the documentation page you mentioned. It's mentioned there, that on GKE you cannot change the default Stackdriver Logging integration. The reason is that GKE maintains this configuration: updates the agent, watches its health and so on. It's not possible to provide the same level of support for all possible configurations.

However, you can always disable the default integration and deploy your own, patched version of DaemonSet. You can find out how to disable the default integration in the GKE documentation:

gcloud beta container clusters update [CLUSTER-NAME] \ --logging-service=none

Note, that after you disabled the default integration, you have to maintain the new deployment yourself: update the agent, set the resources, watch its health.

Dunaway

Thanks, my bad for skipping the prereqs! – Flogging

I create new cluster in UI with default configuration and then i disable the default logging integration I got the error below. ERROR: (gcloud.beta.container.clusters.update) ResponseError: code=400, message=Kubernetes-native resource model in Stackdriver has been previously enabled on this cluster and requires both logging and monitoring parameters to be specified. – Valetudinarian

Then I change from Kubernetes Engine Monitoring(System and workload logging and monitoring) to Kubernetes Engine Monitoring(Legacy logging and monitoring) then I can run that command. Is it risk later or any impact? – Valetudinarian

Answer 2

Here is a solution for using your own fluentd daemonset that is very much like the one included with GKE.

https://cloud.google.com/solutions/customizing-stackdriver-logs-fluentd

Space