How to clear Stackdriver logs in Google Cloud Platform?

Question

I recently realized I was paying way too much for my logs:

As you can see in the image, logs are getting bigger each month

As you also can see I just today put a "limit" on the ingestion. Hopefully this will slow things down.

But as I understand it, my logs have gotten so big that I have to pay for their retention each month. I cannot figure out how to: a) delete logs of a certain period (or just all of them) b) make logs auto delete after x days

I also just today put a quota limit of 100 instead of 6000

Boastful

Answer 1

The logs expire according to the retention policy:

Admin Activity        400 days
System Events         400 days
Data Access           30 days
Access Transparency   30 days
Other Logs            30 days

Note that you're not charged for Admin Activity or System Event logs.

Some solutions to control costs are exclusions and exports, but even if you use timestamp to specify the range of dates in the filter expressions to create an exclusion filter, since it's already loaded, it won't be excluded. The same applies to creating a log sink for exporting data, since it will export future matching logs.

You can use gcloud logging logs delete to delete all the logs for a given project or for a given resource, but you can't specify a range of time.

So, my suggestions are the next ones:

1.- Delete all the existing logs for resources you don't need logging.

2.- Create exclusions to keep only the logs you may need during 30 days.

3.- Create export sinks for all the logs you may need for more than 30 days.

Agranulocytosis

Thank you for your reply. I tried gcloud logging logs delete ERROR: (gcloud.logging.logs.delete) argument LOG_NAME: Must be specified. and as you can see I need a log name. Have no idea how to find out the name? So that was 1. For 2 I did create exclusions to only log 1% of the stuff but coulnd't find the limit to set it for 30 days? I really don't understand how Admin and System events are supposed to be free when they are the only ones that lasts for over 30 days. I pay around 100 USD a month for logs and the logs are just getting bigger (they accumulate) as you can see from the screenshot. – Boastful

For 1, you have to specify the resource, as in the example provided: projects/my-gcp-project-id/logs/compute.googleapis.com%2Factivity_log. You can specify any other resource, to discover the Log ID, go to Stackdriver Logging, select a resource and a log. Then click in the arrow of the bar above and click "Convert to advanced filter", this bar will show logName="LOG_ID". – Leucippus

Regarding 2, you're not accumulating logs, if you did, then you would have TBs of logs since you're creating around 250 GB of logs each month. Maybe your App Engine application creates more logs each month, but those are not accumulating, since by default the retention policy for those logs is 30 days. On this month you already ingested 154.52 GB of logs and with the new filters, either the logs are deleted using 1, or 30 days gone by to flush all the logs according to their retention policy. – Leucippus

1 worked well now, thank you very much. For 2: can I choose for the logs to flush earlier? – Boastful

I'm afraid there isn't an option to configure the days of the retention policy, but there is a Feature Request to allow configuration of retention time, I suggest to upvote and post there. – Leucippus

@Boastful you can see the list of available logs with gcloud logging logs delete – Backgammon