Kubernetes logs dump for some time range

Question

Is it possible to obtain Kubernetes logs for a dedicated time range?

All I can do right now is to make a dump of about the last-hour log for the single pod using kubectl logs > dump.log cmd.

But for debugging reasons, it's necessary to obtain the logs for the last week. I was unable to find any abilities to do this in Kubernetes logs.

The only thought is to attach some external service like Kibana for the logs collection, but maybe built-in Kubernetes remedies allow to do this?

Peevish

it would depend on your log retention and roll over strategy you have in your cluster, generally you would need to consider node space especially when nodes are running multiple pods etc. Generally speaking my prefered strategy is to have short retention period on the node side and push log off to a centralised solution like you mentioned with elk, splunk datadog, loki etc – Jacklyn

Answer 1

...the last-hour log for the single pod

To retrieve last 1 hour log you can do this kubectl logs <pod> --since=1h. Asserted from kubectl help for more options:

--since=0s: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of since-time / since may be used.

--since-time='': Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time / since may be used.

--tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided.

Felicefelicia

Answer 2

AWK is an awesome tool in Unix/Linux systems for these types of logical operations

So, to display logs in between two-time ranges (ex: 10 AM to 11 AM):

1. Using --since-time and awk

kubectl logs pod_name --since-time=2022-04-30T10:00:00Z | awk '$0 < "2022-04-30 11:00:00"'

2. Using only awk

kubectl logs pod_name | awk '$0 > "2022-04-30 10:00:00"' | awk '$0 < "2022-04-30 11:00:00"'

Note: Please format date_time using in awk command based on the logs output.

Bedfast

Answer 3

Is it possible to obtain Kubernetes logs for a dedicated time range?

Yes, it is possible and in many different ways.

The only thought is to attach some external service like Kibana for the logs collection, but maybe built-in Kubernetes remedies allow to do this?

Both are possible. However, it all depends on the specific case which will be better. Chris Doyle put it well in his comment:

it would depend on your log retention and roll over strategy you have in your cluster, generally you would need to consider node space especially when nodes are running multiple pods etc. Generally speaking my preferred strategy is to have short retention period on the node side and push log off to a centralised solution like you mentioned with elk, splunk, datadog, loki etc

Of course, the built-in k8s tools will also be able to help you. You have to use kubectl logs with the proper flags. You can read about all options in the manual:

This could be the most interesting part:

--since=0: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of since-time / since may be used.

--since-time="": Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time / since may be used.

To display pod logs from the last week, you can run the following command:

kubectl logs -n <pod_namespace (optional)>  <pod name> --since 168h

Utter

The only other thing to consider with this command is if your log rotation policy will keep logs that long. For example It might only keep the last n logs and roll over on file size of x. So if your producing very little in the logs then you will be able to go back a long period. But if your policy says rotate logs when log reaches 100Mb and keep the last 10 logs, then your log data will store up to 1gb of data. If your producing 1gb an hour then your only going to be able to look back 1 hour since thats the oldest data you would have in your log retention – Jacklyn