Appending docker stats per day to a single file

Question

I am trying to log the docker stats every minute to a csv file in S3 bucket. The below command is run on putty.

while true; do docker stats --no-stream | aws s3 cp - s3://username/dockerstats/`date -u +"%Y%m%dT%H%M%S.csv"`; sleep 60; done

As it can be seen above, it is creating a new file every minute. I would like to append the docker stats to the same csv file. Can anyone let me know what would be the command to append data to the same file csv file? It would be one csv file per day.

Overshoot

Answer 1

You redirect the output with >> to append to your file.

docker stats --no-stream >> "$(date -u +"%Y%m%d.csv")"

It will create the file if it doesn't exist, so it will work with your dynamic date file names.

If you only use one > it will not append to the file but completely overwrite it.

Convoke

For windows users: docker stats --no-stream >> "%Date:~6,4%_%Date:~3,2%_%Date:~0,2%.csv" – Ravage

Answer 2

Just wanted to add to the above answer from my experience:

You can remove --no-stream flag if you want to get continuous results over 1 sec intervals

docker stats | sed 's/$/\n/' >> "$(date +"%Y%m%d-%H:%M").log"

Note >> is sort of useless here as the file has minute value in the name

Beauchamp