Can we see transfer progress with kubectl cp?

Question

Is it possible to know the progress of file transfer with kubectl cp for Google Cloud?

Subversive

One alternative I recently discovered...Google Cloud Shell shows file upload progress if that fits into your use case. – Calvin

Answer 1

No, this doesn't appear to be possible.

kubectl cp appears to be implemented by doing the equivalent of

kubectl exec podname -c containername \
  tar cf - /whatever/path \
| tar xf -

This means two things:

1. tar(1) doesn't print any useful progress information. (You could in principle add a v flag to print out each file name as it goes by to stderr, but that won't tell you how many files in total there are or how large they are.) So kubectl cp as implemented doesn't have any way to get this out.

2. There's not a richer native Kubernetes API to copy files.

If moving files in and out of containers is a key use case for you, it will probably be easier to build, test, and run by adding a simple HTTP service. You can then rely on things like the HTTP Content-Length: header for progress metering.

Samite

Answer 2

One option is to use pv which will show time elapsed, data transferred and throughput (eg MB/s):

$ kubectl exec podname -c containername -- tar cf - /whatever/path | pv | tar xf -

14.1MB 0:00:10 [1.55MB/s] [       <=>                                   ]

If you know the expected transfer size ahead of time you can also pass this to pv and it will then calculate a % progress and also an ETA, eg for a 100m transfer:

$ kubectl exec podname -c containername -- tar cf - /whatever/path | pv -s 100m | tar xf -

13.4MB 0:00:09 [1.91MB/s] [==>                          ] 13% ETA 0:00:58

You obviously need to have pv installed (locally) for any of the above to work.

Lunseth

This is brilliant and works as advertised. The example copies from a remote container to your local (which has kubectl). How might the inverse operation work? From local to a remote container (which may not have kubectl)? – Quathlamba

@ChrisBeck I don't believe there is a way to do the copy, in either direction, without kubectl. Copying from local to remote is simple enough though: $ tar cf - /whatever/path | pv -s 100m | kubectl exec -i podname -c containername tar xf - note the addition of the -i argument passed to kubectl exec – Lunseth

More recent kubectls prefer -- as in kubectl exec podname -c containername -- tar cf - /whatever/path | pv | tar xf - – Quathlamba

This also worked for me cat local_file | pv | kubectl exec -i podname -c containername -- sh -c 'cat > remote_file' – Flaccid

If tar is not available (required for kubectl cp)on the pod, you can use this technique to copy but you don't get the progress indicator: fabianlee.org/2022/09/10/…. – Quathlamba

Answer 3

On MacOS, there is still the hacky way of opening the "Activity Monitor" on the "Network" tab. If you are copying with kubectl cp from your local machine to a distant pod, then the total transfer is shown in the "Sent Bytes" column. Not of super high precision, but it sort of does the job without installing anything new.

Caduceus

That is a neat idea, can't believe I haven't thought of that. – Graupel

Answer 4

I figured out a hacky way to do this. If you have bash access to the container you're copying to, you can do something like wc -c <file> on the remote, then compare that to the size locally. du -h <file> is another option, which gives human-readable output so it may be better

Breeze

I like it. Super simple. – Carpal

ls -al works. nice. – Trunnel

Answer 5

It's not possible, but you can find here how to implement rsync with kubernetes, rsync shows you the progress of the transfer file. rsync files to a kubernetes pod

Eurasian

Answer 6

I know it doesn't show an active progress of each file, but does output a status including byte count for each completed file, which for multiple files run via scripts, is almost as good as active progress:

kubectl cp local.file container:/path/on/container --v=4

Notice the --v=4 is verbose mode and will give you output. I found kubectl cp output shows from v=3 thru v=5.

Strickman

Answer 7

Showing off a hacky way to achieve this via a hidden flag in scp:

kubectl exec pod-name -- sh -c 'file=/tmp/test1; name=$(basename $file) ; size=$(wc -c $file | cut -d " " -f 1); echo "C0644 $size $name"; dd if=$file bs=8192 2>/dev/null; printf "\0"' | scp -t /tmp

This will copy the file(/tmp/test1) in the container to your /tmp directory with mode 644, with a nice progress bar and speedometer. If you want you can probably read the file mode and set it accordingly in the first line of the SCP message as well.

But we really should be waiting for this PR to be addressed.

ref:

http://nanozen.snert.org/how-the-scp-protocol-works/

Semipermeable