How to best capture and log scp output?

Question

I am trying to capture output from an install script (that uses scp) and log it. However, I am not getting everything that scp is printing out, namely, the progress bar.

screen output:

Copying /user2/cdb/builds/tmp/uat/myfiles/* to server /users/myfiles as cdb

cdb@server's password: myfile 100% |*****************************| 2503 00:00

log output:

Copying /user2/cdb/builds/tmp/uat/myfiles/* to server /users/myfiles as cdb

I'd really like to know that my file got there. Here's what I am trying now to no avail:

myscript.sh 2>&1 | tee mylogfile.log

Does anyone have a good way to capture scp output and log it?

Thanks.

Soutine

Answer 1

It looks like your just missing whether the scp was succesful or not from the log.

I'm guessing the scroll bar doesn't print to stdout and uses ncurses or some other kind of TUI?

You could just look at the return value of scp to see whether it was successful. Like

scp myfile [email protected]:. && echo success!

man scp says

scp exits with 0 on success or >0 if an error occurred.

Freemasonry

Answer 2

scp prints its progress bar to the terminal using control codes. It will detect if you redirect output and thus omit the progress bar.

You can get around that by tricking scp into thinking it runs in a terminal using the "script" command which is installed on most distros by default:

script -q -c "scp server:/file /tmp/" > /tmp/test.txt

The content of test.txt will be:

file    0%    0     0.0KB/s   --:-- ETA
file   18%   11MB  11.2MB/s   00:04 ETA
file   36%   22MB  11.2MB/s   00:03 ETA
file   54%   34MB  11.2MB/s   00:02 ETA
file   73%   45MB  11.2MB/s   00:01 ETA
file   91%   56MB  11.2MB/s   00:00 ETA
file  100%   61MB  10.2MB/s   00:06

...which is probably what you want.

I stumbled over this problem while redirecting the output of an interactive script into a log file. Not having the results in the log wasn't a problem as you can always evaluate exit codes. But I really wanted the interactive user to see the progress bar. This answer solves both problems.

Hillock

Beware: you may want to use the --return (-r) option for script, if it is available, so the return code of the child process will be preserved. Otherwise, even if the command fails, your script command will happily return a success code. – Coelenterate

The single character for --return is actually -e. – Lignite

Answer 3

None of the answers here worked for me, I needed to recursively copy large directory with lot of files over long geo distance, so I wanted to log the progress (&& echo success! was by far not enough).

What I finally engineered and somehow worked was:

scp -vrC root@host:/path/to/directory . 2> copy.log &

With -v doing the trick of verbose logging (-C allows compression and -r recursion).

Grepping the logfile

grep file copy.log | wc -l

allowed me to see the number of files copied so far.

Wurtz

for me its flooding log with many debug lines from scp/ssh per file. (.. would be nice to catch just one logline for success or error per file. so I've added also | grep -v debug – Vespucci

Answer 4

Maybe you can use 'script' to log the terminal session.

Subsequence

Answer 5

scp myfile [email protected]:. && echo success! 

is very helpful but to write the message to a log file I changed it like this

scp myfile [email protected]:. && echo myfile successfully copied! >> logfile 2>&1

and this will write "myfile successfully copied!" message to the log file.

Shagreen

Answer 6

Try:

scp server:/file /tmp/ > /dev/tty

Bettinabettine

Adding an explanation of this answer would be a good idea. (It also strikes me as unlikely to get the output to his log file.) – Archaeopteryx

Why did someone downvoted? This is the CORRECT answer. By default scp will omit output when no tty is available, so you can force with >/dev/tty. – Cavalcade

If this is correct, where is the log file? – Compose

Answer 7

I can't comment yet :( so I'll add an update here...

@Martin had the best solution for me although if your scp command is midway through your script then it's output may appear after commands that actually ran afterwards.

I think that is because script must run the command in a subshell but I am yet to test.

EDIT: it does indeed spawn a shell so if you need things to run (and indeed fail) in a sequential manner (like in a build script) then you would have to add some logic around the use of the script command.

i.e.

script -q -c "your command" && sleep 1

or something similar so that your parent shell waits for the child shell to finish before moving on.

Eardrum

script -qc 'sleep 10' blocks execution on my system whereas echo | script -qc 'sleep 10' does not. Workaround: echo | script -qc 'sleep 10' | cat. I think the shell blocks as long as stdin/err/out are still open. – Hillock

Answer 8

SCP on my server did not have -c option (well -c was to specify a cihper) - I needed the rate, so I used below:

$ scp -q -v nb3510@servername:sftp.dummy . 2>&1 | grep 'Bytes per second'
Bytes per second: sent 52945.7, received 188047087.0

-v is for verbose on my server.

Diamante

-C (capital C) is for compression – Compose

Answer 9

Here is what I did

script -q /dev/stdout -c 'scp -P 3303 -o StrictHostKeyChecking=no -r -i $(sshKey.secureFilePath) LocalFolder myadminadmin@$(HOSTIP):/RemoteFolder/' | tee scp.log

You may not need —o with SSH options, but it was necessary for me to automate something that runs scp on a new VM every time.

Buzzer

Answer 10

$ grep -r "Error" xyz.out > abc.txt

Here in the above command I am storing output into file abc.txt.

This grep command is for searching text containg Error in file xyz.out and storing the output in abc.txt without displaying on console.

Ninnyhammer

This is literally not related at all to scp. – Compose

Answer 11

yes i recently was trying to get output within a php script from proc_open() i lost a quiet a time trying to get output :-) but its a bit late here and i then reading this post here made me realize that i dont really need this junky output to my script

just the exit code will do the job :-)

$exit_code = proc_close($process);

Edieedification

Although the exit code is available, this is not what that author is asking. – Cavalcade