Get list of files from FTP server

Question

I'm trying to get a list of all the files we have on a server ( specifically every pdf file we have there ). I've tried using total commander and search for the files. It worked to some extent, as in, i got a list of every pdf we had there, but no way of exporting the results ( we have 100.000+ files there )

I've tried using a bash script to get the information, but i'm not very experienced with linux, and i don't really know what i'm doing.

My script looks like this :

#!/bin/bash
hostname="<host>"
ftp -i -nv $hostname << EOF
user <username> <password>
ls -R 
EOF

Running the above script i get

?Invalid command
501 Illegal PORT command
ftp: bind: Address already in use
221 Goodbye

Any help or pointing me on what to search would be greatly appreciated.

Formyl

Answer 1

Try to configure ftp to use the PASV (passive) mode for data transfers. This done with the -p switch.

I'm not sure if you will be able to do a recursive file listing with this ftp-client. ls -R in my case just gave the list of files and directories in the current working dir. Maybe Recursive FTP directory listing in shell/bash with a single session (using cURL or ftp) will help you.

Crenelation

Answer 2

With curl, this is handy
curl ftp://yourftpserver/dir/ --user username:password

Siouan

FYI: curl -l returns just the names. – Yokoyama

how do you script this to recurse? – Saphra

I do not understand why most people are using username:password in their command, that's bad practice and can be history hjacked! At least use a (protected) file and use a substitution for it, e.g.: curl ftp://yourftpserver/dir/ --user "$(cat .userpw)" – Ornis

Is there a human readable size output version? – Recorder

Answer 3

ncftpls ftp://yourftpserver/dir/*.pdf

Note that patterns such as *.pdf, etc. in the above command do work as expected.

For recursive, use -R. For more options, see man ncftpls.

ncftpls is provided by the ncftp package. For RHEL, this package is available in the epel repo.

Yokoyama

Thank you @A-B-B, I didn't know about this utility. Now I can do this to get the name of the latest coreutils source code tarball: ncftpls -x "-t" ftp://ftp.gnu.org/gnu/coreutils/coreutils*.xz|head -1 – Shortstop

A shame this does not use .netrc – Kukri

Answer 4

curl ftp://user:password@<ip>/path/

The last / is a must if it is a directory. This worked in curl version 7.29.0

Knockwurst

Answer 5

As mentionend in one of my comments, do not use user:password as plain text in your commands ever, otherwise you are running at risk of beeing history hjacked! Instead use at least a protected/restricted file with the username+password and substitute it in your command, e.g.:

ftp://yourftpserver/dir/ --user "$(cat .userpw)"

Whereas .userpw is your protected/restricted file with the example content: myusername:mypassword

Ornis

Answer 6

I have always run this FTP script in a linux system, after login ls command is submitted to a ftp connection(passive). List of files is written to a stdout stream. Script is inline in a sh script file.

#!/bin/sh

HOST=11.22.33.44
USER=myuser
PWD="mypwd"

ftp -p -n $HOST <<END_SCRIPT
quote USER $USER
quote PASS $PWD
ls
quit
END_SCRIPT
exit 0

Stagecraft

Answer 7

curl --list-only ftp://user:[email protected]/directory/JPG/

with curl request lists files

Kakalina

Answer 8

try .netrc file :

#!/bin/sh
 
HOST=11.22.33.44 
USER=myuser 
PWD="mypwd"
 
cat >> ~/.netrc <<END_SCRIPT 
machine $HOST login $USER password $PWD
macdef init 
ls 
cd upload 
lcd patch 
bin 
hash 
prompt 
mput *.patch 
put update.log 
ls 
quit
 
END_SCRIPT
chmod 600 ~/.netrc 
exit 0

run it once, and

$ chmod 600 ~/.netrc ; ls -la ~/.netrc 

make sure the file perm : -rw------- ... ~/.netrc

$ ftp $HOST >> ~/ftp.log
 

if the .netrc file is already exist, you have only to keyin ftp $HOST, and it will finish all commands in .netrc file.

and, if you have to leave for a while, you can use nohup ftp $HOST , and logout.

The process log will be in the file: ’nohup.out’ ( or '$HOME/nohup.out' )

Store it in your own file: ' nohup ftp $HOST >> ~/MyLogFile.log'

or No Output file needed: ' nohup ftp $HOST > /dev/null '

--

more information about .netrc file, reference :

https://www.unix.com/man-page/ultrix/5/netrc/

-or-

$ man .netrc

in your own *NIX OS

.....hmmm... not support by Microsoft OS...

Countercheck