I am using cURL to try to download all files in a certain directory.
here's what my list of files looks like:
I have tried to do in bash script: iiumlabs.[].csv.pgp and iiumlabs* and I guess curl is not big on wildcards.
curl -u login:pass ftp.myftpsite.com/iiumlabs* -O
question: how do i download this directory of files using cURL?
OK, considering that you are using Windows, the most simple way to do that is to use the standard ftp tool bundled with it. I base the following solution on Windows XP, hoping it'll work as well (or with minor modifications) on other versions.
First of all, you need to create a batch (script) file for the ftp program, containing instructions for it. Name it as you want, and put into it:
curl -u login:pass ftp.myftpsite.com/iiumlabs* -O
open ftp.myftpsite.com
login
pass
mget *
quit
The first line opens a connection to the ftp server at ftp.myftpsite.com. The two following lines specify the login, and the password which ftp will ask for (replace login and pass with just the login and password, without any keywords). Then, you use mget * to get all files. Instead of the *, you can use any wildcard. Finally, you use quit to close the ftp program without interactive prompt.
If you needed to enter some directory first, add a cd command before mget. It should be pretty straightforward.
Finally, write that file and run ftp like this:
ftp -i -s:yourscript
where -i disables interactivity (asking before downloading files), and -s specifies path to the script you created.
Sadly, file transfer over SSH is not natively supported in Windows. But for that case, you'd probably want to use PuTTy tools anyway. The one of particular interest for this case would be pscp which is practically the PuTTy counter-part of the openssh scp command.
The syntax is similar to copy command, and it supports wildcards:
pscp -batch [email protected]:iiumlabs* .
If you authenticate using a key file, you should pass it using -i path-to-key-file. If you use password, -pw pass. It can also reuse sessions saved using PuTTy, using the load -load your-session-name argument.
pscp work just fine (it's practically a scp counter-part). I'll add the simple way of using it to the answer. –
Quincy If you're not bound to curl, you might want to use wget in recursive mode but restricting it to one level of recursion, try the following;
wget --no-verbose --no-parent --recursive --level=1 \
--no-directories --user=login --password=pass ftp://ftp.myftpsite.com/
--no-parent : Do not ever ascend to the parent directory when retrieving recursively.--level=depth : Specify recursion maximum depth level depth. The default maximum depth is five layers.--no-directories : Do not create a hierarchy of directories when retrieving recursively.--delete-after : can be added if you need to delete files after downloading.--no-host-directories : to download right in '.' current folder, not create directory named by domain.--no-clobber : skip downloads that would download to existing files--continue : Continue getting a partially-downloaded file for more stabilitycd : to define the destination directorySo this sample can look like following:
cd /home/destination/folder \
&& wget --no-verbose --no-parent --recursive --level=1 \
--no-directories --no-host-directories \
--no-clobber --continue \
--user=login --password=pass ftp://ftp.myftpsite.com/
--delete-after can be added if you need to delete files after downloading. –
Impaction --no-host-directories to download right in '.' current folder, not create directory named by domain. –
Impaction --user="login" --password="pass" –
Impaction OK, considering that you are using Windows, the most simple way to do that is to use the standard ftp tool bundled with it. I base the following solution on Windows XP, hoping it'll work as well (or with minor modifications) on other versions.
First of all, you need to create a batch (script) file for the ftp program, containing instructions for it. Name it as you want, and put into it:
curl -u login:pass ftp.myftpsite.com/iiumlabs* -O
open ftp.myftpsite.com
login
pass
mget *
quit
The first line opens a connection to the ftp server at ftp.myftpsite.com. The two following lines specify the login, and the password which ftp will ask for (replace login and pass with just the login and password, without any keywords). Then, you use mget * to get all files. Instead of the *, you can use any wildcard. Finally, you use quit to close the ftp program without interactive prompt.
If you needed to enter some directory first, add a cd command before mget. It should be pretty straightforward.
Finally, write that file and run ftp like this:
ftp -i -s:yourscript
where -i disables interactivity (asking before downloading files), and -s specifies path to the script you created.
Sadly, file transfer over SSH is not natively supported in Windows. But for that case, you'd probably want to use PuTTy tools anyway. The one of particular interest for this case would be pscp which is practically the PuTTy counter-part of the openssh scp command.
The syntax is similar to copy command, and it supports wildcards:
pscp -batch [email protected]:iiumlabs* .
If you authenticate using a key file, you should pass it using -i path-to-key-file. If you use password, -pw pass. It can also reuse sessions saved using PuTTy, using the load -load your-session-name argument.
pscp work just fine (it's practically a scp counter-part). I'll add the simple way of using it to the answer. –
Quincy What about something like this:
for /f %%f in ('curl -s -l -u user:pass ftp://ftp.myftpsite.com/') do curl -O -u user:pass ftp://ftp.myftpsite.com/%%f
You can use script like this for mac:
for f in $(curl -s -l -u user:pass ftp://your_ftp_server_ip/folder/)
do curl -O -u user:pass ftp://your_ftp_server_ip/folder/$f
done
Here is how I did to download quickly with cURL (I'm not sure how many files it can download though) :
setlocal EnableDelayedExpansion
cd where\to\download
set STR=
for /f "skip=2 delims=" %%F in ('P:\curl -l -u user:password ftp://ftp.example.com/directory/anotherone/') do set STR=-O "ftp://ftp.example.com/directory/anotherone/%%F" !STR!
path\to\curl.exe -v -u user:password !STR!
Why skip=2 ?
To get ride of . and ..
Why delims= ? To support names with spaces
Oh, I have just the thing you need!
$host = "ftp://example.com/dir/";
$savePath = "downloadedFiles";
if($check = isFtpUp($host)){
echo $ip." -is alive<br />";
$check = trim($check);
$files = explode("\n",$check);
foreach($files as $n=>$file){
$file = trim($file);
if($file !== "." || $file !== ".."){
if(!saveFtpFile($file, $host.$file, $savePath)){
// downloading failed. possible reason: $file is a folder name.
// echo "Error downloading file.<br />";
}else{
echo "File: ".$file." - saved!<br />";
}
}else{
// do nothing
}
}
}else{
echo $ip." - is down.<br />";
}
and functions isFtpUp and saveFtpFile are as follows:
function isFtpUp($host){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $host);
curl_setopt($ch, CURLOPT_USERPWD, "anonymous:[email protected]");
curl_setopt($ch, CURLOPT_FTPLISTONLY, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 3);
$result = curl_exec($ch);
return $result;
}
function saveFtpFile( $targetFile = null, $sourceFile = null, $savePath){
// function settings
set_time_limit(60);
$timeout = 60;
$ftpuser = "anonymous";
$ftppassword = "[email protected]";
$savePath = "downloadedFiles"; // should exist!
$curl = curl_init();
$file = @fopen ($savePath.'/'.$targetFile, 'w');
if(!$file){
return false;
}
curl_setopt($curl, CURLOPT_URL, $sourceFile);
curl_setopt($curl, CURLOPT_USERPWD, $ftpuser.':'.$ftppassword);
// curl settings
// curl_setopt($curl, CURLOPT_FAILONERROR, 1);
// curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_TIMEOUT, $timeout);
curl_setopt($curl, CURLOPT_FILE, $file);
$result = curl_exec($curl);
if(!$result){
return false;
}
curl_close($curl);
fclose($file);
return $result;
}
EDIT:
it's a php script. save it as a .php file, put it on your webserver, change $ip to address(need not be ip) of ftp server you want to download files from, create a directory named downloadedFiles on the same directory as this file.
If you're not bound to curl, you might want to use lftp , try the following:
lftp ftp://ftp.some.domain.com -u [email protected],'password' -e 'set ftp:ssl-allow no; mirror --Remove-source-files --verbose /remote/directory /home/destination/folder; bye'
Where
-u username,'password' : put your password rather in quotas--Remove-source-files : deete remove files after you downloaded them--verbose : good to see info but feel free to delete on productionset ftp:ssl-allow no; : only in case if you do not need to check certificate (not ssl)© 2022 - 2024 — McMap. All rights reserved.
curl? To be honest, it's one of the most hard-to-use tools I've ever seen. What is your platform? Because in the post you mention bash, and in the answer comments you mentioned .bat files... Also, how issftprelevant? And does your example actually use FTP? Because there's noftp://there so I'm pretty sure curl used HTTP... – Quincy