How to script FTP upload and download
Asked Answered
L

9

40

I'm attempting to make a batch file to upload a file to an FTP server. If I type it in manually it works fine, but when I run the batch file it halts after it's connected... It says:

connected to domain.com.

220 microsoft ftp server

User(domain.com:(none)):

And then nothing else. What is going on here?

Below is my batch file:

ftp www.domainhere.com

user useridhere

passwordhere

put test.txt

bye

pause
Lisbethlisbon answered 1/6, 2009 at 18:31 Comment(4)
This hardly even belongs on server fault. Better wait for superuser.com :)Manakin
well there is a command line and ftp category...Lisbethlisbon
Seems like a reasonable question to me.Daughtry
Similar: How can I upload (FTP) files to server in a Bash script?Pyelonephritis
D
71

It's a reasonable idea to want to script an FTP session the way the original poster imagined, and that is the kind of thing Expect would help with. Batch files on Windows cannot do this.

But rather than doing cURL or Expect, you may find it easier to script the FTP interaction with PowerShell. It's a different model, in that you are not directly scripting the text to send to the FTP server. Instead you will use PowerShell to manipulate objects that generate the FTP dialogue for you.

Upload:

$File = "D:\Dev\somefilename.zip"
$ftp = "ftp://username:[email protected]/pub/incoming/somefilename.zip"

"ftp url: $ftp"

$webclient = New-Object System.Net.WebClient
$uri = New-Object System.Uri($ftp)

"Uploading $File..."

$webclient.UploadFile($uri, $File)

Download:

$File = "c:\store\somefilename.zip"
$ftp = "ftp://username:[email protected]/pub/outbound/somefilename.zip"

"ftp url: $ftp"

$webclient = New-Object System.Net.WebClient
$uri = New-Object System.Uri($ftp)

"Downloading $File..."

$webclient.DownloadFile($uri, $File)

You need PowerShell to do this. If you are not aware, PowerShell is a shell like cmd.exe which runs your .bat files. But PowerShell runs .ps1 files, and is quite a bit more powerful. PowerShell is a free add-on to Windows and will be built-in to future versions of Windows. Get it here.

Source: http://poshcode.org/1134

Daughtry answered 1/6, 2009 at 21:42 Comment(9)
Hrrm, looks like good idea seeing how I can't get a batch to run the way i want it... I just finished installing it.. Not exactly sure what I'm doing but I'll give it a few minutes to sink in. Will I be able to incorporate windows scheduler with this to run my script in set intervals?Lisbethlisbon
ok, I've got it to upload and download correctly. How would I go about automating the process?Lisbethlisbon
You're doing fine. If you are doing FTP from within powershell, then it is automated. What you want is to schedule it. Which means schtasks.exe. support.microsoft.com/kb/814596Daughtry
haha, was waiting to get a couple more answers out of you first! =0 I figured it all out, I just used a batch file with powershell.exe /pathto/update.ps1 then put windows schedular on the bat file. Not sure if that's the best way but it's working! Thanks for all your help cheeso!Lisbethlisbon
the way to run powershell on a schedule is to specify this command line to the scheduler: c:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe <fullpath-to-ps1-file> <args-to-ps1-file>Daughtry
How would you modify this to upload several files with different extensions?Flask
Is there a way to use wild cards in the filename? I want to download SEP virus definition updates, and the file name format is YYYYMMDD-*-v5i32.exe where YYYYMMDD is the latest date and * is a 3 digit number you never know what it will be.Caution
Note that the windows ftp program can take a command file, which might be an easier solution if you want to stay with simple batch files (see the other answer about that https://mcmap.net/q/246661/-how-to-script-ftp-upload-and-download )Larue
How would you specify to download the file in ascii mode? I'm currently retrieving some sort of binary/non-text file when retrieving a plaintext file from a particular FTP server.Rashid
F
27

Create a command file with your commands.

I.e., file commands.txt:

open www.domainhere.com
user useridhere
passwordhere
put test.txt
bye

Then run the FTP client from the command line:

ftp -s:commands.txt

Note: This will work for the Windows FTP client.

Facing answered 1/6, 2009 at 18:31 Comment(3)
doesn't work, says password is srequire, tried it a couple different waysLisbethlisbon
@Lisbethlisbon oh c'mon! You just need to skip the user before username, and it works like a dream.Abysm
If only I had more upvotes ;). BTW, on Linux you can do this with .netrc and piping commands into ftp.Abysm
H
6

Batch files don't work that way. They don't just "type" everything - they run system commands, in this case ftp, wait for them to return, and run the next command... so in this case, the interpreter is simply waiting for ftp to exit.

If you must use the ftp command, then prepare a script file (for example, commands.txt and run ftp -s:commands.txt.

But using cURL, or a PHP/Perl/Python/whatever script may be a better idea.

Hibbard answered 1/6, 2009 at 19:9 Comment(0)
F
6

I've done this with PowerShell:

function DownloadFromFtp($destination, $ftp_uri, $user, $pass){
    $dirs = GetDirecoryTree $ftp_uri $user $pass

    foreach($dir in $dirs){
       $path = [io.path]::Combine($destination,$dir)

       if ((Test-Path $path) -eq $false) {
          "Creating $path ..."
          New-Item -Path $path -ItemType Directory | Out-Null
       }else{
          "Exists $path ..."
       }
    }

    $files = GetFilesTree $ftp_uri $user $pass

    foreach($file in $files){
        $source = [io.path]::Combine($ftp_uri,$file)
        $dest = [io.path]::Combine($destination,$file)

        "Downloading $source ..."
        Get-FTPFile $source $dest $user $pass
    }
}

function UploadToFtp($artifacts, $ftp_uri, $user, $pass){
    $webclient = New-Object System.Net.WebClient 
    $webclient.Credentials = New-Object System.Net.NetworkCredential($user,$pass)  

    foreach($item in Get-ChildItem -recurse $artifacts){ 

        $relpath = [system.io.path]::GetFullPath($item.FullName).SubString([system.io.path]::GetFullPath($artifacts).Length + 1)

        if ($item.Attributes -eq "Directory"){

            try{
                Write-Host Creating $item.Name

                $makeDirectory = [System.Net.WebRequest]::Create($ftp_uri+$relpath);
                $makeDirectory.Credentials = New-Object System.Net.NetworkCredential($user,$pass) 
                $makeDirectory.Method = [System.Net.WebRequestMethods+FTP]::MakeDirectory;
                $makeDirectory.GetResponse();

            }catch [Net.WebException] {
                Write-Host $item.Name probably exists ...
            }

            continue;
        }

        "Uploading $item..."
        $uri = New-Object System.Uri($ftp_uri+$relpath) 
        $webclient.UploadFile($uri, $item.FullName)
    }
}

 function Get-FTPFile ($Source,$Target,$UserName,$Password) 
 { 
     $ftprequest = [System.Net.FtpWebRequest]::create($Source) 
     $ftprequest.Credentials = New-Object System.Net.NetworkCredential($username,$password) 
     $ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::DownloadFile 
     $ftprequest.UseBinary = $true 
     $ftprequest.KeepAlive = $false 

     $ftpresponse = $ftprequest.GetResponse() 
     $responsestream = $ftpresponse.GetResponseStream() 

     $targetfile = New-Object IO.FileStream ($Target,[IO.FileMode]::Create) 
     [byte[]]$readbuffer = New-Object byte[] 1024 

     do{ 
         $readlength = $responsestream.Read($readbuffer,0,1024) 
         $targetfile.Write($readbuffer,0,$readlength) 
     } 
     while ($readlength -ne 0) 

     $targetfile.close() 
 } 

#task ListFiles {
#   
#    $files = GetFilesTree 'ftp://127.0.0.1/' "web" "web"
#    $files | ForEach-Object {Write-Host $_ -foregroundcolor cyan}
#}

function GetDirecoryTree($ftp, $user, $pass){
    $creds = New-Object System.Net.NetworkCredential($user,$pass)

    $files = New-Object "system.collections.generic.list[string]"
    $folders = New-Object "system.collections.generic.queue[string]"
    $folders.Enqueue($ftp)

    while($folders.Count -gt 0){
        $fld = $folders.Dequeue()

        $newFiles = GetAllFiles $creds $fld
        $dirs = GetDirectories $creds $fld

        foreach ($line in $dirs){
            $dir = @($newFiles | Where { $line.EndsWith($_) })[0]
            [void]$newFiles.Remove($dir)
            $folders.Enqueue($fld + $dir + "/")

            [void]$files.Add($fld.Replace($ftp, "") + $dir + "/")
        }
    }

    return ,$files
}

function GetFilesTree($ftp, $user, $pass){
    $creds = New-Object System.Net.NetworkCredential($user,$pass)

    $files = New-Object "system.collections.generic.list[string]"
    $folders = New-Object "system.collections.generic.queue[string]"
    $folders.Enqueue($ftp)

    while($folders.Count -gt 0){
        $fld = $folders.Dequeue()

        $newFiles = GetAllFiles $creds $fld
        $dirs = GetDirectories $creds $fld

        foreach ($line in $dirs){
            $dir = @($newFiles | Where { $line.EndsWith($_) })[0]
            [void]$newFiles.Remove($dir)
            $folders.Enqueue($fld + $dir + "/")
        }

        $newFiles | ForEach-Object { 
            $files.Add($fld.Replace($ftp, "") + $_) 
        }
    }

    return ,$files
}

function GetDirectories($creds, $fld){
    $dirs = New-Object "system.collections.generic.list[string]"

    $operation = [System.Net.WebRequestMethods+Ftp]::ListDirectoryDetails
    $reader = GetStream $creds $fld $operation
    while (($line = $reader.ReadLine()) -ne $null) {

       if ($line.Trim().ToLower().StartsWith("d") -or $line.Contains(" <DIR> ")) {
            [void]$dirs.Add($line)
        }
    }
    $reader.Dispose();

    return ,$dirs
}

function GetAllFiles($creds, $fld){
    $newFiles = New-Object "system.collections.generic.list[string]"
    $operation = [System.Net.WebRequestMethods+Ftp]::ListDirectory

    $reader = GetStream $creds $fld $operation

    while (($line = $reader.ReadLine()) -ne $null) {
       [void]$newFiles.Add($line.Trim()) 
    }
    $reader.Dispose();

    return ,$newFiles
}

function GetStream($creds, $url, $meth){

    $ftp = [System.Net.WebRequest]::Create($url)
    $ftp.Credentials = $creds
    $ftp.Method = $meth
    $response = $ftp.GetResponse()

    return New-Object IO.StreamReader $response.GetResponseStream()
}

Export-ModuleMember UploadToFtp, DownLoadFromFtp
Fibrovascular answered 1/4, 2011 at 22:36 Comment(0)
K
3

You can script the ftp command with the -s:filename option. The syntax is just a list of commands to pass to the ftp shell, each terminated by a newline. This page has a nice reference to the commands that can be performed with ftp.

Upload/Download Entire Directory Structure

Using the normal ftp doesn't work very well when you need to have an entire directory tree copied to or from a FTP site. So you could use something like these to handle those situations.

These scripts work with the Windows ftp command and allows for uploading and downloading of entire directories from a single command. This makes it pretty self-reliant when using on different systems.

Basically, they map out the directory structure to be up/downloaded, dump corresponding ftp commands to a file, and then execute those commands when the mapping has finished.

ftpupload.bat

@echo off

SET FTPADDRESS=%1
SET FTPUSERNAME=%2
SET FTPPASSWORD=%3
SET LOCALDIR=%~f4
SET REMOTEDIR=%5

if "%FTPADDRESS%" == "" goto FTP_UPLOAD_USAGE
if "%FTPUSERNAME%" == "" goto FTP_UPLOAD_USAGE
if "%FTPPASSWORD%" == "" goto FTP_UPLOAD_USAGE
if "%LOCALDIR%" == "" goto FTP_UPLOAD_USAGE
if "%REMOTEDIR%" == "" goto FTP_UPLOAD_USAGE

:TEMP_NAME
set TMPFILE=%TMP%\%RANDOM%_ftpupload.tmp
if exist "%TMPFILE%" goto TEMP_NAME

SET INITIALDIR=%CD%

echo user %FTPUSERNAME% %FTPPASSWORD% > %TMPFILE%
echo bin >> %TMPFILE%
echo lcd %LOCALDIR% >> %TMPFILE%

cd %LOCALDIR%

setlocal EnableDelayedExpansion
echo mkdir !REMOTEDIR! >> !TMPFILE!
echo cd %REMOTEDIR% >> !TMPFILE!
echo mput * >> !TMPFILE!
for /d /r %%d in (*) do (
    set CURRENT_DIRECTORY=%%d
    set RELATIVE_DIRECTORY=!CURRENT_DIRECTORY:%LOCALDIR%=!
    echo mkdir "!REMOTEDIR!/!RELATIVE_DIRECTORY:~1!" >> !TMPFILE!
    echo cd "!REMOTEDIR!/!RELATIVE_DIRECTORY:~1!" >> !TMPFILE!
    echo mput "!RELATIVE_DIRECTORY:~1!\*" >> !TMPFILE!
)

echo quit >> !TMPFILE!

endlocal EnableDelayedExpansion

ftp -n -i "-s:%TMPFILE%" %FTPADDRESS%

del %TMPFILE%

cd %INITIALDIR%

goto FTP_UPLOAD_EXIT

:FTP_UPLOAD_USAGE

echo Usage:    ftpupload [address] [username] [password] [local directory] [remote directory]
echo.

:FTP_UPLOAD_EXIT

set INITIALDIR=
set FTPADDRESS=
set FTPUSERNAME=
set FTPPASSWORD=
set LOCALDIR=
set REMOTEDIR=
set TMPFILE=
set CURRENT_DIRECTORY=
set RELATIVE_DIRECTORY=

@echo on

ftpget.bat

@echo off

SET FTPADDRESS=%1
SET FTPUSERNAME=%2
SET FTPPASSWORD=%3
SET LOCALDIR=%~f4
SET REMOTEDIR=%5
SET REMOTEFILE=%6

if "%FTPADDRESS%" == "" goto FTP_UPLOAD_USAGE
if "%FTPUSERNAME%" == "" goto FTP_UPLOAD_USAGE
if "%FTPPASSWORD%" == "" goto FTP_UPLOAD_USAGE
if "%LOCALDIR%" == "" goto FTP_UPLOAD_USAGE
if not defined REMOTEDIR goto FTP_UPLOAD_USAGE
if not defined REMOTEFILE goto FTP_UPLOAD_USAGE

:TEMP_NAME
set TMPFILE=%TMP%\%RANDOM%_ftpupload.tmp
if exist "%TMPFILE%" goto TEMP_NAME

echo user %FTPUSERNAME% %FTPPASSWORD% > %TMPFILE%
echo bin >> %TMPFILE%
echo lcd %LOCALDIR% >> %TMPFILE%

echo cd "%REMOTEDIR%" >> %TMPFILE%
echo mget "%REMOTEFILE%" >> %TMPFILE%
echo quit >> %TMPFILE%

ftp -n -i "-s:%TMPFILE%" %FTPADDRESS%

del %TMPFILE%

goto FTP_UPLOAD_EXIT

:FTP_UPLOAD_USAGE

echo Usage:    ftpget [address] [username] [password] [local directory] [remote directory] [remote file pattern]
echo.

:FTP_UPLOAD_EXIT

set FTPADDRESS=
set FTPUSERNAME=
set FTPPASSWORD=
set LOCALDIR=
set REMOTEFILE=
set REMOTEDIR=
set TMPFILE=
set CURRENT_DIRECTORY=
set RELATIVE_DIRECTORY=

@echo on
Kerrikerrie answered 22/4, 2015 at 19:28 Comment(0)
H
1

I was having a similar issue - like the original poster, I wanted to automate a file upload, but I couldn't figure out how. Because this is on a register terminal at my family's store, I didn't want to install PowerShell (although that looks like an easy option) and I just wanted a simple .bat file to do this.

This is pretty much what grawity and another user said; I'm new to this stuff, so here's a more detailed example and explanation (thanks also to How to Automate FTP Uploads from the Windows Command Line who explains how to do it with just one .bat file).

Essentially you need two files - one .bat and one .txt. The .bat tells ftp.exe what switches to use. The .txt gives a list of commands to ftp.exe. In the text file put this:

username
password
cd whereverYouWantToPutTheFile
lcd whereverTheFileComesFrom
put C:\InventoryExport\inventory.test (or your file path)
bye

Save that wherever you want. In the BAT file put:

ftp.exe -s:C:\Windows\System32\test.txt destinationIP
pause

Obviously change the path after the -s: to wherever your text file is. Take out the pause when you're actually running it - it's just so you can see any errors. Of course, you can use "get" or any other ftp command in the .txt file to do whatever you need to do.

I'm not positive that you need the lcd command in the text file. Like I said, I'm new to using command line for this type of thing, but this is working for me.

Howes answered 7/8, 2013 at 18:2 Comment(0)
C
1

This script generates the command file then pipes the command file to the ftp program, creating a log along the way. Finally print the original bat file, the command files and the log of this session.

@echo on
@echo off > %0.ftp
::== GETmy!dir.bat
>> %0.ftp echo a00002t
>> %0.ftp echo iasdad$2
>> %0.ftp echo help
>> %0.ftp echo prompt
>> %0.ftp echo ascii
>> %0.ftp echo !dir REPORT.CP1C.ROLLEDUP.TXT
>> %0.ftp echo get REPORT.CP1C.ROLLEDUP.TXT
>> %0.ftp echo !dir REPORT.CP1C.ROLLEDUP.TXT
>> %0.ftp echo *************************************************   
>> %0.ftp echo !dir CONTENT.CP1C.ROLLEDUP.TXT
>> %0.ftp echo get CONTENT.CP1C.ROLLEDUP.TXT
>> %0.ftp echo !dir CONTENT.CP1C.ROLLEDUP.TXT
>> %0.ftp echo *************************************************   
>> %0.ftp echo !dir WORKLOAD.CP1c.ROLLEDUP.TXT
>> %0.ftp echo get WORKLOAD.CP1C.ROLLEDUP.TXT
>> %0.ftp echo !dir WORKLOAD.CP1C.ROLLEDUP.TXT
>> %0.ftp echo *************************************************   
>> %0.ftp echo !dir REPORT.TMMC.ROLLEDUP.TXT
>> %0.ftp echo get REPORT.TMMC.ROLLEDUP.TXT
>> %0.ftp echo !dir REPORT.TMMC.ROLLEDUP.TXT
>> %0.ftp echo *************************************************   
>> %0.ftp echo !dir CONTENT.TMMC.ROLLEDUP.TXT
>> %0.ftp echo get CONTENT.TMMC.ROLLEDUP.TXT
>> %0.ftp echo !dir CONTENT.TMMC.ROLLEDUP.TXT
>> %0.ftp echo **************************************************   
>> %0.ftp echo !dir WORKLOAD.TMMC.ROLLEDUP.TXT
>> %0.ftp echo get WORKLOAD.TMMC.ROLLEDUP.TXT
>> %0.ftp echo !dir WORKLOAD.TMMC.ROLLEDUP.TXT
>> %0.ftp echo quit
ftp -d -v -s:%0.ftp 150.45.12.18 > %0.log
type %0.bat 
type %0.ftp 
type %0.log 
Carlita answered 25/11, 2013 at 19:55 Comment(0)
R
0

I had this same issue, and solved it with a solution similar to what Cheeso provided.

"doesn't work, says password is srequire, tried it a couple different ways "

Yep, that's because FTP sessions via a command file don't require the username to be prefaced with the string "user". Drop that, and try it.

Or, you could be seeing this because your FTP command file is not properly encoded (that bit me, too). That's the crappy part about generating a FTP command file at runtime. PowerShell's out-file cmdlet does not have an encoding option that Windows FTP will accept (at least not one that I could find).

Regardless, as doing a WebClient.DownloadFile is the way to go.

Rebbecarebbecca answered 28/9, 2009 at 17:1 Comment(1)
You are getting that problem because OUT-FILE by default uses the Unicode encoding. This will add two bytes to the beginning of the file, and this causes problems as the two bytes are passed to the ftp input stream. Overcome it by using OUT-FILE -Encoding ASCII ...Brest
U
-1

Try manually:

$ ftp www.domainhere.com 
> useridhere
> passwordhere
> put test.txt
> bye
> pause
Unmarked answered 2/9, 2009 at 18:8 Comment(1)
This answer is no different than in the question it-self, so it should be removed.Unpolitic

© 2022 - 2024 — McMap. All rights reserved.