Download most recent file from FTP using PowerShell

Question

I am working on a PowerShell script, which will pull files from an FTP site. The files are uploaded to the FTP site every hour so I need to download the most recent one. The code I currently have downloads all the files from today instead of just one file. How do I make it download only the most recent file?

Here is the code that I am currently using

$ftpPath = 'ftp://***.***.*.*'
$ftpUser = '******'
$ftpPass = '******'
$localPath = 'C:\Temp'
$Date = get-date -Format "ddMMyyyy"
$Files = 'File1', 'File2'

function Get-FtpDir ($url, $credentials)
{
  $request = [Net.FtpWebRequest]::Create($url)
  if ($credentials) { $request.Credentials = $credentials }
  $request.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
  (New-Object IO.StreamReader $request.GetResponse().GetResponseStream()) -split "`r`n" 

}

$webclient = New-Object System.Net.WebClient 
$webclient.Credentials = New-Object System.Net.NetworkCredential($ftpUser,$ftpPass)  
$webclient.BaseAddress = $ftpPath

Foreach ( $item in $Files )
{
    Get-FTPDir $ftpPath $webclient.Credentials |
      ? { $_ -Like $item+$Date+'*' } |
      % {

          $webClient.DownloadFile($_, (Join-Path $localPath $_)) 
      }
}

Readus

Answer 1

It's not easy with the FtpWebRequest. For your task, you need to know file timestamps.

Unfortunately, there's no really reliable and efficient way to retrieve timestamps using features offered by FtpWebRequest/.NET framework/PowerShell as they do not support an FTP MLSD command. The MLSD command provides listing of remote directory in a standardized machine-readable format. The command and the format is standardized by RFC 3659.

Alternatives which you can use, that are supported by .NET framework:

• ListDirectoryDetails method (an FTP LIST command) to retrieve details of all files in a directory and then you deal with FTP server specific format of the details (*nix format similar to ls *nix command is the most common, drawback is that the format may change over time, as for newer files "May 8 17:48" format is used and for older files "Oct 18 2009" format is used)
• GetDateTimestamp method (an FTP MDTM command) to individually retrieve timestamps for each file. Advantage is that the response is standardized by RFC 3659 to YYYYMMDDHHMMSS[.sss]. Disadvantage is that you have to send a separate request for each file, what can be quite inefficient.

Some references:

• C# class to parse WebRequestMethods.Ftp.ListDirectoryDetails FTP response
• Parsing FtpWebRequest ListDirectoryDetails line
• Retrieving creation date of file (FTP)

---

Alternatively, use a 3rd party FTP library that supports the MLSD command, and/or supports parsing of the proprietary listing format.

For example WinSCP .NET assembly supports both.

An example code:

# Load WinSCP .NET assembly
Add-Type -Path "WinSCPnet.dll"

# Setup session options
$sessionOptions = New-Object WinSCP.SessionOptions -Property @{
    Protocol = [WinSCP.Protocol]::Ftp
    HostName = "example.com"
    UserName = "user"
    Password = "mypassword"
}

$session = New-Object WinSCP.Session

# Connect
$session.Open($sessionOptions)

# Get list of files in the directory
$directoryInfo = $session.ListDirectory($remotePath)

# Select the most recent file
$latest =
    $directoryInfo.Files |
    Where-Object { -Not $_.IsDirectory } |
    Sort-Object LastWriteTime -Descending |
    Select-Object -First 1

# Any file at all?
if ($latest -eq $Null)
{
    Write-Host "No file found"
    exit 1
}

# Download the selected file
$sourcePath = [WinSCP.RemotePath]::EscapeFileMask($remotePath + $latest.Name)
$session.GetFiles($sourcePath, $localPath).Check()

For a full code, see Downloading the most recent file (PowerShell).

(I'm the author of WinSCP)

Cosecant

WinSCP is a great utility and very reliable :) – Makebelieve

WinSCP rocks. For FTP transfer with PowerShell it is quite efficient. – Pellet

Answer 2

I tried this, but i get an error:

Error: Exception calling "ListDirectory" with "1" argument(s): "Error listing directory '/path/'.
Could not retrieve directory listing
Can't open data connection for transfer of "/path/"

I read a lot about this problem on the internet, but could not find a solution which seemed fairly simple, and I am not a network setup wizard. So I choose a different approach. In our case the filename of the file which I want to automate the download for, has the date specified in it: backup_2018_08_03_020003_1048387.bak

So we can get the file by using mget *2018_08_03* in a command line ftp session.

Our backup procedure is run every morning at 01.00 AM, so we have a backup each day that we can fetch.

Of course it would have been prettier and nicer to have a script that fetched the latest backup file based on the backup file timestamps, just in case that something went wrong with the latest backup or the backup file naming format changes. The script is just a script to fetch the backup for internal development purposes so its not a big deal if it breaks. I will look into this later and check whether i can make a cleaner solution.

I made a batch script which just asks for todays backup file with the ordinary ftp command prompt scripting.

It is important to get the formatting of todays date right. It must match the formatting of the date in the filename correctly.

If you want to use the script you should replace the variables with your own information. You should also have write access to the directory where you run it from.

This is the script that I made:

@Echo Off
Set _FTPServerName=xxx.xxx.xx.xxx
Set _UserName=Username
Set _Password=Password
Set _LocalFolder=C:\Temp
Set _RemoteFolder="/path/"
Set _Filename=*%date:~-4,4%_%date:~-7,2%_%date:~-10,2%*
Set _ScriptFile=ftptempscript
:: Create script
 >"%_ScriptFile%" Echo open %_FTPServerName%
>>"%_ScriptFile%" Echo %_UserName%
>>"%_ScriptFile%" Echo %_Password%
>>"%_ScriptFile%" Echo lcd %_LocalFolder%
>>"%_ScriptFile%" Echo cd %_RemoteFolder%
>>"%_ScriptFile%" Echo binary
>>"%_ScriptFile%" Echo mget -i %_Filename%
>>"%_ScriptFile%" Echo quit
:: Run script
ftp -s:"%_ScriptFile%"
del "%_ScriptFile%"

Neuralgia