C# class to parse WebRequestMethods.Ftp.ListDirectoryDetails FTP response

Question

I'm creating a service to monitor FTP locations for new updates and require the ability to parse the response returned from a FtpWebRequest response using the WebRequestMethods.Ftp.ListDirectoryDetails method. It would be fairly easy if all responses followed the same format, but different FTP server software provide different response formats.

For example one might return:

08-10-11  12:02PM       <DIR>          Version2
06-25-09  02:41PM            144700153 image34.gif
06-25-09  02:51PM            144700153 updates.txt
11-04-10  02:45PM            144700214 digger.tif

And another server might return:

d--x--x--x    2 ftp      ftp          4096 Mar 07  2002 bin
-rw-r--r--    1 ftp      ftp        659450 Jun 15 05:07 TEST.TXT
-rw-r--r--    1 ftp      ftp      101786380 Sep 08  2008 TEST03-05.TXT
drwxrwxr-x    2 ftp      ftp          4096 May 06 12:24 dropoff

And other differences have been observed also so there's likely to be a number of subtle differences I haven't encountered yet.

Does anyone know of a fully managed (doesn't require access to external dll on Windows) C# class that handles these situations seamlessly?

I only need to list the contents of a directory with the following details: File/directory name, last updated or created timestamp, file/directory name.

Thanks in advance for any suggestions, Gavin

Brutus

Answer 1

One solution I came across is EdtFTPnet

EdtFTPnet seems to be quite a feature packed solution that handles lots of different FTP options so is ideal.

It's the free open source solution that I've how employed for http://www.ftp2rss.com (a little tool I needed myself but figured might be useful to others also).

Brutus

Answer 2

For the first (DOS/Windows) listing this code will do:

FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://ftp.example.com/");
request.Credentials = new NetworkCredential("user", "password");
request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
StreamReader reader = new StreamReader(request.GetResponse().GetResponseStream());

string pattern = @"^(\d+-\d+-\d+\s+\d+:\d+(?:AM|PM))\s+(<DIR>|\d+)\s+(.+)$";
Regex regex = new Regex(pattern);
IFormatProvider culture = CultureInfo.GetCultureInfo("en-us");
while (!reader.EndOfStream)
{
    string line = reader.ReadLine();
    Match match = regex.Match(line);
    string s = match.Groups[1].Value;
    DateTime modified =
        DateTime.ParseExact(s, "MM-dd-yy  hh:mmtt", culture, DateTimeStyles.None);
    s = match.Groups[2].Value;
    long size = (s != "<DIR>") ? long.Parse(s) : 0;
    string name = match.Groups[3].Value;

    Console.WriteLine(
        "{0,-16} size = {1,9}  modified = {2}",
        name, size, modified.ToString("yyyy-MM-dd HH:mm"));
}

You will get:

Version2         size =         0  modified = 2011-08-10 12:02
image34.gif      size = 144700153  modified = 2009-06-25 14:41
updates.txt      size = 144700153  modified = 2009-06-25 14:51
digger.tif       size = 144700214  modified = 2010-11-04 14:45

---

For the other (*nix) listing, see my answer to Parsing FtpWebRequest ListDirectoryDetails line.

---

But, actually trying to parse the listing returned by the ListDirectoryDetails is not the right way to go.

You want to use an FTP client that supports the modern MLSD command that returns a directory listing in a machine-readable format specified in the RFC 3659. Parsing the human-readable format returned by the ancient LIST command (used internally by the FtpWebRequest for its ListDirectoryDetails method) should be used as the last resort option, when talking to obsolete FTP servers, that do not support the MLSD command (like the Microsoft IIS FTP server).

---

For example with WinSCP .NET assembly, you can use its Session.ListDirectory or Session.EnumerateRemoteFiles methods.

They internally use the MLSD command, but can fall back to the LIST command and support dozens of different human-readable listing formats.

The returned listing is presented as collection of RemoteFileInfo instances with properties like:

• Name
• LastWriteTime (with correct timezone)
• Length
• FilePermissions (parsed into individual rights)
• Group
• Owner
• IsDirectory
• IsParentDirectory
• IsThisDirectory

(I'm the author of WinSCP)

---

Most other 3rd party libraries will do the same. Using the FtpWebRequest class is not reliable for this purpose. Unfortunately, there's no other built-in FTP client in the .NET framework.

Orford

Martin, you should be charging for this product. I just donated. You saved me HOURS of tedium. Thank you. – Cutty

Answer 3

I'm facing this same problem and have built a simple (albeit not very robust) solution using a Regex to parse out the relevant information from each line using capture groups:

public static Regex FtpListDirectoryDetailsRegex = new Regex(@".*(?<month>(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec))\s*(?<day>[0-9]*)\s*(?<yearTime>([0-9]|:)*)\s*(?<fileName>.*)", RegexOptions.Compiled | RegexOptions.IgnoreCase);

You can then extract the values out of the capture groups by:

        string ftpResponse = "-r--r--r-- 1 ftp ftp              0 Nov 19 11:08 aaa.txt";
        Match match = FtpListDirectoryDetailsRegex.Match(ftpResponse);
        string month = match.Groups["month"].Value;
        string day = match.Groups["day"].Value;
        string yearTime = match.Groups["yearTime"].Value;
        string fileName = match.Groups["fileName"].Value;

Some things not note are:

• this will only work for directory responses with the format described found in the ftpResponse variable above. In my case I'm lucky to only be accessing the same FTP server each time and so it is unlikely that the response format will change.
• the yearTime variable can represent EITHER the year or the time of the file's timestamp. You will need to parse this manually by looking for an instance of the colon : character which will indicate that this capture group contains a time rather than the year

Deserve

This works great, but one scenario breaks it. If for instance one of the months is actually in the file name then it is unable to grab the fields properly. 975401 Apr 14 10:20 1JUNSGCIE6UN.jpg is an example of a file with the date April 14 but since JUN is in the file name it doesn't pull right. Any suggestions? I'm absolutely horrendous with regex strings. Thanks! – Taft

Hi. When yearTime is the time, it means that the year is always the current year? – Clustered

Answer 4

Take a look at Ftp.dll FTP client.

It includes automatic directory listing parser for most FTP servers on Windows, Unix and Netware platforms.

Please note that this is a commercial product I developed.

Nam

Cheers Pawel, much appreciated. Will stick to the free EdtFTPnet version for now unless I encounter some limits with it's free version. – Brutus