Upload files from folders and sub-folders to webapp

Question

Goal: Allow user to select a folder, then find all files recursively that matches a file pattern and transfer(POST) to my web server

In essence just a more advanced upload dialog...

Standard web technology (we're using plupload) doesn't support this, due to security reasons, afaik.

Additional requirements: Easy to use/install from the webapp. SSL and and app user credentials are needed. Some other data like record ID (get or create from web app) to associate the upload files would be nice.

The web app itself is written in Ruby on Rails, but that shouldn't really matter if I need some kind of a native Mac and Windows (=80% of my users) desktop client.

What are my options?

Code and references to open source libs for doing this is a bonus.

Carlton

"find all files recursively that matches a file pattern" Like, are you trying to upload all pngs (but possibly not jpgs) from a folder? What are you actually trying to process? – Morea

1st: correct. 2nd: what do you mean, what file type? *.dcm – Carlton

Answer 1

I would recommend creating an Adobe Air app.

You can reuse your existing plupload js code, and extend it with adobe air apis. Ideally, it would be a mostly static app but make a server call to create the policy document, and do any book-keeping you want to do on the uploads. Take a peak at Accessing AIR API classes from JavaScript. Then look into the filesystem.File class.

Flash has something like a 99.3% penetration, and with it the user just clicks "Install My Uploader" and the air framework is automatically installed if needed. Air is also on Android and iDevices, so your app would probably be available to 99.999% of users and their moms.

It took me a minute to find it, but here is the Adobe® AIR® API Reference for HTML Developers

Oh look what I found in their examples:

var directory = air.File.documentsDirectory;

try
{
    directory.browseForDirectory("Select Directory");
    directory.addEventListener(air.Event.SELECT, directorySelected);
}
catch (error)
{
    air.trace("Failed:", error.message)
}

function directorySelected(event) 
{
    directory = event.target ;
    var files = directory.getDirectoryListing();
    for(var i = 0; i < files.length; i++)
    {
        air.trace(files[i].name);
    }
}

Morea

Answer 2

I would build a client starting from something like this: https://github.com/ms4720/s3sync so that you can keep it in the ruby family. If you need a friendly gui: http://shoesrb.com/tutorials/

Epimenides

This wouldn't work for the regular users. Ruby is not necessarily installed nor can S3 credentials be given to the public. In essence I just need a more advanced upload dialog – Carlton

Answer 3

I still think an Adobe Air app is a solid solution to your problem. Provides easy install/updates, supports ssl, allows you full control over the user experience, 110% cross platform, etc... However if you hate Adobe or writing UI code, have you looked into WebDav / the dav4rack gem?

https://github.com/bryanrite/dav4rack-example-devise-subdirectories

That GitHub example uses devise in a rails app for authentication, and used the same credentials for a user to access a private bucket where they can upload files through any WebDav client. Your web app could provide a link that includes the username in the URL to make access more convenient.

OS X's Finder natively supports WebDav through the Finder's "Connect to Server" dialog (Since 10.1.1).

If your user uses Linux, I'm sure they can figure WebDav out.

From scanning http://en.wikipedia.org/wiki/WebDAV it looks like Windows supported WebDav in Win98. However, along with every other web standard, Microsoft has decided to be, um, creative. It sounds like if the user is up to date with KB892211, KB907306, Office, Internet Explorer, OLE-DB and/or "Microsoft Update for Web Folders" they should be able to access WebDav folders over SSL with various authentication schemes with ease. I'm sure all of your Windows users are using Genuine Windows with all the latest updates and a complete Office Suite.

Since your rails app is the WebDav Server, you will be able to throw errors (Can't write that file to the server), post-process uploaded files/directories, and do whatever bookkeeping you need (you have the user's account, along with what they uploaded). Dav4rack is pretty nice. You can use an actual filesystem as a backend, or a database, or bounce the uploaded files to S3, or just about anything you can do in ruby.

I don't think this solution addresses your "find all files recursively that matches a file pattern" component as well as an Air app could. Without real-world experience, I'd imagine that the pattern matching would occur after the upload completes (potentially bandwidth inefficient).

Morea

you really want this bounty, Jeremy, haha. Not sure what I think about WedDav though, but thanks for sharing. When a Q is very important to me, I add bounty to try get as many angles as possible. You'll earn it for the adobe AIR answer. Thanks again. – Carlton

My company's been kinda considering uploading tools. I made one as part of a rails app that pushed directly to S3 (not an Air app), but it turned out that there were quite a few quirks with that. We got it working, but only with silverlight. Flash couldn’t get upload progress, there were some issues with corrupt files. HTML5 fails, HTML4 kinda worked w/o progress, and I think it needed custom headers. It got to the point where I pretty much feel direct to S3 isn't ready for production. I've been thinking about reworking things and building more Air apps, but it's on to the next project for me! – Morea