Bad idea to transfer large payload using web services?

Question

I gather that there basically isn't a limit to the amount of data that can be sent when using REST via a POST or GET. While I haven't used REST or web services it seems that most services involve transferring limited amounts of data. If you want to transfer 1-5MB worth of data (in either direction) are web services considered a bad idea?

Update: The apps that we are considering connecting via a REST service are internal apps. We do have the option of picking other connectivity options (ie: RMI)

Few

Don't forget to look into compression - especially if you are sending large XML/text files around. An 800MB XML file will compress (with gzip) down to about 30MB. – Dundalk

Answer 1

1-5mb using rest isn't really that large of a dataset. The limiting factor is likely memory. Depending on how you are generating the xml, you could run low on server resources if you have high traffic on the service.

If you are using streaming xml generation memory won't be as much of a problem. creating a dom tree and then spitting it out will be much more memory intensive.

But even if your dom tree took 10x the memory of the raw xml, you would need 40 conncurrent connections fillup 2gb of server memory. So it may not be an issue in your situation.

Less likely is memory issues on the client side. but 5 de-serializing an 5mb xml dump shouldn't be a problem on any normal computer.

Tyus

Answer 2

1-5 MB is okay, as long as you provide the user with some sort of progress feedback. Webservices run over HTTP, which is okay-ish when it comes to overhead on larger datasets, the real problem is XML. XML, more often than not, adds a huge overhead to the size of the serialized data. I'd consider JSON or Hessian for larger datasets :)

Silsby