Trouble Uploading Large Files to RStudio using Louis Aslett's AMI on EC2

Question

After following this simple tutorial http://www.louisaslett.com/RStudio_AMI/ and video guide http://www.louisaslett.com/RStudio_AMI/video_guide.html I have setup an RStudio environment on EC2.

The only problem is, I can't upload large files (> 1GB).

I can upload small files just fine.

When I try to upload a file via RStudio, it gives me the following error:

Unexpected empty response from server

Does anyone know how I can upload these large files for use in RStudio? This is the whole reason I am using EC2 in the first place (to work with big data).

Kasper

Answer 1

Don't you have shell access to your Amazon server? Don't rely on RStudio's upload (which may have a 2Gb limit, reasonably) and use proper unix dev tools:

rsync -avz myHugeFile.dat [email protected]:

on your local PC command line (install cygwin or other unixy compatibility system) will transfer your huge file to your amazon server, and if interrupted will resume from that point, will compress the data for transfer too.

For a windows gui on something like this, WinSCP was what we used to do in the bad old days before Linux.

Daveta

Answer 2

Ok so I had the same problem myself and it was incredibly frustrating, but eventually I realised what was going on here. The default home directory size for AWS is less than 8-10GB regardless of the size of your instance. As this as trying to upload to home then there was not enough room. An experienced linux user would not have fallen into this trap, but hopefully any other windows users new to this who come across this problem will see this. If you upload into a different drive on the instance then this can be solved. As the Louis Aslett Rstudio AMI is based in this 8-10GB space then you will have to set your working directory outside this, the home directory. Not intuitively apparent from Rstudio server interface. Whilst this is an advanced forum and this is a rookie error I am hoping no one deletes this question as I spent months on this and I think someone else will too. I hope this makes sense to you?

Eiffel

Another question has arisen asking for clarification of "Can someone please explain a little when they say that the directory needs to be something else than "home". How do you implement it? Because there is no permission to do that in Louis AMI" #47309452 I think it might be relevant to this so I have put the link to this question in this comment as it is asking for clarification of an equivalent to the method suggested here – Eiffel

Answer 3

This could have something to do with your web server. Are you using nginx or apache as your web server. If so you can modify the upload feature in your nginx server. If you are running nginx on the front end of the web server I would recommend the following fix in your nginx.conf file.

http {
    ...
    client_max_body_size 100M;
}  

https://www.tecmint.com/limit-file-upload-size-in-nginx/

Accomplish

Answer 4

I had a similar problems with a 5GB file. What worked for me was to use SQLite to create a database with the csv file that I needed. Use SQLite code to bring create the database. Then I used a function in RStudio to communicate with the local database. In that way, I was able to bring in the csv file. I can track down the R code that I used if you like.

Grunter