How to read large files from google drive into gae blobstore

Question

This is the code that I'm currently using to read files from google drive into the blobstore of google app engine.

private BlobKey getBlobKey(File f, DriveObject driveObject)
            throws IOException, MalformedURLException { 

        Drive service = ((GoogleDrive) driveObject).getService();

        byte[] buffer = new byte[(int) f.getFileSize().intValue()];
        GenericUrl url = new GenericUrl(f.getDownloadUrl());
        HttpResponse response = service.getRequestFactory()
                    .buildGetRequest(url).execute();

        InputStream is = response.getContent();

        FileService fileService = FileServiceFactory.getFileService();
        AppEngineFile file = null;
        boolean lock = true;
        try {
            file = fileService.createNewBlobFile("application/zip");
            FileWriteChannel writeChannel = fileService.openWriteChannel(
                        file, lock);

            int len;
            while ((len = is.read(buffer)) >= 0) {
            ByteBuffer bb = ByteBuffer.wrap(buffer, 0, len);
            writeChannel.write(bb);
            }
            writeChannel.closeFinally();

        } catch (IOException e) {
                // TODO Auto-generated catch block
            e.printStackTrace();
        }

            BlobKey bk = fileService.getBlobKey(file);

        return bk;
    }

This works fine for small files but it gives an error if the file is more that ~25MB. Is there a better way to do what I'm trying to do?

This is the error I see

Uncaught exception from servlet
com.google.appengine.api.urlfetch.ResponseTooLargeException: The response from url https://doc-0o-bs-docs.googleusercontent.com/docs/securesc/97o44sjeam7f23b1flie9m3qqidhrago/97q4akk8bmpub24itqhclo1vakjusu84/1368259200000/14350098165097390874/14350098165097390874/0ByS2XCxCEzq-NUVuajAwWGVoR1U?h=16653014193614665626&e=download&gd=true was too large.
    at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:132)
    at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
    at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417)
    at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296)
    at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getResponseCode(URLFetchServiceStreamHandler.java:149)
    at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:95)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:980)

I should add that

Drive service = ((GoogleDrive) driveObject).getService();

returns

new Drive.Builder(httpTransport, jsonFactory, cred)
            .build()

Itin

App Engine restricts the size of HTTP message bodies. – Clack

I realise that's what's going on but is there no way around this? It's pretty crucial that I can read large files into the blobstore. Maybe I need to read it in multiple messages. – Itin

I can upload the files I need from my computer via a blobstoreservice but this is not good enoough as I need to be able to do it via the drive interface. – Itin

Answer 1

You can download the file from Drive in chunks. Use the Range header in your download request as documented in the Google Drive documentation.

Range: bytes=500-999

Bakke

Nate that there is still quota an the number of bytes you can download. I'm currently getting a urlfetch.Fetch() exception at around the 50MB mark. (Note that I'm not a paying user. :-) See developers.google.com/appengine/docs/quotas#UrlFetch – Andreeandrei

Answer 2

This FilsService & FileWriteChannel api has been deprecated in favor of the App Engine GCS client.

Purgation