Downloading large file in ColdFusion using CFHTTP

Question

I'm trying to download a large (600MB) binary file to the server using CFHTTP on Coldfusion 8:

<cfhttp 
  method="get" 
  url="#fileURL#" 
  path="#filePath#" 
  file="#fileName#" 
  timeout="600" 
  getasbinary="auto"
>

It's working fine for smaller files (100something MB) but for the large one's I'm getting the Server 500 error: "java.lang.OutOfMemoryError: Java heap space".

The file is being downloaded from a Dropbox folder - so only available option is to use HTTP GET.

Does anyone have idea how to download it, so it wouldn't kill the server or timeout?

Acuminate

If it's a file in Dropbox, why not just install the Dropbox client on the server, and watch the directory for new/updated files? – Leaguer

Because we have couple separate clients on the same server - can't install multiple Dropbox clients. But thanks for the idea Ben. – Acuminate

Answer 1

You can do this by calling Java from CF code. The buffered input and output stream classees are designed to hold onto chunks of data, rather than the whole thing, avoiding OutOfMemory errors.

getByteArray() is a helper method because there's not way to declare something like byte buf[]=new byte[1024]; in CF directly.

In the example change the source and destination variables.

Example

<cfset source = 'protocol://domain/path/to/file.ext'>
<cfset destination = getDirectoryFromPath(getCurrentTemplatePath()) & listlast(source, "/")>
<cffunction name="getByteArray" access="private" returnType="binary" output="no">
    <cfargument name="size" type="numeric" required="true"/>
    <cfset var emptyByteArray =
        createObject("java", "java.io.ByteArrayOutputStream").init().toByteArray()/>
    <cfset var byteClass = emptyByteArray.getClass().getComponentType()/>
    <cfset var byteArray =
        createObject("java","java.lang.reflect.Array").newInstance(byteClass, arguments.size)/>
    <cfreturn byteArray/>
</cffunction>

<cfscript>
uri = createObject("java", "java.net.URL").init(source);
uis = uri.openStream();
bis = createObject("java", "java.io.BufferedInputStream").init(uis);
fos = createObject("java", "java.io.FileOutputStream").init(destination);
bos = createObject("java", "java.io.BufferedOutputStream").init(fos);
buffer = getByteArray(1024);
len = bis.read(buffer);
while(len > 0) {
    bos.write(buffer,0,len);
    len = bis.read(buffer);
}
bos.close();
bis.close();
fos.close();
uis.close();
</cfscript>

Saw

Nice howto, especially for people without Java background (like me). – Floatfeed

Java is great to know with CF because it allows you to work around some of the problems with built-in functionality. – Saw

What would I need to add to use this to connect through a proxy server? My cf server is Coldfusion9. Thanks. – Mentality

@garv: I'm not working with CF these days. I'd suggest asking a new question. – Saw

Answer 2

The problem is is that it's too large. ColdFusion reads the entire thing into memory before writing it to disk.

You'll be better off using some other method to get the file. wget can do http requests from a command-line. That with judicious use of CFEXECUTE is probably a good way to go.

Vagabondage