Generate and stream a zip-file in a Ring web app in Clojure

Question

I have a Ring handler that needs to:

• Zip a few files
• Stream the Zip to the client.

Now I have it sort of working, but only the first zipped entry gets streamed, and after that it stalls/stops. I feel it has something to do with flushing/streaming that is wrong.

Here is my (compojure) handler:

(GET "/zip" {:as request}
            :query-params [order-id   :- s/Any]
            (stream-lessons-zip (read-string order-id) (:db request) (:auth-user request)))

Here is the stream-lessons-zip function:

(defn stream-lessons-zip
  []
  (let [lessons ...];... not shown

  {:status 200
   :headers {"Content-Type" "application/zip, application/octet-stream"
             "Content-Disposition" (str "attachment; filename=\"files.zip\"")
   :body (futil/zip-lessons lessons)}))

And i use a piped-input-stream to do the streaming like so:

(defn zip-lessons
 "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
[lessons]
(let [paths (map #(select-keys % [:file_path :file_name]) lessons)]
(ring-io/piped-input-stream
  (fn [output-stream]
    ; build a zip-output-stream from a normal output-stream
    (with-open [zip-output-stream (ZipOutputStream. output-stream)]
      (doseq [{:keys [file_path file_name] :as p} paths]
        (let [f (cio/file file_path)]
          (.putNextEntry zip-output-stream (ZipEntry. file_name)) 
          (cio/copy f zip-output-stream)
          (.closeEntry zip-output-stream))))))))

So I have confirmed that the 'lessons' vector contains like 4 entries, but the zip file only contains 1 entry. Furthermore, Chrome doesn't seem to 'finalize' the download, ie. it thinks it is still downloading.

How can I fix this?

Kimberly

I tried adding (.flush zip-output-stream) in the doseq before the closeEntry call, but to no avail. – Kimberly

I tried simplified version of your code and it works fine. I think it might be some of the middlewares causing problems. You could try to run your app with no or minimal set of middlewares and see if it works. – Db

You don't happen to be using http-kit do you? I've had problems with trying to stream downloads with that, I don't believe it's supported, whereas it is by ring-jetty. – Pat

I guess this is a copy/paste error but your implementation of stream-lessons-zip shows a no-arg fn, but when you call it in the handler you pass it three args? – Pat

Did you try flushing output-stream after the with-open? – Tomblin

Aah i do use http-kit! Thanks for mentioning that Russel, also thanks for pointing out that the code is conceptually OK, minus some copy/paste errors. I'll add this to the answer – Kimberly

@MartenSytema perhaps you could update whether/how this issue was resolved? – Hine

Yes, actually, thanks for reminding. I'll put the code in the answer below. – Kimberly

Answer 1

It sounds like producing a stateful stream using blocking IO is not supported by http-kit. Non-stateful streams can be done this way:

http://www.http-kit.org/server.html#async

A PR to introduce stateful streams using blocking IO was not accepted:

https://github.com/http-kit/http-kit/pull/181

It sounds like the option to explore is to use a ByteArrayOutputStream to fully render the zip file to memory, and then return the buffer that produces. If this endpoint isn't highly trafficked and the zip file it produces is not large (< 1 gb) then this might work.

Tomblin

Answer 2

So, it's been a few years, but that code still runs in production (ie. it works). So I made it work back then, but forgot to mention it here (and forgot WHY it works, to be honest,.. it was very much trial/error).

This is the code now:

(defn zip-lessons
  "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
  [lessons {:keys [firstname surname order_favorite_name company_name] :as annotation
            :or {order_favorite_name ""
                 company_name ""
                 firstname ""
                 surname ""}}]
  (debug "zipping lessons" (count lessons))
  (let [paths (map #(select-keys % [:file_path :file_name :folder_number]) lessons)]
    (ring-io/piped-input-stream
      (fn [output-stream]
        ; build a zip-output-stream from a normal output-stream
        (with-open [zip-output-stream (ZipOutputStream. output-stream)]
          (doseq [{:keys [file_path file_name folder_number] :as p} paths]
            (let [f (cio/as-file file_path)
                  baos (ByteArrayOutputStream.)]
              (if (.exists f)
                (do
                  (debug "Adding entry to zip:" file_name "at" file_path)
                  (let [zip-entry (ZipEntry. (str (if folder_number (str folder_number "/") "") file_name))]
                    (.putNextEntry zip-output-stream zip-entry)

                   
                    (.close baos)
                    (.writeTo baos zip-output-stream)
                    (.closeEntry zip-output-stream)
                    (.flush zip-output-stream)
                    (debug "flushed")))
                (warn "File '" file_name "' at '" file_path "' does not exist, not adding to zip file!"))))
          (.flush zip-output-stream)
          (.flush output-stream)
          (.finish zip-output-stream)
          (.close zip-output-stream))))))

Kimberly