Lazy concatenation of sequence in Clojure

Question

Here's a beginner's question: Is there a way in Clojure to lazily concatenate an arbitrary number of sequences? I know there's lazy-cat macro, but I can't think of its correct application for an arbitrary number of sequences.

My use case is lazy loading data from an API via paginated (offseted/limited) requests. Each request executed via request-fn below retrieves 100 results:

(map request-fn (iterate (partial + 100) 0))

When there are no more results, request-fn returns an empty sequence. This is when I stop the iteration:

(take-while seq (map request-fn (iterate (partial + 100) 0)))

For example, the API might return up to 500 results and can be mocked as:

(defn request-fn [offset] (when (< offset 500) (list offset)))

If I want to concatenate the results, I can use (apply concat results) but that eagerly evaluates the results sequence:

(apply concat (take-while seq (map request-fn (iterate (partial + 100) 0))))

Is there a way how to concatenate the results sequence lazily, using either lazy-cat or something else?

Canikin

The lazy-cat macro evaluates each argument only as required. – Mckay

Yes, but how do you apply it to a sequence of arguments? – Sluice

Do you really need to concatenate the results, or do you just want to consume them lazily? – Narva

Having the results sequence concatened would make processing them easier. But it's not a strict requirement. I can also map over the results pages. – Sluice

Answer 1

For the record, apply will consume only enough of the arguments sequence as it needs to determine which arity to call for the provided function. Since the maximum arity of concat is 3, apply will realize at most 3 items from the underlying sequence.

If those API calls are expensive and you really can't afford to make unnecessary ones, then you will need a function that accepts a seq-of-seqs and lazily concatenates them one at a time. I don't think there's anything built-in, but it's fairly straightforward to write your own:

(defn lazy-cat' [colls]
  (lazy-seq
    (if (seq colls)
      (concat (first colls) (lazy-cat' (next colls))))))

Romero

Thank you for the clarification on apply and a suggestion for lazy-cat'. – Sluice

nice answer; btw, when using next this function eagerly evaluated 2 items from colls even when only requiring 1 evaluation, e.g. (first (lazy-cat' aseq)). Using rest instead gave the expected behaviour. – Bridesmaid