Lazily evaluated recursive stream in fp-ts from paginated API

Question

My goal is to request transactions from an API and store them to a database. The response is paginated and I want to read every page and store the transactions in batches.

So for one request/response cycle I wish to then process the result and store it to the database before fetching the next batch.

How do I do this in an idiomatic way in fp-ts? I inject the actual HTTP fetch call as (url:string, init:RequestInit) => TE.TaskEither<Error, Response> in order to keep it testable.

So far I've tested RxJS and fp-ts and that got a bit complicated even if I got it to work.

Did give both recursive function and generators a try but I'm not succeeding in getting it to be lazily evaluated.

Are there any existing examples with fp-ts that shows a lazily evaluated stream where each element is dependent on the previous?

Amieeamiel

Answer 1

You can use StateReaderTaskEither.
Let's say you want to read a list of files, and the next filename is stored in the current file.

import * as TE from 'fp-ts/TaskEither'
import * as RA from 'fp-ts/ReadonlyArray'
import { pipe } from 'fp-ts/function'

const files = {
  file1: 'file2',
  file2: 'file3',
  file3: 'file5',
  file4: 'file6',
  file5: 'file4',
}
const getFile = (filename: string, encoding: string) =>
  Promise.resolve(files[filename])

const fileSRTE: SRTE.StateReaderTaskEither<
  string,
  string,
  unknown,
  string
> = (filename: string) => (encoding: string) =>
  pipe(
    TE.tryCatch(
      () => getFile(filename, encoding),
      (err) => err
    ),
    TE.map((file) => [`file: ${file}`, file])
  )

const fileStream: TE.TaskEither<
  unknown,
  readonly string[]
> = pipe(
  RA.replicate(6, fileSRTE),
  SRTE.sequenceArray,
  SRTE.evaluate('file1')
)('utf-8')

fileStream().then(console.log)
//{ _tag: 'Right',right: [ 'file: file2','file: file3','file: file5','file: file4','file: file6','file: undefined' ] }

You can look more into State and Reader in fp-ts Doc

Matadi