Interpret foreign memory as lisp memory (or vice versa) without copying data

Question

I try to write BLOB into database - chunk by chunk, using database API C-function (say, db-write-chunk). This function takes a pointer to a foreign memory (where chunk is placed) as an argument. So, I make buffer for a chunk: foreign-buffer. I'll take chunk data from a file (or binary stream) by read-sequence into stream-buffer:

(let ((foreign-buffer (foreign-alloc :uchar 1024)))
      (stream-buffer ((make-array 1024 :element-type '(unsigned-byte 8))))
    (loop
        for cnt = (read-sequence stream-buffer MY-STREAM)
        while (> cnt 0)
        do
          ;; copy cnt bytes from stream-buffer into foreign-buffer 
          ;; call db-write-chunk with foreign-buffer

L in BLOB is for Large and loop may iterate many times. Besides that, all this code may be wrapped by the external loop (bulk-insert, for example). So, I want to minimize the count of steps in the loop(s) body.

To have this done I need:

• to be able to read sequence not into stream-buffer, but into foreign-buffer directly, like this:

  (read-sequence (coerce foreign-buffer '(vector/array ...)) MY-STREAM)
  

• or to be able to interpret stream-buffer as foreign memory, like this:

  (db-write-chunk (mem-aptr stream-buffer :uchar 0))
  

Is it possible to solve my problem using single buffer only - native or foreign, without copying memory between them?

Greasy

I think this is highly implementation-dependent. What Lisp are you using? – Trichiasis

maybe github.com/sionescu/static-vectors helps? I use that to send large matrices to fortran routines – Exhilarant

@sds, I use SBCL. But SBCL for Windows is said to be unstable, so I keep in mind CLISP and ECL. – Greasy

@DavidHodge, thanks, I've just downloaded it and try to understand how it works – Greasy

Thanks, @sds, @DavidHodge! Both statements are true. The code of staic-vectors is realy implementation-dependent, main part of sources are implementations for specific LISPs. For not to reinventing the wheel, it's better to get ready solution. Static-vectors looks like working well for my purposes, although it generates one style warning when loading ASDF under SBCL (declaim function FREE-STATIC-VECTOR inline after 2 calls to it were compiled). – Greasy

Answer 1

Like anything else ffi, this is implementation dependent, but cffi has cffi:make-shareable-byte-vector, which is a CL (unsigned-byte 8) array which you can then use with cffi:with-pointer-to-vector-data:

(cffi:defcfun memset :pointer
  (ptr :pointer)
  (val :int)
  (size :int))

(let ((vec (cffi:make-shareable-byte-vector 256)))
  (cffi:with-pointer-to-vector-data (ptr vec)
    (memset ptr 0 (length vec))))

Depending on your use, this might be preferable to static-vectors, because you don't have to remember to free it manually. On SBCL this works by pinning the vector data during with-pointer-to-vector-data.

Phyllode