Is it possible passing bytes vector to C function without copying in SBCL

Question

I'm trying to call C functions from SBCL with its FFI facility. The C functions require buffer of bytes (unsigned char) and process the bytes data. Unfortunately, in most cases the buffer is quite large. I'm wondering if I already have a Lisp bytes vector (a simple-array with element-type '(unsigned-byte 8)), is it possible just pass it to the C functions, rather than allocate alien buffers and copy the bytes there. I think this is a frequently recurring scenario, maybe SBCL team already cover that.

I look into the SBCL manual, it only states that sb-alien:c-string type might be passed directly to C functions without copying, and it requires the character be `base-char' (in the range 0 ~ 127 with unicode support). Wouldn't it be nice that we can also pass byte vectors without copying?

Artiste

See github.com/sionescu/static-vectors – Appenzell

Answer 1

Technically, it is possible - at least if the vector is a simple octet array - using sb-kernel:get-lisp-obj-address, and you would also need sb-sys:with-pinned-objects probably. Other question is how wise idea it is (portability, internal changes in array layout in sbcl...).

Dirty example (ignores pinning, long vs. int, ...):

Let us make a function that would use the Lisp array:

extern char test(const char *t, int idx)
{
  return t[idx];
}

and then (after making and loading shared library) we can pass an address of a vector as an integer:

(sb-alien:alien-funcall
 (sb-c-call:extern-alien "test"
                         (function sb-alien:int sb-alien:unsigned-long sb-alien:unsigned-long))
 (sb-kernel:get-lisp-obj-address (make-array 5 :element-type '(unsigned-byte 8) :initial-contents '(10 20 30 40 50)))
 3)

would return third element (not fourth as would be zero based). The object is a bit more complicated. For example, you get array size (tagged) at -7 address.

Meeting

Answer 2

I don't know whether this is possible in SBCL. To make it possible it would at least be necessary to allocate arrays in such a way that the GC doesn't move them. The SBCL manual contains a reference to a macro sb-sys:with-pinned-objects which might do this, but there doesn't seem to be any other information there.

However when I've needed to do something like this I've done it backwards: create a suitable object at the C level, and then wrap it with Lisp code to access it. The advantage of this is that you know how to deal with it from C, the disadvantage is that it's not an array, so you have to write your own access functions: things like read-sequence aren't going to work and so on.

I ended up using structures which wrapped alien vectors: here is a chunk of the code I used to deal with arrays of signed bytes:

(deftype index ()
  `(integer 0 (,most-positive-fixnum)))

...

(defstruct (foreign-octet-vector
            (:constructor %make-foreign-octet-vector (octets size)))
  (octets (error "no")
   :type (alien (* (signed 8)))
   :read-only t)
  (size 0
   :type index
   :read-only t))

(defun make-foreign-octet-vector (size)
  (%make-foreign-octet-vector
   (make-alien (signed 8) size)
   size))

(defun free-foreign-octet-vector (v)
  (free-alien (foreign-octet-vector-octets v))
  nil)

(declaim (inline fov-ref (setf fov-ref)))

(defun fov-ref (v n)
  (declare (type foreign-octet-vector v)
           (type index n))
  (if (< n (foreign-octet-vector-size v))
      (deref (foreign-octet-vector-octets v) n)
      (error "out of range")))

(defun (setf fov-ref) (value v n)
  (declare (type foreign-octet-vector v)
           (type index n)
           (type (signed-byte 8) value))
  (if (< n (foreign-octet-vector-size v))
      (setf (deref (foreign-octet-vector-octets v) n) value)
      (error "out of range")))

On top of this there was a fairly obvious macro using unwind-protect to make sure things got deallocated.

I don't know if this is either idiosyncratic SBCL code, or actually correct in general, but it worked for me. It may be that CFFI provides some better interface to all this.

Tomtoma

Thanks ignis, I'll definitely try that idea. Unfortunately, the C functions I mentioned are from third party libraries, I don't have control unless I modify their codes. As for the GC, I guess if the base-string can be passed to C functions (as alien:c-string type) without copying, there should be a way to avoid GC move the lisp objects. – Artiste

@xiepan: the C code needs no modification: the thing you pass it is a pointer to an array of whatever C type you defined on the Lisp side (so in this case it's an array of signed octets (which means in effect signed chars). (Note in the code this means you pass the value of the octets slot of the structure, not the structure itself). – Tomtoma