Code runs in SLIME+SBCL but not plain SBCL

Question

I've been trying to build a lispy interface to the CFFI bindings (https://gitorious.org/dh-misc/hdf5/source/cb616fd619a387e3cdc927994b9ad12b6b514236:) but I ran into a situation where code runs correctly in SLIME which has an SBCL instance as the backend, but won't run whenever I run the code just in SBCL.

So, I created a test case file which demonstrates the error:

(asdf:load-system :cffi)
;;(asdf:operate 'asdf:load-op :cffi)

(defpackage :hdf5test
  (:use :cl :cffi)
  (:export :test))

(in-package :hdf5test)

(define-foreign-library hdf5
    (t (:default "libhdf5")))

(use-foreign-library hdf5)

;; hdf types:

(defctype size-t :uint)
(defctype hid-t :int)
(defctype herr-t :int)
(defctype hsize-t :uint64)

;; hdf constants:

;; H5S_UNLIMITED: 2^64-1
(defconstant +H5S-UNLIMITED+ 18446744073709551615)

;; H5F_ACC_TRUNC
(defconstant +H5F-ACC-TRUNC+ 2) ;; we'll see if it works

;; H5P_DEFAULT
(defconstant +H5P-DEFAULT+ 0)

;; H5T types:

(defconstant +H5P-DATASET-CREATE+ 150994953)
(defconstant +H5T-NATIVE-INT+ 50331660)

;; hdf functions:

;; H5Screate_simple
(defcfun "H5Screate_simple" hid-t
  (rank :int)
  (current-dims :pointer) ; const hsize_t*
  (maximum-dims :pointer)) ; cons hsize_t*

;; H5Fcreate
(defcfun "H5Fcreate" hid-t
  (filename :string)
  (flags :uint)
  (fcpl-id hid-t)
  (fapl-id hid-t))

;; H5Pcreate
(defcfun "H5Pcreate" hid-t
  (cls-id hid-t))

;; H5Pset_chunk
(defcfun "H5Pset_chunk" herr-t
  (plist hid-t)
  (ndims :int)
  (dim :pointer)) ;; const hsize_t*

;; H5Pset_deflate
(defcfun "H5Pset_deflate" herr-t
  (plist-id hid-t)
  (level :uint))

;; H5Dcreate1
(defcfun "H5Dcreate1" hid-t
  (loc-id hid-t)
  (name :string)
  (type-id hid-t)
  (space-id hid-t)
  (dcpl-id hid-t))

;; H5Dclose
(defcfun "H5Dclose" herr-t
  (dataset-id hid-t))

;; H5Dwrite
(defcfun "H5Dwrite" herr-t
  (datset-id hid-t)
  (mem-type-id hid-t)
  (mem-space-id hid-t)
  (file-space-id hid-t)
  (xfer-plist-id hid-t)
  (buf :pointer))

;; H5Fclose
(defcfun "H5Fclose" herr-t
  (file-id hid-t))

;; H5Sclose
(defcfun "H5Sclose" herr-t
  (space-id hid-t))

(defparameter *rank* 1)

(defun test (filename)
  (with-foreign-string (dataset-name "dataset")
    (with-foreign-objects ((dim :int 1)
               (dataspace-maxdim :uint64 1)
               (memspace-maxdim :uint64 1)
               (chunkdim :int 1)
               (dataspace 'hid-t)
               (dataset 'hid-t)
               (memspace 'hid-t)
               (cparms 'hid-t))
      (setf (mem-aref dim :int 0) 5)
      (format t "dim: ~a~%" (mem-aref dim :int 0))
      ;;(setf (mem-aref maxdim :int 0) -1)
      (setf (mem-aref dataspace-maxdim :uint64 0) +H5S-UNLIMITED+)
      (setf (mem-aref memspace-maxdim :uint64 0) 5)
      (setf (mem-aref chunkdim :int 0) 1)
      (format t "dataspace-maxdim: ~a~%" (mem-aref dataspace-maxdim :uint64 0))
      (format t "memspace-maxdim: ~a~%" (mem-aref memspace-maxdim :uint64 0))
      ;;(with-open-hdf-file (file filename :direction :output :if-exists :supersede)
      (let ((file (h5fcreate filename +H5F-ACC-TRUNC+ +H5P-DEFAULT+ +H5P-DEFAULT+)))
    (setf cparms (h5pcreate +H5P-DATASET-CREATE+))
    (h5pset-chunk cparms *rank* chunkdim)
    (setf dataspace (h5screate-simple *rank* dim dataspace-maxdim))
    (setf dataset (h5dcreate1
               file
               dataset-name
               +H5T-NATIVE-INT+
               dataspace
               cparms))
    (format t "dataspace: ~a~%" dataspace)
    (format t "dataset: ~a~%" dataset)
    (setf memspace (h5screate-simple *rank* dim memspace-maxdim))
    (with-foreign-object (data :int 5)
      (loop for i from 0 to 4 do (setf (mem-aref data :int i) (* i i)))
      (h5dwrite dataset +H5T-NATIVE-INT+ memspace dataspace +H5P-DEFAULT+ data))
    (h5dclose dataset)
    (h5sclose memspace)
    (h5sclose dataspace)
    (h5fclose file)))))

The output I get from running (hdf5test:test "test.h5") in SLIME+SBCL is

dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
0

The output I get from running (hdf5test:test "test.h5") in just SBCL is

dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0:
  #000: H5S.c line 1388 in H5Screate_simple(): maxdims is smaller than dims
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0:
  #000: H5Dio.c line 233 in H5Dwrite(): not a data space
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0:
  #000: H5S.c line 405 in H5Sclose(): not a dataspace
    major: Invalid arguments to routine
    minor: Inappropriate type
0

So you can see it's something to do with how the array is being passed to the hdf functions, but I have no clue why SLIME+SBCL would handle this but not SBCL.

I've also tried the exact same code with CLISP and it works fine, no issues, so it seems to be an SBCL issue.

Any thoughts on this?

EDIT: I thought I should add to the main post that the resulting files truly are different in each case. In SLIME+SBCL or CLISP the file contains a finite dataset with squared integers inside (no reason really, just a test). But, with plain SBCL the data file is left incomplete; if you try to view the contents with h5dump it is an unending trial of zeros (that's how it handles incomplete datasets).

Zachariahzacharias

what does it mean 'runs'? how do you compile & run the code under SLIME/SBCL and SBCl? – Ground

In SLIME+SBCL, I either load directly in the slime REPL with the (load ...) function, or use the C-c C-l emacs shortcut to load the file into the list image. Under just SBCL, I use the (load ...) function or run sbcl with the --load option. To run I just do (hdf5test:test "test.h5") in the REPL. – Zachariahzacharias

BTW this code does require HDF5 C libraries to be installed in order to run. – Zachariahzacharias

It actually looks like the error is coming through the standard error and sbcl just shows it while clisp and sbcl+slime are hiding it. Do you get dropped into the debugger when this happens in sbcl? Failing that you could try compiling the function with no optimization at all using declare or declaim. – Herson

I am not dropped into the debugger, but the results truly are different. In SLIME+SBCL or CLISP, the resulting file is as it should be with meaningful data inside. However with just SBCL the data file is incorrect with no meaningful data. So something does seem to be different in the way the code is handled; does SLIME do something to the environment? – Zachariahzacharias

Update: Tried various declare options with different values for safety etc. and it doesn't seem to have any effect. – Zachariahzacharias

Another update: I've continued work on some higher-level tools for using a subset of the HDF5 library in CL, and in the code I'm using now there doesn't appear to be any difference between running the code in SBCL or SLIME+SBCL or any other implementation I have. I'll leave the problem open though since this problem is still reproducible. – Zachariahzacharias

I recently came across a similar situation, in which I (had forgotten that I) was invoking SBCL in Slime with a saved core file that was initializing some of the libraries I used, which were then loaded differently (different init call) from the shell. Are (were) you possibly passing args to SBCL in inferior-lisp or similar? – Traumatism

Answer 1

Like @nixeagle said, slime seems to hide the error messages originating in the hdf5 library. Along these lines I'd wager that passing the results from SBCL to emacs in slime is what allows the file to be written.

Now the following needs to be taken with a few grains of salt, as I don't really know anything about hdf5 or cffi and am now just getting back into common lisp, but things started working consistently in both slime and sbcl on my x86_64 linux box, once I replaced all those :int types with :uint64, which seems to make sense, as the declarations resolve to that type anyways.

your code in sbcl:

* (load "temp.lisp")
T
* (hdf5test:test "test2.h5")
dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: H5S.c line 1388 in H5Screate_simple(): maxdims is smaller than dims
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: H5Dio.c line 231 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 332 in H5D__pre_write(): not a data space
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: H5S.c line 405 in H5Sclose(): not a dataspace
    major: Invalid arguments to routine
    minor: Inappropriate type
0

part with changes:

    (with-foreign-objects ((dim :uint64 1)
               (dataspace-maxdim :uint64 1)
               (memspace-maxdim :uint64 1)
               (chunkdim :uint64 1)
               (dataspace 'hid-t)
               (dataset 'hid-t)
               (memspace 'hid-t)
               (cparms 'hid-t))
      (setf (mem-aref dim :uint64 0) 5)
      (format t "dim: ~a~%" (mem-aref dim :uint64 0))
      ;;(setf (mem-aref maxdim :int 0) -1)
      (setf (mem-aref dataspace-maxdim :uint64 0) +H5S-UNLIMITED+)
      (setf (mem-aref memspace-maxdim :uint64 0) 5)
      (setf (mem-aref chunkdim :uint64 0) 1)
      (format t "dataspace-maxdim: ~a~%" (mem-aref dataspace-maxdim :uint64 0))
      (format t "memspace-maxdim: ~a~%" (mem-aref memspace-maxdim :uint64 0))

changed code in sbcl:

* (load "temp.lisp")
T
* (hdf5test:test "test2.h5")
dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
0

result files:

% h5dump test.h5 
HDF5 "test.h5" {
GROUP "/" {
   DATASET "dataset" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 5 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): 0, 1, 4, 9, 16
      }
   }
}
}
% h5dump test2.h5
HDF5 "test2.h5" {
GROUP "/" {
   DATASET "dataset" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 5 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): 0, 1, 4, 9, 16
      }
   }
}
}

Hartsfield

Just checked your solution and it works on my machine as well! – Zachariahzacharias

I don't know why SBCL but not SLIME+SBCL nor CLISP fails, but I can confirm that the dim variables being :int is in fact wrong, so it's probably not that important why different implementations handle the buggy code differently. – Zachariahzacharias