Is there a way to mix a LISP's macro &optional and &key params?

Question

I want to define a LISP macro like dolist that lets me define an optional output argument. In the following case study, this macro will be called doread. It will read lines from a file and return the number of lines found that way.

(let ((lines 0))
  (doread (line file lines)
     ;; do something with line
     (incf lines)))

The problem is that getting that lines to work in the above macro

I can do what I want with &key , but not with &optional &key (and the &key is needed since I want to control how a file is read; e.g with read or read-line or whatever).

Now the following works BUT to works the wrong way. Here the out argument has to be a &key and not a &optional:

;; this way works... 

(defmacro doread ((it f  &key out (take #'read)) &body body)
  "Iterator for running over files or strings."
  (let ((str (gensym)))
    `(with-open-file (,str f)
       (loop for ,it = (funcall ,take ,str nil)
             while ,it do
             (progn ,@body))
       ,out)))

;; lets me define something that reads first line of a file
(defun para1 (f)
  "Read everything up to first blank line."
  (with-output-to-string (s)
    (doread (x f :take #'read-line)
      (if (equalp "" (string-trim '(#\Space #\Tab) x))
        (return)
        (format s "~a~%" x)))))

(print (para1 sometime)) ; ==> shows all up to first blank line

What I'd like to do is this is the following (note that out has now moved into &optional:

(defmacro doread ((it f &optional out &key   (take #'read)) &body body)
  "Iterator for running over files or strings."
  (let ((str (gensym)))
    `(with-open-file (,str f)
       (loop for ,it = (funcall ,take ,str nil)
             while ,it do
             (progn ,@body))
       ,out)))

and if that worked, I could do something like.

(defun para1 (f)
  "Print  everything up to first blank line. 
   Return the lines found in that way"
  (let ((lines 0))
      (doread (x f lines :take #'read-line)
        (if (equalp "" (string-trim '(#\Space #\Tab) x))
            (return)
            (and (incf lines) (format t "~a~%" x)))))

but it I use &optional out I get

 loading /Users/timm/gits/timm/lisp/src/lib/macros.lisp
*** - GETF: the property list (#'READ-LINE) has an odd length

Parang

Note that in DOREAD the f needs to be ,f. – Tankersley

Answer 1

You cannot mix &optional and &key and expect to be able to pass only the keyword arguments. You can however define a syntax that allow for an optional list of arguments associated with the source.

For example:

(defpackage :so (:use :cl :alexandria))
(in-package :so)

(defmacro do-read ((entry source &optional result) &body body)
  (destructuring-bind (source &key (take '#'read)) (ensure-list source)
    (once-only (take)
      `(loop
          :with ,entry
          :do (setf ,entry (handler-case (funcall ,take ,source)
                             (end-of-file () (loop-finish))))
            (progn ,@body)
          :finally (return ,result)))))

The syntax for DO-READ could be written as:

(DO-READ (ENTRY [SOURCE|(SOURCE &KEY TAKE)] &OPTIONAL RESULT) . BODY)

This is not an unusual syntax w.r.t. standard Lisp forms (see LET, keyword synax in lambda-lists, defstruct, etc.). You could add more keyword parameters along with TAKE.

Remarks

• In macros, I prefer to emit LOOP keywords as keywords, not symbols in the macro's definition package; otherwise, when macroexpanding the code, you are likely to get the symbols prefixed by the macro's package (i.e. SO::WITH instead of :WITH), which becomes quickly unreadable.

• Returning NIL from READ-LINE is fine, but not from READ, as NIL could be a successfully read value. In general, since TAKE is provided by the user, you don't know if NIL is an acceptable result or not. That's why I catch END-OF-FILE instead. In case you want to read from other sources you may also check a secondary return value, or document that they signal a condition too.

• The ENTRY variable's scope is extended so that RESULT can be ENTRY itself; in your case, OUT could not be equal to IT, because once you exit the loop, you don't have access to it anymore. This is a minor point, but that can be useful.

• I did not include WITH-OPEN-FILE, in case you want to read from something else than files (streams).

• #'READ is quoted, this is not important here but a good habit to have in macros, so that you actually evalute things at evaluation time, not at macroexpansion time.

Examples

(with-input-from-string (in "abcdef")
  (do-read (char (in :take #'read-char) char)
    (print char)))

Print all characters and return #\f.

(with-input-from-string (in (format nil "~{~a~%~}" *features*))
  (let ((lines 0))
    (do-read (line in lines)
      (incf lines))))

Print the number of lines in a string.

Hosmer

This is great, thanks. destructuring-bind? Very cool. Did not think to use that. But once-only leaves me confused. And other people. common-lisp.net/project/cl-utilities/doc/once-only.html Can you clarify? – Parang

once-only is like with-gensyms so is for macro hygiene. But ensures that the variable which is replaced by a gensym - will be executed just once in the macro body. So a very practical macro - and mostly one wants once-only if one uses with-gensyms. – Lafountain

There is a very nice in-depth step-by-step explanation of once-only here: malisper.me/once-only – Lafountain

The function here could be any expression, and generally you do not expect an expression written once to be evaluated multiple times under the hood. So in a normal function, you just bind a temp. variable to to the expression and then use the variable. Once-only automatically adds this tmp var. This is needed here because the loop eval the expression each time. – Hosmer

'You cannot mix &optional and &key' ? – Tankersley

Note that (although this probably can't work here since you don't know what take is and what its conventions are) a good EOF-detecting trick with read and functions with compatible arguments is (eql (read in nil in) in) since one value they can never return is the stream they are reading from. This avoids potentially expensive exceptions. – Caroche

Answer 2

Works for me:

(defmacro doread ((it f &optional out &key (take #'read)) &body body)
  "Iterator for running over files or strings."
  (let ((str (gensym)))
    `(with-open-file (,str ,f)
       (loop for ,it = (funcall ,take ,str nil)
             while ,it do
             (progn ,@body))
       ,out)))

(defun para1 (f)
  "Print  everything up to first blank line. 
   Return the lines found in that way"
  (let ((lines 0))
    (doread (x f lines :take #'read-line)
      (if (equalp "" (string-trim '(#\Space #\Tab) x))
          (return)
        (and (incf lines) (format t "~a~%" x))))))

Using it in LispWorks:

CL-USER 104 > (para1 (capi:prompt-for-file "text file"))
;;; -*- mode: Lisp; Base: 10 ; Syntax: ANSI-Common-Lisp ; buffer-read-only: t; -*-
;;; This is ASDF 3.3.3: Another System Definition Facility.
;;;
;;; Feedback, bug reports, and patches are all welcome:
;;; please mail to <[email protected]>.
;;; Note first that the canonical source for ASDF is presently
;;; <URL:http://common-lisp.net/project/asdf/>.
;;;
;;; If you obtained this copy from anywhere else, and you experience
;;; trouble using it, or find bugs, you may want to check at the
;;; location above for a more recent version (and for documentation
;;; and test files, if your copy came without them) before reporting
;;; bugs.  There are usually two "supported" revisions - the git master
;;; branch is the latest development version, whereas the git release
;;; branch may be slightly older but is considered `stable'
15

It's just that once you want to specify :take you also need to provide the optional arg. That's a common pitfall and that's why the combination of optional and keyword args is not liked.

Tankersley