Common lisp: Can I define a function with arbitrary number of args and optional keyword args?

Question

I am fairly new to CL coming from R and Python.

I want to define a function to which I can pass an arbitrary number of arguments and also have keywords with default values that I can set if I want different values from the defaults.

In R I can do this like so:

foo <- function(..., a = 1, b = 2){
    list(a = a, b = b, ...)
}

foo(1, 2, 3)
foo(1, 2, 3, a = 2)
foo(b = 10, 1, 2, 3)

In PCL it says you can combine &key and &rest arguments but it doesn't work in the way I would expect if I try something like

(defun foo (&rest rest &key (a 1) (b 2))
   (list rest a b))

Here, I get an unknown &key error if I specify anything other than the two keyword args:

(foo :a 100 12 3)
>> unknown &KEY argument: 12

What I want is similar functionality to this:

(defun bar (&optional (a 1) (b 2) &rest rest)
   (list rest a b))

(bar 5 4 1 2 3 4)
>>((1 2 3 4) 5 4)

But I want to choose whether or not I supply to argument to :a and :b. I am using sbcl.

Hoedown

Answer 1

There is no standard way to do this in CL. You could make the function take everything in the rest parameter, and parse the keyword arguments yourself, if you don't want to design the API differently.

That said, here are a few points for further exploration. They might be useful in specific use cases, but are rather limited and exploit implementation-dependent behavior.

You can use &allow-other-keys in the lambda list or :allow-other-keys t when calling foo to prevent the unknown key error, but rest will also include the keys and values of your keyword arguments:

CL-USER> (defun foo (&rest rest &key (a 1) (b 2))
           (list rest a b))
FOO
CL-USER> (foo)
(NIL 1 2)
CL-USER> (foo :a 100 12 3 :allow-other-keys t)
((:A 100 12 3 :ALLOW-OTHER-KEYS T) 100 2)
CL-USER> (defun foo (&rest rest &key (a 1) (b 2) &allow-other-keys)
           (list rest a b))
FOO
CL-USER> (foo)
(NIL 1 2)
CL-USER> (foo :a 100 12 3)
((:A 100 12 3) 100 2)

As acelent correctly points out in a comment below, this might signal an error. It works for me in CLISP, SBCL, and CCL under the default optimize settings, but by the standard, keyword argument names (i.e. the first of each pair of arguments) must be symbols. Whether or not this works depends on the safety level and is implementation-dependent. It should signal an error (on conforming implementations) in safe code (safety level of 3).

In general, allowing other keys can be useful for passing keyword arguments through, but is not exactly what you wanted. One quick and dirty way might be filtering for keyword parameters in rest and just dropping them and their succeeding elements. Something like this:

CL-USER> (defun foo (&rest rest &key (a 1) (b 2) &allow-other-keys)
           (let ((rest (loop for (key value) on rest by #'cddr
                             unless (keywordp key)
                               append (list key value))))
             (list rest a b)))
FOO
CL-USER> (foo)
(NIL 1 2)
CL-USER> (foo :a 100 12 3)
((12 3) 100 2)

Which will, alas, by the standard only work for even numbers of arguments, as coredump points out in his answer. (It might work with an odd number of arguments under some implementations for some safety levels, but it didn't work in the implementations I tested.) Also, obviously it isn't robust in other ways (different positions of keyword arguments etc.), and not meant for production use, but just as a starting point for possible exploration.

The proper solution would involve writing your own keyword argument parser, and use it with rest, or, as pointed out in coredump's answer, using a different API. Another point worth mentioning is that in CL, applying large numbers of arguments is generally not a good idea since it might lead to inefficient code. What's worse, it is also not very reliable, since the number of allowed arguments is implementation-dependent. For example, under CCL on my system, call-arguments-limit is 65536. It may be significantly – even orders of magnitude – smaller under other implementations and systems. So, in general, prefer reduceing to applying large numbers of arguments.

Airlift

In the loop accumulation append (list key value) should probably be nconc (list key value); you're generating the new list structure, so there's no need to copy it so many times. Alternatively, you could also collect key collect value, I suppose. – Slut

@JoshuaTaylor I thought about it when writing the example, and decided against it. OP is a beginner, and append is the more self-documenting option. Using nconc, consing to reversed lists etc. have their uses, but in this case, it's not necessary, since one shouldn't apply huge numbers of arguments anyway. It isn't reliable. (Prefer reduce) On my system, call-arguments-list is 60k, but it may be significantly (orders of magnitude) smaller. Double collect doesn't read as nice IMO. You have a point, but I don't think this example is the right place for such optimizations. – Airlift

Keyword argument keys must be symbols, so your example call with the pair 12 3 may signal an error. On the other hand, keyword argument keys may be any symbol, not just keywords. – Collencollenchyma

@Collencollenchyma Yes, you're right. I wanted to show a quick and dirty example for further exploration, but it really feels pretty kludgy. I tried it with three implementations, none of which signalled an error under default safety, but it's not reliable, and probably shouldn't be used. I'll edit the answer to clarify that. – Airlift

Answer 2

Basically, here is how variable numbers of arguments are handled in conjunction with keywords:

• First, mandatory and optional arguments are bound to variables.
• Then, the remaining supplied arguments are bound to the &rest parameter, if given; let's say the parameter is named rest. It is the list of all remaining arguments.
• Now, if &key was also supplied in the lambda form, keywords are extracted from rest, which is then expected to have an even number of arguments (cf. specification), where keywords and values alternate, as in (:k1 v1 :k2 v2 ...).
• Unless you append &allow-other-keys in the lambda form definition or supply :allow-other-keys T when calling it, the number of supplied arguments must match exactly the number (and name) of expected keywords parameters you can only provide the keyword parameters declared in the lambda form (see @acelant's comments for details).

So, what can you do?

The simplest approach would be to define your functions so that all parameters are bound to keywords, with default values. You may also allow the caller to pass additional arguments:

    (defun foo (&rest rest &key (a 1) (b 2) &allow-other-keys) ...)

This is very similar to your definition, but you cannot just pass values; all arguments must be given along with a key:

    (foo :a 100 :max-depth 12 :max-try 3) ;; for example

CL is not R nor Python: maybe you don't need to pass so many arguments to your functions, maybe you can use special variables (dynamic scope), generic methods, ... Functions in existing packages usually mix mandatory, optional and keyword parameters. Take the time to read standard APIs (e.g. cl-ppcre, drakma) and see how you could define your functions in a more idiomatic way.

To conclude, maybe you really need to define functions with such R-like arguments. If this is the case, you can try to define your own macros for lambda lists in function definitions forms, so that

    (r-defun FOO ((a 1) (b 2) &rest rest) ...)

gets translated to something like:

    (defun FOO (&rest args)
       (let* ((rest (copy-seq args)) ;; must not modify "args"
              (a (find-keyword-and-remove-from-rest :a rest :default 1)
              (b (find-keyword-and-remove-from-rest :b rest :default 2))
          ... ;; function body
    ))

Do this if you want to have fun with macroexpansion, but really, I'am pretty sure this is not needed.

---

EDIT I originally modified the args argument (previously named rest) instead of doing a copy, but that was a bad example. The specification says :

When the function receives its arguments via &rest, it is permissible (but not required) for the implementation to bind the rest parameter to an object that shares structure with the last argument to apply. Because a function can neither detect whether it was called via apply nor whether (if so) the last argument to apply was a constant, conforming programs must neither rely on the list structure of a rest list to be freshly consed, nor modify that list structure.

Squatter

The keyword arguments don't have to match the number of keyword parameters. For instance, you may provide less keyword arguments than the declared keyword parameters, or have duplicate keys and the first such value is used. And for keyword arguments to be valid, not only must there be an even amount of remaining arguments, but the keys must be symbols. – Collencollenchyma

Thanks. So would it be more idiomatic to just wrap the possible args in one list? Say if I wanted to write a function that concatenates strings with a default joining character for within lists and another for between lists of strings the call would be something like (my-join '("This" "is" "a" ("list" "of") "strings") :within " " :between ", ") rather than having the strings forming the &rest. – Hoedown

@Hoedown Yes, exactly. Many functions operate on lists, or more generally sequences, and your example definitely fits the "LISt Processing" approach. Just bear in mind that this is not the only way to structure your data. – Squatter