How do keywords work in Common Lisp?

Question

The question is not about using keywords, but actually about keyword implementation. For example, when I create some function with keyword parameters and make a call:

(defun fun (&key key-param) (print key-param)) => FUN
(find-symbol "KEY-PARAM" 'keyword) => NIL, NIL   ;;keyword is not still registered
(fun :key-param 1) => 1
(find-symbol "KEY-PARAM" 'keyword) => :KEY-PARAM, :EXTERNAL

How a keyword is used to pass an argument? Keywords are symbols whos values are themselves, so how a corresponding parameter can be bound using a keyword?

Another question about keywords — keywords are used to define packages. We can define a package named with already existing keyword:

(defpackage :KEY-PARAM) => #<The KEY-PARAMETER package, 0/16 ...
(in-package :KEY-PARAM) => #<The KEY-PARAMETER package, 0/16 ...
(defun fun (&key key-param) (print key-param)) => FUN
(fun :KEY-PARAM 1) => 1

How does system distinguish the usage of :KEY-PARAM between package name and function parameter name? Also we can make something more complicated, if we define function KEY-PARAM and export it (actually not function, but name):

(in-package :KEY-PARAM)
(defun KEY-PARAM (&key KEY-PARAM) KEY-PARAM) => KEY-PARAM
(defpackage :KEY-PARAM (:export :KEY-PARAM))  
   ;;exporting function KEY-PARAM, :KEY-PARAM keyword is used for it
(in-package :CL-USER) => #<The COMMON-LISP-USER package, ...
(KEY-PARAM:KEY-PARAM :KEY-PARAM 1) => 1
   ;;calling a function KEY-PARAM from :KEY-PARAM package with :KEY-PARAM parameter...

The question is the same, how Common Lisp does distinguish the usage of keyword :KEY-PARAM here?

If there is some manual about keywords in Common Lisp with explanation of their mechanics, I would be grateful if you posted a link here, because I could find just some short articles only about usage of the keywords.

Puttergill

Answer 1

See Common Lisp Hyperspec for full details of keyword parameters. Notice that

(defun fun (&key key-param) ...)

is actually short for:

(defun fun (&key ((:key-param key-param)) ) ...)

The full syntax of a keyword parameter is:

((keyword-name var) default-value supplied-p-var)

default-value and supplied-p-var are optional. While it's conventional to use a keyword symbol as the keyword-name, it's not required; if you just specify a var instead of (keyword-name var), it defaults keyword-name to being a symbol in the keyword package with the same name as var.

So, for example, you could do:

(defun fun2 (&key ((myoption var))) (print var))

and then call it as:

(fun 'myoption 3)

The way it works internally is when the function is being called, it steps through the argument list, collecting pairs of arguments <label, value>. For each label, it looks in the parameter list for a parameter with that keyword-name, and binds the corresponding var to value.

The reason we normally use keywords is because the : prefix stands out. And these variables have been made self-evaluating so that we don't have to quote them as well, i.e. you can write :key-param instead of ':key-param (FYI, this latter notation was necessary in earlier Lisp systems, but the CL designers decided it was ugly and redundant). And we don't normally use the ability to specify a keyword with a different name from the variable because it would be confusing. It was done this way for full generality. Also, allowing regular symbols in place of keywords is useful for facilities like CLOS, where argument lists get merged and you want to avoid conflicts -- if you're extending a generic function you can add parameters whose keywords are in your own package and there won't be collisions.

The use of keyword arguments when defining packages and exporting variables is again just a convention. DEFPACKAGE, IN-PACKAGE, EXPORT, etc. only care about the names they're given, not what package it's in. You could write

(defpackage key-param)

and it would usually work just as well. The reason many programmers don't do this is because it interns a symbol in their own package, and this can sometimes cause package conflicts if this happens to have the same name as a symbol they're trying to import from another package. Using keywords divorces these parameters from the application's package, avoiding potential problems like this.

The bottom line is: when you're using a symbol and you only care about its name, not its identity, it's often safest to use a keyword.

Finally, about distinguishing keywords when they're used in different ways. A keyword is just a symbol. If it's used in a place where the function or macro just expects an ordinary parameter, the value of that parameter will be the symbol. If you're calling a function that has &key arguments, that's the only time they're used as labels to associate arguments with parameters.

Incised

@Incised "(defun fun (&key key-param) ...) is actually short for: (defun fun (&key ((:key-param key-param)) ) ...)" with the exception that after the first, a symbol named "KEY-PARAM" hasn't been interned in the keyword package yet (at least according to OP's transcript). If it were a shorthand where the arglist were being expanded (and maybe an implementation could do that; I'm not sure if it'd be allowed or not), then the keyword would have been interned at macro-expansion time. (This isn't particularly important, but it does point "leak" (or not) some of defun's implementation.) – Shower

Answer 2

The good manual is Chapter 21 of PCL.

Answering your questions briefly:

• keywords are exported symbols in keyword package, so you can refer to them not only as :a, but also as keyword:a

• keywords in function argument lists (called lambda-lists) are, probably, implemented in the following way. In presence of &key modifier the lambda form is expanded into something similar to this:

  (let ((key-param (getf args :key-param)))
    body)
  

• when you use a keyword to name a package it is actually used as a string-designator. This is a Lisp concept that allows to pass to a certain function dealing with strings, that are going to be used as symbols later (for different names: of packages, classes, functions, etc.) not only strings, but also keywords and symbols. So, the basic way to define/use package is actually this:

  (defpackage "KEY-PARAM" ...)
  

  But you can as well use:

  (defpackage :key-param ...)
  

  and

  (defpackage #:key-param ...)
  

  (here #: is a reader macro to create uninterned symbols; and this way is the preferred one, because you don't create unneeded keywords in the process).

  The latter two forms will be converted to upper-case strings. So a keyword stays a keyword, and a package gets its named as string, converted from that keyword.

To sum up, keywords have the value of themselves, as well as any other symbols. The difference is that keywords don't require explicit qualification with keyword package or its explicit usage. And as other symbols they can serve as names for objects. Like, for example, you can name a function with a keyword and it will be "magically" accessible in every package :) See @Xach's blogpost for details.

Stoecker

Thank you! (getf :key-param args) - is really nice explanation. Also, I tried keyword-named functions, in Lispworks something like (defun :test () "hi") causes an error Defining function :TEST visible from package KEYWORD. (but there is a debugger option "Define it anyway"), while in SBCL everything works. – Puttergill

Welcome! I also have to say, that I've accidentally wrote this in wrong order - the proper way is (getf args :key-param) :) – Stoecker

Answer 3

There is no need for "the system" to distinguish different uses of keywords. They are just used as names. For example, imagine two plists:

(defparameter *language-scores* '(:basic 0 :common-lisp 5 :python 3))
(defparameter *price* '(:basic 100 :fancy 500))

A function yielding the score of a language:

(defun language-score (language &optional (language-scores *language-scores*))
  (getf language-scores language))

The keywords, when used with language-score, designate different programming languages:

CL-USER> (language-score :common-lisp)
5

Now, what does the system do to distinguish the keywords in *language-scores* from those in *price*? Absolutely nothing. The keywords are just names, designating different things in different data structures. They are no more distinguished than homophones are in natural language – their use determines what they mean in a given context.

In the above example, nothing prevents us from using the function with a wrong context:

(language-score :basic *prices*)
100

The language did nothing to prevent us from doing this, and the keywords for the not-so-fancy programming language and the not-so-fancy product are just the same.

There are many possibilities to prevent this: Not allowing the optional argument for language-score in the first place, putting *prices* in another package without externing it, closing over a lexical binding instead of using the globally special *language-scores* while only exposing means to add and retrieve entries. Maybe just our understanding of the code base or convention are enough to prevent us from doing that. The point is: The system's distinguishing the keywords themselves is not necessary to achieve what we want to implement.

The specific keyword uses you ask about are not different: The implementation might store the bindings of keyword-arguments in an alist, a plist, a hash-table or whatever. In the case of package names, the keywords are just used as package designators and the package name as a string (in uppercase) might just be used instead. Whether the implementation converts strings to keywords, keywords to strings, or something entirely different internally doesn't really matter. What matters is just the name, and in which context it is used.

Sagittal

Thank you! One more question, is it possible to check whether impleentation converts keywords to strings or not? Is is possible to reduce keywords to strings? I mean when Lisp reader finds a keyword it parses it just as a special kind of string. – Puttergill

In order to check how an implementation works internally, you'd have to take a look at the source code (or detailed implementation-specific documentation). In the case of package names, I'd assume implementations to compare the strings, because otherwise (with default reader settings) case-information would be lost. While two packages "foo" and "Foo" are different, two keywords :foo and :Foo are not. They are both interned with the same symbol-name, "FOO". (But then, there's also :|Foo| to prevent that.) – Sagittal