In Lisp, is it more idiomatic to use let* or setf?

Question

When multiple steps are required in the calculation of a value, I have a tendency to use (let*) to declare multiple versions of a variable, thus:

(let* ((var 1)
       (var (* var 2))
       (var (- var)))
 (format t "var = ~a~%" var))

Rather than the more imperative style one would expect in other languages:

(let ((var 1))
 (setf var (* var 2))
 (setf var (- var))
 (format t "var = ~a~%" var))

(Obviously this is an oversimplified example, that I would usually combine into a single declaration.)

I guess I just prefer the first version because I'm never actually mutating state. It seems a little more "functional" or clean - and of course, potentially more thread-safe.

But it occurs to me that the latter might be "cheaper" in terms of memory allocation or execution cycles.

Is one of these practices [a] more idiomatic than the other; or [b] more efficient in terms of memory usage or clock cycles?

EDIT: Real examples from my code.

(let* ((config-word      (take-multibyte-word data 2 :skip 1))
       (mem-shift-amount ...)
       (config-word      (translate-incoming-data config-word mem-shift-amount blank-value)))

  ...)

(let* ((reserve-byte-count (get-config :reserve-bytes))
       (reserve-byte-count (and (> reserve-byte-count 0) reserve-byte-count))
  ...)

(let* ((licence                (decipher encrypted-licence decryption-key))
       (licence-checksum1      (coerce (take-last 16 licence) '(vector (unsigned-byte 8))))
       (licence                (coerce (drop-last 16 licence) '(vector (unsigned-byte 8))))
       (licence-checksum2      (md5:md5sum-sequence (coerce licence '(vector (unsigned-byte 8)))))
       (licence                (and (equalp licence-checksum1 licence-checksum2)
                                    (decode licence))))
  ...)

Bag

The two forms, with the modern optimizing compilers, do not have practical significant differences in terms of execution times or space occupation. You can choose between them according to your idea of program readability and your stylistic preferences. Personally I would prefer the second solution since it conveys better the idea of applying sequentially a set of trasformations to a certain data. Note that let* is a sequential construct, so no multithread considerations can be applied to this case. Moreover, I've seen in other people's code more commonly the second form than the first one. – Internship

@Internship wouldn't things concerning memory change if the calculation in the let* clauses would contain free variables. Then setf would be more memory efficient? Are the compilers really optimize everything away? – Sampan

Answer 1

As mentioned in a comment, I would not worry about performance, at all, until you know it's a problem. It's not safe to assume you know what compilers will do with your code, or indeed what the machine will do with the code the compiler produces.

I should point out in advance that my Lisp style is probably idiosyncratic: I've written Lisp for a long time and fairly long ago stopped caring what other people think about my code (yes, I work on my own).

Also note that this whole answer is an opinion about style: I think questions of style in programming are interesting and important since programs are about communicating with people as much as they are about communicating with machines, and that this is particularly important in Lisp since Lisp people have always understood this. But still, opinions on style vary, and legitimately so.

I would find a form like

(let* ((var ...)
       (var ...)
       ...)
  ...)

awkward to read at best, because it smells like a mistake. The alternative

(let ((var ...))
  (setf var ...)
  ...)

seems more honest to me, although it's obviously grotty (any assignment makes me twitch, really, although obviously it's sometimes needed and I'm not some kind of functional purist). However in both cases I'd ask myself why is this being done? In particular if you have something like this:

(let* ((var x)
       (var (* var 2))
       (var (- var)))
  ...)

This is your original form except I've bound var to x so that the whole thing is not known at compile-time. Well, why not just write the expression you mean rather than bits of it?

(let ((var (- (* x 2))))
  ...)

which is easier to read and the same thing. Well, there are cases where you can't do that so easily:

(let* ((var (f1 x))
       (var (+ (f2 var) (f3 var))))
  ...)

More generally, if the value of some intermediate expression is used more than once then it needs to be bound to something. It's tempting to say that, well, you can turn the above form into this:

(let ((var (+ (f2 (f1 x)) (f3 (f1 x)))))
  ...)

But that's not safe: If we assume that f1 is expensive, then the compiler may or may not be able to coalesce the two calls to f1, but it certainly can only do so if it knows that f1 is side-effect free and deterministic, and that probably requires heroic optimisation strategies. Worse, if f1 is not actually a function, then this expression is not even equivalent to the previous one:

(let ((x (random 1.0)))
  (f x x))

Is not the same as

(f (random 1.0) (random 1.0))

This is true even without considering side-effects!

However I think the only time where you need to have intermediate bindings is where the value of some subexpression is used more than once in an expression: if it's only used once, then you can just splice it into the later expression as I did in your example above, and I would always do that unless the expressions were really stupidly large.

And in the case where either the subexpression is used more than once or the expressions were really complicated, what I would do is either use a different name for the intermediate value (it's not that hard to think up names for them) or just bind it where you need it. So starting from this:

(let* ((var (f1 x))
       (var (+ (f2 var) (f3 var))))
  ...)

I would turn it either into this:

(let* ((y (f1 x))
       (var (+ (f2 y) (f3 y))))
  ...)

This has the problem that y is bound in the body for no good reason, so I would probably turn it into this:

(let ((var (let ((y (f1 x)))
             (+ (f2 y) (f3 y)))))
  ...)

This binds the intermediate value exactly and only where it is needed. I think its main problem is that people from an imperative programming background who are not really comfortable with expression languages find it hard to read. But, well, as I said, I don't care about them.

---

One case where I probably would be OK with the (let* ((var ...) (var ... var ...)) ...) idiom is in macro-generated code: I might write macros which did this. But I'd never expect to have to read that code.

A case where I would never find (let* ((var ...) (var ...)) ...) (or, using my successively macro below (successively (var ... ...) ...) acceptable is where the multiple bindings of var refer to semantically different things, with the possible exception of some clearly-intermediate expression with a semantically-anonymous name (like, say, x in some hairy bit of arithmetic, in a context where x didn't naturally mean a coordinate). Using the same name for different things in the same expression is just horrible, I think.

As an example of that, I would take this expression from the question:

(let* ((licence                (decipher encrypted-licence decryption-key))
       (licence-checksum1      (coerce (take-last 16 licence) '(vector (unsigned-byte 8))))
       (licence                (coerce (drop-last 16 licence) '(vector (unsigned-byte 8))))
       (licence-checksum2      (md5:md5sum-sequence (coerce licence '(vector (unsigned-byte 8)))))
       (licence                (and (equalp licence-checksum1 licence-checksum2)
                                    (decode licence))))
  ...)

and turn it into this expression

(let* ((licence-with-checksum (decipher encrypted-licence decryption-key))
       (given-checksum (coerce (take-last 16 license-with-checksum)
                               '(vector (unsigned-byte 8))))
       (encoded-license (coerce (drop-last 16 license-with-checksum)
                        '(vector (unsigned-byte 8))))
       (computed-checksum (md5:md5sum-sequence encoded-license))
       (license (if (equalp given-checksum computed-checksum)
                            (decode encoded-license)
                          nil)))
  ...)

(I have removed at least one unneeded call to coerce as well, and I would worry about some of the others: is the license-with-checksum already of the right type? If so what are take-last and drop-last getting wrong that their results need to be coerced again?)

This has four distinct bindings which mean four different things which are named for what they mean:

• license-with-checksum is the license with attached checksum we're going to check and decode;
• given-checksum is the checksum we were given from license-with-checksum
• encoded-license is the encoded license from license-with-checksum;
• computed-checksum is the checksum we computed from encoded-license;
• license is the decoded license if the checksums match, or nil if they don't.

I might change some of these names if I knew more about the semantics of the operations involved.

And of course, if license turns out to be nil, we can now report the error using any or all of this information.

---

Thinking about this some more I wrote a macro called successively which does some of this in a way I find readable. If you say

(successively (x y
                 (* x x) 
                 (+ (sin x) (cos x)))
  x)

Then the macro expands to

(let ((x y))
  (let ((x (* x x)))
    (let ((x (+ (sin x) (cos x))))
      x)))

And even better you can say:

(successively ((x y) (values a b) (values (+ (sin x) (sin y))
                                          (- (sin x) (sin y))))
  (+ x y))

And you'll get

(multiple-value-bind (x y) (values a b)
  (multiple-value-bind (x y) (values (+ (sin x) (sin y)) (- (sin x) (sin y)))
    (+ x y)))

Which is quite nice.

Here's the macro:

(defmacro successively ((var/s expression &rest expressions) &body decls/forms)
  "Successively bind a variable or variables to the values of one or more
expressions.

VAR/S is either a single variable or a list of variables.  If it is a
single variable then it is successively bound by a nested set of LETs
to the values of the expression and any more expressions.  If it is a
list of variables then the bindings are done with MULTIPLE-VALUE-BIND.

DECLS/FORMS end up in the body of the innermost LET.

You can't interpose declarations between the nested LETs, which is
annoying."
  (unless (or (symbolp var/s)
              (and (listp var/s) (every #'symbolp var/s)))
    (error "variable specification isn't"))
  (etypecase var/s
    (symbol
     (if (null expressions)
         `(let ((,var/s ,expression))
            ,@decls/forms)
       `(let ((,var/s ,expression))
          (successively (,var/s ,@expressions)
            ,@decls/forms))))
    (list
     (if (null expressions)
         `(multiple-value-bind ,var/s ,expression
            ,@decls/forms)
       `(multiple-value-bind ,var/s ,expression
          (successively (,var/s ,@expressions) 
            ,@decls/forms))))))

I think this demonstrates just how easy it is to adapt Lisp to be the language you want it to be: it took me about five minutes to write this macro.

Mistress

You know, I never really thought about using let inside another let in this manner. You reckon that would be the preferable option? (Lisp is such an alien language.) – Bag

@SodAlmighty: I think to know what I'd write I'd need a more realistic example than yours (which I tried to invent in my answer). I think Lisp is only alien if you have significant exposure to imperative programming languages. – Mistress

Will add a real example. – Bag

I wish I was as good at writing macros as you. I'm not awful, but I do tend to feel my head unscrewing when I try to do things like that. How difficult would it be to modify the macro to automatically return x without having to be told? After all, presumably you'd always want to return the first variable - else why use the macro? – Bag

I do have significant exposure to other languages. I find it very difficult to think in lisp. I love the language, but it's hard. Oh, and I get no help from the compiler at all when debugging. Debugging compiler errors in SBCL is a veritable nightmare. – Bag

@SodAlmighty: for successively I don't want to return the variable(s) for the same reason I don't in let: it's bound so I can use it in the body of the expression. – Mistress

@SodAlmighty: I think your license example is a prime case of where I'd just cringe at the the same name being used for semantically-different objects (I've added a note about this to my answer now). I'm not meaning to be rude about your code: this is ultimately a style question, but I'd never do that. – Mistress

Mmm...they're not really semantically different. It's always a license - it's just in various stages of processing and decoding. And it's always a sequence of bytes. But I do see what you mean about it looking strange. This is, in part, why I asked the question. As to successively, wouldn't you in 99% of cases be returning x after threading it through the various operations? – Bag

Would you say that successively offered any tangible advantage over arrows in particular cases? Or should just I adopt the practice of using arrows in these situations? In fact, if you look at my licensing example above, you will note that I'm interleaving references to the license with other operations to extract and validate parts of the license. I'm not sure successively would help me there. – Bag

@SodAlmighty: no, I would not be using the bindings from successively as its value in most cases: if I wanted to do that I'd use some kind of composition macro. successively and arrows are not equivalent: successively is a binding form like let &c, the macros defined by arrows aren't: they're forms for threading values through operations (personally I find the implicit argument thing horrid in all these macros). I've added an example from your license code where I've given the various objects names I think are semantically more meaningful. – Mistress

Answer 2

I would say, none of the above; but rther:

(let* ((v0 1)
       (v1 (* v0 2))
       (v2 (- v1)))
 (format t "var = ~a~%" v2)

Substitute meaningful names, as needed. Good Lisp compilers have to be able to eliminate temporary variables, because macros copiously generate temporary variables in the form of gensyms. Often, those gensyms occur in instances when they are not actually needed; they are bound to something which doesn't cause any name clash and has no multiple evaluation issues.

For instance, in the following expansion, X itself could safely be used in place of the #:LIMIT-3374 gensym. Multiple evaluations of X cause no issues, and the loop doesn't mutate X:

[6]> (ext:expand-form '(loop for i below x collect i))
(BLOCK NIL
 (LET ((#:LIMIT-3374 X) (I 0))
  (LET ((#:ACCULIST-VAR-3375 NIL))
   (TAGBODY SYSTEM::BEGIN-LOOP (WHEN (>= I #:LIMIT-3374) (GO SYSTEM::END-LOOP))
    (SETQ #:ACCULIST-VAR-3375 (CONS I #:ACCULIST-VAR-3375)) (PSETQ I (+ I 1))
    (GO SYSTEM::BEGIN-LOOP) SYSTEM::END-LOOP
    (RETURN-FROM NIL (SYSTEM::LIST-NREVERSE #:ACCULIST-VAR-3375)))))) ;

There is rarely a good reason to bind the same name twice in the same construct, even if it is allowed in that type of construct.

You're only saving a few bytes of space in the compiler's environment by not interning additional symbols, and that's only if those symbols already don't occur somewhere else.

The one good reason is that the variable involved is a special variable, which has to be bound several times in order to appear in the dynamic scope of functions invoked from the init expressions:

(let* ((*context-variable* (foo)) 
       (*context-variable* (bar)))
  ...)

Both foo and bar react to the current value of the *context-variable* variable; foo calculates a value that can be used as *context-variable* and we would like bar to see that value rather than the one which foo was exposed to.

Or something like:

(let* ((*stdout* stream-x)
       (a (init-a)) ;; the output of these goes to stream-x
       (b (frob a))
       (*stdout* stream-y)
       (c (extract-from b)) ;; the output of this goes to stream-y
       ...)
  ...)

Lucania

Clever point about the special variables, but wouldn't you actually write (let ((*special* ...)) (let ((*special* ...)) ...))? I'd find that clearer! – Mistress

Answer 3

I'd either use significant names for the intermediate steps, or, if no sensible names present themselves, use a threading macro (e. g. from arrows (shameless plug)). I'd usually not setf local variables, but that's mostly a stylistic choice. (I do sometimes use setf on structures contained in local variables in order to avoid consing.)

Some compilers are better at compiling local let bindings than setf modifications, but that's maybe not the best reason to choose one or the other either.

Yalonda

I went and looked at arrows. It seems interesting, if a little confusing. I will check it out. – Bag