strange interaction between lexical-binding and defvar in emacs lisp

Question

The following emacs lisp file is about seeing what happens when Alice uses a lexically bound local variable foo in her init file and Bob defines foo as a global special variable with defvar in his init file and Alice borrows part of Bob's init file code into her own init file not knowing that foo is going to turn special.

;; -*- lexical-binding: t; -*-
;; Alice init file

;; Alice defining alice-multiplier
(defun alice-multiplier-1 (foo)
  (lambda (n) (* n foo)))
(defun alice-multiplier-2 (num)
  (let ((foo num))
    (lambda (n) (* n foo))))

;; Alice using alice-multiplier
(print
 (list
  :R1 (mapcar (alice-multiplier-1 10) (list 1 2 3))
  :R2 (mapcar (alice-multiplier-2 10) (list 1 2 3))))

;; from Bob's code
;; ...    
(defvar foo 1000)
;; ...

;; Alice using alice-multiplier
(print
 (list
  :R3 (mapcar (alice-multiplier-1 10) (list 1 2 3))
  :R4 (mapcar (alice-multiplier-2 10) (list 1 2 3))))

Output:

(:R1 (10 20 30) :R2 (10 20 30))

(:R3 (10 20 30) :R4 (1000 2000 3000))

Results R1 and R2 are just as what I expect. Result R4 is consistent with defvar documentation, although it may surprise Alice unless she reads Bob's code.

1. I find R3 surprising. Why is R3 like that?

2. Speaking of R4, what can Alice do to protect her foo from turning special by others? For example, foo may be a lexical local variable she uses in her init file or one of her emacs package, and (defvar foo "something") may be in some of the packages she happens to use, or foo could be one of the new special variable names introduced by a future version of Emacs. Is there something Alice can put in her file that says to Emacs "In this file, foo should be always lexical, even if some code from outside happens to use a special variable of the same name"?

Craftsman

That's not "code from outside", though, is it? – Antivenin

@Antivenin I think that “code from outside” is used in the sense that I might copy and paste code from the web into my .emacs file, not that it's from outside the file. – Brescia

Actually, this can get messy, can't it? I know the byte compiler complains if dynamic variable names do not contain some kind of prefix for name-spacing, and the naming conventions recommend name-spacing all of your variables this way. I suspect that's ultimately the answer, but it does make one wish that some kind of proper name spacing had been introduced to complement the lexical binding changes... – Antivenin

@Antivenin "code from outside" can be an emacs package written by Bob superdired which defvars foo and Mallory may happen to install two packages superdired and supercalc, the latter being a package written by Alice to provide functions like alice-multiplier. When Mallory does something with a superdired buffer, and then uses alice-multiplier in another unrelated task, Mallory will see R4 if Alice has written alice-multiplier like alice-multiplier-2. – Craftsman

Answer 1

What is going on

From the "theoretical" (Scheme/Common Lisp) point of view, as soon as you enable lexical bindings, for all practical purposes alice-multiplier-1 and alice-multiplier-2 are identical. Any difference in their behavior is a bug in Emacs Lisp and should be reported as such.

Compiled

If you put your code (i.e., the 2 defuns and the ;; -*- lexical-binding: t; -*- line) into a file, emacs-list-byte-compile-and-load it, then you can test my claim by evaluating these 4 forms:

(disassemble 'alice-multiplier-1)
(disassemble 'alice-multiplier-2)
(disassemble (alice-multiplier-1 10))
(disassemble (alice-multiplier-2 10))

you will see that 3 and 4 are identical and 1 and 2 differ by one instruction (which should be reported as a bug to the Emacs maintainers, but does not affect the behavior).

Note that none of the disassemblies mention foo, which means that the defvar will not affect their behavior.

All is good!

Interpreted

Indeed, the behavior you see is incorrect; the correct result after defvar is

(:R1 (10000 20000 30000) :R2 (10000 20000 30000))

please report this to the emacs maintainers.

Different???

Yes, defvar does (and should!) affect the behavior of interpreted code and does not (and should not!) affect the behavior of compiled code.

What you should do

There is no way to "protect" your foo from being proclaimed special by others - except by prefixing "your" symbols with alice-.

However, if you byte-compile the file with the alice-multiplier-1 definition, the compiled file does not even contain foo and thus future declarations of foo will not affect you.

Locket

Answer 2

As to the second question, as far as I know, none. But one can still work around. That is to adopt two naming conventions: which I will call yellow and green.

yellow naming convention

All special variables must have yellow names. A yellow name is a name that include at least one hyphen. For example hello-world and ga-na-da are yellow names. The official Emacs Lisp manual and the byte compiler encourage this convention.

green naming convention

A green name is a name that is not yellow. For example, helloworld and ganada are green names.

All lexical nonlocal/free variables must have green names. What is a nonlocal variable? The body of the anonymous function within alice-multiplier-2 mentions three names, n, foo, num. Of these three, only n has a declaration within the function body (of the anonymous function). The other two are nonlocal from the point of view of the anonymous function. They are nonlocal variables.

As long as Alice and Bob stick to the two naming conventions, all is well. Even if they don't now, it's likely that the two people will converge to these conventions in the end on their own without mutual communication even, through these stages:

1. Alice and Bob stick to none of the conventions.

2. Since the manual and the byte compiler encourage yellow naming convention, there comes a point when both Alice and Bob start sticking to the yellow convention.

3. Alice adopts green convention which at least protects her code from yellow special variables by others.

4. Alice and Bob stick to both conventions.

For precise conditions in which collisions can occur, see Invasion of special variables

Craftsman