Scope of functions and variables within a function and/or let statement for lisp

Question

I'm having some trouble understanding the scope of variables and functions when defined inside a function call. I tried searching for this scope, but could not find a suitable answer (or maybe was searching for the wrong thing), so I decided to write a few functions to test things myself:

(defun test-scope1 ()
  (setf myvar 1)
  (defun set-var1 ()
    (setf myvar 2))
  (set-var1))

With this function I just wanted to see if anything gets set global. I would expect myvar and set-var to be defined globally because I have no scope here. As expected, before calling (test-scope1) the commands myvar and (set-var) give me errors. After calling (test-scope1), I can run myvar and (set-var) in the interpreter to yield 2.

Thinking to myself it would be nice to encapsulate my variables, I decided to add a let inside my function, therefore getting my next test:

(defun test-scope2 ()
  (let ((myvar 10))
    (defun set-var2 ()
      (setf myvar 20))
    (set-var2)))

I would expect myvar to be stuck in the scope of the let block, but wouldn't be able to guess for set-var2. It could be stuck in the let block or it could be defined globally. After running (test-scope2) I try to access myvar and get 2. That means this new function has its own myvar since it's still 2 from the previous function. I try running (set-var2)and get 20.

I'm not completely surprised that the function is defined globally after being run in the let block, but now I'm very confused what myvar variable its accessing. Since it didn't change my global copy of myvar, it would seem there's some variable floating around it still refers to.

Now I want to see if I can manipulate that floating variable, so I create this third function.

(defun test-scope3 ()
  (let ((myvar (if (ignore-errors myvar)
                   myvar
                   100)))
    (defun set-var3 ()
      (setf myvar (+ myvar 100)))
    (set-var3)))

Instead of just setting the variable to a fixed value, I want to increment it based on the previous value. I check two things here. The first is when test-scope3 is called. I wanted to see if I could pull up a "previous value" of myvar, since if it's floating around somewhere maybe I can access it again. It probably wouldn't be good practice, but that's not the point here. Ignore-errors is there in case there wasn't really a previous value floating around, in which case I choose a default of 100.

The second thing I test is to have set-var3 add 100 to the previous the value of myvar. I want to see if this will adjust that floating variable, or if somehow is something static. I have no idea what my function "should" return.

After running (test-scope3) I'm completely surprised to see 102. So apparently my test-scope3 found the value of myvar from running test-scope1. But, after checking the value of myvar in the interpreter, it is still 2. Next I run (set-var3) and get a return value of 202. Okay, so it added 100 again to my previous value. Calling it again returns 302 and so on. But, calling (test-scope3) again resets this value back down to 102.

I wrote one more function as a double nested let command. I just ran this in two ways: with no myvar definition for the let arguments. This function returns 10002. Then, I tried setting the local myvar to 50 and had 1050 return.

(defun test-scope4 ()
  (let () ; or ((myvar 50))
    (let ((myvar (if (ignore-errors myvar)
                      myvar
                      2000)))
      (defun set-var4 ()
        (setf myvar (+ myvar 1000)))
    (set-var4))))

So, with all this together, here are some of my specific questions:

1. When I do a defun inside another defun, even in a let block, why does that function become accessible globally?
2. When I call set-var#, what variables (or within what scope) is this accessing? Or maybe more appropriately, when I define a function inside another statement, what am I actually binding?
3. When I use the variable myvarin a function, where does this pull from? From my 4th example conjecture is that it looks for the symbol myvar in the current scope, then checks up one level higher until it finds a value for it.

---

Sorry if everything was very wordy and my questions ill-formed. I tried to investigate things to the best of my ability. Really this all leads to my real question, which after writing everything up I realize might be beyond the scope (no pun intended) of this question as I've set it up so far.

The issue of hiding my inner functions as given above could be handled with a lambda expression; however, what I'd really love to do is to have a recursive function inside of a bigger block that uses the block to store values without feeding them directly into the function. To my knowledge, this is not possible with a lambda expression. For example, consider the following function which does this.

(defun outer-function (start)
  (let ((x start))
    (defun increment-to-ten ()
      (setf x (+ x 1))
      (if (< x 10)
          (increment-to-ten)))
    (increment-to-ten)
    (print x)))

Which could be instead implemented recursively with arguments as

(defun increment-to-ten-recursive (x)
  (if (< x 10)
      (increment-to-ten-recursive (+ x 1))
      10))

If there's a solution for this, that would be great, or if my thinking is completely wrong and there's a better way to do this that would be great to. It just seems convenient to have a block store data for you and then just call a recursive function with no arguments to work on that data.

Twitty

Answer 1

Why it is difficult to discuss the effects you see:

You are doing things which are undefined in Common Lisp: setting undeclared variables in test-scope1: myvar. It's from then on unclear how the following code behaves.

Undeclared Variables

It is undefined what effect it has when an undeclared variable foo is set. Implementations allow it. Some will warn. SBCL:

* (setf foo 10)
; in: SETF FOO
;     (SETF FOO 10)
; ==>
;   (SETQ FOO 10)
; 
; caught WARNING:
;   undefined variable: FOO
; 
; compilation unit finished
;   Undefined variable:
;     FOO
;   caught 1 WARNING condition

Global variables are defined with DEFPARAMETER and DEFVAR. Local variables are defined by LET, LET* and by functions parameters. Since DEFPARAMETER and DEFVAR define global special (using dynamic binding) variables, it is common to write them as *some-variable* - note the * around it, which are part of the symbol name and not special syntax. It's a convention for special variables.

Nested DEFUNs

Nested DEFUNs are not used in Common Lisp. DEFUN is a top-level form, which sets a global function. To define local functions use FLET and LABELS. You can nest DEFUN forms, but it is bad style and you won't find it used in actual Lisp code. Don't nest DEFUN forms. Note: this is different from Scheme, where one can nest DEFINE forms. Since nested DEFUNs are not used, it makes little sense to discuss the effects. Convert the examples to use local functions defined by FLET or LABELS.

• When I do a defun inside another defun, even in a let block, why does that function become accessible globally?

Because DEFUN defines global functions. That's its purpose.

You need to rewrite your examples to make it possible to discuss the effects:

• declare all variables

• don't use nested DEFUNs

We could try to understand the code you have in your current examples, but much depends on the implementation and the sequence of actions. We are either not in the realm of portable Common Lisp or doing stuff (nested DEFUNs), which has no practical use in Common Lisp.

Zebra

The issue with defun being a top-level form and me thinking that let would keep anything I made in it to that block is the issue that I was having. I am a bit unsure what you mean when you say my variables are undeclared though. Do you mean because my defuns are top level functions, using (setf myvar #) inside of them has an undefined effect since I didn't declare myvar at the top level? – Twitty

@Nyles: any SETF on an undefined variable has undefined effects. Make sure your variables are defined first. SETF does not define variables, it just sets them. – Zebra

Are the problems with undeclared variables which you mention from the the (setf myvar #) statements within my nested defuns? For let blocks (without nested defuns), If I have a something like (let ((myvar 10)) (setf myvar 20)), is this okay? Is the initial argument for the let block considered a declaration for myvar or not (in the scope of the let block)? – Twitty

@Nyles: see my text above. LET defines variables. SETF not. If a variable has been defined, for example by an enclosing LET or by DEFUN, LAMBDA, DEFMETHOD, DEFPARAMETER, DEFVAR, ... , you can set them with SETF. – Zebra

Thank you very much for you quick and detailed responses, I appreciate them. It looks labels are what I need for my recursive uses. – Twitty

@Twitty Perhaps a useful note: (defun foo (...) ...) is essentially (but an implementation may do additional things) (setf (symbol-function 'foo) (lambda (...) ...)). Now compare with what would happen if you return a closure over variables from let, e.g., (let ((a nil)) (lambda () (setf a (+ 1 a)))). You can call that function again and again, and you'll get increasing numbers. The function is a closure over the lexical variable a. – Adige

Answer 2

As Rainer explained, the answer to your first question is that defun defines global functions. Use labels to define local functions.

Rainer is also right about much being dependent on the configuration of your lisp. Many Lisps know about two types of scope, lexical and dynamic. Lexical scope is the one you can see when you look at a page of code, and what you're used to from most other programming languages. Dynamic scope is what you get when you a lisp looks for the value of a variable in the environment where the function was called instead of the one where it was defined.

Finally, your code makes usse of closures. When you put a lambda (defun) inside a let, you create a closure, e.g. your variables "hang or float around" for the function to use in the future even when the function that contains them returns to it's caller.

So here's what might be going on (as Rainer said, it's hard to know since you're doing things that usually aren't done, such as nesting defuns):

So... probably, test-scope1 uses a global myvar and it's set-var references the global myvar since we are dealing with lexical scope. test-scope2 builds a closure, which creates a myvar with lexical scope only accessible to set-var2. Since this myvar has lexical scope, the call to set-var in test-scope2 will not use this myvar but still the global one.

I'm not sure why calling test-scope3 gets you 102, since test-scope3 just calls set-var, which should set the global myvar to 2, not 102. The only code that could explain 102 is in set-var3, and this isn't being called when you call test-scope3. Here's what might be going on if you got 2 instead of 102 when calling test-scope3:

(defun test-scope3 ()
  (let ((myvar (if (ignore-errors myvar)
                   myvar
                   100)))
    (defun set-var3 ()
      (setf myvar (+ myvar 100)))
    (set-var)))

With the let, you create a local binding for myvar with lexical scope, but you refer to the global myvar in it's binding form - the (if (ignore-errors myvar) myvar). Later on you call set-var, which should again reference the global myvar since test'scope3's let uses lexical scope for the binding of myvar. Then, wen you call set-var3 for the first time, it takes the myvar bound by the above let, which currently has a value of 2, and adds 100 to it. When you call it the second time, you again add 100 to the lexically bound myvar which still hangs around in the closure you built in test-scope3, and so on. But when you call test-scope3 again, you create a new closure (but replace the old one because you use defun to give the same name to it...) which again sets the initial value of the closed-over myvar to the value of the global myvar, which is 2.

I suggest you read Paul Graham's excellent ANSI Common Lisp and On Lisp, which explain special variables (dynamic scope), lexical scope and closures in depth.

Here's how you might have several closures share variables:

(let ((myvar 0))
  (defun reset () 
    (setf myvar 0)
  (defun inc ()
    (setf myvar (+ myvar 1))))

This builds two closures and makes them share myvar, and since myvar has lexical scope you can't access this copy of myvar from anywhere else. So you've basically given exclusive control of myvar to reset and inc.

This could be one way to build your recursive function that doesn't need a value passed into it with each iteration.

Carmelacarmelia

My bad on the accidental (set-var) in some of the functions. Each of the numbers should always match. I edited my post, so it should read much clearer now. Also, I didn't know those closures hung around. That's what my investigation let me to believe, but I wasn't quite sure if it was exactly true or not. – Twitty

For example CMUCL will by default (last I looked) declare an undefined variable to be special. Thus after the first (setf myvar ...), all the following myvars are special - using dynamic binding. This can alter the semantics of the program completely. – Zebra

From the effects Nyles describes, I guessed his Lisp uses no special variables in the examples. (so only lexical, no dynamic scope). You're right of course that this is in no way portable. – Carmelacarmelia

Yes, but it makes no sense to discuss these effects, since they depend on undefined behavior in Common Lisp. One might look at two different implementations of Common Lisp (say CMUCL and SBCL) and they may do something completely different. – Zebra