SaveDefinitions considered dangerous

Question

SaveDefinitions is a nice option of Manipulate. It causes Manipulate to store any definitions used for its creation inside the Manipulate panel. A Manipulate made this way can be copied to an empty notebook and will still work on its own. Additionally, your working notebook containing many such Manipulates also doesn't turn into a flurry of pink boxes with printed error messages below it upon opening. Great!

However, all this goodness has its dark side which can bite you real hard if you are not aware of it. I've had this in a notebook I had been working on for a few days, but I present you with a step-by-step toy example scenario which recreates the problem.

In this scenario you want to create a Manipulate showing a plot of a nice wavy function, so you define this (please make a window size like this, this is important):

The definition is nice, so we keep it for the next time and make it an initialization cell. Next we add the Manipulate, and execute it too.

f[x_] := x^2

Manipulate[
 Plot[n f[x], {x, -3, 3}],
 {n, 1, 4},
 SaveDefinitions -> True
 ]

All works great, the Manipulate really shines, it is a good day.

Just being your paranoid self you check whether the definition is OK:

Yeah, everything still checks out. Fine. But now it occurs to you that a better wavy function would be a sine, so you change the definition, execute, and being paranoid, check:

Everything still fine. You're ready from a day's hard work you save your work and quit. [Quit kernel]

Next day. You start your work again. You evaluate the initialization cells in your notebook. Definition still good? Check.

Now, you scroll down to your Manipulate box (no need to re-execute thanks to the SaveDefinitions), play a little with the slider. And scroll back up.

Being the paranoid you, you once more check the definition of f:

Lo and behold, someone has changed the definition behind your back! And nothing executed between your first and second Information(?) check according to the In[] numbers (In[1]: def of f, In[2] first ?, In[3] second ?).

What happened? Well, it's the Manipulate of course. A FullForm reveals its internal structure:

Manipulate[Plot[n*f[x],{x, -3, 3}],{{n, 2.44}, 1, 4},Initialization:>{f[x_] := x^2}]

There you have the culprit. The initialization part of the box defines f again, but it's the old version because we didn't re-evaluate the Manipulate after modifying its definition. As soon as the manipulate box gets on the screen, it is evaluated and you've got your old definition back. Globally!

Of course, in this toy example it is immediately clear something strange is happening. In my case, I had a larger module in a larger notebook in which I, after some debugging, had changed a small part. It seemed to work, but the next day, the same bug that had bugged me before hit again. It took me a couple of hours before I realized that one of the several Manipulates that I used to study the problem at hand from all sides was doing this.

Clearly, I'm tempted to say, this is unwanted behavior. Now, for the obligatory question: what can we do to prevent this behind-your-back behavior of Manipulate from occurring other than re-executing every Manipulate in your notebook each time you change a definition that might be used by them?

Materfamilias

I see no real problem here. What you showed is a conflict of two methods of initialization. If you use SaveDefinitions -> True option in Manipulate you just should not use initialization cell for the same task! It is similar to creating two initialization cells in the same Notebook with different definitions for the same symbol... – Doelling

@alexey I don't agree. Both initialisations serve a different purpose. Clearly, SaveDefinitions is not meant to globally introduce definitions. And what it actually saves is hidden. Not nice for something with a global effect. – Materfamilias

In this sense I agree with you. It is just bad design of Manipulate. – Doelling

@Sjoerd, although what you describe makes complete sense (and I agree it's very bad), I can't seem to be able to reproduce it here, despite following your steps! Anyone else managed to reproduce? – How

This has happened to me many times, too... – Montevideo

@How which version/system do you have? Did you take care that the manipulate panel was out of sight during the crucial steps? – Materfamilias

@szabolcs Did you quit the kernel where indicated? – Materfamilias

@Sjoerd, it was out of site, but I haven't scrolled far enough apparently. "Works" now. – How

Answer 1

Here is an attempt. The idea is to identify symbols with DownValues or some other ...Values inside your manipulated code, and automatically rename them using unique variables / symbols in place of them. The idea here can be executed rather elegantly with the help of cloning symbols functionality, which I find useful from time to time. The function clone below will clone a given symbol, producing a symbol with the same global definitions:

Clear[GlobalProperties];
GlobalProperties[] :=
  {OwnValues, DownValues, SubValues, UpValues, NValues, FormatValues, 
      Options, DefaultValues, Attributes};


Clear[unique];
unique[sym_] :=
 ToExpression[
    ToString[Unique[sym]] <> 
       StringReplace[StringJoin[ToString /@ Date[]], "." :> ""]];


Attributes[clone] = {HoldAll};
clone[s_Symbol, new_Symbol: Null] :=
  With[{clone = If[new === Null, unique[Unevaluated[s]], ClearAll[new]; new],
        sopts = Options[Unevaluated[s]]},
     With[{setProp = (#[clone] = (#[s] /. HoldPattern[s] :> clone)) &},
        Map[setProp, DeleteCases[GlobalProperties[], Options]];
        If[sopts =!= {}, Options[clone] = (sopts /. HoldPattern[s] :> clone)];
        HoldPattern[s] :> clone]]

There are several alternatives of how to implement the function itself. One is to introduce the function with another name, taking the same arguments as Manipulate, say myManipulate. I will use another one: softly overload Manipulate via UpValues of some custom wrapper, that I will introduce. I will call it CloneSymbols. Here is the code:

ClearAll[CloneSymbols];
CloneSymbols /: 
Manipulate[args___,CloneSymbols[sd:(SaveDefinitions->True)],after:OptionsPattern[]]:=
   Unevaluated[Manipulate[args, sd, after]] /.
     Cases[
       Hold[args],
       s_Symbol /; Flatten[{DownValues[s], SubValues[s], UpValues[s]}] =!= {} :> 
          clone[s],
       Infinity, Heads -> True];

Here is an example of use:

f[x_] := Sin[x];
g[x_] := x^2;

Note that to use the new functionality, one has to wrap the SaveDefinitions->True option in CloneSymbols wrapper:

Manipulate[Plot[ f[n g[x]], {x, -3, 3}], {n, 1, 4}, 
          CloneSymbols[SaveDefinitions -> True]]

This will not affect the definitions of original symbols in the code inside Manipulate, since it were their clones whose definitions have been saved and used in initialization now. We can look at the FullForm for this Manipulate to confirm that:

Manipulate[Plot[f$37782011751740542578125[Times[n,g$37792011751740542587890[x]]],
   List[x,-3,3]],List[List[n,1.9849999999999999`],1,4],RuleDelayed[Initialization,
     List[SetDelayed[f$37782011751740542578125[Pattern[x,Blank[]]],Sin[x]],
       SetDelayed[g$37792011751740542587890[Pattern[x,Blank[]]],Power[x,2]]]]]

In particular, you can change the definitions of functions to say

f[x_]:=Cos[x];
g[x_]:=x;

Then move the slider of the Manipulate produced above, and then check the function definitions

?f
Global`f
f[x_]:=Cos[x]

?g
Global`g
g[x_]:=x

This Manipulate is reasonably independent of anything and can be copied and pasted safely. What happens here is the following: we first find all symbols with non-trivial DownValues, SubValues or UpValues (one can probably add OwnValues as well), and use Cases and clone to create their clones on the fly. We then replace lexically all the cloned symbols with their clones inside Manipulate, and then let Manipulate save the definitions for the clones. In this way, we make a "snapshot" of the functions involved, but do not affect the original functions in any way.

The uniqueness of the clones (symbols) has been addressed with the unique function. Note however, that while the Manipulate-s obtained in this way do not threaten the original function definitions, they will generally still depend on them, so one can not consider them totally independent of anything. One would have to walk down the dependency tree and clone all symbols there, and then reconstruct their inter-dependencies, to construct a fully standalone "snapshot" in Manipulate. This is doable but more complicated.

EDIT

Per request of @Sjoerd, I add code for a case when we do want our Manipulate-s to update to the function's changes, but do not want them to actively interfere and change any global definitions. I suggest a variant of a "pointer" technique: we will again replace function names with new symbols, but, rather than cloning those new symbols after our functions, we will use the Manipulate's Initialization option to simply make those symbols "pointers" to our functions, for example like Initialization:>{new1:=f,new2:=g}. Clearly, re-evaluation of such initialization code can not harm the definitions of f or g, and at the same time our Manipulate-s will become responsive to changes in those definitions.

The first thought is that we could just simply replace function names by new symbols and let Manipulate initialization automatically do the rest. Unfortunately, in that process, it walks the dependency tree, and therefore, the definitions for our functions would also be included - which is what we try to avoid. So, instead, we will explicitly construct the Initialize option. Here is the code:

ClearAll[SavePointers];
SavePointers /: 
Manipulate[args___,SavePointers[sd :(SaveDefinitions->True)],
after:OptionsPattern[]] :=
Module[{init},
  With[{ptrrules = 
    Cases[Hold[args], 
      s_Symbol /; Flatten[{DownValues[s], SubValues[s], UpValues[s]}] =!= {} :> 
         With[{pointer = unique[Unevaluated[s]]},
            pointer := s;
            HoldPattern[s] :> pointer], 
            Infinity, Heads -> True]},
           Hold[ptrrules] /. 
              (Verbatim[HoldPattern][lhs_] :> rhs_ ) :> (rhs := lhs) /. 
               Hold[defs_] :> 
                 ReleaseHold[
                      Hold[Manipulate[args, Initialization :> init, after]] /. 
                            ptrrules /. init :> defs]]]

With the same definitions as before:

ClearAll[f, g];
f[x_] := Sin[x];
g[x_] := x^2;

Here is a FullForm of produced Manipulate:

In[454]:= 
FullForm[Manipulate[Plot[f[n g[x]],{x,-3,3}],{n,1,4},
     SavePointers[SaveDefinitions->True]]]

Out[454]//FullForm=   
Manipulate[Plot[f$3653201175165770507872[Times[n,g$3654201175165770608016[x]]],
List[x,-3,3]],List[n,1,4],RuleDelayed[Initialization,
List[SetDelayed[f$3653201175165770507872,f],SetDelayed[g$3654201175165770608016,g]]]]

The newly generated symbols serve as "pointers" to our functions. The Manipulate-s constructed with this approach, will be responsive for updates in our functions, and at the same time harmless for the main functions' definitions. The price to pay is that they are not self-contained and will not display correctly if the main functions are undefined. So, one can use either CloneSymbols wrapper or SavePointers, depending on what is needed.

Conform

Thanks, Leonid. I have to study this more closely than I have time now, but one quick remark about your remark that Module might cause name conflicts in new sessions: I understand that using DynamicModule should prevent that (see: reference.wolfram.com/mathematica/tutorial/…) – Materfamilias

@Sjoerd Indeed, the option to use DynamicModule crossed my mind. But the variables created by DynamicModule belong to the FrontEnd, and somehow I better like the idea of using kernel variables for function definitions. I may be completely wrong with this, I even seem to remember some MathGroup discussions where this practice was discouraged. – Conform

@Sjoerd I simplified the implementation of clone, and implemented a custom unique function that should address the uniqueness problem, also for different Mathematica sessions. The code now should be easier to digest. – Conform

OK, I have studied this for some time. It's not the easiest to grasp, and I'm not sure whether I understand every bit of it (the new parameter in clone for instance is not entirely clear to me), but it looks like it will prevent the Manipulate panel from doing sneaky things. Good work! OTOH, some of the functionality of Manipulate seems to be lost now. If you change the function outside of the Manipulate environment, the Manipulate does not update to reflect this as it did before. Not sure whether this is good or bad. – Materfamilias

@Sjoerd The new parameter is just in case if user wants to supply a specific symbol to be the clone of the original symbol (I originally wrote this function for the case where this option was needed). It is then cleared and used, otherwise unique symbol is created. Regarding the loss of useful functionality, I have some ideas on that, but they need to be elaborated. Hope to get something better soon. – Conform

@Sjoerd I added a new version which produces Manipulate-s that are responsive to changes in functions, see my edit. I also suspect that my clone-based version will fail if some functions depend on (call) other user-defined functions, since apparently the initialization mechanism behind SaveDefinitions->True walks the dependency tree and collects also definitions on which the functions called inside Manipulate depend, and so on. Have no time to fix that right now, but it can be done. – Conform

currently I only have iPhone access to SO so I'll get back to this when i'm back. – Materfamilias

I deeply respect your efforts in this area. I'll mark this as the accepted answer, but I hope WRI will take up this issue somewhere in the future and come up with a robust, internalized answer. – Materfamilias

@Sjoerd Thanks! I also agree that the design which introduces or modifies global definitions is unfortunate, and share your hopes for a better built-in solution. But I don't see a simple and universal solution for this problem off the top of my head - which does not mean much, since my Dynamic - related skills are wanting. – Conform

@Leonid Your clone function will create all clones with delayed definitions only. So it will not create exact clones of the original functions. For example: a = 2; clone[Unevaluated[a], b]; Definition[b] gives b := 2. I brought up this problem earlier but had not got any solution without using Definition. – Doelling

@Alexey You are right, one can construct examples where OwnValues and possibly other ...Values do not fully convey the information available during the assignment, in particular whether or not it was delayed. This is a good point, but there are just a few cases in practice where this difference will be important. I wish there would be a completely programmatic way to get a full access to the information available during the creation of new global rules. – Conform

@Leonid Does not Definition give such a way? – Doelling

@Alexey Try to parse it programmatically, not just read it off the screen. – Conform

@Leonid Yes, it is not straightforward. One way to start is MakeExpression[ToBoxes[Definition[symbol]], StandardForm]. – Doelling

@Alexey I am afraid this does not work, as it only does an idle boxes->expression cycle with Definition. Try FullForm on your code output to see what I mean. – Conform

@Leonid You are right. But we still can hack BoxForms representation of Definition - it is sufficient just to remove InterpretationBox: a[1] = 1; a[2] = 2; ToExpression[ First@ToBoxes[Definition[a]], StandardForm, Hold] gives Hold[{{{{a[1] = 1}, {Null}, {a[2] = 2}}}}]. Evaluating this held expression will probably re-create all definitions associated with a. – Doelling

@Alexey Ok, good point, may be this can be an option. I will consider it for the future, whenever I have time to improve clone and test. – Conform

@Leonid The only problem with this method I see is that it evaluates the r.h. sides of immediate assignments. This problem can be avoided when defining ...Values. For example Block[{Rule}, SetAttributes[Rule, HoldRest]; OwnValues[a] = {Rule[HoldPattern[a], Print[a]]};]; Definition[a] gives a = Print[a]. It is special case of course but probably important for advanced users of Mathematica. – Doelling

Answer 2

The answer is to use initialization cell as initialization for the Manipulate:

Manipulate[
 Plot[n f[x], {x, -3, 3}], {n, 1, 4}, 
 Initialization :> FrontEndTokenExecute["EvaluateInitialization"]]

---

You can also use DynamicModule:

DynamicModule[{f},
 f[x_] := x^2;
 Manipulate[Plot[n f[x], {x, -3, 3}], {n, 1, 4}]]

You do not need SaveDefinitions -> True in this case.

EDIT

In response to Sjoerd's comment. With the following simple technique you do not need to copy the definition everywhere and update all copies if you change the definition (but you still need to re-evaluate your code to get updated Manipulate):

DynamicModule[{f}, f[x_] := x^2;
  list = Manipulate[Plot[n^# f[x], {x, -3, 3}], {n, 2, 4}] & /@ Range[3]];
list // Row

Doelling

OK. I see how the first one could help. However, it requires that the definition in question is in an initialization cell. I used that in my example, but that's not necessary to get this behind-your-back effect. The second one helps, but only if there aren't any other Manipulates that make use of the same function. You would have to copy the definition everywhere. And update all copies if you change the definition. – Materfamilias

i'm currently traveling. I'll check in two weeks. – Materfamilias