What are the benefits of switching from Rule and /. to OptionsPattern[] and OptionValue in a large application?

Question

Old habits die hard, and I realise that I have been using opts___Rule pattern-matching and constructs like thisoption /. {opts} /. Options[myfunction] in the very large package that I'm currently developing. Sal Manango's "Mathematica Cookbook" reminds me that the post-version-6 way of doing this is opts:OptionsPattern[] and OptionValue[thisoption]. The package requires version 8 anyway, but I had just never changed the way I wrote this kind of code over the years.

Is it worth refactoring all that from my pre-version-6 way of doing things? Are there performance or other benefits?

Regards

Verbeia

EDIT: Summary

A lot of good points were made in response to this question, so thank you (and plus one, of course) all. To summarise, yes, I should refactor to use OptionsPattern and OptionValue. (NB: OptionsPattern not OptionPattern as I had it before!) There are a number of reasons why:

1. It's a touch faster (@Sasha)
2. It better handles functions where the arguments must be in HoldForm (@Leonid)
3. OptionsPattern automatically checks that you are passing a valid option to that function (FilterRules would still be needed if you are passing to a different function (@Leonid)
4. It handles RuleDelayed (:>) much better (@rcollyer)
5. It handles nested lists of rules without using Flatten (@Andrew)
6. It is a bit easier to assign multiple local variables using OptionValue /@ list instead of having multiple calls to someoptions /. {opts} /. Options[thisfunction] (came up in comments between @rcollyer and me)

EDIT: 25 July I initially thought that the one time using the /. syntax might still make sense is if you are deliberately extracting a default option from another function, not the one actually being called. It turns out that this is handled by using the form of OptionsPattern[] with a list of heads inside it, for example: OptionsPattern[{myLineGraph, DateListPlot, myDateTicks, GraphNotesGrid}] (see the "More information" section in the documentation). I only worked this out recently.

Fatten

@Sasha, @rcollyer, @AndrewMoylan: these are all great points, thanks! To summarise so far, yes I should refactor if I get time because (1) it's faster (@Sasha); (2) it handles nested lists (@Andrew); and (3) it handles RuleDelayed. I should add a fourth: it simplifies the code to set local variables for lots of options at once (i.e. {opt1, opt2, opt3} = {OptionValue[myOption1], OptionValue[myOption2],OptionValue[myOption2]}). – Fatten

missed that one. The "lazy person" in me says that can be simplified further: {opt1, opt2, opt3} = OptionValue /@ { ... }. Also, to add to your summary, OptionsPattern[] and OptionValue[] produce cleaner code. – Wirer

duh! of course :-) Anyway I'll sit on these for a day, since it is not obvious which of these answers is the "best". They are all helpful, thanks! I am convinced to refactor now. – Fatten

Regardless of which way you end up with to handle options, if your package is very large, then it may make sense to split it into two or more packages according to the "low coupling - high cohesion" principle. I found empirically that for me it starts becoming difficult to maintain and evolve a package once it grows to more than 1000 lines of code. And b.t.w., WorkBench helps a lot for large projects. – Enjoy

Thank you to whoever pushed this question over the line to 10 up votes. – Fatten

@Fatten - Fabulous question and summary. Current documentation really should include some more examples and explain the evolution from OptionQ to OptionsPattern. (1) Does anyone else find it odd that Workbench 2.0's Option 'quick-fix' steers users toward OptionQ (without explaining why)?? (2) Sometimes examples of OptionsPattern actually 'name' the pattern---e.g., opts:OptionsPattern[]---what are the benefits and pitfalls of doing so? (I rarely name patterns used in the heads of expressions; I think due to some global conflicts that arose... should named patters be Protected?) – Heyes

@Heyes I continue to name my OptionsPatterns - always opts - out of habit, and for those rare cases where I want to pass a whole bunch of options to a supporting function, e.g. in a hypothetical mySpecialPlot, you just want to filter the options that are valid for ListPlot or whatever, and pass that to the underlying function. You can't use OptionValue for that because you don't know in advance which options the user has set. – Fatten

@Fatten - Thanks. I'm starting to get it now. :) – Heyes

Answer 1

While several answers have stressed different aspects of old vs. new way of using options, I'd like to make a few additional observations. The newer constructs OptionValue - OptionsPattern provide more safety than OptionQ, since OptionValue inspects a list of global Options to make sure that the passed option is known to the function. The older OptionQ seems however easier to understand since it is based only on the standard pattern-matching and isn't directly related to any of the global properties. Whether or not you want this extra safety provided by any of these constructs is up to you, but my guess is that most people find it useful, especially for larger projects.

One reason why these type checks are really useful is that often options are passed as parameters by functions in a chain-like manner, filtered, etc., so without such checks some of the pattern-matching errors would be very hard to catch since they would be causing harm "far away" from the place of their origin.

In terms of the core language, the OptionValue - OptionsPattern constructs are an addition to the pattern-matcher, and perhaps the most "magical" of all its features. It was not necessary semantically, as long as one is willing to consider options as a special case of rules. Moreover, OptionValue connects the pattern-matching to Options[symbol] - a global property. So, if one insists on language purity, rules as in opts___?OptionQ seem easier to understand - one does not need anything except the standard rule-substitution semantics to understand this:

f[a_, b_, opts___?OptionQ] := Print[someOption/.Flatten[{opts}]/.Options[f]]

(I remind that the OptionQ predicate was designed specifically to recognize options in the older versions of Mathematica), while this:

f[a_, b_, opts:OptionsPattern[]] := Print[OptionValue[someOption]]

looks quite magical. It becomes a bit clearer when you use Trace and see that the short form of OptionValue evaluates to a longer form, but the fact that it automaticaly determines the enclosing function name is still remarkable.

There are a few more consequences of OptionsPattern being a part of the pattern language. One is the speed improvements discussed by @Sasha. However, speed issues are often over-emphasized (this is not to detract from his observations), and I expect this to be especially true for functions with options, since these tend to be the higher-level functions, which will likely have non-trivial body, where most of the computation time will be spent.

Another rather interesting difference is when one needs to pass options to a function which holds its arguments. Consider a following example:

ClearAll[f, ff, fff, a, b, c, d];
Options[f] = Options[ff] = {a -> 0, c -> 0};
SetAttributes[{f, ff}, HoldAll];
f[x_, y_, opts___?OptionQ] :=
   {{"Parameters:", {HoldForm[x], HoldForm[y]}}, {" options: ", {opts}}};
ff[x_, y_, opts : OptionsPattern[]] :=
   {{"Parameters:", {HoldForm[x], HoldForm[y]}}, {" options: ", {opts}}};

This is ok:

In[199]:= f[Print["*"],Print["**"],a->b,c->d]
Out[199]= {{Parameters:,{Print[*],Print[**]}},{ options: ,{a->b,c->d}}}

But here our OptionQ-based function leaks evaluation as a part of pattern-matching process:

In[200]:= f[Print["*"],Print["**"],Print["***"],a->b,c->d]
During evaluation of In[200]:= ***
Out[200]= f[Print[*],Print[**],Print[***],a->b,c->d]

This is not completely trivial. What happens is that the pattern-matcher, to establish a fact of match or non-match, must evaluate the third Print, as a part of evaluation of OptionQ, since OptionQ does not hold arguments. To avoid the evaluation leak, one needs to use Function[opt,OptionQ[Unevaluated[opt]],HoldAll] in place of OptionQ. With OptionsPattern we don't have this problem, since the fact of the match can be established purely syntactically:

In[201]:= ff[Print["*"],Print["**"],a->b,c->d]
Out[201]= {{Parameters:,{Print[*],Print[**]}},{ options: ,{a->b,c->d}}}

In[202]:= ff[Print["*"],Print["**"],Print["***"],a->b,c->d]
Out[202]= ff[Print[*],Print[**],Print[***],a->b,c->d]

So, to summarize: I think choosing one method over another is largely a matter of taste - each one can be used productively, and also each one can be abused. I am more inclined to use the newer way, since it provides more safety, but I do not exclude that there exist some corner cases when it will surprise you - while the older method is semantically easier to understand. This is something similar to C-C++ comparison (if this is an appropriate one): automation and (possibly) safety vs. simplicity and purity. My two cents.

Enjoy

Answer 2

It seems like relying on pattern-matcher yields faster execution than by using PatternTest as the latter entails invocation of the evaluator. Anyway, my timings indicate that some speed-ups can be achieved, but I do not think they are so critical as to prompt re-factoring.

In[7]:= f[x__, opts : OptionsPattern[NIntegrate]] := {x, 
  OptionValue[WorkingPrecision]}

In[8]:= f2[x__, opts___?OptionQ] := {x, 
  WorkingPrecision /. {opts} /. Options[NIntegrate]}

In[9]:= AbsoluteTiming[Do[f[1, 2, PrecisionGoal -> 17], {10^6}];]

Out[9]= {5.0885088, Null}

In[10]:= AbsoluteTiming[Do[f2[1, 2, PrecisionGoal -> 17], {10^6}];]

Out[10]= {8.0908090, Null}

In[11]:= f[1, 2, PrecisionGoal -> 17]

Out[11]= {1, 2, MachinePrecision}

In[12]:= f2[1, 2, PrecisionGoal -> 17]

Out[12]= {1, 2, MachinePrecision}

Paronomasia

Interesting results. While this is logical, it never occurred to me that the difference may be so significant. +1 – Enjoy

Answer 3

A little known (but frequently useful) fact is that options are allowed to appear in nested lists:

In[1]:= MatchQ[{{a -> b}, c -> d}, OptionsPattern[]]

Out[1]= True

The options handling functions such as FilterRules know about this:

In[2]:= FilterRules[{{PlotRange -> 3}, PlotStyle -> Blue, 
  MaxIterations -> 5}, Options[Plot]]

Out[2]= {PlotRange -> 3, PlotStyle -> RGBColor[0, 0, 1]}

OptionValue takes it into account:

In[3]:= OptionValue[{{a -> b}, c -> d}, a]

Out[3]= b

But ReplaceAll (/.) doesn't take this into account of course:

In[4]:= a /. {{a -> b}, c -> d}

During evaluation of In[4]:= ReplaceAll::rmix: Elements of {{a->b},c->d} are a mixture of lists and nonlists. >>

Out[4]= a /. {{a -> b}, c -> d}

So, if you use OptionsPattern, you should probably also use OptionValue to ensure that you can consume the set of options the user passes in.

On the other hand, if you use ReplaceAll (/.), you should stick to opts___Rule for the same reason.

Note that opts___Rule is also a little bit too forgiving in certain (admittedly obscure) cases:

Not a valid option:

In[5]:= MatchQ[Unevaluated[Rule[a]], OptionsPattern[]]

Out[5]= False

But ___Rule lets it through:

In[6]:= MatchQ[Unevaluated[Rule[a]], ___Rule]

Out[6]= True

Update: As rcollyer pointed out, another more serious problem with ___Rule is that it misses options specified with RuleDelayed (:>). You can work around it (see rcollyer's answer), but it's another good reason to use OptionValue.

Bligh

I think it makes more sense to compare OptionsPattern[] with ___?OptionQ, which was designed to check for options and handles the above examples fine, rather than with ___Rule (although this is indeed the construct that the OP mentions). Also, in the past, there was a recommended way to deal with nested lists of options, which was to use OptionName/.Flatten[{opts}]/.Options[function]`, as described e.g. in the book of Roman Maeder "Programming in Mathematica" - so this is not a limitation of an older approach either. – Enjoy

@Leonid, I forgot about OptionQ. You're right, definitely superior to ___Rule. – Wirer

@Leonid: very good point. I remember switching from OptionQ to Rule` at some stage in the last 15 years in response to what I thought was the prefered way of doing things, around version 5, I guess. Looks like I need to catch up. :-) – Fatten

Answer 4

Your code itself has a subtle, but fixable flaw. The pattern opts___Rule will not match options of the form a :> b, so if you ever need to use it, you'll have to update your code. The immediate fix is to replace opts___Rule with opts:(___Rule | ___RuleDelayed) which requires more typing than OptionsPattern[]. But, for the lazy among us, OptionValue[...] requires more typing than the short form of ReplaceAll. However, I think it makes for cleaner reading code.

I find the use of OptionsPattern[] and OptionValue to be easier to read and instantly comprehend what is being done. The older form of opts___ ... and ReplaceAll was much more difficult to comprehend on a first pass read through. Add to that, the clear timing advantages, and I'd go with updating your code.

Wirer