What is open recursion?

Question

What is open recursion? Is it specific to OOP?

(I came across this term in this tweet by Daniel Spiewak.)

Disinfection

Came here from one of his other more recent tweet "My definition of OO is message passing married with open recursion." – Kafiristan

@letronje, you might want yo have a look at the paper I just linked. – Disinfection

Given the fact this is the first entry when I google "open recusion", it derserves some more comprehensive answers. Before I've done that, there are some other closing discussions probably worth reading, e.g.: stackoverflow.com/questions/17803621, news.ycombinator.com/item?id=20495597. – Delwyn

Answer 1

just copying http://www.comlab.ox.ac.uk/people/ralf.hinze/talks/Open.pdf: "Open recursion Another handy feature offered by most languages with objects and classes is the ability for one method body to invoke another method of the same object via a special variable called self or, in some langauges, this. The special behavior of self is that it is late-bound, allowing a method defined in one class to invoke another method that is defined later, in some subclass of the first. "

Casease

"open recursion" just seems like a fancy name for the state monad in Haskell, or doing recursion with regular object methods (the first argument is self) in python...? Is there something new and meaningful this term brings to the table? – Kitty

Yes, regular object methods in Python (or Ruby or C++ or Smalltalk or Java or C# or ...) can exhibit open recursion. You might not think it's a big deal, but if you ever formally study the semantics of OO you'll see it's actually quite tricky. – Ignorance

This anwser seem a bit misleading to me. Open recursion is the ability for one method body to invoke another method of the same object (mutually recursive definitions, that's why recursive) and allowing a method defined in one class to invoke another method that is defined later in some subclass of the first ( by opening the closure of the base class and letting the derived class in so the base class can see it e.g. making it virtual in C++). See journal.stuffwithstuff.com/2013/08/26/what-is-open-recursion for a well written article bout this topic. – Forayer

Answer 2

This paper analyzes the possibility of adding OO to ML, with regards to expressivity and complexity. It has the following excerpt on objects, which seems to make this term relatively clear –

3.3. Objects

The simplest form of object is just a record of functions that share a common closure environment that carries the object state (we can call these simple objects). The function members of the record may or may not be defined as mutually recursive. However, if one wants to support inheritance with overriding, the structure of objects becomes more complicated. To enable open recursion, the call-graph of the method functions cannot be hard-wired, but needs to be implemented indirectly, via object self-reference. Object self-reference can be achieved either by construction, making each object a recursive, self-referential value (the fixed-point model), or dynamically, by passing the object as an extra argument on each method call (the self-application or self-passing model).5 In either case, we will call these self-referential objects.

Disinfection

Open recursion is more of a side effect brought by late-binding and inheritance, which is why Daniel Spiewak can coin the saying "OO is message passing married with open recursion", given inheritance and subtyping polymorphism are among the pillars of OOP. – Sheik

Answer 3

The name "open recursion" is a bit misleading at first, because it has nothing to do with the recursion that normally is used (a function calling itself); and to that extent, there is no closed recursion. It basically means, that a thing is referring to itself. I can only guess, but I do think that the term "open" comes from open as in "open for extension". In that sense an object is open to extension, but still referring to itself.

Perhaps a small example can shed some light on the concept.

Imaging you write a Python class like this one:

class SuperClass:
   def method1(self):
       self.method2()
   def method2(self):
       print(self.__class__.__name__)

If you ran this by

s = SuperClass()
s.method1()

It will print "SuperClass".

Now we create a subclass from SuperClass and override method2:

class SubClass(SuperClass):
def method2(self):
    print(self.__class__.__name__)

and run it:

sub = SubClass()
sub.method1()

Now "SubClass" will be printed.

Still, we only call method1() as before. Inside method1() the method2() is called, but both are bound to the same reference (self in Python, this in Java). During sub-classing SuperClass method2() is changed, which means that an object of SubClass refers to a different version of this method.

That is open recursion.

In most cases, you override methods and call the overridden methods directly. This scheme here is using an indirection over self-reference.

P.S.: I don't think this has been invented but discovered and then explained.

Extraversion

This is a direct application of the term "recursion" in the normal sense. OTOH, it is a popular misconception that recursion is about recursive calls. Recursion of calls is one instance of more general recursion. And here the recursion is exactly about the calls in the meta language (e.g. in the formal semantics, or an interpreter) which implementing the name lookup rules of the object language. BTW, "open" is the opposite of "closed" which has the same etymology of "closure", i.e. the "closed terms" (in rewrite systems like lambda calculi). – Delwyn

Answer 4

In short, open recursion is about something actually not related to OOP, but more general.

The relation with OOP comes from the fact that many typical "OOP" PLs have such properties, but it is essentially not tied to any distinguishing features about OOP.

So there are different meanings, even in same "OOP" language. I will illustrate it later.

Etymology

As mentioned here, the terminology is likely coined in the famous TAPL by BCP, which illustrates the meaning by concrete OOP languages.

TAPL does not define "open recursion" formally. Instead, it points out the "special behavior of self (or this) is that it is late-bound, allowing a method defined in one class to invoke another method that is defined later, in some subclass of the first".

Nevertheless, neither of "open" and "recursion" comes from the OOP basis of a language. (Actually, it is also nothing to do with static types.) So the interpretation (or the informal definition, if any) in that source is overspecified in nature.

Ambiguity

The mentioning in TAPL clearly shows "recursion" is about "method invocation". However, it is not that simple in real languages, which usually do not have primitive semantic rules on the recursive invocation itself. Real languages (including the ones considered as OOP languages) usually specify the semantics of such invocation for the notation of the method calls. As syntactic devices, such calls are subject to the evaluation of some kind of expressions relying on the evaluations of its subexpressions. These evaluations imply the resolution of method name, under some independent rules. Specifically, such rules are about name resolution, i.e. to determine the denotation of a name (typically, a symbol, an identifier, or some "qualified" name expressions) in the subexpression. Name resolution often respects to scoping rules.

OTOH, the "late-bound" property emphasizes how to find the target implementation of the named method. This is a shortcut of evaluation of specific call expressions, but it is not general enough, because entities other than methods can also have such "special" behavior, even make such behavior not special at all.

A notable ambiguity comes from such insufficient treatment. That is, what does a "binding" mean. Traditionally, a binding can be modeled as a pair of a (scoped) name and its bound value, i.e. a variable binding. In the special treatment of "late-bound" ones, the set of allowed entities are smaller: methods instead of all named entities. Besides the considerably undermining the abstraction power of the language rules at meta level (in the language specification), it does not cease the necessity of traditional meaning of a binding (because there are other non-method entities), hence confusing. The use of a "late-bound" is at least an instance of bad naming. Instead of "binding", a more proper name would be "dispatching".

Worse, the use in TAPL directly mix the two meanings when dealing with "recusion". The "recursion" behavior is all about finding the entity denoted by some name, not just specific to method invocation (even in those OOP language).

The title of the chapter (Case Study: Imperative Objects) also suggests some inconsistency. Obviously, the so-called late binding of method invocation has nothing to do with imperative states, because the resolution of the dispatching does not require mutable metadata of invocation. (In some popular sense of implementation, the virtual method table need not to be modifiable.)

Openness

The use of "open" here looks like mimic to open (lambda) terms. An open term has some names not bound yet, so the reduction of such a term must do some name resolution (to compute the value of the expression), or the term is not normalized (never terminate in evaluation). There is no difference between "late" or "early" for the original calculi because they are pure, and they have the Church-Rosser property, so whether "late" or not does not alter the result (if it is normalized).

This is not the same in the language with potentially different paths of dispatching. Even that the implicit evaluation implied by the dispatching itself is pure, it is sensitive to the order among other evaluations with side effects which may have dependency on the concrete invocation target (for example, one overrider may mutate some global state while another can not). Of course in a strictly pure language there can be no observable differences even for any radically different invocation targets, a language rules all of them out is just useless.

Then there is another problem: why it is OOP-specific (as in TAPL)? Given that the openness is qualifying "binding" instead of "dispatching of method invocation", there are certainly other means to get the openness.

One notable instance is the evaluation of a procedure body in traditional Lisp dialects. There can be unbound symbols in the body and they are only resolved when the procedure being called (rather than being defined). Since Lisps are significant in PL history and the are close to lambda calculi, attributing "open" specifically to OOP languages (instead of Lisps) is more strange from the PL tradition. (This is also a case of "making them not special at all" mentioned above: every names in function bodies are just "open" by default.)

It is also arguable that the OOP style of self/this parameter is equivalent to the result of some closure conversion from the (implicit) environment in the procedure. It is questionable to treat such features primitive in the language semantics.

(It may be also worth noting, the special treatment of function calls from symbol resolution in other expressions is pioneered by Lisp-2 dialects, not any of typical OOP languages.)

More cases

As mentioned above, different meanings of "open recursion" may coexist in a same "OOP" language.

C++ is the first instance here, because there are sufficient reasons to make them coexist.

In C++, name resolution are all static, normatively name lookup. The rules of name lookup vary upon different scopes. Most of them are consistent with identifier lookup rules in C (except for the allowance of implicit declarations in C but not in C++): you must first declare the name, then the name can be lookup in the source code (lexically) later, otherwise the program is ill-formed (and it is required to issue an error in the implementation of the language). The strict requirement of such dependency of names are considerable "closed", because there are no later chance to recover from the error, so you cannot directly have names mutually referenced across different declarations.

To work around the limitation, there can be some additional declarations whose sole duty is to break the cyclic dependency. Such declarations are called "forward" declarations. Using of forward declarations still does not require "open" recursion, because every well-formed use must statically see the previous declaration of that name, so each name lookup does not require additional "late" binding.

However, C++ classes have special name lookup rules: some entities in the class scope can be referred in the context prior to their declaration. This makes mutual recursive use of name across different declarations possible without any additional "forward" declarations to break the cycle. This is exactly the "open recursion" in TAPL sense except that it is not about method invocation.

Moreover, C++ does have "open recursion" as per the descriptions in TAPL: this pointer and virtual functions. Rules to determine the target (overrider) of virtual functions are independent to the name lookup rules. A non-static member defined in a derived class generally just hide the entities with same name in the base classes. The dispatching rules kick in only on virtual function calls, after the name lookup (the order is guaranteed since evaulations of C++ function calls are strict, or applicative). It is also easy to introduce a base class name by using-declaration without worry about the type of the entity.

Such design can be seen as an instance of separate of concerns. The name lookup rules allows some generic static analysis in the language implementation without special treatment of function calls.

OTOH, Java have some more complex rules to mix up name lookup and other rules, including how to identify the overriders. Name shadowing in Java subclasses is specific to the kind of entities. It is more complicate to distinguish overriding with overloading/shadowing/hiding/obscuring for different kinds. There also cannot be techniques of C++'s using-declarations in the definition of subclasses. Such complexity does not make Java more or less "OOP" than C++, anyway.

Other consequences

Collapsing the bindings about name resolution and dispatching of method invocation leads to not only ambiguity, complexity and confusion, but also more difficulties on the meta level. Here meta means the fact that name binding can exposing properties not only available in the source language semantics, but also subject to the meta languages: either the formal semantic of the language or its implementation (say, the code to implement an interpreter or a compiler).

For example, as in traditional Lisps, binding-time can be distinguished from evaluation-time, because program properties revealed in binding-time (value binding in the immediate contexts) is more close to meta properties compared to evaluation-time properties (like the concrete value of arbitrary objects). An optimizing compiler can deploy the code generation immediately depending on the binding-time analysis either statically at the compile-time (when the body is to be evaluate more than once) or derferred at runtime (when the compilation is too expensive). There is no such option for languages blindly assume all resolutions in closed recursion faster than open ones (and even making them syntactically different at the very first). In such sense, OOP-specific open recursion is not just not handy as advertised in TAPL, but a premature optimization: giving up metacompilation too early, not in the language implementation, but in the language design.

Delwyn

Answer 5

Open recursion allows to call another methods of object from within, through special variable like this or self.

Rajasthani