Why does method inheritance kill additional arguments?

Question

I want to set some flags in my generic (before calling UseMethod(), I know that much :)), and then use and/or update these flags in the methods.

Like so:

g <- function(x) {
  y <- 10
  UseMethod("g")
}
g.default <- function(x) {
  c(x = x, y = y)
}
g.a <- function(x) {
  y <- 5  # update y from generic here
  NextMethod()
}

This works when jumping directly to the default method:

g(structure(.Data = 1, class = "c"))  # here y is never updated
# x  y 
# 1 10 

But when I go through NextMethod(), y mysteriously disappears:

g(structure(.Data = 1, class = "a"))  # here y is updated, but cannot be found
# Error in g.default(structure(.Data = 1, class = "a")) : 
#  object 'y' not found

I have figured out how to fix this, by just passing around y a lot:

f <- function(x, ...) {
  y <- 10
  UseMethod("f")
}
f.default <- function(x, ..., y = 3) {
  c(x = x, y = y)
}
f.a <- function(x, ...) {
  y <- 5
  NextMethod(y = y)
}

which yields

f(structure(.Data = 1, class = "c"))
# x y 
# 1 3 
f(structure(.Data = 1, class = "a"))
# x y 
# 1 5 

My question is: Why does NextMethod() in the above g().*-example kill the additional y argument?

I thought the whole point of UseMethod() and NextMethod() was to pass on any and all objects from call to call, without having to manually pass them on:

NextMethod works by creating a special call frame for the next method. If no new arguments are supplied, the arguments will be the same in number, order and name as those to the current method but their values will be promises to evaluate their name in the current method and environment.

• help("UseMethod")

I'm particularly confused that UseMethod() does seem to pass on y, but NextMethod() doesn't.

Spiers

In contrast to the paragraph about UseMethod the quote does not say that local variables are passed. It only refers to arguments and more specifically about them being passed as promises. Passing local variables probably would decrease performance and is something unusual (why is this needed?). There is nothing killed; the content of the function environment is simply not passed. – Marguritemargy

It's always better to be explicit regarding what you want to pass to functions, so your solution seems good to me. – Marguritemargy

thanks @Roland, I was just confused by the wording of the help. Also, just explain my confusion, I thought UseMethod() and NextMethod() behave differently. UseMethod() does pass on y from the content of its function environment automatically, correct (here from the generic to the default). NextMethod() however, does not do this special trickery. – Spiers

@Marguritemargy am I correct to note that there is – to the naive observer – inconsistent (though documented) behavior between UseMethod() and NextMethod() in this regard? (no criticism, there may well be reasons). – Spiers

@Marguritemargy I posted a (draft) answer based on your pointers; hope I got it right. Will wait whether someone comes up with a more profound explanation. – Spiers

My mental model is that the generic is doing the work of dispatching. The method then signals to the generic that its neighbor will do the work. From this point of view this is not inconsistent (although it's not what actually happens). – Marguritemargy

I think the key to this may be that "NextMethod works by creating a special call frame for the next method". UseMethod calls the generic as a calling environment so its variables are on the search path, but NextMethod is making some sort of separate environment, apparently. I think? – Shig

E.g. it's the difference between (function(){x = 5; (function() x)()})() and (function(){x = 5; eval(expression((function() x)()), envir = globalenv())})() – Shig

@Shig If you add print(parent.frame(n = 1)) to the method definitions you'll see that this is not the case. – Marguritemargy

@Marguritemargy Oof, now I don't know what I don't understand. But from ?parent.frame: "Strictly, sys.parent and parent.frame refer to the context of the parent interpreted function. So [...] S3 methods can also do surprising things." – Shig

Answer 1

As @Roland pointed out, this behavior is documented:

From help("UseMethod") the paragraph about UseMethod() notes (emphasis added):

UseMethod creates a new function call with arguments matched as they came in to the generic. Any local variables defined before the call to UseMethod are retained (unlike S). Any statements after the call to UseMethod will not be evaluated as UseMethod does not return.

The respective paragraph about NextMethod() (already cited above) merely notes:

NextMethod works by creating a special call frame for the next method. If no new arguments are supplied, the arguments will be the same in number, order and name as those to the current method but their values will be promises to evaluate their name in the current method and environment. Any named arguments matched to … are handled specially: they either replace existing arguments of the same name or are appended to the argument list. They are passed on as the promise that was supplied as an argument to the current environment. (S does this differently!) If they have been evaluated in the current (or a previous environment) they remain evaluated. (This is a complex area, and subject to change: see the draft ‘R Language Definition’.)

In short then UseMethod() does something special and extraordinary: it passes on local variables. NextMethod(), as per usual, doesn't do this.

UseMethod() is the exception, not NextMethod().

Spiers