How can a non-imported method in a not-attached package be found by calls to functions not having it in their namespace?
Asked Answered
A

1

30

An R namespace acts as the immediate environment for all functions in its associated package. In other words, when function bar() from package foo calls another function, the R evaluator first searches for the other function in <environment: namespace:foo>, then in "imports.foo", <environment: namespace:base>, <environment: R_GlobalEnv>, and so on down the search list returned by typing search().

One nice aspect of namespaces is that they can make packages act like better citizens: unexported functions in <environment: namespace:foo> and functions in imports:foo are available only: (a) to functions in foo; (b) to other packages that import from foo; or (c) via fully qualified function calls like foo:::bar().

Or so I thought until recently...

The behavior

This recent SO question highlighted a case in which a function well-hidden in its package's namespace was nonetheless found by a call to a seemingly unrelated function:

group <- c("C","F","D","B","A","E")
num <- c(12,11,7,7,2,1)
data <- data.frame(group,num)

## Evaluated **before** attaching 'gmodels' package
T1 <- transform(data, group = reorder(group,-num))

## Evaluated **after** attaching 'gmodels
library(gmodels)
T2 <- transform(data, group = reorder(group,-num))

identical(T1, T2) 
# [1] FALSE

Its immediate cause

@Andrie answered the original question by pointing out that gmodels imports from the the package gdata, which includes a function reorder.factor that gets dispatched to inside the second call to transform(). T1 differs from T2 because the first is calculated by stats:::reorder.default() and the second by gdata:::reorder.factor().

My question

How is it that in the above call to transform(data, group=reorder(...)), the dispatching mechanism for reorder finds and then dispatches to gdata:::reorder.factor()?

(An answer should include an explanation of the scoping rules that lead from a call involving functions in the stats and base packages to a seemingly well-hidden method in gdata.)


Further possibly helpful details

  1. Neither gdata:::reorder.factor, nor the gdata package as a whole are explicitly imported by gmodels. Here are the import* directives in gmodels' NAMESPACE file:

    importFrom(MASS, ginv)
    importFrom(gdata, frameApply)
    importFrom(gdata, nobs)
    
  2. There are no methods for reorder() or transform() in <environment: namespace:gmodels>, nor in "imports:gmodels":

    ls(getNamespace("gmodels"))
    ls(parent.env(getNamespace("gmodels")))
    
  3. Detaching gmodels does not revert reorder()'s behavior: gdata:::reorder.factor() still gets dispatched:

    detach("package:gmodels")
    T3 <- transform(data, group=reorder(group,-num))
    identical(T3, T2)
    # [1] TRUE
    
  4. reorder.factor() is not stored in the list of S3 methods in the base environment:

    grep("reorder", ls(.__S3MethodsTable__.))
    # integer(0)
    

R chat threads from the last couple of days include some additional ideas. Thanks to Andrie, Brian Diggs, and Gavin Simpson who (with others) should feel free to edit or add possibly impt. details to this question.

Alwin answered 12/6, 2012 at 20:6 Comment(4)
I see it's your first question. Welcome to SO! :)Frumpy
+1 for good stewardship of your own question.Grunt
@BrandonBertelsen -- And you think that your problem was driving you batty! Figuring out what was going on here obsessed me for several days, before I realized "Oh, duh. I should ask this on SO and get some better minds working on this for me!"Papst
I've learned to give up after 30 minutes. It's cheaper to pay people in checkmarks and upvotes :)Grunt
M
18

I'm not sure if I correctly understand your question, but the main point is that group is character vector while data$group is factor.

After attaching gmodels, the call for reorder(factor) calls gdata:::reorder.factor. so, reorder(factor(group)) calls it.

In transform, the function is evaluated within the environment of the first argument, so in T2 <- transform(data, group = reorder(group,-num)), group is factor.

UPDATED

library attaches the import packages into loaded namespace.

> loadedNamespaces()
 [1] "RCurl"     "base"      "datasets"  "devtools"  "grDevices" "graphics"  "methods"  
 [8] "stats"     "tools"     "utils"    
> library(gmodels) # here, namespace:gdata is loaded
> loadedNamespaces()
 [1] "MASS"      "RCurl"     "base"      "datasets"  "devtools"  "gdata"     "gmodels"  
 [8] "grDevices" "graphics"  "gtools"    "methods"   "stats"     "tools"     "utils"    

Just in case, the reorder generic exists in namespace:stats:

> r <- ls(.__S3MethodsTable__., envir = asNamespace("stats"))
> r[grep("reorder", r)]
[1] "reorder"            "reorder.default"    "reorder.dendrogram"

And for more details

The call of reorder will search the S3generics in two envs:

see ?UseMethod

first in the environment in which the generic function is called, and then in the registration data base for the environment in which the generic is defined (typically a namespace).

then, loadNamespace registers the S3 functions to the namespace.

So , in your case, library(gmodels) -> loadNamespace(gdata) -> registerS3Methods(gdata).

After this, you can find it by:

> methods(reorder)
[1] reorder.default*    reorder.dendrogram* reorder.factor*    

   Non-visible functions are asterisked

However, as the reorder.factor is not attached on your search path, you cannot access it directly:

> reorder.factor
Error: object 'reorder.factor' not found

Probably this is whole scenario.

Messenger answered 12/6, 2012 at 22:38 Comment(5)
Could you explain how starting from (either of) the two environments mentioned in ?UseMethod, a search for a function even looks in gdata's Namespace? It is loaded, but I thought not on the search path (which I thought was the difference between being loaded and attached).Socio
@BrianDiggs -- Here's what happens. When gdata is loaded, all of the methods that it defines are "registered" in data bases in a variety of different namespaces. The rule is that a method gets registered in the namespace of the package that defines its S3 generic. Since reorder is defined in stats, that's where it gets registered, in an environment called .__S3MethodsTable__.. Check this out: library(gmodels); head(ls(getNamespace("stats"), all=TRUE)); ls(stats:::.__S3MethodsTable__., pattern="reorder"); get("reorder.factor", stats:::.__S3MethodsTable__.). Cool, eh?Papst
@BrianDiggs -- The upshot of this is that if you importFrom() even a single function from a package, you also register all of the methods in that package. (Quite an unexpected side-effect, though I can see why R-core did it that way.) This also explains why detaching gmodels had no impact on which reorder() method gets called: detaching does not go around and clean up all of the methods that were registered when the package was loaded. (In fact, even unloadNamespace() doesn't do that cleanup. I wonder if there's some function that does...)Papst
@JoshO'Brien Thanks for the clarification. It is worth noting that ls(.__S3MethodsTable__., envir=asNamespace("stats")) is not the same as ls(stats:::.__S3MethodsTable__.) (the former in kohske's answer, the latter in your comment). The latter seems more appropriate, and shows that the method functions are loaded in the namespace of the generic without regard to the original namespace. Also, I think I have heard (but can't cite right now) that there is no way to fully guarantee a complete detachment of a package once loaded; this may be one of the reasons why.Socio
@BrianDiggs -- True. ls(.__S3MethodsTable__., envir=asNamespace("stats")) is essentially just ls(asNamespace("stats"), since there's no .__S3MethodsTable__. in .GlobalEnv. I bet you're also right about this being a (or perhaps the) reason that detaching is not reliably complete.Papst

© 2022 - 2024 — McMap. All rights reserved.