What exactly is "broken" with Microsoft Visual C++'s two-phase template instantiation?
Asked Answered
S

5

39

Reading questions, comments and answers on SO, I hear all the time that MSVC doesn't implement two-phase template lookup / instantiation correctly.

From what I understand so far, MSVC++ is only doing a basic syntax check on template classes and functions and doesn't check that names used in the template have atleast been declared or something along those lines.

Is this correct? What am I missing?

Sydel answered 8/6, 2011 at 0:56 Comment(6)
I have no reference for this, but I believe that has been implemented now, likely in VC 2005 or 2008.Chick
@Simon : I don't think that's the case.Fullbodied
@ildjarn: I retract my statement! Template compilation has been significantly improved, but two-phase lookup specifically is apparantly still overly lax. I think broken is probably overly strong a term, looking at the definition.Chick
My experince with VC2010 gave me an impression that compiler does a lot less looking up names in the first phase than required. At least I found more errors with gcc, which VC skipped.Gleanings
You may find the recently submitted bug to Visual Studio interesting: MSVC needs namespace even if it is hoisted - connect.microsoft.com/VisualStudio/feedback/details/715626 Perhaps, if more folks voted, it would be fixed sooner.Chordate
Page not found for connect.microsoft.com/VisualStudio/feedback/details/715626. Does this link is still valid for you?Carlow
A
42

I'll just copy an example from my "notebook"

int foo(void*);

template<typename T> struct S {
  S() { int i = foo(0); }
  // A standard-compliant compiler is supposed to 
  // resolve the 'foo(0)' call here (i.e. early) and 
  // bind it to 'foo(void*)'
};

void foo(int);

int main() {
  S<int> s;
  // VS2005 will resolve the 'foo(0)' call here (i.e. 
  // late, during instantiation of 'S::S()') and
  // bind it to 'foo(int)', reporting an error in the 
  // initialization of 'i'
}

The above code is supposed to compile in a standard C++ compiler. However, MSVC (2005 as well as 2010 Express) will report an error because of incorrect implementation of two-phase lookup.


And if you look closer, the issue is actually two-layered. At the surface, it is the obvious fact that Microsoft's compiler fails to perform early (first phase) lookup for a non-dependent expression foo(0). But what it does after that does not really behave as a proper implementation of the second lookup phase.

The language specification clearly states that during the second lookup phase only ADL-nominated namespaces get extended with additional declarations accumulated between the point of definition and point of instantiation. Meanwhile, non-ADL lookup (i.e. ordinary unqualified name lookup) is not extended by the second phase - it still sees those and only those declarations that were visible at the first phase.

That means that in the above example the compiler is not supposed to see void foo(int) at the second phase either. In other words, the MSVC's behavior cannot be described by a mere "MSVC postpones all lookup till the second phase". What MSVC implements is not a proper implementation of the second phase either.

To better illustrate the issue, consider the following example

namespace N {
  struct S {};
}

void bar(void *) {}

template <typename T> void foo(T *t) {
  bar(t);
}

void bar(N::S *s) {}

int main() {
  N::S s;
  foo(&s);
}

Note that even though bar(t) call inside the template definition is a dependent expression resolved at the second lookup phase, it should still resolve to void bar(void *). In this case ADL does not help the compiler to find void bar(N::S *s), while the regular unqualified lookup is not supposed to get "extended" by the second phase and thus is not supposed to see void bar(N::S *s) either.

Yet, Microsoft's compiler resolves the call to void bar(N::S *s). This is incorrect.

The problem is still present in its original glory in VS2015.

Attrition answered 8/6, 2011 at 2:2 Comment(15)
Confirmed the same behaviour in VC2010 SP1. I wouldn't expect MS to change this behavior now, too much of a compatibility burden for the relatively small portability loss. I'd actually recommend that the standard is changed to make code like this (with ambiguous intent) require a diagnostic. But that's a different rabbithole...Chick
@Simon: when a compiler is found to fail Standards compliance, it's reasonable to add a command line option for backwards compatibility and default to future Standards compliance. It's not MS or any compiler writer's place to accept "relatively small portability loss" that helps lock their clients in... that's open to deliberate abuse, and history shows oft cynically embraced. Otherwise, there's nothing more ambiguous about intent here than in 100 other parts of name lookup - it's ridiculous to "recommend" a change to the Standard for this.Jacobina
@Tony: What I want this to do is independent of my guess at what MS will do, which I don't really feel very strong about either way. Even if every compiler implemented this correctly, I would still think two-phase lookup behavior is a misfeature of the specification, probably copied from Bjarne's compiler, and my guess at the easiest way to fix it would be to error on when two-phase and late lookup disagree. I would guess about the same amount of code would be broken as for the >> C++11 change, and it is quite likely to be already wrong.Chick
@Simon: I can only disagree... in code such as Andrey's example, the template writer has a conception of what foo() will do and wants to call it on that basis. To substitute some other foo() later allows client code to affect the template instantiation, which has two serious issues: 1) objects may have different instantiations for the same template/parameters, which is beyond what the name mangling/linker is meant to handle, 2) client code can substitute arbitrary functionality into the template implementation, breaking the conceptual "encapsulation" and introducing dependencies.Jacobina
@Simon: further, non-template functions don't suddenly start calling "better" matches for the functions they call when such functions appear later in the translation unit, why complain about templated functions working the same way as non-templates? The Standard's status quo doesn't seem particularly confusing or error-prone to me, quite the opposite. Re >> - there's strong incentive to change that as it's a FAQ-level issue that's proven troublesome to and/or needed explanation for thousands of newbie C++ programmers.Jacobina
@Tony: actually, the "misselection" of foo would occur with a perfectly compliant compiler if you included the declaration of void foo(int) before included the template. C++ has no concept of modularity because of the fundamentally broken include idea, therefore a template / inline function writer cannot expect to know the non-dependent functions that will be called... and the unsuspecting user will have an ill-formed program if this happen in one TU but not the other! Solution ? namespace, namespace, namespace.Demoralize
@Matthieu: yes, a legitimate dig at functions in the global namespace, but again - no more applicable to templated functions than other (inline) functions. (BTW / VC++ does at least prefer a match in the template's namespace to a better match outside it, so it's only pretty bodgy (namespace-less) code that gets bitten by VC++'s bug).Jacobina
@Tony: I don't want it to select the 'better' foo(), I want it to fail at instatiation time with an ambiguous name error! And templates work differently because they are "implemented" at instantiation point, not declaration point.Chick
@Xeo: "Overload on the return type"? What exactly are you referring to?Attrition
@AndreyT: Forget my comment, I indeed forgot something because it's a "feature" I never use / that shouldn't really be used. Function overloads are able to have different return types so long they are still unambiguous. So, nevermind me. :)Sydel
@Simon: "And templates work differently because they are "implemented" at instantiation point, not declaration point." - a circular argument - summarising your misconception as a justification for itself; this whole discussion is about the Standard dictating that aspects of the template implementation be locked in at declaration, which makes that assertion wrong. Your argument boils down to "I'd find it more intuitive if templates errored when lookups vary between declaration and instantiation time" - nobody can deny you your intuitions, but personally I think the status quo sensible.Jacobina
@Tony: Ehh. I don't know where misconception comes from - this is a hypothetical change to the standard to make reading templates safer and more obvious to normal humans (like me!). I could have been clearer: templates, unlike non-templates, can currently change their meaning depending on what is defined after them. I think that's dumb, confusing and dangerous - it should be all or nothing. Currently conformant code that would be broken by this (that already wouldn't work in VC) can be easily be fixed and made easier to understand. But this is just an off the cuff guess at a quick fix.Chick
@Simon: in "templates, unlike non-templates, can currently change their meaning depending on what is defined after them. I think that's dumb..." you describe the non-Standard VC++ behaviour ;-). The Standard does dictate "nothing", but you want to change it from all or nothing to "neither - give me an error". Anyway, this exchange is perhaps better left to a natural death, as in the end I'm confident we do understand each other's position, and my quips about the soundness of logic used to derive and promote your position don't change the fact that our positions do and will differ. Cheers.Jacobina
@Tony: Sure. And to be super-clear, I don't hate the standard behaviour, I just think it could be nicer - and C++11 has given me hope that weird areas of the language could be improved in the future.Chick
@TonyD not quite. Dependent names can change their meaning. The problem with this is us humans being unable to tell at the first glance which names are dependent.Brumaire
C
22

The Clang project has a pretty good writeup of two-phase lookup, and what the various implementation differences are: http://blog.llvm.org/2009/12/dreaded-two-phase-name-lookup.html

Short version: Two phase lookup is the name for the C++ standard defined behavior for name lookup within template code. Basically, some names are defined as dependent (the rules for which are a bit confusing), these names must be looked up when instantiating the template, and independent names must be looked up when parsing the template. This is both hard to implement (apparently), and confusing for developers, so compilers tend to not implement it to the standard. To answer your question, it looks like Visual C++ delays all lookups, but searches both the template context and the instantiation context, so it accepts a lot of code that the standard says it shouldn't. I'm not sure whether it doesn't accept code it should, or worse, interprets it differently, but it seems possible.

Chick answered 8/6, 2011 at 1:51 Comment(1)
It will interpret it differently- for example, if you declare better-match overloads after the template declaration, the Standard defines the worse match but MSVC will pick the better match.Anishaaniso
D
9

Historically gcc didn't implement the two-phase name lookup correctly either. It's apparently very difficult to get to, or at least there wasn't much incentive...

  • gcc 4.7 claims to implement it correctly, at last
  • CLang aims at implementing it, baring bugs, it's done on ToT and will get into 3.0

I don't know why VC++ writers never chose to implement this correctly, implementation of a similar behavior on CLang (for microsoft compabitility) hints that there might be some performance gain to delaying the instantiation of templates at the end of the translation unit (which does not mean implementing the look-up incorrectly, but make it even more difficult). Also, given the apparent difficulty of a correct implementation it may have been simpler (and cheaper).

I would note that VC++ is first, and foremost, a commercial product. It is driven by the need to satisfy its clients.

Demoralize answered 8/6, 2011 at 6:39 Comment(3)
Before C++ was standardized, it was a lot less clear how name lookup should work in templates. MS just implemented one way. Now they won't change it because it too much work for them to maintain both ways of name lookup - and they would have to, because of the amazing amount of code (both inside MS and outside) that depends on the broken behavior. I actually heard a story that some guy went ahead and implemented correct name lookup, but it broke so much code that they never integrated the change.Gmur
@SebastianRedl: Backward compatibility hurts kitten :( But I do understand they would be unwilling to break their clients code.Demoralize
Microsoft engineers have indicated that they plan to implement two-phase lookup correctly eventually (See the VC++ Conformance Update table). This feature has been difficult for them to implement because their compiler does not use complete ASTs.Kaltman
R
5

short answer

Disable language extensions with /Za

longer answer

I was investigating this issue lately and was amazed that under VS 2013 following example from standard [temp.dep]p3 produces wrong result:

typedef double A;
template<class T> class B {
public:
    typedef int A;
};
template<class T> struct X : B<T> {
public:
    A a;
};

int main()
{
    X<int> x;
    std::cout << "type of a: " << typeid(x.a).name() << std::endl;
}

will print:

type of a: int

while it should print double. The solution to make VS standard conformant is to disable language extensions (option /Za), now type of x.a will resolve to double, also other cases of using dependent names from base classes will be standard conformant. I am not sure if this enables two phase lookup.

[Update Jul-2019] Its also true for vs 2015 - https://rextester.com/YOH81784, but VS2019 shows correctly double. According to this article Two-phase name lookup support comes to MSVC, it was fixed since VS 2017.

Romero answered 16/12, 2014 at 9:42 Comment(2)
/Za doesn't help in this case, tested in VS2015Hackle
Nitpicking: "It should print double" No, it should print typeid(double).name(), which isn't necessarily double.Woodborer
B
3

Now that MSVC has most of two-phase name lookup implemented, I'm hoping this blog post answers this question completely: Two-phase name lookup comes to MSVC (VC++ blog)

Boadicea answered 17/11, 2017 at 19:8 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.