Is the compiler treatment of implicit interface variables documented?

Question

I asked a similar question about implicit interface variables not so long ago.

The source of this question was a bug in my code due to me not being aware of the existence of an implicit interface variable created by the compiler. This variable was finalized when the procedure that owned it finished. This in turn caused a bug due to the lifetime of the variable being longer than I had anticipated.

Now, I have a simple project to illustrate some interesting behaviour from the compiler:

program ImplicitInterfaceLocals;

{$APPTYPE CONSOLE}

uses
  Classes;

function Create: IInterface;
begin
  Result := TInterfacedObject.Create;
end;

procedure StoreToLocal;
var
  I: IInterface;
begin
  I := Create;
end;

procedure StoreViaPointerToLocal;
var
  I: IInterface;
  P: ^IInterface;
begin
  P := @I;
  P^ := Create;
end;

begin
  StoreToLocal;
  StoreViaPointerToLocal;
end.

StoreToLocal is compiled just as you would imagine. The local variable I, the function's result, is passed as an implicit var parameter to Create. The tidy up for StoreToLocal results in a single call to IntfClear. No surprises there.

However, StoreViaPointerToLocal is treated differently. The compiler creates an implicit local variable which it passes to Create. When Create returns, the assignment to P^ is performed. This leaves the routine with two local variables holding references to the interface. The tidy up for StoreViaPointerToLocal results in two calls to IntfClear.

The compiled code for StoreViaPointerToLocal is like this:

ImplicitInterfaceLocals.dpr.24: begin
00435C50 55               push ebp
00435C51 8BEC             mov ebp,esp
00435C53 6A00             push $00
00435C55 6A00             push $00
00435C57 6A00             push $00
00435C59 33C0             xor eax,eax
00435C5B 55               push ebp
00435C5C 689E5C4300       push $00435c9e
00435C61 64FF30           push dword ptr fs:[eax]
00435C64 648920           mov fs:[eax],esp
ImplicitInterfaceLocals.dpr.25: P := @I;
00435C67 8D45FC           lea eax,[ebp-$04]
00435C6A 8945F8           mov [ebp-$08],eax
ImplicitInterfaceLocals.dpr.26: P^ := Create;
00435C6D 8D45F4           lea eax,[ebp-$0c]
00435C70 E873FFFFFF       call Create
00435C75 8B55F4           mov edx,[ebp-$0c]
00435C78 8B45F8           mov eax,[ebp-$08]
00435C7B E81032FDFF       call @IntfCopy
ImplicitInterfaceLocals.dpr.27: end;
00435C80 33C0             xor eax,eax
00435C82 5A               pop edx
00435C83 59               pop ecx
00435C84 59               pop ecx
00435C85 648910           mov fs:[eax],edx
00435C88 68A55C4300       push $00435ca5
00435C8D 8D45F4           lea eax,[ebp-$0c]
00435C90 E8E331FDFF       call @IntfClear
00435C95 8D45FC           lea eax,[ebp-$04]
00435C98 E8DB31FDFF       call @IntfClear
00435C9D C3               ret 

I can guess as to why the compiler is doing this. When it can prove that assigning to the result variable will not raise an exception (i.e. if the variable is a local) then it uses the result variable directly. Otherwise it uses an implicit local and copies the interface once the function has returned thus ensuring that we don't leak the reference in case of an exception.

But I cannot find any statement of this in the documentation. It matters because interface lifetime is important and as a programmer you need to be able to influence it on occasion.

So, does anybody know if there is any documentation of this behaviour? If not does anyone have any more knowledge of it? How are instance fields handled, I have not checked that yet. Of course I could try it all out for myself but I'm looking for a more formal statement and always prefer to avoid relying on implementation detail worked out by trial and error.

Update 1

To answer Remy's question, it mattered to me when I needed to finalize the object behind the interface before carrying out another finalization.

begin
  AcquirePythonGIL;
  try
    PyObject := CreatePythonObject;
    try
      //do stuff with PyObject
    finally
      Finalize(PyObject);
    end;
  finally
    ReleasePythonGIL;
  end;
end;

As written like this it is fine. But in the real code I had a second implicit local which was finalized after the GIL was released and that bombed. I solved the problem by extracting the code inside the Acquire/Release GIL into a separate method and thus narrowed the scope of the interface variable.

Mic

Don't know why this was downvoted, other than that the question is really complex. Upvoted for being way over my head. I do know that exactly this bit of arcanum resulted in some subtle reference counting bugs in an app I worked on a year ago. One of our best geeks spent hours figuring it out. In the end we worked around it but never understood how the compiler was intended to work. – Iman

Why does interface lifetime matter in this situation? StoreViaPointerToLocal() will have two references to the interface, and two calls to IntfClear(), but the interface will still be freed when StoreViaPointerToLocal() exits since its reference count still falls 0. Can you provide a better example of where interface lifetime is a real problem when such implicit variables are involved? – Torpor

When you work with reference-counting interfaces you should not assume that there are no other clients (the compiler in you case) that have their own interface references. If all clients do AddRef/Release correctly, all should work OK. Otherwise it is a bug (your bug since I assume that the compiler do it's reference counting correctly). – Medallion

@Serg The compiler did its reference counting perfectly. The problem was that there was an extra variable holding a reference that I could not see. What I want to know is what provokes the compiler to take such an extra, hidden, reference. – Mic

I understand you, but a good practice is to write code that does not depend on such extra variables. Let the compiler create these variables as much as it likes, a solid code should not depend on it. – Medallion

It looks like a PInterface enforces refcounting too, although I may not understand the question. Anyway, I think you can never rely on the moment an interface will be released. You will know it won't be released until its last reference is gone, but I don't think its guaranteed to be freed immediately when the last reference goes out of scope. Although I do think the Delphi documentation says that the interface is released if you explicitly set its last reference to nil. – Closet

@Serg Agreed. 999/1000 times this sort of hidden ref is benign. But consider my example with the Python GIL. That pretty much trapped me with no way out. – Mic

I can't see any interface reference in your example. As an ad hoc solution probably you may change the scope of a hidden variable by extracting part of your code that works with interface references into procedure. – Medallion

@Serg the interface references are there. As I explained, I did indeed extract code into separate procedure to narrow scope. What I want is to understand when the compiler takes hidden references. I have no trouble dealing with that eventuality. – Mic

Another example when this is happening: procedure StoreViaAbsoluteToLocal; var I: IInterface; I2: IInterface absolute I; begin I2 := Create; end; – Sebastian

I don't think it's documented, but the compiler seems to generate code not only to assign to the implicit temporary variable, but also copy-assign to the local variable (of the relevant type) with the same address, if there is one. – Sebastian

Out of curiosity: is the implicit variable cleared sooner if you set I or P^ to nil? – Svensen

@Svensen No. Only at the end of the procedure is it cleared. – Mic

I'm tempted to call this a compiler bug...temporaries should be cleared after they go out of scope, which should be the end of the assignment (and not the end of the function). Not doing so produces subtle errors as you have discovered. – Svensen

Another question (sorry, don't have a Delphi compiler handy): if you do I := Create; I := Create; (i.e. create twice) in StoreToLocal, does the implicit variable show up? – Svensen

Wait, two? Doesn't it produce zero implicit locals with only one I := Create? – Svensen

@Svensen You are right. Sorry, I hadn't caught up on the details. The I := Create case is not interesting. No implicit locals. It only gets interesting when there are implicit locals. – Mic

let us continue this discussion in chat – Svensen

Perhaps it is made just as quick "hacky" solution to deal with multiple (logical) vars pointing to interface. Looks like a bug for me. Anyway, it seems to be undocumented – Nidia

However, QC has bugreport of similar behavior – Nidia

Answer 1

If there is any documentation of this behavior, it will probably be in the area of compiler production of temporary variables to hold intermediate results when passing function results as parameters. Consider this code:

procedure UseInterface(foo: IInterface);
begin
end;

procedure Test()
begin
    UseInterface(Create());
end;

The compiler has to create an implicit temp variable to hold the result of Create as it is passed into UseInterface, to make sure that the interface has a lifetime >= the lifetime of the UseInterface call. That implicit temp variable will be disposed at the end of the procedure that owns it, in this case at the end of the Test() procedure.

It's possible that your pointer assignment case may fall into the same bucket as passing intermediate interface values as function parameters, since the compiler can't "see" where the value is going.

I recall there have been a few bugs in this area over the years. Long ago (D3? D4?), the compiler didn't reference count the intermediate value at all. It worked most of the time, but got into trouble in parameter alias situations. Once that was addressed there was a follow up regarding const params, I believe. There was always a desire to move disposal of the intermediate value interface up to as soon as possible after the statement in which it was needed, but I don't think that ever got implemented in the Win32 optimizer because the compiler just wasn't set up for handling disposal at statement or block granularity.

Mesquite

Answer 2

You can not guarantee that compiler will not decide to create a temporal invisible variable.

And even if you do, the turned off optimization (or even stack frames?) may mess up your perfectly checked code.

And even if you manage to review your code under all possible combinations of project options - compiling your code under something like Lazarus or even new Delphi version will bring hell back.

A best bet would be to use "internal variables can not outlive routine" rule. We usually do not know, if compiler would create some internal variables or not, but we do know, that any such variables (if created) would be finalized when routine exists.

Therefore, if you have code like this:

// 1. Some code which may (or may not) create invisible variables
// 2. Some code which requires release of reference-counted data

E.g.:

Lib := LoadLibrary(Lib, 'xyz');
try
  // Create interface
  P := GetProcAddress(Lib, 'xyz');
  I := P;
  // Work with interface
finally
  // Something that requires all interfaces to be released
  FreeLibrary(Lib); // <- May be not OK
end;

Then you should just wrap "Work with interface" block into subroutine:

procedure Work(const Lib: HModule);
begin
  // Create interface
  P := GetProcAddress(Lib, 'xyz');
  I := P;
  // Work with interface
end; // <- Releases hidden variables (if any exist)

Lib := LoadLibrary(Lib, 'xyz');
try
  Work(Lib);
finally
  // Something that requires all interfaces to be released
  FreeLibrary(Lib); // <- OK!
end;

It is a simple, but effective rule.

Claiborne

In my scenario, I := CreateInterfaceFromLib(...) was resulting in an implicit local. So what you suggest won't help. In any case, I already demonstrated clearly a workaround in the question. One based upon lifetime of implicit locals being controlled by function scope. My question concerned the scenarios which would lead to the implicit locals. – Mic

My point was that this is a wrong question to ask in the first place. – Claiborne

You are welcome to that view point but you should express it as a comment. Adding code that attempts (unsuccessfully) to reproduce the workarounds of the question, seems odd to me. – Mic