Why can Delphi DLLs use WideString without using ShareMem?

Question

David's answer to another question shows a Delphi DLL function returning a WideString. I never thought that was possible without the use of ShareMem.

My test DLL:

function SomeFunction1: Widestring; stdcall;
begin
  Result := 'Hello';
end;

function SomeFunction2(var OutVar: Widestring): BOOL; stdcall;
begin
  OutVar := 'Hello';
  Result := True;
end;

My caller program:

function SomeFunction1: WideString; stdcall; external 'Test.dll';
function SomeFunction2(var OutVar: Widestring): BOOL; stdcall; external 'Test.dll';

procedure TForm1.Button1Click(Sender: TObject);
var
  W: WideString;
begin
  ShowMessage(SomeFunction1);
  SomeFunction2(W);
  ShowMessage(W);
end;

It works, and I don't understand how. The convention I know of is the one used by the Windows API, for example Windows GetClassNameW:

function GetClassNameW(hWnd: HWND; lpClassName: PWideChar; nMaxCount: Integer): Integer; stdcall;

Meaning the caller provides the buffer, and the maximum length. The Windows DLL writes to that buffer with the length limitation. The caller is allocates and deallocates the memory.

Another option is that the DLL allocate the memory for example by using LocalAlloc, and the Caller deallocates the memory by calling LocalFree.

How does the memory allocation and deallocation work with my DLL example? Does the "magic" happen because the result is WideString(BSTR)? And why aren't Windows APIs declared with such convenient convention? (Are there any known Win32 APIs that uses such convention?)

---

EDIT:

I Tested the DLL with C#.
Calling SomeFunction1 causes an AV (Attempted to read or write protected memory).
SomeFunction2 works fine.

[DllImport(@"Test.dll")]
[return: MarshalAs(UnmanagedType.BStr)]
static extern string SomeFunction1();

[DllImport(@"Test.dll")]
[return: MarshalAs(UnmanagedType.Bool)]
static extern bool SomeFunction2([MarshalAs(UnmanagedType.BStr)] out string res);

...

string s;
SomeFunction2(out s);
MessageBox.Show(s); // works ok
MessageBox.Show(SomeFunction1()); // fails with AV!

Here is a followup.

Surpass

Answer 1

A WideString is the same as a BSTR, it's just the Delphi name for it. The memory allocation is handled by the shared COM allocator, CoTaskMemAlloc. Because all parties use the same allocator you can safely allocate in one module and deallocate in another.

So, the reason you don't need to use Sharemem is that the Delphi heap is not being used. Instead the COM heap is used. And that is shared between all modules in a process.

If you look at the Delphi implementation of WideString you will see calls to the following APIs: SysAllocStringLen, SysFreeString and SysReAllocStringLen. These are the system provided BSTR API functions.

Many of the Windows APIs you refer to pre-date the invention of COM. What's more, there are performance benefits to using a fixed length buffer, allocated by the caller. Namely that it can be allocated on the stack rather than a heap. I also can imagine that the Windows designers don't want to force every process to have to link to OleAut32.dll and pay the price of maintaining the COM heap. Remember that when most of the Windows API was designed, the performance characteristics of the typical hardware was very different from now.

Another possible reason for not using BSTR more widely is that the Windows API is targeted at C. And managing the lifetime of BSTR from C is very much more tricky than from higher level languages like C++, C#, Delphi etc.

There is an extra complication however. The Delphi ABI for WideString return values is not compatible with Microsoft tools. You should not use WideString as a return type, instead return it via an out parameter. For more details see Why can a WideString not be used as a function return value for interop?

Pick

Thanks for answering. Look here. my understanding is that the DLL should is call CoTaskMemAlloc. does Delphi call it when allocation WideString? also, when does the CoTaskMemFree called? – Surpass

The SysXXX functions make the necessary calls to the COM allocator. – Pick

The link in your comment merely states the fact that the p/invoke marshaller assumes that a return value of type string was allocated with CoTaskMemAlloc and so it calls CoTaskMemFree when it has finished marshalling. – Pick

Not sure what is COM allocator. where is the COM interface? surely my Dll is not a COM server or ActiveX. – Surpass

The COM allocator is a shared heap that is accessed through CoTaskMemXXX, IMalloc etc. It is implemented by Ole32.dll – Pick

Understood :) finally, do you know of any known API that uses such convention? – Surpass

Well, COM uses this convention! That's the very obvious example. – Pick

@Surpass I don't know of any such examples. – Pick

@kobik, David, that's an interesting question/discussion... Thanks! – Ovum

@kobik: most Win32 API functions DO NOT use CoTaskMem...() to manage memory. They use GlobalAlloc() or LocalAlloc() instead, both of which are local to the calling process and do not need sharing across process boundaries. Documentation for functions usually states how memory is alloated and how it must be freed. – Hera

@Remy Most Win32 functions do not use GlobalAlloc or LocalAlloc. Most Win32 function ask the caller to provide a buffer. – Pick

@DavidHeffernan: Yes, and many functions allocate their own memory and return it to the caller instead. Usually those functions have counterpart functions to free the memory so the allocation details are hidden from the caller. But there are some functions that use GlobalAlloc() or LocalAlloc() and require the caller to manually call GlobalFree() or LocalFree(), respectively. – Hera

@remy Clipboard functions are the prime example. Those type of apis will not want to impose ole dlls on their clients. But if you are creating your own api especially if you are using p/invoke, then BSTR makes coding very simple. – Pick

@DavidHeffernan: the clipboard uses memory that is based on GlobalAlloc(). Look at the documentation for the SetClipboardData() function: "If the hMem parameter identifies a memory object, the object must have been allocated using the GlobalAlloc() function with the GMEM_MOVEABLE flag." – Hera

@Remy You are repeating what I just said! I was continuing your thread. I'm not sure what point you are trying to make. – Pick

@DavidHeffernan: sorry, I had misread your comment about the clipboard as trying to state it did the opposite of what I said about how many APIs do use Global/Local(). – Hera

@DavidHeffernan, your conclusions here made me very confused. – Surpass

@Surpass I'm confused too. I think Delphi handles WideString return values incorrectly. I am about to ask a question on that very subject. What I stated above about memory management and the COM heap is quite correct. – Pick

@DavidHeffernan, but it works with my D5. Have you tested it with XE2? – Surpass

@Surpass I'm running with XE2 yes. Perhaps this is a regression. – Pick

@DavidHeffernan, tested with C#. Same results for me. when using var or out it works. returning WideString as function result fails. – Surpass

@Surpass I think return value should fail in D5 too. I guess the return value will fail for all callers apart from delphi callers. See my question: #9350030. – Pick