Asynchronous TADOQuery's OnFetchComplete not synchronized to main thread

Question

When using TADOQuery with [eoAsyncFetchNonBlocking] and attaching to OnFetchComplete event I found that OnFetchComplete is not executing in the main thread (tested in XE4 and XE8). I assume this a bug*, since most of us will do work in the UI on these type of event. I believe this to be the source of some problems in a larger project and I need a workaround.

[EDIT] *After reading the ADO documentation I know concede that this may not be a bug, but the multithreading problem remains.

Is there an elegant way to force to have code in this handler to execute on the main thread? I don't want to use a timer (but if that's the only solution I'll take it). Alternatively, is there an ADO Synchronization object I can wait for here or some other form of signaling to the ADO provider?

Here is a simplified sample that shows that the problem. My project is more complex with a factory creating and filling these datasets, but it would be analogous here to attaching the dataset to a grid inside ADOQuery1FetchComplete.

unit Unit4;

interface

uses
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
  Vcl.Controls, Vcl.Forms, Vcl.Dialogs, Data.DB, Data.Win.ADODB, Vcl.StdCtrls;

type
  TForm4 = class(TForm)
    Button1: TButton;
    Button2: TButton;
    ADOQuery1: TADOQuery;
    ADOConnection1: TADOConnection;
    procedure Button1Click(Sender: TObject);
    procedure Button2Click(Sender: TObject);
    procedure ADOQuery1FetchComplete(DataSet: TCustomADODataSet;
      const Error: Error; var EventStatus: TEventStatus);
    procedure FormCreate(Sender: TObject);
  private
    { Private declarations }
    FMainThreadID : DWORD;
  public
    { Public declarations }
  end;

var
  Form4: TForm4;

implementation

{$R *.dfm}

procedure TForm4.ADOQuery1FetchComplete(DataSet: TCustomADODataSet;
  const Error: Error; var EventStatus: TEventStatus);
begin
  Assert(FMainThreadID = GetCurrentThreadId); //this assertion fails!
  // I need UI code here to run  FMainThreadID
end;

procedure TForm4.Button1Click(Sender: TObject);
begin
   ADOQuery1.Open;
end;


procedure TForm4.FormCreate(Sender: TObject);
begin
    FMainThreadID := GetCurrentThreadId;
end;

end.

And the dfm simply has the query set with ExecuteOptions = [eoAsyncFetchNonBlocking] and OnFetchComplete handled.

object Form4: TForm4
  Left = 0
  Top = 0
  Caption = 'Form4'
  ClientHeight = 186
  ClientWidth = 258
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'Tahoma'
  Font.Style = []
  OldCreateOrder = False
  OnCreate = FormCreate
  PixelsPerInch = 96
  TextHeight = 13
  object Button1: TButton
    Left = 24
    Top = 88
    Width = 75
    Height = 25
    Caption = 'Button1'
    TabOrder = 0
    OnClick = Button1Click
  end
  object ADOQuery1: TADOQuery
    Connection = ADOConnection1
    ExecuteOptions = [eoAsyncFetchNonBlocking]
    OnFetchComplete = ADOQuery1FetchComplete
    Parameters = <>
    SQL.Strings = (
      'SELECT * FROM TABLENAME')
    Left = 144
    Top = 16
  end
  object ADOConnection1: TADOConnection
    Connected = True
    ConnectionString = 
      'Provider=SQLOLEDB.1;Integrated Security=SSPI;Persist Security In' +
      'fo=False;Initial Catalog=DBNAME;Data Source=.\INSTANCENAME'
    LoginPrompt = False
    Provider = 'SQLOLEDB.1'
    Left = 40
    Top = 16
  end
end

[EDIT] A suggestion was made to use TThread.Sychronize, but this is not a Delphi Thread.

If the GetCurrentThreadId is not sufficient evidence that the handler is called from another thread here are the callstacks of the main and problematic thread (I added a sleep in the main thread for good measure)

Main thread sleeping

:77d0c7bc ntdll.ZwDelayExecution + 0xc
:7745104f KERNELBASE.Sleep + 0xf
Unit6.TForm6.btnQueryClick($32BC00)
Vcl.Controls.TControl.Click
Vcl.StdCtrls.TCustomButton.Click
Vcl.StdCtrls.TCustomButton.CNCommand(???)
Vcl.Controls.TControl.WndProc((48401, 1344, 7275840, 0, 1344, 0, (), 1344, 111, (), 0, 0, ()))
Vcl.Controls.TWinControl.WndProc((48401, 1344, 7275840, 0, 1344, 0, (), 1344, 111, (), 0, 0, ()))
Vcl.StdCtrls.TButtonControl.WndProc((48401, 1344, 7275840, 0, 1344, 0, (), 1344, 111, (), 0, 0, ()))
Vcl.Controls.TControl.Perform(???,???,7275840)
Vcl.Controls.DoControlMsg(???,(no value))
Vcl.Controls.TWinControl.WMCommand((273, (), 1344, 0, (), 7275840, 0))
Vcl.Forms.TCustomForm.WMCommand((273, (), 1344, 0, (), 7275840, 0))
Vcl.Controls.TControl.WndProc((273, 1344, 7275840, 0, 1344, 0, (), 1344, 111, (), 0, 0, ()))
Vcl.Controls.TWinControl.WndProc((273, 1344, 7275840, 0, 1344, 0, (), 1344, 111, (), 0, 0, ()))
Vcl.Forms.TCustomForm.WndProc((273, 1344, 7275840, 0, 1344, 0, (), 1344, 111, (), 0, 0, ()))
Vcl.Controls.TWinControl.MainWndProc(???)
System.Classes.StdWndProc(2829362,273,1344,7275840)
:759b8e71 user32.CallNextHookEx + 0xb1
:759b90d1 ; C:\windows\SysWOW64\user32.dll
:759b932c ; C:\windows\SysWOW64\user32.dll
:759b9529 ; C:\windows\SysWOW64\user32.dll
:77d107d6 ntdll.KiUserCallbackDispatcher + 0x36
:759be4a9 ; C:\windows\SysWOW64\user32.dll
:711f19e4 ; C:\windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.9600.17810_none_a9edf09f013934e0\comctl32.dll
:711f1a7b ; C:\windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.9600.17810_none_a9edf09f013934e0\comctl32.dll
:759b8e71 user32.CallNextHookEx + 0xb1
:759b90d1 ; C:\windows\SysWOW64\user32.dll
:759bddd5 user32.CallWindowProcW + 0x95
Vcl.Controls.TWinControl.DefaultHandler(???)
:00532947 TWinControl.DefaultHandler + $EB
:00532836 TWinControl.WndProc + $5CA
:00544cdd TButtonControl.WndProc + $71
:004c9162 StdWndProc + $16
:759b8e71 user32.CallNextHookEx + 0xb1
:759b90d1 ; C:\windows\SysWOW64\user32.dll
:759ba66f ; C:\windows\SysWOW64\user32.dll
:759ba6e0 user32.DispatchMessageW + 0x10
:005bb158 TApplication.ProcessMessage + $F8
:00040000

Other thread calling the handler

Unit6.TForm6.QueryFetchComplete($288B3E0,nil,esOK)
Data.Win.ADODB.TCustomADODataSet.FetchComplete(nil,89849068,Pointer($3299D8) as _Recordset)
:6b7ab81d ; C:\Program Files (x86)\Common Files\System\ado\msado15.dll
:6b7ab4b6 ; C:\Program Files (x86)\Common Files\System\ado\msado15.dll
:6b7a17c8 ; C:\Program Files (x86)\Common Files\System\ado\msado15.dll
:6b7b616f ; C:\Program Files (x86)\Common Files\System\ado\msado15.dll
:69038991 ; C:\Program Files (x86)\Common Files\System\msadc\msadce.dll
:69038bd6 ; C:\Program Files (x86)\Common Files\System\msadc\msadce.dll
:69038d54 ; C:\Program Files (x86)\Common Files\System\msadc\msadce.dll
:69037a02 ; C:\Program Files (x86)\Common Files\System\msadc\msadce.dll
:69021205 ; C:\Program Files (x86)\Common Files\System\msadc\msadce.dll
:69038034 ; C:\Program Files (x86)\Common Files\System\msadc\msadce.dll
:77a07c04 KERNEL32.BaseThreadInitThunk + 0x24
:77d2ad1f ntdll.RtlInitializeExceptionChain + 0x8f
:77d2acea ntdll.RtlInitializeExceptionChain + 0x5a

Hawkinson

GetCurrentThreadId identifies the calling thread. If the IDs don't match it must be a separate thread. – Hawkinson

You can use the form's window handle (or allocate one) to PostMessage with your own user message in the OnFetchComplete event handler, which in fact runs outside the context of the main thread. – Pentameter

This event runs indeed in another thread, like Kobik said, your best option is to use PostMessage, you can find a sample here. – Geographical

@Geographical That is actually a reasonable solution. Please post it as an answer. I also considered setting a TEvent in the handler and having a helper TThread that waits for this event and synchronizes with the main thread to call the proper action, but your solution is less code. I'm new, can I post my solution as a self answer ? As to compete with others for most elegant – Hawkinson

Sure you can answer your own question... – Geographical

Answer 1

In my experience the easier way is to use either:

Synchronize or TThread.Queue

This is not a bug or at least not a VCL bug. This behavior is handled by the provider and we cannot say it is not following the specification because there is no specification about how to manage the asynchrony of those events. All the spec says is the following:

adAsyncFetchNonBlocking

Indicates that the main thread never blocks while retrieving. If the requested row has not been retrieved, the current row automatically moves to the end of the file.

This is an example of code warning the main thread that the execution is completed:

procedure TForm1.ADOQuery1FetchComplete(DataSet: TCustomADODataSet;
  const Error: Error; var EventStatus: TEventStatus);
begin
  TThread.Synchronize(nil,
    procedure
    begin
      ShowMessage('FetchData Completed');
    end
    );
end;

Update:

I confirmed this. It will work for versions 6, 7, XE4 and XE7 (I don't have other version here). There is nothing wrong with using Synchronize to inject your code in order to execute into the main thread context. Also, I want to get your attention to the fact that DataSet is merely a pointer (actually a reference) to your ADOQuery Object, so you don't necessarily have to reference it on your anonymous method, this is an important fact for older versions like 6 or 7, because anonymous methods does not exists.

BONUS READING: EVENTS

Qp

OnFetchProgress and OnFetchComplete are the only two events that exhibit this behavior. The problem is that I don't know how to synchronize in this case. This at least does not seem to originate as a Delphi TThread, but as one originated by the OLEDB provider. If it was a Delphi TThread I could use TThread.Synchronize, but I have no flow control here. – Hawkinson

The thread is not originated from Delphi, as I said. There is nothing wrong with use Synchronize or TThread.Queue because the provider thread is created within your process address space, you will still have access to your Main Form controls – Qp

What kind of additional flow control do you need? – Qp

I cannot work with the dataset in that event handler, I need to find a safe way or point to work with it. Reading the spec (on the events esp.) I concede that it probably not a bug, but it is still a multithreading problem. Access to the dataset in event handlers here - even in OnFetchComplete - cause unexpected results. A timer to do synchronization with the main thread resolves those problems, but its awfully crude – Hawkinson

Then Synchronize is the way, the code will be injected to run in the context of the main thread, I don't understand your point, though. Synchronize will garantee the original thread will be still active when the code is run, since it will block the caller thread until the code's execution is finished. – Qp

Much to my surprise.Tests reveals that the static method TThread.Synchronize when passed a nil will block the thread even though it is of unknown origin. Thank you very much! I learned something – Hawkinson

@JasperSchellingerhout I knew it would work. But you were so convicted, then I had to check by myself in the newer version, hence the delay to undelete the answer. – Qp

All rep due @Qp I was overly skeptical, thanks for your patience. I tested your suggestion this morning at work and saw how the static method handled that case by creating a TMonitor and realized they handle anonymous threads appropriately. – Hawkinson