Adding UIAutomation Providers to Delphi controls (specifically grids)

Question

Our VCL Delphi application has a number of grids that we need to start to interact with via UIAutomation. There are a number of issues, not least that the TStringGrid doesn't implement any of the IUIAutomation patterns (IGridProvider or ITableProvider, or for that matter even IValueProvider).

I am trying to find out what I need to added to a TStringGrid to allow it to implement the providers (which in the System.Windows.Automation.Provider namespace in .NET).

Expiation

Does this SO Question help? – Potshot

@Raw No, that's the other side of the interface, consuming a provider. The asker wants to implement a provider. – Tarpon

@Mmarquee I've been told that the FMX framework implements UI Automation patterns. I know you are using VCL, but you could look at the FMX source code for inspiration. – Tarpon

Hmm. the FMX grid (and indeed any FMX controls) just are not seen via UISpy or Inspect Object - all I get is the Form itself. I will dig about in the source code though, in case there are clues, – Expiation

I must admit I was surprised when I heard that FMX implemented UI Automation. Perhaps it was added in later versions of FMX. Or perhaps I was just mis-informed. Perhaps I've just given you a bad steer. – Tarpon

I am using Delphi XE5 - as you said, it might be in XE6,7 or 8. All the more reason for me to push for the new version – Expiation

Well, on the other hand, perhaps my information is wrong. Indeed, I can't find anything in an XE8 FMX app. I should have known better. Sorry. – Tarpon

Maybe it is in the accessibility pack? docwiki.embarcadero.com/RADStudio/XE8/en/… – Altruism

In another track, I have found the header files for the library in Visual Studio - UIAutomationCore.idl. So I ran MIDL to create a TLB, and then converted that to a pascal file using TLIBIMP, and although it looks as I would expect, when I use the code, I still see nothing in Inspect. I'll do it again and make sure I've not done it wrong – Expiation

I will also look at the accessibility pack – Expiation

The accessibility pack is for XE6 and above, so I can't get it for XE5. – Expiation

@Altruism thanks, that's probably it! – Tarpon

I have found this article - which shows how to do it using C# - codemag.com/article/0810112 - however, it ends up calling functions from inside the Managed coder in UIAutomationCore.dll, which are not exported in the IDL or TLB. Looks like it might not be possible to add it to controls via Delphi. – Expiation

@MMarquee - I think you have misunderstood that article. The article expressly states that it applies primarily to Win32, unmanaged C++ code. Porting to Delphi VCL should be relatively straightforward – Furlani

I agree that it should be straightforward, but it comes down to a couple of calls that end up (I think) using code that is not exported to the idl / TLB and hence not available after the import into Delphi. I have more time this week to investigate, so may be able to make progress - for example 'UiaHostProviderFromHwnd' doesn't seem to end up anywhere useful to me. – Expiation

Answer 1

Although I cannot provide the specific steps required to implement the automation capabilities you require on TStringGrid, I can say that based on the comments you have almost everything you need.

The article you found describing the basic implementation of UI Automation support for Win32 Unmanaged code is a good place to start.

The questions over what is and is not exposed through the IDL in UIAutomationCore.DLL are then addressed by the fact that the DLL in question is itself intended to be consumed by unmanaged code. It contains no managed code itself. At least not that is involved in an unmanaged use case.

What is does contain is a COM interface described by IDL, but also some functions simply exported by the DLL. As far as I know, IDL does not describe the exports table of a DLL. Even if it is capable of doing so, in the case of this DLL it does not (at least not in all cases).

For example, the UiaHostProviderFromHwnd() function that you have mentioned is a simple DLL export. Some of the additional functions exported in this way are described in this MSDN blog post describing creating a .net interop interface for this library. In that article they are called "flat API methods".

Using PE Explorer I can see 81 such functions exported by the UIAutomationCore.dll library.

Unfortunately a DLL exports table does not describe the parameters or return types of any exported function, only the names. So, in addition to the type library (produced from the IDL) you will also need to locate and convert the UIAutomationCore.h header file for use with Delphi (i.e. Pascal).

You should then have everything need to implement the UI Automation capabilities for any VCL control you desire.

Furlani

After messing about last night I have a partial answer, i.e. I have managed to implement the pattern providers - in so much that the control shows up as having the patterns - I now need to implement the same in a 'GridItem' type thing, so I can actually get at the contents of the grid. I will post what I have as answer to this question, as I think the actual implementation of these GridItems is a different question. – Expiation

Answer 2

Here are my steps ...

(The actual files are too large to post all of them, so this is a distillation of the major points).

ALSO - This still has major issues, probably of my own making, but it is enough for me to make progress.

1. Get the UIAutomationCore.idl (mine was as part of a Visual Studio installation).
2. Run midl.exe to create the type library.
3. Run tlibimp.exe from the command-line (as Delphi doesn't seem to like the .tlb created in step 3), and create the UIAutomationCore_TLB.pas file. This ends up being a rather large file, with all of the COM parts of UIAutomationCore defined in pascal.
4. There are methods in the original DLL that are not COM, and these need to be defined as well. These I added to the generated file from Step 3 - although probably they should be defined elsewhere in case this file is regenerated.

    function UiaHostProviderFromHwnd(hwnd: HWND; provider: IRawElementProviderSimple): LRESULT; stdcall; external 'UIAutomationCore.dll' name 'UiaHostProviderFromHwnd';    
    function UiaReturnRawElementProvider(hwnd: HWND; wParam: WPARAM; lParam: LPARAM; element : IRawElementProviderSimple) : LRESULT; stdcall; external 'UIAutomationCore.dll' name 'UiaReturnRawElementProvider';

5. The component needs to implement the IRawElementProviderSimple interface, as well as any other providers - in the example case I have used ISelectionProvide, in order to illustrate what I did.

    // IRawElementProviderSimple
    function Get_ProviderOptions(out pRetVal: ProviderOptions): HResult; stdcall;
    function GetPatternProvider(patternId: SYSINT; out pRetVal: IUnknown): HResult; stdcall;
    function GetPropertyValue(propertyId: SYSINT; out pRetVal: OleVariant): HResult; stdcall;
    function Get_HostRawElementProvider(out pRetVal: IRawElementProviderSimple): HResult; stdcall;

    // ISelectionProvider
    function GetSelection(out pRetVal: PSafeArray): HResult; stdcall;
    function Get_CanSelectMultiple(out pRetVal: Integer): HResult; stdcall;
    function Get_IsSelectionRequired(out pRetVal: Integer): HResult; stdcall;

These are implemented as follows ..

function TAutomationStringGrid.Get_ProviderOptions(
  out pRetVal: ProviderOptions): HResult;
begin
  pRetVal:= ProviderOptions_ClientSideProvider;
  Result := S_OK;
end;

function TAutomationStringGrid.GetPatternProvider(patternId: SYSINT;
  out pRetVal: IInterface): HResult;
begin
  pRetval := nil;
  if (patternID = UIA_SelectionPatternId) then
  begin
    result := QueryInterface(ISelectionProvider, pRetVal);
  end
  else
    result := S_OK;
end;

function TAutomationStringGrid.GetPropertyValue(propertyId: SYSINT;
  out pRetVal: OleVariant): HResult;
begin
  if(propertyId = UIA_ControlTypePropertyId) then
  begin
    TVarData(pRetVal).VType := varWord;
    TVarData(pRetVal).VWord := UIA_DataGridControlTypeId;
  end;
  result := S_OK;
end;

function TAutomationStringGrid.Get_HostRawElementProvider(
  out pRetVal: IRawElementProviderSimple): HResult;
begin
  result := UiaHostProviderFromHwnd (self.Handle, pRetVal);
end;

function TAutomationStringGrid.GetSelection(out pRetVal: PSafeArray): HResult;
begin
end;
function TAutomationStringGrid.Get_CanSelectMultiple(
  out pRetVal: Integer): HResult;
begin
end;

function TAutomationStringGrid.Get_IsSelectionRequired(
  out pRetVal: Integer): HResult;
begin
end;

In order to actually get the control, the WM_GETOBJECT message needs to be handled ...

    procedure WMGetObject(var Message: TMessage); message WM_GETOBJECT;

This is implemented as follows ..

procedure TAutomationStringGrid.WMGetObject(var Message: TMessage);
begin
  if (Message.Msg = WM_GETOBJECT) then
  begin
    QueryInterface(IID_IRawElementProviderSimple, FRawElementProviderSimple);

    message.Result := UiaReturnRawElementProvider(self.Handle, Message.WParam, Message.LParam, FRawElementProviderSimple);
  end
  else
    Message.Result := DefWindowProc(self.Handle, Message.Msg, Message.WParam, Message.LParam);
end;

Expiation

I have still not actually got this to do what I want yet - I have fixed the fact that it seems to go into a massive loop looking up the same interface until it finally crashes - still much more work to get it to do what I want, it looks like – Expiation