Does Microsoft UI Automation Framework work with Chrome, Python and Java Apps?

Question

I am working on an automation project, in which I need to capture the activities [ application launched, data entered, input type etc.] user performs on a desktop. I came across Microsoft UI Automation framework which so far works well for native windows based applications like MS Office, .NET apps etc. However I did not find any useful information / samples of capturing the information from different web browsers [Chrome is a must], Python apps, Java Apps etc. Can someone please confirm whether MS UI Automation Framework supports such apps. Any working example to extract user activities from these apps would be highly appreciated. Thanks.

Weary

Answer 1

Chrome only supports UI Automation for toolbars, tabs, menu, buttons around the web page. Everything that's rendered as a web page is not seen by UIA.

For the web page content, the easiest way is to use Selenium (driven by the ChromeDriver), which is kind of a de facto standard for browsers, and has nothing to do with UIA.

To test if an app supports UIA, and how far it does, it's very easy, just run UIA's Inspect tool and check the UI tree over that application.

Chase

UIA should see the chrome document, as long as the document is at least partially visible. You can test this with Inspect tool on a partially visible chrome window, vs minimized chrome window. This is in the year 2021. – Weir

@BarmakShemirani - some (!) answers are dated on SO yes, but not everybody uses 2021 technologies, why don't you make your own answer? – Chase

This answer is correct except for odd behavior of Chrome. Firefox for example works as expected. – Weir

Answer 2

Some additions to Simon's answer...

Chrome page content can be seen by UIA if you run chrome --force-renderer-accessibility. Only for existing Chrome process it won't work. Though user can create a new tab chrome://accessibility manually and enable UIA for all or some chosen pages. This method also works for AT-SPI accessibility technology on Linux. Of course, Selenium WebDriver is an industry standard here. But another way exists. Both Mozilla and IE support UIA by default.

Inspect.exe can be simply downloaded from this GitHub repo.

Regarding Java apps it depends on the app type. Your chances is about 50/50.

WxPython or PyQt5 are good for UIA. TkInter or Kivy apps are not.

P.S. There is an example how to drag a file from explorer.exe and drop to Google Drive in Chrome using Python library pywinauto.

Cavorelievo

Need some help. You say: "Chrome page content can be seen by UIA". I want to get events performed on a webapp on chrome (like keep track of keys pressed and button clicked and on what element). Can I do this by using UIA on chrome?. – Talion

Could you please help me with: #68703914 – Talion

@Talion THIS. THIS HELPED ME SO, SO MUCH!!! In order to Force Chrome to allow your application to see it, you have to call Chrome from a Command Line such as follows: Start "Chrome" Chrome.Exe --force-renderer-accessibility "https://yourURLhere.com" and it will allow your application to view the objects using UIA. An important note though! It must open a new window with this flag, and using --new-window does NOT work - I just confirmed. You must open a completely new instance of Chrome for it to manifest. – Nancinancie

Answer 3

I'm a bit late to the party.. But Chromes accessibility features are only activated once something tries to access it's accessibility.

If you call AccessibleObjectFromWindow ([DllImport("oleacc.dll")]) with the window handle an existing chrome window will have its accessibility activated (and you'll see the actual web page content in UIA!).

If the chrome window is opened after your app is running - Chrome pings open processes for any open accessibility apps... for that you use AccessibleObjectFromEvent and the event you're responding to comes from the windows pipeline: EVENT_SYSTEM_ALERT = 0x0002 .

The bottom line is - you have to tell chrome that there's something installed that wants to access it's web page content.

Oh! and your application has to be signed!! Unsigned apps won't be able to access web content - I think that's the same in firefox too.

I hope this helps someone in the future.

See: https://www.chromium.org/developers/design-documents/accessibility

Rumpus

Could you please help me with : #68703914 – Talion

This is good to know as well. This would be the legit way to build an application I suppose, vs the answer I'm currently using where I force Chrome open w/ the accessibility flag turned on – Nancinancie