CGWindowID from AXUIElement

Question

I'm trying to automate a foreign OSX application using the accessibility API. Some of the state of the application isn't available through the API, so I acquire it through screen scraping. To do this, I need to get CGWindowID for an accessibility object with a 'Window Role'.

Is there any direct way of acquiring CGWindowID of a 'Window Role' accessibility object? I can get it heuristically, by matching various attributes of the window, such as the size, title and location, but this is really hacky, and I'd feel better if my application would also support the corner cases, even if they are unlikely.

Warrington

Answer 1

There’s no way to do that; the accessibility hierarchy is completely decoupled from the actual window/view hierarchy. I think your matching will work best.

Phosphorism

Thanks for the answer. I already suspected that, but just nice to have someone else confirm it. Anyway, I have my matching logic working good enough for my purposes through position and size matching, as well as some additional logic used to move windows a pixel left/right/up/down in case on conflicts. – Warrington

Answer 2

There is a function from at least Leopard and up (and still around as of 10.7.3):

extern "C" AXError _AXUIElementGetWindow(AXUIElementRef, CGWindowID* out);

Usual caveats about using something like this apply though. It may change as soon as the next OS update!

Meteor

how did you find this api? – Pedagogy

@Pedagogy Mostly just otool (the Xcode equivalent of nm, ldd, objdump and the like), and some hacks based on it that help make sense of disassembly etc. This stackoverflow thread is helpful: #9494995 – Meteor

Does this still work? How would I use this from Swift? I tried and got the error "Use of unresolved identifier '_AXUIElementGetWindow'" – Lorgnon

Still works on SDK 10.15 objective-c. But undocumented API still makes me worried. – Papal

_AXUIElementGetWindow still works under the latest macOS version. I tried to find an inverse function that maps AXUIElement -> CGWindowID but I couldn't find anything. I searched for private function names using a command like grep --files-with-matches "_AXUIElementGetWindow" /Library/Developer/CommandLineTools/SDKs/MacOSX14.2.sdk/System/Library/**/*.tbd, but I didn't find anything. If I had found a promising function name that way I could have reverse engineered the framework with an app like IDA, Ghidra, or Hopper to figure out the function arguments and return type. – Dynah

Oh and I search through the .tbd files using Xcode with a regex like _AXUI.*Element.*(Create|Copy) – Dynah

Answer 3

The correct current declaration in a Swift Bridging Header is:

#import <AppKit/AppKit.h>

AXError _AXUIElementGetWindow(AXUIElementRef element, uint32_t *identifier);

As used here: https://github.com/rxhanson/Rectangle/blob/master/Rectangle/Rectangle-Bridging-Header.h

Swifter