How to get global screen coordinates of currently selected text via Accessibility APIs.

Question

I need help to find out, how Dictionary app showing following popup dialog for selected text on pressing CMD+CTRL+D on any application. I want to implement the same kind of functionality for my cocoa app, where my app will run in background and showing suggestions on some hot key press for the selected text.

I have already implemented hot key capturing, i just need to have code to get the rectangle area of selected text on screen, so i can show the dialog like dictionary app.

Thanks

Clapboard

how did you show the popover out of the limits of your own app? – Assurgent

@Clapboard ,How did you display suggestion window on any another application? – Cormorant

Answer 1

You can use the accessibility APIs for that. Make sure that the "Enable access for assistive devices" setting is checked (in System Preferences / Universal Access).

The following code snippet will determine the bounds (in screen coordinates) of the selected text in most applications. Unfortunately, it doesn't work in Mail and Safari, because they use private accessibility attributes. It's probably possible to get it to work there as well, but it requires more work and possibly private API calls.

AXUIElementRef systemWideElement = AXUIElementCreateSystemWide();
AXUIElementRef focussedElement = NULL;
AXError error = AXUIElementCopyAttributeValue(systemWideElement, kAXFocusedUIElementAttribute, (CFTypeRef *)&focussedElement);
if (error != kAXErrorSuccess) {
    NSLog(@"Could not get focussed element");
} else {
    AXValueRef selectedRangeValue = NULL;
    AXError getSelectedRangeError = AXUIElementCopyAttributeValue(focussedElement, kAXSelectedTextRangeAttribute, (CFTypeRef *)&selectedRangeValue);
    if (getSelectedRangeError == kAXErrorSuccess) {
        CFRange selectedRange;
        AXValueGetValue(selectedRangeValue, kAXValueCFRangeType, &selectedRange);
        AXValueRef selectionBoundsValue = NULL;
        AXError getSelectionBoundsError = AXUIElementCopyParameterizedAttributeValue(focussedElement, kAXBoundsForRangeParameterizedAttribute, selectedRangeValue, (CFTypeRef *)&selectionBoundsValue);
        CFRelease(selectedRangeValue);
        if (getSelectionBoundsError == kAXErrorSuccess) {
            CGRect selectionBounds;
            AXValueGetValue(selectionBoundsValue, kAXValueCGRectType, &selectionBounds);
            NSLog(@"Selection bounds: %@", NSStringFromRect(NSRectFromCGRect(selectionBounds)));
        } else {
            NSLog(@"Could not get bounds for selected range");
        }
        if (selectionBoundsValue != NULL) CFRelease(selectionBoundsValue);
    } else {
        NSLog(@"Could not get selected range");
    }
}
if (focussedElement != NULL) CFRelease(focussedElement);
CFRelease(systemWideElement);

Watercourse

Hi, thanks for your code. I checked your code and seem to be working and i can get the selection bound in TextEdit, but the problem is when i try to show the popup dialog on that particular rect, then popup is showing wrong. Basically, its x position is working fine, but not y position. Please let me know, do i need to do some conversion to get the correct x and y locations relative to screen. – Clapboard

The coordinates are vertically flipped. To get a frame for your panel, you'll have to convert the y-coordinate like this: selectionBounds.origin.y = [[NSScreen mainScreen] frame].size.height - selectionBounds.origin.y - selectionBounds.size.height (simplified a bit, assumes that you only have one screen). – Watercourse

Perfect. Thanks again for helping me. I am wondering if i can use the same code to get the insertion point position, just like the selection rect. – Clapboard

Generally, it's possible, but some cases are quite difficult to handle. Without a selection, the selection range has a length of 0. You can't get the bounds of an empty range, so you could construct the range of the previous character (length 1) and get the bounds of that (the right edge of it being the insertion point). However, this won't work if the text is empty or the insertion point is at the start of a new line. You could synthesize a keypress (like space), get the bounds of the character just entered and delete it again... Not really a beautiful solution though... – Watercourse

Thanks again, you are right, i can get the position by doing what you have suggested. But, my requirement is to show popup while user is typing and we found a word from collection to show all the suggestions related with that word. I am not sure, but i have read there is Text Services Manager in Carbon, which provides events to track the insertion point location. Thanks – Clapboard

I've never used it, but the docs state that "almost all Text Services Manager (TSM) functions are not available to 64-bit applications", so this might be a dead end. – Watercourse

@Watercourse I am using the same code and trying to get the selected text range in Sandboxed environment but it is not working. I am getting getSelectionBoundsError == kAXErrorAttributeUnsupported – Remittent

@Watercourse , can you please help me out finding how to get the selected text range for Safari and mail? Any reference link or document I can refer – Cormorant

@Remittent Accessibility is not meant to work in sandboxed apps, so App Store will not be an option if you choose this route. – Hiphuggers

Answer 2

Here you go for @omz answer in Swift

let systemWideElement = AXUIElementCreateSystemWide()
var focusedElement : AnyObject?

let error = AXUIElementCopyAttributeValue(systemWideElement, kAXFocusedUIElementAttribute as CFString, &focusedElement)
if (error != .success){
    print("Couldn't get the focused element. Probably a webkit application")
} else {
    var selectedRangeValue : AnyObject?
    let selectedRangeError = AXUIElementCopyAttributeValue(focusedElement as! AXUIElement, kAXSelectedTextRangeAttribute as CFString, &selectedRangeValue)
    if (selectedRangeError == .success){
        var selectedRange : CFRange?
        AXValueGetValue(selectedRangeValue as! AXValue, AXValueType(rawValue: kAXValueCFRangeType)!, &selectedRange)
        var selectRect = CGRect()
        var selectBounds : AnyObject?
        let selectedBoundsError = AXUIElementCopyParameterizedAttributeValue(focusedElement as! AXUIElement, kAXBoundsForRangeParameterizedAttribute as CFString, selectedRangeValue!, &selectBounds)
        if (selectedBoundsError == .success){
            AXValueGetValue(selectBounds as! AXValue, .cgRect, &selectRect)
            //do whatever you want with your selectRect
            print(selectRect)
        }
    }
}

Cloddish

Answer 3

What you're looking for is a Service. With services, your app doesn't even have to be running or capture global hotkeys.

For example, the functionality of the dictionary app you described is actually a service, observable in the Services menu.

Apple's Service Implementation Guide is probably the best info on services out there.

Loading

Thanks for your answer. I know about the services and already used in my app for grabbing text selection. But the requirement is to show the popup dialog like dictionary app on selected/highlighted on screen, just like dictionary app is showing in my post. Please let me know did i get the selection area relative to screen in service delegate? Thanks – Clapboard