UI Automation events stop being received after a while monitoring an application and then restart after some time

Question

We are using Microsoft's UIAutomation framework to develop a client that monitors events of a specific application and responds to them in different ways. We've started with the managed version of the framework, but due to delay issues, moved to the native version wrapped in UIACOMWrapper. After more issues with performance inside our (massive) WPF application, we decided to move it to a separate terminal application (transfer the events to our WPF app through UDP) which seemed to fix all the performance issues. The only problem is that it seems that every several minutes, the events for TabSelection, StructureChanged, WindowOpened and WindowClosed stop being captured for a few minutes. Surprisingly PropertyChanged events are still received and handled while this happens. I will post the relevant code of our event monitor, but this is probably irrelevant as we have seen similar behavior when using Microsoft's own AccEvent utility. I can't post the code of the monitored application as it is proprietary and confidential as well, I can say that it is a WinForms application that hosts WPF windows and also quite massive. Has anyone seen this sort of behavior while working with the UI Automation framework? Thank you for your time.

Here's the monitor code (I know the event handling is on the UI Automation threads here but moving it to a dedicated thread did not change anything):

        public void registerHandlers()
    {
        //Register on structure changed and window opened events 
        System.Windows.Automation.Automation.AddStructureChangedEventHandler(
            this.getMsAutomationElement(), System.Windows.Automation.TreeScope.Subtree, this.handleStructureChanged);
        System.Windows.Automation.Automation.AddAutomationEventHandler(
            System.Windows.Automation.WindowPattern.WindowOpenedEvent,
            this.getMsAutomationElement(),
            System.Windows.Automation.TreeScope.Subtree,
            this.handleWindowOpened);
        System.Windows.Automation.Automation.AddAutomationEventHandler(
            System.Windows.Automation.WindowPattern.WindowClosedEvent,
            System.Windows.Automation.AutomationElement.RootElement,
            System.Windows.Automation.TreeScope.Subtree,
            this.handleWindowClosed);

        this.registerValueChanged();
        this.registerTextNameChange();
        this.registerTabSelected();
        this.registerRangeValueChanged();
    }

    private void registerRangeValueChanged()
    {
        if (this.getMsAutomationElement() != null)
        {
            System.Windows.Automation.Automation.AddAutomationPropertyChangedEventHandler(
                    this.getMsAutomationElement(),
                    System.Windows.Automation.TreeScope.Subtree, this.handlePropertyChange,
                    System.Windows.Automation.RangeValuePattern.ValueProperty);
        }
    }

    private void unregisterRangeValueChanged()
    {
        System.Windows.Automation.Automation.RemoveAutomationPropertyChangedEventHandler(
                this.getMsAutomationElement(),
                this.handlePropertyChange);
    }

    private void registerValueChanged()
    {
        if (this.getMsAutomationElement() != null)
        {
            System.Windows.Automation.Automation.AddAutomationPropertyChangedEventHandler(
                this.getMsAutomationElement(),
                System.Windows.Automation.TreeScope.Subtree, this.handlePropertyChange,
                System.Windows.Automation.ValuePattern.ValueProperty);
        }
    }

    private void unregisterValueChanged()
    {
        System.Windows.Automation.Automation.RemoveAutomationPropertyChangedEventHandler(
                            this.getMsAutomationElement(),
                            this.handlePropertyChange);
    }

    private void registerTextNameChange()
    {
        if (this.getMsAutomationElement() != null)
        {
            System.Windows.Automation.Automation.AddAutomationPropertyChangedEventHandler(
            this.getMsAutomationElement(),
            System.Windows.Automation.TreeScope.Subtree, this.handlePropertyChange,
                System.Windows.Automation.AutomationElement.NameProperty);
        }
    }

    private void unregisterTextNameChange()
    {
        System.Windows.Automation.Automation.RemoveAutomationPropertyChangedEventHandler(
        this.getMsAutomationElement(),
        this.handlePropertyChange);
    }
    private void handleWindowOpened(object src, System.Windows.Automation.AutomationEventArgs e)
    {
        Console.ForegroundColor = ConsoleColor.Magenta;
        Console.WriteLine(DateTime.Now.ToShortTimeString() + " " + "Window opened:" + " " + 
            (src as System.Windows.Automation.AutomationElement).Current.Name);

        System.Windows.Automation.AutomationElement element = src as System.Windows.Automation.AutomationElement;
        //this.sendEventToPluginQueue(src, e, element.GetRuntimeId(), this.getAutomationParent(element).GetRuntimeId());
        //Fill out the fields of the control added message
        int[] parentId = this.getAutomationParent(element).GetRuntimeId();
        this.copyToIcdArray(parentId,
            this.protocol.getMessageSet().outgoing.ControlAddedMessage.Data.controlAdded.parentRuntimeId);
        this.copyToIcdArray(element.GetRuntimeId(),
            this.protocol.getMessageSet().outgoing.ControlAddedMessage.Data.controlAdded.runtimeId);
        //Send the message using the protocol
        this.protocol.send(this.protocol.getMessageSet().outgoing.ControlAddedMessage);
    }

    private void copyToIcdArray(int[] runtimeId, ICD.UI_AUTOMATION.RuntimeId icdRuntimeId)
    {
        icdRuntimeId.runtimeIdNumberOfItems.setVal((byte)runtimeId.Count());
        for (int i = 0; i < runtimeId.Count(); i++)
        {
            icdRuntimeId.runtimeIdArray.getElement(i).setVal(runtimeId[i]);
        }
    }

    private void handleWindowClosed(object src, System.Windows.Automation.AutomationEventArgs e)
    {
        if (src != null)
        {
            Console.ForegroundColor = ConsoleColor.Cyan;
            Console.WriteLine(DateTime.Now.ToShortTimeString() + " " + "Window closed:" + " " +
                (src as System.Windows.Automation.AutomationElement).GetRuntimeId().ToString());

            System.Windows.Automation.AutomationElement element = src as System.Windows.Automation.AutomationElement;
            this.copyToIcdArray(element.GetRuntimeId(),
                this.protocol.getMessageSet().outgoing.ControlRemovedMessage.Data.controlRemoved.runtimeId);
            //Send the message using the protocol
            this.protocol.send(this.protocol.getMessageSet().outgoing.ControlRemovedMessage);

            //this.sendEventToPluginQueue(src, e, element.GetRuntimeId());
        }
    }

EDIT: I forgot to mention that I strongly suspect that the issue is that one of the UI-Automation event handler threads gets stuck somehow. The reason I believe this, is that when the problem occurred in my monitor, I started an instance of AccEvent and it received all the missing events that my monitor was not getting. This means that the events are being fired but not passed to my monitor.

EDIT2: I forgot to mention that this happens running in Windows 8 with the specific target application, I have not seen this phenomenon on my own Windows 7 machine with other applications. Another interesting thing is that it seems to happen periodically more or less, but regardless of when I subscribe to events, i.e. it can happen almost immediately after subscribing but then it takes several minutes to reoccur.

Quillen

Having no UIAutomation framework knowledge, I do want to state some obvious things: Did you check the eventlogs of the machines involved (time outs, stackoverflows, etc). Have you looked at runtime WPF binding errors (e.g. in the output pane in VS (turn on WPF tracing), do they coincide with the missing events? – Commotion

Can you add logging to the event you suspect. how long does each test take to run approx. ? is it 7 minutes exactly or variable by a minute or two ? does it have the ability to take screenshots? is this machine dedicated to ui automation? – Sabo

@Quillen any chance you have insight on #32540942 – Irizarry

@Irizarry Actually Guy Barker is far more authoritative on the subject and I see he has already answered your question. I will look into it when I have more time in case I have a suggestion anyway. – Quillen

@MichelvanEngelen Sorry for taking so much time to respond. There is only one machine at the moment, running both the tested application and the monitoring (terminal) application. I am assuming you're talking about WPF binding errors in our application (is there a way to show these for external applications for which we have no code?) but for this case it is a terminal application so no WPF binding that I know of. Your idea to check the eventlogs of the machine is interesting, I may try it even though it will be hard to synchronize with our logs. – Quillen

@Sabo Same apology as given to Michel. Did not know about Test Library. Do you think it may show a different behavior than my basic implementation or Microsoft's AccEvent utility? Are you suggesting I run a generic test against the root element of the application?I have not timed the period of this phenomenon and I do not need to run any test scenario, just monitor events and manually open and close a window every once in a while to see if it is stuck. Why 7 minutes? Do you suspect anything? – Quillen

prob is similar no want to post can you please answer that too codeproject.com/Questions/1114347/… – Burrton

@noumanarshad You can just add a link to this question in the code project one, but as you can see, there is no absolute answer for this issue, just a workaround that may or may not be satisfactory. – Quillen

YOU mean no solution ok tell me alternative of Console.Read(); in code because this is implemented in servoce and service used in forms so it gives error and possibly the reason – Burrton

///I strongly suspect that the issue is that one of the UI-Automation event handler threads gets stuck somehow/// - Since I referenced UIAComWrapper in my WinForms app the next methods stopped to return: Automation.RemoveAllEventHandlers() and Automation.RemoveAutomationEventHandler() - both wrapped by this library. – Kalli

@TarasKozubski I have seen occasional hangups when trying to remove event handlers but on Windows 7 and newer systems they get released after 2 minutes which I think is the default timeout of the framework. What OS are you running on? Do you always experience this hangup? From my testing, the phenomenon of events that stopped being received also occurred when running the SDK's AccEvent but starting a new instance received the events, so that wasn't the wrapper's issue. – Quillen

@Quillen My OS is Win 8.0 and I always experience this hangup (100% of tries) But it was OK with original UIA libraries. – Kalli

@Quillen I just tried and didn't get any 2 minutes timeout. It just hangs – Kalli

@TarasKozubski Were you using the native or managed version of UIA before switching to UIAComWrapper? If you were using the managed version, it could explain the difference. – Quillen

@Quillen I tried both. With no luck. My last question on stackoverflow is dedicated to this problem. There is a small example with source code there. – Kalli

Answer 1

I'm afraid I don't know the cause of the delays that you're seeing, but here are some thoughts on this...

Everything I say below relates to the native UIA API in Windows, not the managed .NET UIA API. All improvements to UIA in recent years have been made to the Windows UIA API. So whenever I write UIA client C# code, I call UIA through a managed wrapper that I generate with the tlbimp.exe SDK tool.

That is, I first generate the wrapper with a command like...

"C:\Program Files (x86)\Microsoft SDKs\Windows\v8.1A\bin\NETFX 4.5.1 Tools\x64\tlbimp.exe" c:\windows\system32\uiautomationcore.dll /out:Interop.UIAutomationCore.dll

Then I include a reference to the Interop.UIAutomationCore.dll in my C# project, add "using Interop.UIAutomationCore;" to my C# file, and then I can do things like...

IUIAutomation uiAutomation = new CUIAutomation8();

IUIAutomationElement rootElement = uiAutomation.GetRootElement();

uiAutomation.AddAutomationEventHandler(
    20016, // UIA_Window_WindowOpenedEventId
    rootElement,
    TreeScope.TreeScope_Descendants,
    null,
    this);

...

public void HandleAutomationEvent(IUIAutomationElement sender, int eventId)
{
    // Got a window opened event...
}

In Windows 7, there were some important constraints around UIA event handlers. It was easy to write event handlers which didn't account for those constraints, and that could lead to long delays when interacting with UIA. For example, it was important to not add or remove a UIA event handler from inside an event handler. So at the time, I intentionally made no UIA calls at all from inside my event handlers. Instead, I'd post myself a message or add some action to a queue, allow my event handler to return, and take whatever action I wanted to in response to the event shortly afterwards on another thread. This required some more work on my part, but I didn't want to risk hitting delays. And any threads I created would be running in an MTA.

An example of the action described above is in my old focus tracking sample up at https://code.msdn.microsoft.com/windowsapps/Windows-7-UI-Automation-6390614a/sourcecode?fileId=21469&pathId=715901329. The file FocusEventHandler.cs creates the MTA thread and queues messages to avoid making UIA calls inside the event hander.

Since Window 7, I know the constraints in UIA relating to threading and delays have been relaxed, and the likelihood of encountering delays has been reduced. More recently, there were some improvements between Windows 8.1 and Windows 10 in this area, so if it'd be practical to run your code on Windows 10, it would be interesting to see if the delays still repro there.

I know this is time consuming, but you might be interested in removing the interaction with UIA inside your event handlers and seeing if the delays go away. If they do, it'd be a case of determining which action seems to trigger the problem, and seeing if there's an alternative way of achieving your goals without performing the UIA interaction in the event handlers.

For example, in your event handler, you call...

this.getAutomationParent(element).GetRuntimeId();

I expect this will lead to two calls back into the provider app which generated the event. The first call is to get the parent of the source element, and the second call is to get the RuntimeId of that parent. So while UIA is waiting for your event handler to return, you've called twice back into UIA. While I don't know that that's a problem, I'd avoid it.

Sometimes you can avoid a cross-proc call back to the provider process by having some data of interest cached with the event itself. For example, say I know I'm going to want the RuntimeId of an element that raised a WindowOpened event. I can ask UIA to cache that data with the events I receive, when I register for the events.

int propertyRuntimeId = 30000; // UIA_RuntimeIdPropertyId

...

IUIAutomationCacheRequest cacheRequestRuntimeId = uiAutomation.CreateCacheRequest();
cacheRequestRuntimeId.AddProperty(propertyRuntimeId);

uiAutomation.AddAutomationEventHandler(
    20016, // UIA_Window_WindowOpenedEventId
    rootElement,
    TreeScope.TreeScope_Descendants,
    cacheRequestRuntimeId,
    this);

...

public void HandleAutomationEvent(IUIAutomationElement sender, int eventId)
{
    // Got a window opened event...

    // Get the RuntimeId from the source element. Because that data is cached with the
    // event, we don't have to call back through UIA into the provider process here.
    int[] runtimeId = sender.GetCachedPropertyValue(propertyRuntimeId);
}

On a side note, when practical, I always cache data when dealing with events or accessing elements through UIA, (by using calls such as FindFirstBuildCache(),) as I want to avoid as many cross-proc calls as possible.

So my advice would be:

1. Use the native Windows UIA API with a managed wrapper generated by tlbimp.exe.
2. Cache as much data as possible with the events, to avoid having to call back into the provider process unnecessarily later.
3. Avoid calls back into UIA from inside a UIA event handler.

Thanks,

Guy

Platinotype

Hi guy, I now noticed that I had already used a synchronization queue to avoid calling any code on the event handler thread, sorry for misleading you. So basically nothing is performed inside the event handler except for passing the action we want to perform to a different thread along with the parameters supplied by the event handler which is exactly as you recommended. Unfortunately, the same phenomenon still exhibits itself (certain events simply stop for long periods of time). Do you think that caching all of the properties needed may have an impact anyway? – Quillen

Hi, I do think it'd be worth getting as much useful data cached as possible when accessing the UIA elements. It can feel like hassle setting up the cache requests, but once you have a few in the code, it's pretty quick to copy/paste/edit those around as your need to. The perf cost of a cross-process call can be pretty high compared to other things that UIA does, so I always try to avoid them where practical. I'm afraid I really don't know whether adding more use of caching to avoid some of the existing cross-proc calls will impact the delays you're seeing, but I think it'd be worth exploring. – Platinotype

Actually, I tried caching the event registering as per your example. To my great surprise there was a very noticeable performance hit that I really can't explain, all events were being received with a big delay. I feel my initial question may have been misunderstood: there are no delays, structure change events (and window open/close) stop being received after a while for several minutes at a time. – Quillen

Answer 2

I have seen this behavior in my project. The solution was unsubscribes and resubscribe to the events using a timer. In addition, I set off any action following the events in a new task (running in an STA thread pool).

Stratocumulus

Thanks for the advice, we've considered doing what you suggest (unsubscribe, subscribe) but that means that our monitoring might miss some of the events while it is performing this re-subscription making the entire testing scenario not 100% reliable. Why are you using an STA thread when the documentation clearly states that you should use MTA threads and specifically use the same one for subscribing\unsubscribing? Have you seen any difference by using this method? – Quillen

1) I'm using the STA thread to perform the handlers actions not to do the subscription. This way I can make sure it is internally synchronized, I don't think it is a must, hence the parenthesis. you can still spawn other threads from your action. – Stratocumulus

2) you can subscribe before you unsubscribe. As long as the handler is a new object. You can create a new action delegate that wraps the same method. – Stratocumulus

Do you know of any way to guarantee that the re-subscription process will be triggered before the events stop being received? – Quillen

I do not. We just do the resubscribe process often enough – Stratocumulus