How do I use UI Automation to retrieve text from Edge Browser

Question

It seems that this used to work, but no longer does. Perhaps there is some toggle somewhere that enables it? Using this code

private static async Task<string> getText(double x, double y)
{
    try
    {
        var location = new System.Windows.Point(x, y);
        AutomationElement element = AutomationElement.FromPoint(location);

        object patternObj;
        if (element.TryGetCurrentPattern(TextPattern.Pattern, out patternObj))
        {
            var textPattern = (TextPattern)patternObj;

            var range = textPattern.RangeFromPoint(location);
            range.ExpandToEnclosingUnit(TextUnit.Word);

            var text = range.GetText(-1).Trim();
            return text;
        }
        else
        {
            return "no text found";
        }
    }
    catch (Exception ex)
    {
        return ex.Message;
    }
}

It does work on Metro apps with browers (though a bit flaky if you scroll too fast). For the manifest I am using uiAccess=true, AsInvoker. When run as Administrator, it does not help.

Update. A solution that uses WebDriver is acceptable if it can do the same thing.

Petulant

At what point does the code no longer work as expected? (For example, the Text pattern is not supported, or RangeFromPoint() returns an unexpected range.) There should be an element in Edge which supports the Text pattern. So perhaps FromPoint() is not returning that element. Could you check the properties of that element to determine which element you have. If it's not the element that supports the Text pattern, maybe you could navigate to the Text pattern element from the element that is returned by FromPoint(). – Breath

It is possible that from point does not work, though I need that. – Petulant

Your question is too broad as it could/should work in the general case. Please provide a full repro case. – Spoilt

Could you tell us what text you're trying to get out? The url, the html, the titlebar? Ref to this questions for the url scraping using windows automation: https://mcmap.net/q/1629835/-reading-out-edge-browser-title-amp-url-with-system-windows-automation – Yaekoyael

Answer 1

At the time of writing Microsoft Edge is not supported by CodedUI, they have mentioned they are evaluating support but currently you can't use it: This link shows a filed bug on the issue: https://connect.microsoft.com/VisualStudio/feedback/details/1551238/vs2015-supports-codedui-automation-test-for-edge-browser

WebDriver is the currently the best way to do automation of Microsoft Edge. However looking at the code above you are not able to do exactly the same thing. With WebDriver you can locate an element by Id, ClassName, TagName, Name, LinkText, Partial Link Text, CSS, Xpath but as far as I am aware you can't locate an object from x, y co-ordinates as you do in the example above.

To get started with Webdriver. Create a console app. Install the following packages:

Install-Package Selenium.RC
Install-Package Selenium.WebDriver
Install-Package Selenium.WebDriverBackedSelenium
Install-Package Selenium.Support

Install The correct Microsoft WebDriver depending on your operating system:

• For Windows 10 Build 10240, install this version of Microsoft WebDriver.
• For Windows 10 Fall 2015 Update, install Microsoft WebDriver Fall 2015 Update.
• For the latest preview build from the Windows Insider Program, install this version of Microsoft WebDriver.

More Information on Microsoft WebDriver can be found here.

You can then add some code to drive WebDriver, the following example goes to my blog and gets an element via it's css class name:

using System;
using System.IO;
using OpenQA.Selenium;
using OpenQA.Selenium.Edge;
using OpenQA.Selenium.Remote;
using OpenQA.Selenium.Support.UI;

namespace SampleGetText
{
    public class Program
    {
        static void Main(string[] args)
        {
            var text = GetText();
        }

        public static string GetText()
        {
            RemoteWebDriver driver = null;
            string serverPath = "Microsoft Web Driver";
            // Makes sure we uses the correct ProgramFiles depending on Enviroment
            string programfiles = Environment.Is64BitOperatingSystem ? "%ProgramFiles(x86)%" : "%ProgramFiles%";

            try
            {
                // Gets loaction for MicrosoftWebDriver.exe
                serverPath = Path.Combine(System.Environment.ExpandEnvironmentVariables(programfiles), serverPath);

                //Create a new EdgeDriver using the serverPath
                EdgeOptions options = new EdgeOptions();
                options.PageLoadStrategy = EdgePageLoadStrategy.Eager;
                driver = new EdgeDriver(serverPath, options);

                //Set a page load timeout for 5 seconds
                driver.Manage().Timeouts().SetPageLoadTimeout(TimeSpan.FromSeconds(5));

                // Navigate to my blog
                driver.Url = "https://blogs.msdn.microsoft.com/thebeebs/";

                // Find the first element on my screen with CSS class entry-title and return the text
                IWebElement myBlogPost = driver.FindElement(By.ClassName("entry-title"));
                return myBlogPost.Text;
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                return "";
            }
            finally
            {
                if (driver != null)
                {
                    driver.Close();
                }
            }
        }
    }
}

Zephyrus

Please check links. Most are not working. or just reference the generic page learn.microsoft.com/en-us/microsoft-edge/dev-guide/tools/… – Batt