JavaFX WebEngine stuck in "Running" state

Question

While using the WebEngine in JavaFX2, I've noticed it sometimes just gets stuck. Assume I were making a crawler that simply finds hyperlinks on a page and then visits them to do the same recursively, keeping track of which links we have visited and which are already on the frontier. While running my code, the execution would sometimes hang at arbitrary moments.

I've added some debug code to my project in the form of listeners to the workDoneProperty and exceptionProperty and by printing every transition of the loadWorker's stateProperty. Then I noticed sometimes the engine would stop mid-loading of a URL (the state is stuck in RUNNING and there are no more workDone updates). I'm assuming this is because of a time out or something, but I've stopped waiting to see if it is indeed a timeout after 5 minutes.

The exceptionProperty doesn't seem to generate any events nor does the webEngine transition into FAILED or CANCELLED, it just stops. I'm wondering if this is potentially a race condition in the library or maybe there's something I'm missing... Has anyone encountered this who knows how this can be fixed? It's quite important for my app that the engine doesn't just stop randomnly...

EDIT: added output from my console:

Work done: -1
Engine Load Worker transitioning into state: READY
Work done: 0
Engine Load Worker transitioning into state: SCHEDULED
Engine Load Worker transitioning into state: RUNNING
Work done: 21
Work done: 24
Work done: 24
Work done: 57
Work done: 72
BUILD STOPPED (total time: 9 minutes 32 seconds)

Biscuit

Answer 1

I've encountered the same problem. Seems like it happened when I've created a local "WebView" instance inside a method without keeping a hard reference to it (so after the method call ended - it was probably GC-ed.)

I fixed the problem by using a static variable for my WebView instance (that I'm initializing in a JAVAFX thread - otherwise I get an exception)

private static WebView webview;
public static void someMethod() {


    try {
        if (webview == null){
            webview = new WebView();
        }
        WebEngine webEngine = webview.getEngine();
        webEngine.getLoadWorker().stateProperty().addListener(
                new ChangeListener<State>() {
                    public void changed(ObservableValue ov, State oldState, State newState) {
                        System.out.println("newState = " + newState);
                        if (newState == State.SUCCEEDED) {
                            System.out.println(webEngine.getLocation());
                        }
                    }
                });
        webEngine.load("http://javafx.com");
    } catch (Exception ex) {
        System.err.print("error " + ex.getMessage());
        ex.printStackTrace();
    }
}

Platform

Answer 2

The WebEngine remain stuck in Running state when you try to load the same site twice sequientially. The solution is to load after/before every page the blank page. Unless you are already on the blank page.

I don't recommend writing a crawler with JAvaFX WebEngine. Is seriously bugged. (I know, I did it for a university project)

Also if you use the load pages concurrently use a AtomicReference.

Reference for an already written WebCrawler: https://github.com/llde/crawly

Zachariah