How to write a python script to keep trying a webpage until it opens

Question

We are waiting for my sister's result. And as it happens, a lot, with the Indian govt. the server is slow, traffic is heavy.

So, I thought of writing a python program, to keep trying until the server responds to the Http request. But the program:

import urllib
i=1
f = open("C:/Users/DELL/Desktop/neetpg.html",'w')
while(True):
    try:
        page = urllib.urlopen("http://www.nbe.gov.in/asr/neet_pdf/")
        print "Done"
        break
    except:
        print i
        i += 1
        continue
f.write(page.read())

print "check"

But the program, doesn't run properly. I tried replacing the url with facebook.com, it still prints out numbers.

Moreover what I'd like, is to achieve that if the server does respond, the webpage loads the js and css files along with the html file and all this should open in a browser.

I also took a hint from http://docs.python.org/2/library/webbrowser.html and changed the program to:

import webbrowser
i=1
while(True):
    try:
        webbrowser.open("http://www.nbe.gov.in/asr/neet_pdf/")
        print "Done"
        break
    except:
        print i
        i += 1
        continue
print "check"

But all this does, is opens a new window in my default web browser, and sets the url to what is given, and "opens it". Meanwhile, printing Done and Check on the python shell.

The web browser having not received a response from the server, displays could not connect to www.nbe.gov.in.

How to achieve this ?

EDIT: Just saw that the facebook.com script worked after all. It took it approx 15 tries, and then it happened. The .html file is written properly. With all the CSS and probably the JS too.

Why is it so, that it took so many tries, while, I can simply open facebook.com from the browser easily.

Pinch

ooo.. Library upgrade. Btw, what all better things are there ? – Pinch

Hmm. strange. The code works on my machine – Eulaeulachon

it works ? Is the website opening? What county ? :D can u check out the result for us ? – Pinch

pastebin.com/vF2W7diM – Eulaeulachon

thanks for the source code of the page, but entering queries in it doesn't work! :) – Pinch

You'd need the all the code of the site to get that to work, but it wouldn't help you, because you don't have acces from your computer to their databases. You could maybe modify the code to submit to their site, but you would end up with the same problem. Good luck :) – Eulaeulachon

Answer 1

Give selenium a try.

The idea is to keep opening the page until the driver sees the right title. And if it's there, just break the loop and leave the page opened:

from selenium import webdriver


driver = webdriver.Firefox()
while True:
    driver.get("http://www.nbe.gov.in/asr/neet_pdf/")

    if 'NEET-PG' in driver.title:
        break

Hope that helps.

Admissible

Selenium ? Where can I downlaod this from ? – Pinch

Just read the docs, should be easy to install. – Admissible

@TehTris that's not what implicit waits do. – Sisco

thanks, Deleting comment. I now know this is not what this does. Lol. – Ascarid