Avoid opening browser on remote server during selenium call
Asked Answered
I

3

3

I have written a selenium application using webdriver. I wish to run it on a remote server. When I do that by logging into the server via putty (along with Xming), the selenium tries opening the browser on the server only and load the pages through the external display. However in doing that, it takes a lot of time than if I would have been able to get the browser open on my localhost only (and not the server). Is it possible for such thing to happen or opening on the server only is the only option (which is painfully slow). Kindly tell me if I am missing something as well.

Thanks in advance.

Ileostomy answered 4/6, 2013 at 7:0 Comment(0)
V
2

Try using Selenium Grid, instead of Putty, to run your Selenium application on a remote server. The Selenium website has an excellent Quick Start guide for using the Selenium Grid: http://code.google.com/p/selenium/wiki/Grid2.

Veneration answered 5/6, 2013 at 18:18 Comment(2)
I implemented the code but it still tries opening it from putty so takes a lot of time to get the work done. The code is as follows:Ileostomy
i found the following suggestion at link. Even if I did that, it shows me a Error: cannot open display: :0Ileostomy
V
1

You can run Selenium with a"headless" driver, HtmlUnitDriver, that does not actually open a browser:

http://code.google.com/p/selenium/wiki/HtmlUnitDriver

Note: HtmlUnitDriver will accept an argument, so that it can emulate a specific driver.

Veneration answered 11/9, 2013 at 17:0 Comment(0)
I
0

@Lori

I implemented the code but it still tries opening it from putty so takes a lot of time to get the work done. The code is as follows: 'code'

import sys
from scrapy.spider import BaseSpider
from scrapy.http import FormRequest
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from scrapy.item import Item
from scrapy.http import Request

from selenium import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

class DmozSpider(BaseSpider):
name = "linkedin_crawler"
#defence news 
global company
global query
companyFilename = '<filename>'
f=open(companyFilename,"r")
f.seek(0)
company = f.readline().strip()
f.close()
queryFilename = '/var/www/Symantec/recon/recon/' +company+ '/Spider/LinkedIn/query.txt'
f = open(queryFilename)
f.seek(0)    
query=f.readline().strip()
f.close()

start_urls = ['https://www.linkedin.com/uas/login'];


def __init__(self):
    BaseSpider.__init__(self)
capabilities = webdriver.DesiredCapabilities()
self.selenium = webdriver.Remote(command_executor = 'http://localhost:5000/wd/hub', desired_capabilities = capabilities.FIREFOX)

def __del__(self):
    self.selenium.quit()


def parse(self, response):
sel= self.selenium
    sel.get(response.url)
global query

elem1 = sel.find_element_by_name("session_key")
elem2 = sel.find_element_by_name("session_password")
elem1.send_keys("myemailid")
elem2.send_keys("mypassword")
elem2.send_keys(Keys.RETURN)

    return Request(query, callback=self.page_parse)


def page_parse(self,response):
global query
global company


sel= self.selenium
    sel.get(query)

for i in xrange(10):
#for i in xrange(5):

    nameFilename = ''

    #print hxs
    nlist = sel.find_elements_by_xpath('//ol[@class="search-results"]/li/div/h3/a')


    fh = open(nameFilename,"a") 

    for j in xrange(len(nlist)):

        url = nlist[j].get_attribute("href").encode('utf-8')
        name =  nlist[j].text.encode('utf-8')

        fh.write(name)
        fh.write("<next>")
        fh.write(url)
            fh.write('\n')


    fh.close()

    next = sel.find_elements_by_xpath('//a[@class="page-link"]')
    next[0].click()
    time.sleep(5)

To tun this script on server, I am using putty to fire the command. But then it again uses Xming to open the browser which makes the process slow again. So, how to run the script without opening the browser on my local machine via Xming so that this does not become the bottleneck. Thanks

Ileostomy answered 7/6, 2013 at 6:15 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.