I'm using faraday with net-http-persistent adapter to make HTTP requests.

I want to optimise my requests by making them execute asynchronously but as I need a persistent connection I keep getting errors such as too many connections reset which I assume is due to the fact that I have multiple threads creating new connections.

I tried changing my adapter to typhoeus but as the connection is not persistent the final result of executing all request is not as expected.

My goal is to add items to a basket by making this HTTP requests. Without the persistent connection items are not added to the basket.

So, my question is:

Is it possible to make persistent HTTP requests reusing the connection between threads? If so, how can this be achieved?

Here is a piece of my code:

Create the connection:

Faraday.new do |c|
        c.use :cookie_jar, jar: cookie_jar
        c.options.open_timeout = 5
        c.options.timeout = 10
        c.request :url_encoded
        c.response :logger, logger
        c.adapter :net_http_persistent do |http| # yields Net::HTTP::Persistent
          http.idle_timeout = 2
        end
      end

Creating threads and getting the result of each one of them

  result = []
  threads = []
  total_items = items.size

  items.each_slice(5) do |sliced_items|

    # Create a thread for a batch of 5 items and store its result
    threads << Thread.new do
      Thread.current[:output] = browser.add_all_items(sliced_items)
    end
  end

  # Wait for all threads to finish their work and store their output into result
  threads.each do |t|
     t.join
     result << t[:output]
  end

add_all_items and add_to_cart methods:

 # Add a batch of items by the key passed (id, gtin, url)
    def add_all_items(items_info, key)
      results = []
      items_info.each do |item|
        begin
          add_to_cart(item[key], item[:quantity])
          item[:message] = nil
        rescue => e
          item[:message] = e.message
          puts "---------- BACKTRACE -------------- \n #{e.backtrace}"
        end
        puts "\n--------- MESSAGE = #{item[:message]} --------- \n"
        results << item
        puts "-------- RESULTS  #{results}"
      end
      results
    end

def add_to_cart(url, count = 1)
      response = connection.get(url) do |req|
        req.headers["User-Agent"] =  @user_agent
      end
      doc = Nokogiri::HTML(response.body)
      stoken = doc.search('form.js-oxProductForm input[name=stoken]').attr('value').value
      empty_json = '""'
      product_id = get_item_id(url)
      data = { #removed payload for security reasons }

      # Using example.com for question purposes
      response = connection.post('https://www.example.com/index.php?') do |req|
        req.headers["Origin"] = "https://www.example.com"
        req.headers["Content-Type"] = "application/x-www-form-urlencoded; charset=UTF-8"
        req.headers["Accept"] = "application/json, text/javascript, */*; q=0.01"
        req.headers["Referer"] = url
        req.headers["Pragma"] = "no-cache"
        req.headers["Accept-Language"] = "de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"
        req.headers["User-Agent"] = @user_agent
        req.headers["Cache-Control"] = "no-cache"
        req.headers["Connection"] = "keep-alive"
        req.headers["DNT"] ="1"
        req.headers["Content-Length"] = data.size.to_s
        req.headers["Accept"] = "*/*"
        req.headers["X-Requested-With"] = "XMLHttpRequest"
        req.headers["Connection"] = "keep-alive"
        req.body = data
      end

      begin
        json = JSON.parse(response.body)
        raise "Could not add item: #{json['message']}" if json['success'] != 1 || json['item'] != product_id
      rescue JSON::ParserError => e
        puts "JSON Error"
      end

    end

def get_item_id(url)
      response = connection.get(url) do |req|
        req.headers["User-Agent"] =  @user_agent
      end
      doc = Nokogiri::HTML(response.body)
      doc.search('.js-oxProductForm input[name=aid]').attr('value').value
end

Thanks in advance.

Is it possible to make persistent HTTP requests reusing the connection between threads? If so, how can this be achieved?

Recommended topics

Hot tags