I am attempting to deploy a Sinatra streaming SSE response application on the Cedar stack. Unfortunately while it works perfectly in development, once deployed to Heroku the callback or errback never get called when a connection is called, leading to the connection pool getting filled up with stale connections (that never time out because data is still being sent to them on the server side.)
Relvant info from the Heroku documentation:
Long-polling and streaming responses
Cedar supports HTTP 1.1 features such as long-polling and streaming responses. An application has an initial 30 second window to respond with a single byte back to the client. However, each byte transmitted thereafter (either received from the client or sent by your application) resets a rolling 55 second window. If no data is sent during the 55 second window, the connection will be terminated.
If you’re sending a streaming response, such as with server-sent events, you’ll need to detect when the client has hung up, and make sure your app server closes the connection promptly. If the server keeps the connection open for 55 seconds without sending any data, you’ll see a request timeout.
This is exactly what I would like to do -- detect when the client has hung up, and close the connection promptly. However, something about the Heroku routing layer seems to prevent Sinatra from detecting the stream close event as it would normally.
Some sample code that can be used to replicate this:
require 'sinatra/base'
class MyApp < Sinatra::Base
set :path, '/tmp'
set :environment, 'production'
def initialize
@connections = []
EM::next_tick do
EM::add_periodic_timer(1) do
@connections.each do |out|
out << "connections: " << @connections.count << "\n"
end
puts "*** connections: #{@connections.count}"
end
end
end
get '/' do
stream(:keep_open) do |out|
@connections << out
puts "Stream opened from #{request.ip} (now #{@connections.size} open)"
out.callback do
@connections.delete(out)
puts "Stream closed from #{request.ip} (now #{@connections.size} open)"
end
end
end
end
I've put a sample app up at http://obscure-depths-3413.herokuapp.com/ using this code that illustrates the problem. When you connect, the amount of connections will increment, but when you disconnect they never go down. (Full source of demo with Gemfile etc is at https://gist.github.com/mroth/5853993)
I'm at wits end trying to debug this one. Anyone know how to fix it?
P.S. There appears to have been a similar bug in Sinatra but it was fixed a year ago. Also this issue only occurs on production in Heroku, but works fine when run locally.
P.S.2. This occurs when iterating over the connections objects as well, for example adding the following code:
EM::add_periodic_timer(10) do
num_conns = @connections.count
@connections.reject!(&:closed?)
new_conns = @connections.count
diff = num_conns - new_conns
puts "Purged #{diff} connections!" if diff > 0
end
Works great locally, but the connections never appear as closed on Heroku.
foreman start? – Forefather