How to avoid nginx "upstream sent too big header" errors?
Asked Answered
T

6

36

I'm running nginx, Phusion Passenger and Rails.

I am running up against the following error:

upstream sent too big header while reading response header from upstream, client: 87.194.2.18, server: xyz.com, request: "POST /user_session HTTP/1.1", upstream: "passenger://unix:/tmp/passenger.3322/master/helper_server.sock

It is occuring on the callback from an authentication call to Facebook Connect.

After googling, and trying to change nginx settings including proxy_buffer_size and large_client_header_buffers is having no effect.

How can I debug this?

Talishatalisman answered 21/2, 2010 at 19:52 Comment(0)
R
32

Came across this error recently.

Since Passenger 3.0.8 there is now a setting that allows you to set the buffers and buffer size. So now you can do

http {
    ...
    passenger_buffers 8 16k;
    passenger_buffer_size 32k;
}

That resolved the issue for me.

Rubino answered 26/9, 2011 at 17:37 Comment(0)
F
35

Try to add this to the config:

http {
    ...
    proxy_buffers 8 16k;
    proxy_buffer_size 32k;
    }
Frisco answered 21/2, 2010 at 23:46 Comment(4)
since he's using Phusion Passenger atop of nginx, your solution will NOT help him. because his quoted error message above comes from passenger itself and almost definitely not from nginx directly. However, it might be safe to still do what you said, but also add the two passenger-pendants (passenger_buffers & passenger_buffer_size) as stated below.Kike
@trapni May not help answer this question, but is also a solution for other people using nginx proxyFitter
solved my problem with nginx reverse proxy to apache2. thanks :)Petulia
Winner winner chicken dinnerAustralia
R
32

Came across this error recently.

Since Passenger 3.0.8 there is now a setting that allows you to set the buffers and buffer size. So now you can do

http {
    ...
    passenger_buffers 8 16k;
    passenger_buffer_size 32k;
}

That resolved the issue for me.

Rubino answered 26/9, 2011 at 17:37 Comment(0)
H
26

Maybee adding this will make it work, how are you connecting to upstream? http, fastcgi or something else?

http {
    ...
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;
}
Hiram answered 15/8, 2011 at 14:55 Comment(5)
phusion passenger is not using fastcgi at all, it is a a direct part of the nginx process, so this will NOT help at al in his case.Kike
Oh shit! You're absolutely right, stupid mistake from my side :)Johnnie
Thanks for this. It worked great for my fastcgi implementation.Unguentum
This helped me two with my Wordpress nGinX configuration.Viperine
Works great for php-fpm header issues.Giraffe
N
6
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
Nyeman answered 17/1, 2012 at 19:27 Comment(0)
V
1

This is everything I have come to understand about this error in the last 2 years:

upstream sent too big header while reading response header from upstream is nginx's generic way of saying "I don't like what I'm seeing"

  1. Your upstream server thread crashed
  2. The upstream server sent an invalid header back
  3. The Notice/Warnings sent back from STDERR broke their buffer and both it and STDOUT were closed

3: Look at the error logs above the message, is it streaming with logged lines preceding the message? PHP message: PHP Notice: Undefined index: Example snippet from a loop my log file:

2015/11/23 10:30:02 [error] 32451#0: *580927 FastCGI sent in stderr: "PHP message: PHP Notice:  Undefined index: Firstname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:  Undefined index: Lastname in /srv/www/classes/data_convert.php on line 1090
... // 20 lines of same
PHP message: PHP Notice:  Undefined index: Firstname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:  Undefined index: Lastname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:
2015/11/23 10:30:02 [error] 32451#0: *580927 FastCGI sent in stderr: "ta_convert.php on line 1090
PHP message: PHP Notice:  Undefined index: Firstname

you can see in the 3rd line (from the 20 previous errors) the buffer limit was hit, broke, and the next thread wrote in over it. Nginx then closed the connection and returned 502 to the client.

2: log all the headers sent per request, review them and make sure they conform to standards (nginx does not permit anything older than 24 hours to delete/expire a cookie, sending invalid content length because error messages were buffered before the content counted...)

examples include:

<?php
//expire cookie
setcookie ( 'bookmark', '', strtotime('2012-01-01 00:00:00') );
// nginx will refuse this header response, too far past to accept
....
?>

and this:

<?php
header('Content-type: image/jpg');
?>

<?php   //a space was injected into the output above this line
header('Content-length: ' . filesize('image.jpg') );
echo file_get_contents('image.jpg');
// error! the response is now 1-byte longer than header!!
?>

1: verify, or make a script log, to ensure your thread is reaching the correct end point and not exiting before completion.

Vaticinal answered 18/9, 2013 at 14:29 Comment(0)
M
0

I thought I'd chime in with my solution since I don't see it currently listed. Turns out I was unintentionally putting a large object into the session, as shown below.

session["devise.#{provider}_data"] = env["omniauth.auth"]

This only happened when somebody first authenticated with GitHub OAuth and subsequently tried to authenticate with another social profile that used the same email (why I couldn't originally figure out the issue).

Here's the full OmniauthCallbacksController for contextual reference:

class OmniauthCallbacksController < Devise::OmniauthCallbacksController

  def self.provides_callback_for(provider)
    class_eval %Q{
      def #{provider}
        @user = User.from_omniauth(request.env["omniauth.auth"])
        if @user.persisted?
          sign_in_and_redirect @user, event: :authentication
          set_flash_message(:notice, :success, kind: "#{provider}".capitalize) if is_navigational_format?
        else
          auth = request.env["omniauth.auth"]
          if User.exists?(email: auth.info.email)
            set_flash_message(:notice, :failure, kind: "#{provider}".capitalize, reason: "email " + auth.info.email + " already exists") if is_navigational_format?
          else
            set_flash_message(:notice, :error, kind: "#{provider}".capitalize) if is_navigational_format?
          end
          session["devise.#{provider}_data"] = env["omniauth.auth"] <----- Remove this line
          redirect_to new_user_registration_path
        end
      end
    }
  end

  [:github, :linkedin, :google_oauth2].each do |provider|
    provides_callback_for provider
  end
end

All was well once I removed the offending line. I am guessing I had it in there for debugging purposes.

Mystic answered 20/5, 2016 at 15:34 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.