Rails when served by unicorn is generating the wrong fingerprint for precompiled assets

Question

I have a production server with nginx -> unicorn -> rails. I precompile assets, which puts assets in public/assets with a fingerprint (hash) appended to the filename. However, when a web page is requested, references to application.css and application.js assets will have the wrong fingerprint. For example, the rails helper stylesheet_link_tag will generate a filename that does not exist in public/assets on the server, because the requested fingerprint does not match the precompiled one. Image assets work fine (fingerprints match).

Troubleshooting this, I precompiled assets on my local machine, and the fingerprint matches the precompiled fingerprint on my server. Moreover, when running locally with webrick in production mode, everything works. I then tried running webrick on my server, which worked, making me think unicorn is the source of the problem.

I have solved this problem by starting unicorn with the --no-default-middleware (or -N) option, and this makes unicorn behave as expected with respect to precompiled assets. My understanding is that this tells unicorn not to load a default set of Rack middleware that it would otherwise load. However, I don't really understand why this solves the problem, or what is going wrong in the first place. What is going on?

Some specifics: Ubuntu 12.04, Rails 4.0.1, Ruby 2.1.0, bootstrap 3.0 with a third party theme

Update:

The reason that I don't believe this is an nginx problem is that when I request the page from a browser, and then view source, the filename for the application.css file has a fingerprint, so it actually looks like application-3855b1928b94aa5bff5e1dac1aa56882.css. This does not match the real fingerprint of the file on my server. I am convinced that the fingerprint on the server is correct because I get the same fingerprint on my local development machine. So the client is loading the web page, then asking the server for the .css file...nginx receives that request, but can't find the .css file, because it actually isn't there, so nginx is behaving as expected.

The fingerprint is generated twice: Once when I precompile assets, which is working correctly (pretty sure), and is happening with no unicorn involvement. And a second time, when rails in the context of a unicorn worker sees the stylesheet_link_tag helper and on-the-fly calculates the fingerprint (I think this is what it's doing), which for some reason is generating an incorrect fingerprint. This same process happens if I replace unicorn with webrick on the exact same server, but in this case the fingerprints match. If I start unicorn with the -N flag, the fingerprints match, but I don't know why this makes a difference, and I don't know why nobody else seems to have to do this.

Mystify

Do you have any specific rack middleware which handles assets? – Clarkclarke

@Nicolay - only what Rails gives you by default. Only non-standard gems are stripe, twilio-ruby, aws-s3, pg, zip, and unicorn. – Mystify

Are you doing the precompilation on the server, or locally and then uploading the manifest? – Clarkclarke

@Nicolay - on the server--but during troubleshooting I also precompiled locally just to make sure the locally-generated fingerprints match the server-generated fingerprints, which they do. – Mystify

Sorry for my ignorance, but why use a middleware when using unicorn? – Hackery

@WhitKemmey - I don't know much about nginx, but could something in this question be of help? #9826237 – Clarkclarke

@RubyRacer - I'm not purposely using any middleware, other than what rails or unicorn might itself use, which I am not knowledgeable about. However, I've never seen anyone else (in examples on the web) set the --no-default-middleware flag when starting unicorn, so what is going on that I have to? – Mystify

@Nicolay - I looked at the link you suggested, and that is not my problem (my nginx config does not have that particular mistake). I'm going to update the question to explain why I don't think this is an nginx error. – Mystify

A shot in the dark: does the fingerprints in your manifest.json match the delivered files? – Dowling

Are you using the USR2 signal to reload the unicorn? If so, try stop and start it again. – Caruso

having a similar problem here since i updated to rails4. Have you found the reason for it? On my system the fingerprints inside the app seem to be one version "behind". if i do another deployment without cleaning assets, the assets will load (from the last deploy) – Wingless

@Rodrigo's comment solved exact same problem on my side – Boil

Answer 1

A wild guess: check that Unicorn is actually run in production environment on your sever.

Reprehensible

Answer 2

Even we faced the same problem where assets compilation succeeded but things did not work as expected. We found that unicorn was the main culprit here. Same as your case, even we used to restart unicorn after compiling the assets. It was noticed that when unicorn is restarted, only its worker processes are restarted and not the master process. This is the main reason the correct assets are not served. Later, after compiling assets, we stopped and started unicorn so that the unicorn master process is also restarted and the correct assets were getting served. Stopping and starting unicorn brings around 10 secs on downtime when compared to restarting the unicorn. This is the workaround that can be used where as long term solution is move to puma from unicorn.

Arsenault