Capistrano Symlinks Being Cached?

Question

I've been setting up PHP deployments with Capistrano on CentOS 6 and have run into an interesting issue. The way capistrano works, it sets up folders like this:

• /var/www/myapp.com/
  • current (symlink to latest release in /releases)
  • shared
  • releases
    • 20130826172737
    • 20130826172114

When I look at the "current" symlink, it points to the most recent release. At first, when opening my web app, everything worked fine. After deploying a new release, the current folder correctly points to the new release, but the web application tries to load files from the old release (which has been deleted in a Capistrano cleanup process). Also, the virtual host is configured to point at /var/www/myapp.com/current/Public.

Are symlinks cached in any way?

The specific PHP code that fails (which initializes my framework) is this:

require_once dirname(dirname(__FILE__)) . '/App/App.php';
App\App::run();

That is in index.php currently located at /var/www/app.com/current/Public/index.php.

My Apache error logs show:

PHP Fatal error: require_once(): Failed opening required '/var/www/myapp.com/releases/20130826172237/App/App.php' (include_path='.:/usr/share/pear:/usr/share/php') in /var/www/myapp.com/releases/20130826172237/Public/index.php

And the current symlink shows:

current -> /var/www/zverse/releases/20130826172641

Obviously 20130826172641 != 20130826172237 which the latter was the previous version.

Any ideas or areas I can look at?

Cummings

Seems to have been a hiccup. I waited about 10 minutes and tried again, and it worked. I wonder if it was some sort of Apache optimization? – Cummings

Disable PHP's realpath cache or use mod_realdoc and this issue will go away without having to restart the server. – Odyssey

same issue with nginx server on Centos, the 'current' symlink won't update to the newest one. – Cumae

Answer 1

I can't verify this, but it seems that there is some unpredictable behaviour with Apache following / caching the old location of symlinks:

• Is there a way to mimic symlink behavior with an apache configuration?
• Case Against Using Symlinks For Code Promotion

The only thing that would absolutely clear up this issue was to cycle Apache, which we would prefer not to do on every deployment. -- Mike Brittain

He suggests moving the whole directory, instead of updating the symlink.

Cowardice

Thanks for the information. I still encounter the issue quite frequently, but have made part of my Capistrano deployment process do a service httpd restart which seems to fix the issue. – Cummings

service httpd reload/systemctl reload apache2.service involves no downtime and has the same effect. – Causal

thanks @Causal - would you like to edit the answer / make it a community answer? I'm not in a position to test this currently – Cowardice

I think I'll leave it as a comment if that's OK. I haven't tested this for Fedora/RHEL, only for Debian/Ubuntu flavors, so I may have the syntax off. – Causal

Answer 2

Have you checked the realpath_cache_size and realpath_cache_ttl directives? By default, php > 5.1 caches the real paths of symlinked files for 120 seconds. This will cause problems with capistrano deployments. The main problems are caching - that even if you clear your cache, your old php files will continue to be served for two minutes, repopulating it with old data - and interaction between php and static files. Static files are served directly by Apache, so will be updated immediately. The php code will still be from the previous release for two minutes after deploying though, so it will be expecting the old versions of any changed static files. That's especially a problem if you use a cache breaking procedure that changes the names of those files; in that case the php code won't be able to find the files it's expecting at all.

Anyway, there are two solutions. The first is to set realpath_cache_size to 0 in php.ini. (Note: setting realpath_cache_ttl to 0 does not disable the cache.) Or, if you want to keep it enabled, you should be able to use the clearstatcache function to clear the realpath cache immediately after deploying your symlink, using a capistrano hook. Be aware though, if you're using mod_php, the php cli and apache runtimes are separate, so you would need to call that function using a php script invoked by apache, similarly to what's done for clearing the APC cache here. I haven't tested that though, as I didn't notice a significant performance impact from simply disabling the cache.

Farmyard

Any thoughts on the exact interaction between the realpath cache and the filling of the APC cache, post-clearance? apc_clear_cache() will clear the php-fpm pool's entire (shared memory) cache, but clearstatcache(true) will only clear that specific php-fpm process's realpath cache. If I have 4 php-fpm processes running on a busy system, the other 3 could still populate from old data again? – Touchmenot

@Touchmenot I'm not sure exactly what you're asking. The APC cache and the realpath cache are two separate issues. If you are using APC for your classmap, or anything like that, that will be invalidated when you reploy a new revision of code, you will need to clear it when you deploy. The realpath cache is separate, and refers to php caching symlinks, so it will use the old locations of files after you re-deploy, unless you run clearstatcache. If indeed that only clears the realpath cache for that one php process (as I said, I haven't tested) you would need a way to clear it for all. – Farmyard

@Touchmenot not sure how you would accomplish that without restarting the web server. (Personally I found that just disabling the realpath cache didn't affect performance in any noticeable way. Again, that's distinct from the APC cache.) One other thing to note is if you are using APC or some other cache to cache your classmap or any php directories, you will need to clear that immediately after clearing the realpath cache; otherwise it will get repopulated from the old values. – Farmyard

right... we've been diligently clearing my APC opcode and user caches after a Capifony deploy ("after 'deploy'"), but still sometimes having problems. e.g. new Assetic asset files, but references to the old ones in the delivered HTML. Then I found your notes about the realpath cache. I shall experiment some more, with it set to 0 size, as I think clearing it just can't be done reliably across multiple php-fpm processes without some modifications to php-fpm. – Touchmenot

APC & OpCache are disabled, and realpath_cache_* vars are set to 0, but paths still mixing... Can't find the solution. Btw we're using nginx not apache, but problem still exists. – Frayda