Prevent RequireJS from Caching Required Scripts

Question

RequireJS seems to do something internally that caches required javascript files. If I make a change to one of the required files, I have to rename the file in order for the changes to be applied.

The common trick of appending a version number as a querystring param to the end of the filename does not work with requirejs <script src="jsfile.js?v2"></script>

What I am looking for is a way to prevent this internal cacheing of RequireJS required scripts without having to rename my script files every time they are updated.

Cross-Platform Solution:

I am now using urlArgs: "bust=" + (new Date()).getTime() for automatic cache-busting during development and urlArgs: "bust=v2" for production where I increment the hard-coded version num after rolling out an updated required script.

Note:

@Dustin Getz mentioned in a recent answer that Chrome Developer Tools will drop breakpoints during debugging when Javascript files are continuously refreshed like this. One workaround is to write debugger; in code to trigger a breakpoint in most Javascript debuggers.

Server-Specific Solutions:

For specific solutions that may work better for your server environment such as Node or Apache, see some of the answers below.

Mongolia

You are sure this is not the server or client doing the caching? (have been using required js for a few months now and haven't noticed anything similar) IE was caught caching MVC action results, chrome was caching our html templates but js files all seem to refresh when the browsers cache has been reset. I suppose if you were looking to make use of caching but you cant do the usual because the requests from required js were removing the query string that might cause the problem? – Enid

I am not sure if RequireJS removes appended version nums like that. It may have been my server. Interesting how RequireJS has a cache-buster setting though, so you may be right about it removing my appended version nums on required files. – Mongolia

i updated my answer with a potential caching solution – Firetrap

Now I can add the following to the litany I put forth in my blog post this morning: codrspace.com/dexygen/… And that is, not only do I have to add cache busting, but require.js ignores a hard refresh. – Corker

I am confused about the use case for this.... Is this for hot-reloading AMD modules into the front-end or what? – Broad

During development I just disable the cache in the browser. Imo the easiest for development only. I think most browsers have this feature; I use Firefox. – Bronez

Answer 1

RequireJS can be configured to append a value to each of the script urls for cache busting.

From the RequireJS documentation (http://requirejs.org/docs/api.html#config):

urlArgs: Extra query string arguments appended to URLs that RequireJS uses to fetch resources. Most useful to cache bust when the browser or server is not configured correctly.

Example, appending "v2" to all scripts:

require.config({
    urlArgs: "bust=v2"
});

For development purposes, you can force RequireJS to bypass the cache by appending a timestamp:

require.config({
    urlArgs: "bust=" + (new Date()).getTime()
});

Dundalk

Very helpful, thank you. I am using urlArgs: "bust=" + (new Date()).getTime() for automatic cache-busting during development and urlArgs: "bust=v2" for production where I increment the hard-coded version num after rolling out an updated required script. – Mongolia

...also as performance optimizer you can use Math.random() instead of (new Date()).getTime(). It more beauty, not create object and a bit faster jsperf.com/speedcomparison. – Belldas

Thanks! Works for me. And I'm also using maven-resources-plugin to include the build number or a timestamp when a new version is built. – Mairamaire

You can get it just a bit smaller: urlArgs: "bust=" + (+new Date) – Mileage

Note that once you add this to your config, you will have to get the change from your web server, so you might need to clear your cache manually one more time! – Ataghan

I think busting the cache every single time is a terrible idea. Unfortunately RequireJS does not offers another alternative. We do use urlArgs but don't use a random or timestamp for this. Instead we use our current Git SHA, that way only changes when we deploy new code. – Loveless

If only one file changes will the cache be busted for all the other files also? That would be really inefficient. – Dogie

We cannot use urlArgs for building release. According to the doc: "During development it can be useful to use this, however be sure to remove it before deploying your code." requirejs.org/docs/api.html#config – Hispanic

How can this "v2" variant work in production, if the file that provides the "v2" string itself is cached? If I release a new application into production, adding "v3" is not going to do anything, as the application happily keeps working with the cached v2 files, including the old config with v2 urlArgs. – Rugging

I don't think this is the most practical solution, in chrome the debugger uses the url to identify the content and preserve breakpoint across refresh, but with this configuration the url changes on every refresh and you thus loose the breakpoints. – Explain

@Hispanic Then how can I cachebust js files included by requirejs? – Germicide

Help me understand this caching issue: Is there a reason why requirejs loaded scripts do not use headers such as If-Modified-Since, If-None-Match, etags etc to help your browser figure out if it should use cached data or not? – Timotheus

Sorry, my comment is wrong. RequireJS does use the appropriate cache headers. It's when I start nesting require calls, I run into issues. The browser fails to see modifications made to the second, nested require call – Timotheus

It should be noted that using a query string for cache busting (which is what urlArgs parameter is doing) isn't foolproof. For example, some web proxies don't cache resources with a query string by default. See this article for more details: stevesouders.com/blog/2008/08/23/… – Latitudinarian

Answer 2

Do not use urlArgs for this!

Require script loads respect http caching headers. (Scripts are loaded with a dynamically inserted <script>, which means the request looks just like any old asset getting loaded.)

Serve your javascript assets with the proper HTTP headers to disable caching during development.

Using require's urlArgs means any breakpoints you set will not be preserved across refreshes; you end up needing to put debugger statements everywhere in your code. Bad. I use urlArgs for cache-busting assets during production upgrades with the git sha; then I can set my assets to be cached forever and be guaranteed to never have stale assets.

In development, I mock all ajax requests with a complex mockjax configuration, then I can serve my app in javascript-only mode with a 10 line python http server with all caching turned off. This has scaled up for me to a quite large "enterprisey" application with hundreds of restful webservice endpoints. We even have a contracted designer who can work with our real production codebase without giving him access to our backend code.

Firetrap

Care to comment on why using urlArgs makes debugging harder? – Polymerize

@JamesP.Wright, because (in Chrome at least) when you set a breakpoint for something that happens at page load, then click refresh, the breakpoint isn't hit because the URL has changed and Chrome has dropped the breakpoint. I'd love to know a client-only workaround to this. – Herrod

Thanks, Dustin. If you find a way around this please post. In the meantime you can use debugger; in your code wherever you want a breakpoint to persist. – Mongolia

For anyone using http-server on node (npm install http-server). You can also disable caching with -c-1 (i.e. http-server -c-1). – Peisch

You can disable caching in Chrome to get around the debugging problem during development: #5690769 – Purkey

+1 to !!!NOT USE urlArgs IN PRODUCTION!!! . Imagine the case that your web site has 1000 JS files (yes, possible!) and their load is controlled by requiredJS. Now you release v2 or your site where only few JS files are changed! but by adding urlArgs=v2, you force to reload all 1000 JS files! you will pay a lot of traffic! only modified files should be re-loaded, all others should be responded with status 304 (Not Modified). – Lifework

You should not use a modulating (maybe date oriented) when developing. Chrome (and the others as well) can disable caching when the dev tools are open. If you use this your breakpoints will stay and you will always refresh your javascript files. For production scenario's I do use a bust. Works wonders. – Decorator

@walf, what can you do instead? If you set caching forever (or for a long time) - browser will not even know that it has to update any files – Studdingsail

Proper HTTP headers is not going to save us. There's always the possibility of a faulty caching Proxy in the line somewhere that neither the host or the client can resolve. Rotating URL names with random component is the only way to be sure a client gets the latest resources. – Rugging

Bust the cache with the version number of the production release, not a time stamp. Best of both worlds, caching for basic reloads and busts the cache when a customer updates their server. – Toussaint

Why then does a nested require call on my end not do a cache validation request when refreshing the browser? All other scripts will validate fine (and return a 304 status if no changes were found) but nested calls will still be served from local cache – Timotheus

Answer 3

The urlArgs solution has problems. Unfortunately you cannot control all proxy servers that might be between you and your user's web browser. Some of these proxy servers can be unfortunately configured to ignore URL parameters when caching files. If this happens, the wrong version of your JS file will be delivered to your user.

I finally gave up and implemented my own fix directly into require.js. If you are willing to modify your version of the requirejs library, this solution might work for you.

You can see the patch here:

https://github.com/jbcpollak/requirejs/commit/589ee0cdfe6f719cd761eee631ce68eee09a5a67

Once added, you can do something like this in your require config:

var require = {
    baseUrl: "/scripts/",
    cacheSuffix: ".buildNumber"
}

Use your build system or server environment to replace buildNumber with a revision id / software version / favorite color.

Using require like this:

require(["myModule"], function() {
    // no-op;
});

Will cause require to request this file:

http://yourserver.com/scripts/myModule.buildNumber.js

On our server environment, we use url rewrite rules to strip out the buildNumber, and serve the correct JS file. This way we don't actually have to worry about renaming all of our JS files.

The patch will ignore any script that specifies a protocol, and it will not affect any non-JS files.

This works well for my environment, but I realize some users would prefer a prefix rather than a suffix, it should be easy to modify my commit to suit your needs.

Update:

In the pull request discussion, the requirejs author suggest this might work as a solution to prefix the revision number:

var require = {
    baseUrl: "/scripts/buildNumber."
};

I have not tried this, but the implication is that this would request the following URL:

http://yourserver.com/scripts/buildNumber.myModule.js

Which might work very well for many people who can use a prefix.

Here are some possible duplicate questions:

RequireJS and proxy caching

require.js - How can I set a version on required modules as part of the URL?

Godman

I really would like to see your update make its way in to the official requirejs build. The principal author of requirejs might be interested, too (see @Louis's answer up above). – Mongolia

@Mongolia - feel free to comment on the PullRequest (github.com/jrburke/requirejs/pull/1017), jrburke did not seem interested. He does suggest a solution using a filename prefix, I'll update my answer to include that. – Godman

Nice update. I think I do like this suggestion by the author, but that's only because I've been using this naming convention lately: /scripts/myLib/v1.1/. I tried adding postfix (or prefix) to my filenames, probably because that's what jquery does, but after a while of that I [got lazy and] started incrementing a version number on the parent folder. I think it's made maintenance easier for me on a big website but, now you have me worrying about URL rewrite nightmares. – Mongolia

Modern front end systems just rewrite the JS files and with an MD5 sum in the filename and then rewrite the HTML files to use the new filenames when building, but that gets tricky with legacy systems where the front end code is served by the server side. – Godman

is this works when i require some js inside jspx file?, like this <script data-main="${pageContext.request.contextPath}/resources/scripts/main" src="${pageContext.request.contextPath}/resources/scripts/require.js"> <jsp:text/> </script> <script> require([ 'dev/module' ]); </script> – Uncertainty

@Uncertainty - yes, it will work, but you instead of using data-main, you may need to do this: <script> var require = { baseUrl: "/scripts/", cacheSuffix: ".buildNumber" }</script><script src="${pageContext.request.contextPath}/resources/scripts/require.js"> <jsp:text/> </script> <script> require([ 'main', 'dev/module' ]); </script> – Godman

Answer 4

Inspired by Expire cache on require.js data-main we updated our deploy script with the following ant task:

<target name="deployWebsite">
    <untar src="${temp.dir}/website.tar.gz" dest="${website.dir}" compression="gzip" />       
    <!-- fetch latest buildNumber from build agent -->
    <replace file="${website.dir}/js/main.js" token="@Revision@" value="${buildNumber}" />
</target>

Where the beginning of main.js looks like:

require.config({
    baseUrl: '/js',
    urlArgs: 'bust=@Revision@',
    ...
});

Interim

Answer 5

In production

urlArgs can cause problems!

The principal author of requirejs prefers not to use urlArgs:

For deployed assets, I prefer to put the version or hash for the whole build as a build directory, then just modify the baseUrl config used for the project to use that versioned directory as the baseUrl. Then no other files change, and it helps avoid some proxy issues where they may not cache an URL with a query string on it.

[Styling mine.]

I follow this advice.

In development

I prefer to use a server that intelligently caches files that may change frequently: a server that emits Last-Modified and responds to If-Modified-Since with 304 when appropriate. Even a server based on Node's express set to serve static files does this right out the box. It does not require doing anything to my browser, and does not mess up breakpoints.

Ineducable

Good points but, your answer is specific to your server environment. Maybe a good alternative for anyone else stumbling across this is the recent recommendation for adding a version number to the filename instead of a querystring parameter. Here's more info on that subject: stevesouders.com/blog/2008/08/23/… – Mongolia

We are running in to this specific problem in a production system. I recommend reving your filenames rather than using a parameter. – Godman

Answer 6

I took this snippet from AskApache and put it into a seperate .conf file of my local Apache webserver (in my case /etc/apache2/others/preventcaching.conf):

<FilesMatch "\.(html|htm|js|css)$">
FileETag None
<ifModule mod_headers.c>
Header unset ETag
Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
Header set Pragma "no-cache"
Header set Expires "Wed, 11 Jan 1984 05:00:00 GMT"
</ifModule>
</FilesMatch>

For development this works fine with no need to change the code. As for the production, I might use @dvtoever's approach.

Luhe

Answer 7

Quick Fix for Development

For development, you could just disable the cache in Chrome Dev Tools (Disabling Chrome cache for website development). The cache disabling happens only if the dev tools dialog is open, so you need not worry about toggling this option every time you do regular browsing.

Note: Using 'urlArgs' is the proper solution in production so that users get the latest code. But it makes debugging difficult because chrome invalidates breakpoints with every refresh (because its a 'new' file being served each time).

Purkey

Answer 8

I don't recommend using 'urlArgs' for cache bursting with RequireJS. As this does not solves the problem fully. Updating a version no will result in downloading all the resources, even though you have just changes a single resource.

To handle this issue i recommend using Grunt modules like 'filerev' for creating revision no. On top of this i have written a custom task in Gruntfile to update the revision no wherever required.

If needed i can share the code snippet for this task.

Galanti

I use a combination of grunt-filerev and grunt-cache-buster to rewrite Javascript files. – Bushweller

Answer 9

This is how I do it in Django / Flask (can be easily adapted to other languages / VCS systems):

In your config.py (I use this in python3, so you may need to tweak the encoding in python2)

import subprocess
GIT_HASH = subprocess.check_output(['git', 'rev-parse', 'HEAD']).strip().decode('utf-8')

Then in your template:

{% if config.DEBUG %}
     require.config({urlArgs: "bust=" + (new Date().getTime())});
{% else %}
    require.config({urlArgs: "bust=" + {{ config.GIT_HASH|tojson }}});
{% endif %}

• Doesn't require manual build process
• Only runs git rev-parse HEAD once when the app starts, and stores it in the config object

Yalu

Answer 10

Dynamic solution (without urlArgs)

There is a simple solution for this problem, so that you can load a unique revision number for every module.

You can save the original requirejs.load function, overwrite it with your own function and parse your modified url to the original requirejs.load again:

var load = requirejs.load;
requirejs.load = function (context, moduleId, url) {
    url += "?v=" + oRevision[moduleId];
    load(context, moduleId, url);
};

In our building process I used "gulp-rev" to build a manifest file with all revision of all modules which are beeing used. Simplified version of my gulp task:

gulp.task('gulp-revision', function() {
    var sManifestFileName = 'revision.js';

    return gulp.src(aGulpPaths)
        .pipe(rev())
        .pipe(rev.manifest(sManifestFileName, {
        transformer: {
            stringify: function(a) {
                var oAssetHashes = {};

                for(var k in a) {
                    var key = (k.substr(0, k.length - 3));

                    var sHash = a[k].substr(a[k].indexOf(".") - 10, 10);
                    oAssetHashes[key] = sHash;
                }

                return "define([], function() { return " + JSON.stringify(oAssetHashes) + "; });"
            }
        }
    }))
    .pipe(gulp.dest('./'));
});

this will generate an AMD-module with revision numbers to moduleNames, which is included as 'oRevision' in the main.js, where you overwrite the requirejs.load function as shown before.

Benoni

Answer 11

This is in addition to @phil mccull's accepted answer.

I use his method but I also automate the process by creating a T4 template to be run pre-build.

Pre-Build Commands:

set textTemplatingPath="%CommonProgramFiles(x86)%\Microsoft Shared\TextTemplating\$(VisualStudioVersion)\texttransform.exe"
if %textTemplatingPath%=="\Microsoft Shared\TextTemplating\$(VisualStudioVersion)\texttransform.exe" set textTemplatingPath="%CommonProgramFiles%\Microsoft Shared\TextTemplating\$(VisualStudioVersion)\texttransform.exe"
%textTemplatingPath% "$(ProjectDir)CacheBuster.tt"

T4 template:

Generated File:

Store in variable before require.config.js is loaded:

Reference in require.config.js:

Beautician

Likely because your solution to a JavaScript problem is a bunch of C# code. This is also a bunch of extra work and code to do something that is ultimately done the exact same way as the accepted answer. – Communize

@Communize You could do this with any server side language... doesn't have to be c#. Also this automates the process, updating the cache bust when you build a new version of the project ensuring that customer is always caching the latest version of the scripts. I don't think that deserves a negative markdown. It is just extending the answer by offering an automated approach to the solution. – Beautician

I basically created this because when maintaining an application that used require.js, I found it quite annoying to have to manually comment out the "(new Date()).getTime()) and uncomment the static cachebuster each time I updated the application. Easy to forget. All of a sudden the customer is verifying changes and seeing cached script so they think nothing has changed.. All because you simply forgot to change the cachebuster. This little bit of extra code erases the chance of that happening. – Beautician

I didn't mark it down, I don't know what actually happened. I'm just suggesting the code that's not only not js, but a C# templating language, is not going to be that helpful to JS devs trying to get an answer to their problem. – Communize

Answer 12

In my case I wanted to load the same form each time I click, I didn't want the changes I've made on the file stays. It may not relevant to this post exactly, but this could be a potential solution on the client side without setting config for require. Instead of sending the contents directly, you can make a copy of the required file and keep the actual file intact.

LoadFile(filePath){
    const file = require(filePath);
    const result = angular.copy(file);
    return result;
}

Reluctant