How to prevent Meteor from watching files?

Question

I would like to use Dojo Toolkit with Meteor.

1. I first copy the whole Dojo Toolkit tree in /public

2. Then, I include it on the client side with:

   <script src="/dojo/dojo.js" data-dojo-config="async: true"></script>`
   

Everything works fine, except Meteor is actually monitoring every single file in /public for changes, so that it can restart the server. This is actually causing a very long delay during the first request on localhost:3000.

Is there a way of preventing Meteor from watching files from a certain directory?

---

Dojo Toolkit is 10k+ files so I get the EMFILE error stated here, corrected with

sudo sh -c 'echo 16384 > /proc/sys/fs/inotify/max_user_watches'

Ha

I finally found a workaround. I'm putting everything in /public/lib/. Then, line 286 of /usr/lib/meteor/app/run.js, I'm adding the folder I don't want Meteor to watch: self.exclude_paths = [ path.join(app_dir, '.meteor', 'local'), path.join(app_dir, 'public', 'lib') ]; This way I can have as much files as I want in lib, and they don't slow everything down. include path is '/lib/dojo/dojo.js'. – Ha

nice fix, you should send them a patch and post this as an answer below – Throne

The workaround @Ha suggests should work fine for this purpose, but Meteor should support something like this natively. This is also filed as a GitHub Issue, which is the appropriate place to continue discussing this matter, or to track its resolution. github.com/meteor/meteor/issues/437 – Pamulapan

#15936591 – Triturable

@BjoernRennhak you know that the answer in the question you linked is in fact a quote from this topic right? – Madagascar

Can't hurt to link them or does it? – Triturable

I don't have a /usr/lib/meteor. Running OS X and installed through the Meteor script. – Bipod

Not sure if this may help: #14824283 – Bipod

Answer 1

realised this is duplicate to: generating and serving static files with Meteor

see: https://github.com/meteor/meteor/issues/437

This was a major problem for me. I have to serve ~12000 static images, which I initially put into the public folder. This caused node to use nearly 100% of one CPU core, constantly. With limited memory the app crashes.

The workaround I'm using for the moment

• create the folder public/.#static/ and move all static assets into it. This folder isn't watched by meteor
• prefix urls with static (/img/cat.png -> /static/img/cat.png)

• install the mime npm package

  cd ~/.meteor/tools/latest/lib/node_modules/
  npm install mime
  

• create a rawConnectionHandler to serve the assets (credits to: https://stackoverflow.com/a/20358612) server/static_files_handler.coffee

  fs = Npm.require('fs')
  mime = Npm.require('mime')
  WebApp.rawConnectHandlers.use (req, res, next) ->
    re = /^\/static\/(.*)$/.exec(req.url)
    if re isnt null # Only handle URLs that start with /static/*
      filePath = process.env.PWD + "/public/.#static/" + re[1]
      type = mime.lookup(filePath)
      data = fs.readFileSync(filePath, data)
      res.writeHead 200,
        "Content-Type": type
  
      res.write data
      res.end()
    else # Other urls will have default behaviors
      next()
    return
  

Limitations of this approach:

• not suitable to serve assets with query parameters. The regex would also match /static/html/image.html?src=/static/img/cat.png trying to serve a file with the filename including the parameters. This is easy to change.
• Meteor is completely unaware of the files, they therefore don't get included into the appcache manifest. If you want to make them available offline, check out the addPaths option I added to https://github.com/buildhybrid/appcache-extra

If you don't want to work around the problems, consider serving the assets from an external service (ex. AWS S3).

Builder

there is a problem if you will go and deploy this app on *.meter.com - this will cause an error - thats because of Meteor ignores (and actually have no) .#static directory – Hild

Answer 2

This is a big issue for large scale applications built in Meteor. I talked to Matt over at Meteor, and he said that their team is working on a solution to this problem for one of their upcoming releases. So get on their newsletter, and you'll be notified when it is available.

Mathieu, your comment:

I finally found a workaround. I'm putting everything in /public/lib/. Then, line 286 of /usr/lib/meteor/app/run.js, I'm adding the folder I don't want Meteor to watch: self.exclude_paths = [ path.join(app_dir, '.meteor', 'local'), path.join(app_dir, 'public', 'lib') ]; This way I can have as much files as I want in lib, and they don't slow everything down. include path is '/lib/dojo/dojo.js'.

is a good hack for now, and here are the other issues that relate that will also be covered in the upcoming release:

• Some packages have overlapping dependencies, but do not exclude them like in PHP's require_once().

• A native require / define dynamic script loader, so not all files are loaded on entry one that can calculate dependency order without having to go levels, lib or main.*

• An official way to create and deploy packages to a repository. Currently it is using Meteorite (mrt) and Atmosphere.

Rosalindrosalinda

The goal is to have meteor serve but not watch the asset files in a directory, preferably all the large video files I have in public. Does your exclude_paths line continue to serve the files? – Jessee