Deep, one-way synchronization of two directories using grunt-contrib-watch. Code works, but grunt-contrib-watch re-init time is too slow

Question

I have two directories src and compiled. I would like to ensure one-way data synchronization from src to compiled with Grunt Watch. As an intermediary step, I would like to compile *.less files as well as a subset of *.js files which are written with ES6 syntax.

I've successfully written the tasks which do what I need:

// NOTE: Spawn must be disabled to keep watch running under same context in order to dynamically modify config file.
watch: {
  // Compile LESS files to 'compiled' directory.
  less: {
    options: {
      interrupt: true,
      spawn: false,
      cwd: 'src/less'
    },
    files: ['**/*.less'],
    tasks: ['less']
  },
  // Copy all non-ES6/LESS files to 'compiled' directory. Include main files because they're not ES6. Exclude LESS because they're compiled.
  copyUncompiled: {
    options: {
      event: ['added', 'changed'],
      spawn: false,
      cwd: 'src'
    },
    files: ['**/*', '!**/background/**', '!**/common/**', '!contentScript/youTubePlayer/**/*', '!**/foreground/**', '!**/test/**', '!**/less/**', '**/main.js'],
    tasks: ['copy:compileSingle']
  },
  // Compile and copy ES6 files to 'compiled' directory. Exclude main files because they're not ES6.
  copyCompiled: {
    options: {
      event: ['added', 'changed'],
      spawn: false,
      cwd: 'src/js'
    },
    files: ['background/**/*', 'common/**/*', 'contentScript/youTubePlayer/**/*', 'foreground/**/*', 'test/**/*', '!**/main.js'],
    tasks: ['babel:compileSingle']
  },
  // Whenever a file is deleted from 'src' ensure it is also deleted from 'compiled'
  remove: {
    options: {
      event: ['deleted'],
      spawn: false,
      cwd: 'src'
    },
    files: ['**/*'],
    tasks: ['clean:compiledFile']
  }
}

  grunt.event.on('watch', function(action, filepath, target) {
    // Determine which task config to modify based on the event action.
    var taskTarget = '';
    if (action === 'deleted') {
      taskTarget = 'clean.compiledFile';
    } else if (action === 'changed' || action === 'added') {
      if (target === 'copyCompiled') {
        taskTarget = 'babel.compileSingle';
      } else if (target === 'copyUncompiled') {
        taskTarget = 'copy.compileSingle';
      }
    }

    if (taskTarget === '') {
      console.error('Unable to determine taskTarget for: ', action, filepath, target);
    } else {
      // Drop src off of filepath to properly rely on 'cwd' task configuration.
      grunt.config(taskTarget + '.src', filepath.replace('src\\', ''));
    }
  });

These tasks watch the appropriate files. The event handler dynamically modifies clean copy and babel tasks such that they work upon the files being added/changed/removed.

However, I am watching several thousand files and the watch task takes a non-trivial amount of time to initialize. On my high-end development PC initialization takes 6+ seconds. This issue is exacerbated by the fact that the watch task reinitializes after every task.

This means that if I have two files, fileA and fileB, and I modify fileA and save then there is a 6+ second period where watch fails to detect modifications to fileB. This results in de-synchronization between my two directories.

I found this GitHub issue regarding my problem, but it is still open and unanswered: https://github.com/gruntjs/grunt-contrib-watch/issues/443

The discussion on GitHub highlights that the issue may only occur when spawn: false has been set, but, according to the Grunt Watch documentation:

If you need to dynamically modify your config, the spawn option must be disabled to keep the watch running under the same context.

As such, I believe I need to continue using spawn: false.

I have to assume this is a pretty standard procedure for Grunt tasks. Am I missing something obvious here? Is the Watch task inappropriate for this purpose? Other options?

Evangelin

Did you take a look at grunt-newer ? Sounds like this does exactly what lowkay mentioned in the opened issue. – Furthermost

Been playing with grunt-newer for the past hour or two. It works well for compiled code, but if a file is copied from 'src' to 'compiled' without needing changes then the 'last modified' time of the file isn't updated. This results in the file being included every time the task runs. If I can find a way to fix that then grunt-newer will suffice. – Evangelin

Do you use jit-grunt ? That would help in speeding up watch task – Jessy

I'll check it out! Thank you for the suggestion. Did not know it existed. – Evangelin

Do you have a detailed log of all the function to know how much it take for each? To be clear, each time something happen, you copy the whole directory? Not just the affected file? – Flamingo

Answer 1

Alright, so I have a working solution, but it's not pretty.

I did end up using grunt-newer to assist with the solution. Unfortunately, it doesn't play well with grunt-contrib-copy because copying a file does not update its last modified time and so grunt-newer will execute 100% of the time.

So, I forked grunt-contrib-copy and added an option to allow updating the last modified time: https://github.com/MeoMix/grunt-contrib-copy

With that, I'm now able to write:

// NOTE: Spawn must be disabled to keep watch running under same context in order to dynamically modify config file.
watch: {
  // Compile LESS files to 'compiled' directory.
  less: {
    options: {
      interrupt: true,
      cwd: 'src/less'
    },
    files: ['**/*.less'],
    tasks: ['less']
  },
  // Copy all non-ES6/LESS files to 'compiled' directory. Include main files because they're not ES6. Exclude LESS because they're compiled.
  copyUncompiled: {
    options: {
      event: ['added', 'changed'],
      cwd: 'src'
    },
    files: ['**/*', '!**/background/**', '!**/common/**', '!contentScript/youTubePlayer/**/*', '!**/foreground/**', '!**/test/**', '!**/less/**', '**/main.js'],
    tasks: ['newer:copy:compiled']
  },
  // Compile and copy ES6 files to 'compiled' directory. Exclude main files because they're not ES6.
  copyCompiled: {
    options: {
      event: ['added', 'changed'],
      cwd: 'src/js'
    },
    files: ['background/**/*', 'common/**/*', 'contentScript/youTubePlayer/**/*', 'foreground/**/*', 'test/**/*', '!**/main.js'],
    tasks: ['newer:babel:compiled']
  },
  // Whenever a file is deleted from 'src' ensure it is also deleted from 'compiled'
  remove: {
    options: {
      event: ['deleted'],
      spawn: false,
      cwd: 'src'
    },
    files: ['**/*'],
    tasks: ['clean:compiledFile']
  }
}

grunt.event.on('watch', function(action, filepath) {
  if (action === 'deleted') {
    // Drop src off of filepath to properly rely on 'cwd' task configuration.
    grunt.config('clean.compiledFile.src', filepath.replace('src\\', ''));
  }
});

Now copying of ES6 files as well as non-LESS/non-ES6 files will only occur if 'src' is newer than 'dest.'

Unfortunately, grunt-newer doesn't really have support for syncing a delete operation when deleted from 'src'. So, I continue to use my previous code for 'delete' operations. This still has the same flaw where after a delete occurs the watch task will be defunct for a moment.

Evangelin