Incremental gulp less build
Asked Answered
H

3

15

In my office we are using gulp to build our less files. I wanted to improve the build task as it took over a second to build on a large project we recently worked on. The idea was to cache the files and only pass the one that changed. So I started with google and found incremental builds for javascript ang thought it would be easy to rewrite them for less. Here's the one I started with: https://github.com/gulpjs/gulp/blob/master/docs/recipes/incremental-builds-with-concatenate.md

After a few unsuccessful tries I ended up with following code (tested with the latest bootstrap distribution):

var gulp            = require('gulp');
var less            = require('gulp-less');
var concat          = require('gulp-concat');
var remember        = require('gulp-remember');
var cached          = require('gulp-cached');

var fileGlob = [
    './bootstrap/**/*.less',
    '!./bootstrap/bootstrap.less',
    '!./bootstrap/mixins.less'
];

gulp.task('less', function () {
    return gulp.src(fileGlob)
        .pipe(cached('lessFiles'))
        .pipe(remember('lessFiles'))
        .pipe(less())
        .pipe(gulp.dest('output'));
});

gulp.task('watch', function () {
    var watcher = gulp.watch(fileGlob, ['less']);
    watcher.on('change', function (e) {
        if (e.type === 'deleted') {
            delete cached.caches.scripts[e.path];
            remember.forget('lessFiles', e.path);
        }
    });
});

But this passes only the changed file and the less compiler fails because of the variable definitions missing. If I pipe the concat plugin before the less task, gulp gets stuck in a (seemingly) endless loop.

gulp.task('less', function () {
    return gulp.src(fileGlob)
        .pipe(cached('lessFiles'))
        .pipe(remember('lessFiles'))
        .pipe(concat('main.less')
        .pipe(less())
        .pipe(gulp.dest('output'));
});

Has anyone experience with those plugins or managed to create an incremental less build in an other way. Here is a (messy) github repository for testing: https://github.com/tuelsch/perfect-less-build

PS: I'm planning on adding linting, sourcemaps, minification, evtl. cache busting and autoprefixer later on.

Helfant answered 29/10, 2014 at 10:50 Comment(3)
I was investigating the exact same thing. Unfortunately there doesn't seem to be a convenient solution. However I've found an article which is covering that topic (still is of no avail, though): io.pellucid.com/blog/…Doronicum
While digging around, I stumbled across broccoli (solitr.com/blog/2014/02/broccoli-first-release), yet another "task runner". It is still a young project, but they seem to implement the above idea as a core feature. Will keep an eye on this.Helfant
Is cached necessary? Does removing it from the pipeline make things work? They work for me, but I don't know if I am doing the same thing as you. Could you provide the step to duplicate your errors?Reticular
O
20

Like Ashwell, I've found it useful to use imports to ensure that all my LESS files have access to the variables and mixins that they need. I also use a LESS file with imports for bundling purposes. This has a few advantages:

  1. I can leverage LESS's features to do complex things like overriding variable values to produce multiple themes, or prepending a class to every rule in another LESS file.
  2. There's no need for the concat plugin.
  3. Tools like Web Essentials for Visual Studio can give syntax help and output previews, because every LESS file is fully capable of being rendered by itself.

Where you want to import variables, mixins, etc, but you don't want to actually output the entire contents of another file, you can use:

@import (reference) "_colors.less";

After a few days of effort, I was finally able to get an incremental build that correctly rebuilds all the objects that depend on the LESS file I changed. I documented the results here. This is the final gulpfile:

/*
 * This file defines how our static resources get built.
 * From the StaticCommon root folder, call "gulp" to compile all generated
 * client-side resources, or call "gulp watch" to keep checking source 
 * files, and rebuild them whenever they are changed. Call "gulp live" to 
 * do both (build and watch).
 */

/* Dependency definitions: in order to avoid forcing everyone to have 
 * node/npm installed on their systems, we are including all of the 
 * necessary dependencies in the node_modules folder. To install new ones,
 * you must install nodejs on your machine, and use the "npm install XXX" 
 * command. */
var gulp = require('gulp');
var less = require('gulp-less');
var LessPluginCleanCss = require('less-plugin-clean-css'),
    cleanCss = new LessPluginCleanCss();
var sourcemaps = require('gulp-sourcemaps');
var rename = require('gulp-rename');
var cache = require('gulp-cached');
var progeny = require('gulp-progeny');
var filter = require('gulp-filter');
var plumber = require('gulp-plumber');
var debug = require('gulp-debug');

gulp.task('less', function() {
    return gulp
        // Even though some of our LESS files are just references, and 
        // aren't built, we need to start by looking at all of them because 
        // if any of them change, we may need to rebuild other less files.
        .src(
        ['Content/@(Theme|Areas|Css)/**/*.less'],
        { base: 'Content' })
        // This makes it so that errors are output to the console rather 
        // than silently crashing the app.
        .pipe(plumber({
            errorHandler: function (err) {
                console.log(err);
                // And this makes it so "watch" can continue after an error.
                this.emit('end');
            }
        }))
        // When running in "watch" mode, the contents of these files will 
        // be kept in an in-memory cache, and after the initial hit, we'll
        // only rebuild when file contents change.
        .pipe(cache('less'))
        // This will build a dependency tree based on any @import 
        // statements found by the given REGEX. If you change one file,
        // we'll rebuild any other files that reference it.
        .pipe(progeny({
            regexp: /^\s*@import\s*(?:\(\w+\)\s*)?['"]([^'"]+)['"]/
        }))
        // Now that we've set up the dependency tree, we can filter out 
        // any files whose
        // file names start with an underscore (_)
        .pipe(filter(['**/*.less', '!**/_*.less']))
        // This will output the name of each LESS file that we're about 
        // to rebuild.
        .pipe(debug({ title: 'LESS' }))
        // This starts capturing the line-numbers as we transform these 
        // files, allowing us to output a source map for each LESS file 
        // in the final stages.
        // Browsers like Chrome can pick up those source maps and show you 
        // the actual LESS source line that a given rule came from, 
        // despite the source file's being transformed and minified.
        .pipe(sourcemaps.init())
        // Run the transformation from LESS to CSS
        .pipe(less({
            // Minify the CSS to get rid of extra space and most CSS
            // comments.
            plugins: [cleanCss]
        }))
        // We need a reliable way to indicate that the file was built
        // with gulp, so we can ignore it in Mercurial commits.
        // Lots of css libraries get distributed as .min.css files, so
        // we don't want to exclude that pattern. Let's try .opt.css 
        // instead.
        .pipe(rename(function(path) {
            path.extname = ".opt.css";
        }))
        // Now that we've captured all of our sourcemap mappings, add
        // the source map comment at the bottom of each minified CSS 
        // file, and output the *.css.map file to the same folder as 
        // the original file.
        .pipe(sourcemaps.write('.'))
        // Write all these generated files back to the Content folder.
        .pipe(gulp.dest('Content'));
});

// Keep an eye on any LESS files, and if they change then invoke the 
// 'less' task.
gulp.task('watch', function() {
    return gulp.watch('Content/@(Theme|Areas|Css)/**/*.less', ['less']);
});

// Build things first, then keep a watch on any changed files.
gulp.task('live', ['less', 'watch']);

// This is the task that's run when you run "gulp" without any arguments.
gulp.task('default', ['less']);

We can now simply run gulp live to build all our LESS files once, and then allow each subsequent change to just build those files that depend on the changed files.

Orthopter answered 5/2, 2015 at 22:4 Comment(8)
This is amazing, progeny and using imports in every file seem to be the parts I was missing. Let me run a few test builds before accepting this as the correct answer.Helfant
@phippu: Glad my effort might be able to help others. Feel free to reach out to me if you run into any snags.Orthopter
@phippu: I updated my blog post and the code above with some additional error-handling lines needed to keep watch working after it runs into an error.Orthopter
I updated the git (github.com/tuelsch/perfect-less-build) with a working example of incremental builds thanks to includes and progeny. Allthough I cannot get the less extensions autoprefixer and clean css to work properly. I get the error "Object has no method run".Helfant
@Orthopter this works fine but when you stop the gulp task and start it again it is compiling all the less files again.Lisettelisha
@classydraught: Yeah, we're six years since my last edit (to the day) and I've kind of moved on to other tech. If you have a solution that figures out how to cache results between runs, feel free to add an answer of your own. :-)Orthopter
@Orthopter Sure will be doing that :) Also could you let me know which technology you're using.Lisettelisha
We're using Angular now, which has its own build system. If I were using a framework without its own build system, I'd go with Webpack today.Orthopter
T
2

So when I want to do incremental builds in gulp I do it by abstracting out the inner process of the gulp task, this way I don't have to worry about keeping a cache.

// Create a function that does just the processing
var runCompile = function( src, dest, opts ){
  return gulp.src( src )
    .pipe(less( opts ))
    .pipe(gulp.dest( dest ));
};

// Leverage the function to create the task
gulp.task( 'less', function(){
  return runCompile( fileGlob, 'output', {} );
});

// Use it again in the watch task
gulp.task( 'less:watch', function(){
  return gulp.watch( fileGlob )
    .on( "change", function( event ){
      // might need to play with the dest dir here
      return runCompile( event.path, 'output', {} );
    });
});

This works great for me and I use this pattern all over my gulp tasks. However I have noticed that sometime gulp will squash paths during the watch "on change" if it gets a single file. In that case I do the path manipulation my self, something like path.dirname(srcPath.replace( srcDir, outputDir )) as the argument dest for the runCompile function.

Edit: Just realized this probably isn't going to solve your "lost variables" problem. I don't have anything off the top of my head to solve that one since I organize my LESS files with a heavy use of imports, so every file that would need a set of variables would have an import statement insuring they are there.

Triplenerved answered 24/1, 2015 at 4:35 Comment(3)
You could also take it up a notch and use something like LazyPipe, npmjs.com/package/lazypipeTriplenerved
Thanks for your approach, but I don't see how this only compiles the file that changed and still results in one single CSS file. Allthough I like the idea with the imports.Helfant
Yeah, I'm sorry I should have read the question a little more throughly.Triplenerved
L
1

We can actually use gulp-newer and gulp-progeny-mtime for this task. The approach by Stripling is almost the best one expect every time you run the gulp less task it will compile everything again from scratch and then will start watching the files. This will cost you a lot of time if you are working with a lot of less style sheets. gulp-progeny-mtime is similar to gulp-progeny except it does the real hardcore stuff. Every time a files passes through gulp-progeny-mtime, it checks for any modification in imports, and if so it will adjust the mtime of the current file in the stream which leads it to pass through gulp-newer. This I feel is better because we are not even caching anything.

   //Compile less for deployment 
   gulp.task("less", () => {
      return gulp
        .src(["static/less/**/*.less"])
        .pipe(progenyMtime())
        .pipe(
          plumber({
            errorHandler: function (err) {
              log(chalk.bgRed.white.bold(err.message));
            },
          })
        )
        .pipe(filter(["**/*.less", "!**/_*.less", "!static/less/includes*/**"]))
        .pipe(newer({ dest: "static/css/", ext: ".css" }))
        .pipe(debug({ title: "LESS" }))
        .pipe(
          less({
            plugins: [cleanCss, autoprefix],
          })
        )
        .pipe(gulp.dest("static/css/"));
    });

    //Watch changes is less and compile if changed.
    gulp.task("watch-less", () => {
      return gulp.watch("static/less/**/*.less", gulp.series("less"));
    });
    
    //Compile all less files on first run ( if changed ) then compile only modified files from next run
    gulp.task("live-less", gulp.series("less", "watch-less"));
Lisettelisha answered 10/2, 2021 at 4:28 Comment(1)
That's quite clever. Also, this question is really old and I don't have the setup to test this anymore.Helfant

© 2022 - 2024 — McMap. All rights reserved.