NodeJS & Gulp Streams & Vinyl File Objects- Gulp Wrapper for NPM package producing incorrect output

Question

Goal

I am currently trying to write a Gulp wrapper for NPM Flat that can be easily used in Gulp tasks. I feel this would be useful to the Node community and also accomplish my goal. The repository is here for everyone to view , contribute to, play with and pull request. I am attempting to make flattened (using dot notation) copies of multiple JSON files. I then want to copy them to the same folder and just modify the file extension to go from *.json to *.flat.json.

My problem

The results I am getting back in my JSON files look like vinyl-files or byte code. For example, I expect output like "views.login.usernamepassword.login.text": "Login", but I am getting something like {"0":123,"1":13,"2":10,"3":9,"4":34,"5":100,"6":105 ...etc

My approach

I am brand new to developing Gulp tasks and node modules, so definitely keep your eyes out for fundamentally wrong things.

The repository will be the most up to date code, but I'll also try to keep the question up to date with it too.

Gulp-Task File

var gulp = require('gulp'),
    plugins = require('gulp-load-plugins')({camelize: true});
var gulpFlat = require('gulp-flat');
var gulpRename = require('gulp-rename');
var flatten = require('flat');

gulp.task('language:file:flatten', function () {

return gulp.src(gulp.files.lang_file_src)
    .pipe(gulpFlat())
    .pipe(gulpRename( function (path){
        path.extname = '.flat.json'
    }))
    .pipe(gulp.dest("App/Languages"));
});

Node module's index.js (A.k.a what I hope becomes gulp-flat)

var through = require('through2');
var gutil = require('gulp-util');
var flatten = require('flat');
var PluginError = gutil.PluginError;

// consts
const PLUGIN_NAME = 'gulp-flat';


// plugin level function (dealing with files)
function flattenGulp() {

    // creating a stream through which each file will pass
    var stream = through.obj(function(file, enc, cb) {
        if (file.isBuffer()) {

             //FIXME: I believe this is the problem line!!
            var flatJSON = new Buffer(JSON.stringify(
                flatten(file.contents)));
            file.contents = flatJSON;
    }

    if (file.isStream()) {

        this.emit('error', new PluginError(PLUGIN_NAME, 'Streams not supported! NYI'));
        return cb();
    }

    // make sure the file goes through the next gulp plugin
    this.push(file);
    // tell the stream engine that we are done with this file
    cb();
});

// returning the file stream
return stream;
}

// exporting the plugin main function
module.exports = flattenGulp;

Resources

• https://github.com/gulpjs/gulp/blob/master/docs/writing-a-plugin/README.md
• https://github.com/gulpjs/gulp/blob/master/docs/writing-a-plugin/using-buffers.md
• https://github.com/substack/stream-handbook

Somatoplasm

Good on you for making an npm package. – Affection

Answer 1

You are right about where the error is. The fix is simple. You just need to parse file.contents, since the flatten function operates on an object, not on a Buffer.

...
var flatJSON = new Buffer(JSON.stringify(
  flatten(JSON.parse(file.contents))));
file.contents = flatJSON;
...

That should fix your problem.

And since you are new to the Gulp plugin thing, I hope you don't mind if I make a suggestion. You might want to consider giving your users the option to prettify the JSON output. To do so, just have your main function accept an options object, and then you can do something like this:

...
var flatJson = flatten(JSON.parse(file.contents));
var jsonString = JSON.stringify(flatJson, null, options.pretty ? 2 : null);
file.contents = new Buffer(jsonString);
...

You might find that the options object comes in useful for other things, if you plan to expand on your plugin in future.

Feel free to have a look at the repository for a plugin I wrote called gulp-transform. I am happy to answer any questions about it. (For example, I could give you some guidance on implementing the streaming-mode version of your plugin if you would like).

Update

I decided to take you up on your invitation for contributions. You can view my fork here and the issue I opened up here. You're welcome to use as much or as little as you like, and in case you really like it, I can always submit a pull request. Hopefully it gives you some ideas at least.

Thank you for getting this project going.

Lurie

I will look more closely at the rest of the answer in the morning. One note is that I realize that gulp requests a paradigm of doing one thing well. This is why I show the "gulp task file" above, which gives some scope into the problem. I never intended to "bake in" gulp-rename for example; that was just the problem in full. Thanks! – Somatoplasm

@BlazeBiker OK, I see the gulp task file now. Somehow I was under the impression you wanted to make the renaming part of the project, but I see now that I was mistaken. – Lurie