child_process.exec/spawn triggers callback/close with npm install command (via Gulp / Shipit)

Question

• I'm using Shipit for deployment.
• On deploy, Shipit checks out the current Git Sha, to a tmp directory, then I run npm install followed by gulp build, then proceed with the deploy.
• Shipit uses Orchestrator for it's task flow, as Gulp does.
• Shipit has it's own CLI, so I can deploy with shipit development deploy.

Everything thing above works. What I'm trying to do is create a gulp deploy task that will initialize Shipit directly, instead of using the CLI. Looks something like this:

gulp.task('shipit:deploy', function() {
  var deployToEnv = argv['deploy-to'] || false;
  var shipit;
  return inquirer.prompt([{
    type: 'list',
    name: 'deployToEnv',
    default: deployToEnv,
    message: 'Deploy to environment:',
    choices: envs
  }]).then(function(answers) {
    deployToEnv = answers.deployToEnv;
    shipit = new Shipit({environment: deployToEnv});
    shipit.initialize();
    shipit.start('deploy');
  });
});

Corresponding shipit config:

  shipit.initConfig(config);
  shipit.blTask('build', function() {
    return shipit.local('npm install --silent', {
      cwd: shipit.config.workspace
    }).then(function() {
      return shipit.local('gulp build', {
        cwd: shipit.config.workspace
      });
    });
  });

  shipit.on('fetched', function() {
    shipit.start('build');
  });

Things appear to work with one problem: it doesn't actually perform the npm install!

Running "npm install --silent" on local. Running "gulp build" on local.

So, it would seem something in the npm install command is prematurely resolving the promise, but I'm not sure how or why.

I had a similar problem (just using shipit cli) with npm warnings, which is when I discovered using the --silent arg solved that.

As a test, I left the code as is, but replaced npm install --silent with sleep 10. Sure enough, it waited for 10 seconds before executing gulp build. So, it would seem it is something specific with the npm install command.

Any help is appreciated!

Update #1:

shipit.local uses child_process.exec. I tried converting this to use child_process.spawn, but had the same result.

Update #2:

If I change the command to sudo npm install, things work as expected! So...what does this mean, and how can I avoid running it with sudo?

Update #3:

Still unable to do this without sudo, but I tried adding the --verbose flag with these results:

Without sudo:

@ npm info it worked if it ends with ok
@ npm verb cli [ '/Users/timkelty/.nvm/versions/node/v0.12.0/bin/node',
@ npm verb cli   '/Users/timkelty/.nvm/versions/node/v0.12.0/bin/npm',
@ npm verb cli   'install',
@ npm verb cli   '--verbose' ]
@ npm info using [email protected]
@ npm info using [email protected]
@ npm verb install where, deps [ '/Users/timkelty/tmp/edwards-garment-website', [] ]
@ npm verb install where, peers [ '/Users/timkelty/tmp/edwards-garment-website', [] ]
@ npm info preinstall [email protected]
@ npm info build /Users/timkelty/tmp/edwards-garment-website
@ npm verb linkStuff [ false, false, false, '/Users/timkelty/tmp' ]
@ npm info linkStuff [email protected]
@ npm verb linkBins [email protected]
@ npm verb linkMans [email protected]
@ npm verb rebuildBundles [email protected]
@ npm info install [email protected]
@ npm info postinstall [email protected]
@ npm verb exit [ 0, true ]
@ npm info ok
Running "gulp build" on local.
@ [15:14:32] Local gulp not found in ~/tmp/edwards-garment-website
@ [15:14:32] Try running: npm install gulp
'build' errored after 755 ms
Error: Command failed: /bin/sh -c gulp build

With sudo:

Running "sudo npm install --verbose" on local.
@ npm info it worked if it ends with ok
@ npm verb cli [ '/Users/timkelty/.nvm/versions/node/v0.12.0/bin/node',
@ npm verb cli   '/Users/timkelty/.nvm/versions/node/v0.12.0/bin/npm',
@ npm verb cli   'install',
@ npm verb cli   '--verbose' ]
@ npm info using [email protected]
@ npm info using [email protected]
@ npm verb install where, deps [ '/Users/timkelty/tmp/edwards-garment-website',
@ npm verb install   [ 'Select2',
@ npm verb install     'autoprefixer-core',
@ npm verb install     'babel',

...so for some reason, when running without sudo, the npm install command seems to think it doesn't have any dependencies to install, so it finishes without error and continues to my next task.

Dichromaticism

Can you use npm install without --silent and print in then console.log(res.stdout); and console.log(res.stderr)? – Soothfast

@Soothfast in the child_process.exec callback, both stdout and stderr are empty strings: gist.github.com/4c0c905e2ab517806915 – Dichromaticism

just to make sure that is not the permission error can you try make change permission to 755 and run script? – Soothfast

@Soothfast Sure, but on what path? I did chmod -R 755 on the dir I'm running the npm install command from, same result. – Dichromaticism

i have antoher idea after a good sleep. Can we make task echo "$(which npm) test" >> test.file instead of sleep 10. Why it is needed? First we know if gulp can create file, second we know if gulp can use npm and which one. – Soothfast

@Soothfast the contents of test.file are /Users/timkelty/.nvm/versions/node/v0.12.0/bin/npm test – Dichromaticism

@Soothfast see update #2 about sudo – Dichromaticism

Answer 1

I found the solution!

The problem was that I was trying to install devDependencies, but when I ran the gulp deploy command, NODE_ENV was set to production, which does not install devDependencies.

Therefore, changing the command to NODE_ENV=development npm install seems to fix it!

Dichromaticism

Answer 2

After update #2, I have more options to write why this combination don't work/ First of all, we need to figure out which user a execute the commands. To do this, you need to execute command whoami and see result (you can do it through ssh or shipit task). The direct question is what priveleges don't have user to install packages.

Possible solution #1:

User don't have rights on tmp folder. We can check it via ls -ld /path/to/tmp, expected output:

drwxrwxrwt 13 root root 280 May  8 19:42 /tmp

You need to verify, that all users can write in tmp dir: drwxrwxrwx. If not, you can set by chmod -R 755.

Possible solution #2:

Execute on npm is granted only on owner user/group, and our user don't have ones. We can check it via ls -lL $(which npm). Expected output:

-rwxr-xr-x 1 root root 1881 Apr 19 07:12 /usr/../lib/node_modules/npm/bin/npm-cli.js

If there is no permission x on file, then execute sudo chmod +x $(which npm).

UPDATE: After this question there is might be also problem not with your tmp folder, but with debug folder for npm. Try to search for one & check permission.

Hope it's all options that we can met.

Soothfast

perms of /tmp: lrwxr-xr-x 1 root wheel – Dichromaticism

@TimKelty it's symlink that is writable only to root/sudo, try ls -lLd /tmp – Soothfast

ls -lLd /tmp drwxr-xr-x 22 root wheel 748 May 11 07:44 /tmp – Dichromaticism

To avoid that whole permsissions issue, I changed the tmp location to ~/tmp/. Still having the issue. Trying solution #2 now. – Dichromaticism

ls -lL $(which npm) -rwxr-xr-x 1 timkelty staff 1928 Feb 6 15:42 /Users/timkelty/.nvm/versions/node/v0.12.0/bin/npm – Dichromaticism

Debug folder: ls -lLd ~/.npm/debug drwxr-xr-x 14 timkelty staff 476 Mar 13 17:12 /Users/timkelty/.npm/debug – Dichromaticism

Anything else I should try, @vanadium23? – Dichromaticism

@TimKelty as far as I can see, if with sudo it installs, then it definitely a file permission issue. Guess we don't notice the elephant. :-( – Soothfast

see update #3 above. It would seem that npm install doesn't think it has any deps to install when not running as sudo. That make any sense? – Dichromaticism

I seem to getting to the root of this: All the deps that are supposed to be installed in this case are devDependencies. I discovered that if I switch them to just dependencies, it works. – Dichromaticism