Access node_modules from another folder

Question

Recently started working with Gulp and I can't figure out is it really necessary to have a copy of node_modules directly in folder with current project? E.g. I have this structure:

mysite
└─builder
      └──node_modules
└─work
└─work2

How can I access node_modules in folder 'builder' from folder 'work' or 'work2' without copying it? It is quite large, about 100mb, and seems to me it has no sense to have a copy of it for every new project.

I tried this line export NODE_PATH='D:\OpenServer\domains\mysite\build' in file package.json and then tried command gulp but it replied
[10:24:27] Local gulp not found in d:\OpenServer\domains\mysite\work [10:24:27] Try running: npm install gulp

Ennui

Found a partial solution in putting folders 'work', 'work2'... inside folder 'builder' as node_modules can be recognized in parent folder. – Ennui

why do you put node_modules inside builder?, why don't you put in mysite directly? – Parachronism

though it is not recommended, you can always use the relative path, i.e. require('./builder/node_modules/....') – Inobservance

@Eduardo Yáñez Parareda hmm I didn't know the logics of how searching node_modules is going on. If I knew that it searches in parent directory also... but it's a problem to access a neighbour directory/.. Well, now I will do that. – Ennui

It traverses up the path from the current script until it finds node_modules/ - if the directory doesn't exist when it reaches root, it fails. require() also accepts a path, which either starts with ./ or just /. In that case, it treats it as a file path. Keep in mind, NPM and Node.js are two separate systems. The node_modules/ convention is a Node.js thing, and the package.json convention is an NPM thing. – Intervene

This has also been an ongoing discussion over at the NPM repository on Github. There is a cache of modules that lives in your home directory, though currently everything is copied anyway. In my opinion, modules should be symlinked from that directory, which is a constant time operation. But the NPM owners don't seem to agree :) – Intervene

#32261034 – Throat

Answer 1

Short answer

Don't do it. Let NPM work the way it's designed to. However, to save space, you can delete the node_modules folder on projects that are currently dormant, and recreate it with a single shot of npm install when you switch back to them.

---

Justification

Even if you share your node_modules, you'll probably have redundancies in it anyway. What will you do about them next ?

It is the essence of NPM to replicate modules per project. If you dig into the node_modules folder tree, you may notice that it can even contain several replications of a same library under one given dependencies tree. Say you requested two modules explicitely, and both these modules themselves pulled a dependency that takes care of a lot of things, and is therefore called lib_DADDYMUMMY :

node_modules
    + a_module_you_use v0.5
        + lib_DADDYMUMMY v0.1 (pulled as a dependency of this module)
    + another_module_that_you_requested v0.3
        + lib_DADDYMUMMY v0.1 (again ! pulled as a dependency of this other module)

This comes in handy when your two module start needing different versions of lib_DADDYMUMMY. This comes in handy when you maintain long-lived projects ! And hell knows that in the JavaScript world, with fast changing APIs, you can consider most any decent project as long-lived. :)

One could imagine having all dependencies being shared by everyone, living in a flat structure, with several versions of a library living next to each other and every one finding what he needs there. That repository could be called, say, .m2. But that's just not the way NPM works unfortunately.

NPM considers that storage space is cheap. That's its price for helping you manage versions in dependencies, dependencies of dependencies, and dependencies of dependencies of dependencies. I consider that it's an affordable price for taking care of the dirty jobs the day when work and work2, as their lives go on, take diverging maintenance paths. I wouldn't try getting in its way by forcing a half-Maven-like folder model.

Unequivocal

Thanks :) Yes, I understand. Probably various complex projects need own node modules for each. But I mostly need to run similar tasks in Gulp for several frontend projects, like compiling sass, optimizing images, merging several files into one etc... Isn't it better to share one node_modules to all these similar projects? – Ennui

In the beginning it is. And if your project all have the same structure, and always all remain the same, then it will be like that. However if work and work2 merge files in very different looking source trees, and a year down the road work2 has been maintained and now uses new SaSS features while work relies on instructions that were OK when you wrote it but that the SaSS guys decided were obsolete, you'll be in trouble, and face a lot of unnecessary maintenance overhead. :) – Unequivocal

Thanks but this solution is not the best for me... :) Because npm install takes a lot of time. Not worth if i just need to make a little change in one file. I see here a lot of warnings to NOT share one node_modules for several projects... but I will try :D well, if I will face troubles in future it will be experience that I shouldn't be so stubborn :D but now it seems to me that it's best solution for fast access to similar projects and creating new ones. – Ennui

Stubbornness is a quality, until one becomes stubbornly stubborn. ;) Take care ! – Unequivocal

Every time copying 100MB to another folder or spending 10 minutes installing all from scratch is space/time wasting. – Heisler

Answer 2

Maybe you should put your package.json into your root directory(mysite/package.json),
then try to install node_modules on the root.
In addition, you write gulpfile on the same dir.

eg.

mysite  
|- package.json  
|- node_modules  
|- gulpfile.js  
└─builder  
└─work  
└─work2

However, I recommend that you write one single gulpfile for each project.

Bartel

Thanks. It's becoming more clear for me now. I think best solution for my situation is to put node_modules into mysite and then create subdirectories of mysite with own package.json and gulpfile.js – Ennui

Answer 3

One problem why you shouldn't do this is because of versioning. If your modules require different versions of the same package, you're going to run into problems. One package is going to win, and it might break another package.

Further, you get into the problem of having to merge the dependency lists in some way - meaning, you'll have to get the dependencies from work/package.json, work2/package.json, etc. and then install all of them at once.

Merging node_modules/ won't solve your problem, either - believe me, don't try.

Intervene

Ok, but atleast when two projects require all the same modules, why not give same modules folder to both of them? – Ennui

@Ennui because it's hard to manage that. The second one of the two modules requires another dependency, the package.json diverges. – Intervene

Answer 4

Paste the node_modules folder inside your mySite directory.

All npm packages such as gulp will work in your work or work2 directory.

But, now(your folder structure) work folders can't find node_modules in their parent directory.

Throat

Answer 5

I.

Use pnpm - a drop in alternative of npm.

It solves your problem by having only one copy of all packages, and symlinking to them in all other folders.

This also makes it much faster, allows for dependencies that are of different versions (you can have e.g. two versions of underscore), is a little more secure (as a malicious pkg cannot scan your other packages), and more.

It also can be used to manage different Node versions, and to setup global packages without using sudo.

II.

You can use a relative or absolute path to other folders, as:

const path = require('node:path');
const modules = path.join(__dirname, '../otherfolder/node_modules');
require(`${modules}/mypkg`); 

III.

Alternatively, NPM can be used to setup symlinks with npm link [<package-spec>], see here, or you can use the OS [ln -s ...]

Scagliola