Having project_name/node_modules as a symlink?

Question

Related: single node_modules folder for multiple projects

If npm install -g everything is not recommended, and I do not want to link individual modules, can I possibly symlink <some project>/node_modules to a common directory to be shared by multiple projects?

Valorize

Answer 1

Node can handle the symlinks perfectly fine. How to achieve this is going to depend on some of your objectives. The most important being: what experience do you want to have for other developers who download your project(s) from version control?

When designing this experience, it is super helpful to read about the Node module loading algorithm, to get insight on what is possible.

• Official documentation

• Explanation of require() internals

In general, my recommendation is to not be concerned with duplicated dependencies between projects. "Fixing" this is not worth the maintenance cost, which includes dependency gridlock (conflicting needs of the subprojects) and needing custom tooling in some cases to account for your custom structure.

With that warning out of the way, how do we do it? The simplest way is to create a superproject that encapsulates various subprojects. The subprojects will effectively inherit the dependencies of the superproject.

superproject/
|-- node_modules/
|   +-- socket.io/
|-- package.json
|-- subprojectA/
|   |-- node_modules/
|   |   +-- browserify/
|   |-- package.json
|   +-- app/
|       +-- client.js
+-- subprojectB/
    |-- node_modules/
    |   +-- express/
    |-- package.json
    +-- lib/
        +-- server.js

This structure works how you might expect, the files within the subprojects can require() their own modules and any of those in superproject/node_modules, but they will not easily require() the modules within their sibling subprojects (it is still possible to do so via explicit paths). In other words, client.js can require() browserify and socket.io without a path, but it would need to use a path to require() express.

An important aspect of this is that npm does a "find up" search for a package.json and deals with modules in a node_modules directory as a sibling to that file when installing, etc. This means that your current working directory needs to be superproject in order to install modules in it, unless your subproject does not have a package.json file.

Collarbone

The "duplicated dependencies" is a huge concern, because the files take up quite a lot of space even for very small projects, and also lead to indirect waste of bandwidth when setting up new projects that are using (mostly) the same dependencies. – Valorize

I can relate to those desires. But I think you will find that tightly coupling various projects in this manner creates a different sort of burden. One where you can't update a dependency because project x needs a certain behavior. But you really want/need to update it because project y is using it differently and needs something new. In the old days, all npm modules were global, this practice stopped because the community realized it isn't worth it. nodejs.org/en/blog/npm/npm-1-0-global-vs-local-installation All that said, there are still ways to do pseudo-global modules. – Collarbone

The superproject is a brilliant solution! Thanks a lot. In my case I'm working on several Laravel PHP projects, which use node for development-only tasks (gulp). Gulp and require() only work when installed locally, but duplicating 200MB of node_modules for every 5MB PHP project is completely unreasonable, especially since I'm not doing any node development myself. The superproject is the only sensible solution I've found: I can have a laravel-5.2 folder with a single copy of the tools required by 5.2, alongside all 5.2 projects, and so on. Brilliant! – Nolly

@SethHolladay People have already faced this problem and it has been successfully solved in the history of OS development. There are literally hundreds (if not thousands) of shared libraries in any OS and multiple different versions of the same library can happily coexist just by using a simple rule - include the version in the name of the library. So a clever developer would have used the convention of node_modules/[email protected]/library.js and then we would not have to fight against the walls .... – Senlac

@IVOGELOV "multiple different versions of the same library can happily coexist" - right, and npm can do this, too. The only difference is that npm chooses to nest dependencies in a tree format when that becomes necessary, rather than always putting everything flat and duplicating version information in the module ID. That works better for local dependencies, which is recommended. To make it work for global shared dependencies, the module loader would need version information in the module ID. But no one wants to require('[email protected]'), because that belongs in only one place: package.json – Collarbone

@SethHolladay I admit that my experience with NPM and Node is not enough so please forgive me if my words sound stupid. In Linux when you install new version of a library the old version is not removed but a symlink (which does not contain version information in its name) is created to point to the latest version of the library. However, if you upgrade a library - current version is not kept. Thus consumers can use the latest version (whatever it is) through the symlink - or explicitly point to the desired version of the library. And it is the loader's job to check package.json – Senlac

@SethHolladay and determine if require or import needs a specific version or whichever is the currently installed latest one. It also looks counter-intuitive to me that NPM refuses to work when node_modules in the project folder is a symlink. – Senlac

node_modules being a symlink should work just fine. I haven't specifically tested this, but if it doesn't work, that sounds like a bug. I'm not aware of any design choice or limitation that would impose a constraint where node_modules must not be a symlink. That said, maybe npm is naively clobbering it. As for, "it is the loader's job to check package.json", that would be nice. Unfortunately, it doesn't. Sounds like a good solution, though, and one that could potentially be implemented in userland. – Collarbone

Not to hijack the post but I'm facing a similar issue, see my SO post here, and have tried the solution of creating a symlink from an external node-modules directory that puts it in the root of my project directory. My point is not to share the node_modules directory across different projects but to deal with a space constrained environment of a disk partitioned Raspberry Pi. – Gonorrhea

Yeah this is why Deno goes another way, and stores all dependencies centrally... – Cowpuncher

Accepting this because I finally resign to the fact that I don't have time for "fighting" the tools (npm, bundlers etc) any more than I already do. Trying get some random Node project with a well-specified package.json to work is already hard enough... – Valorize

Answer 2

In npm >= 7.21.0 you can't. npm will delete a symlink called node_modules when you do npm install. I'm using yarn as a workaround, which handles a symlinked node_modules folder fine.

Sacttler

Still looking into it, but this does seem to be working for me. The steps to try it would be 1) remove node_modules and package-lock.json, 2) create node_modules as a symlink: ln -s /tmp/node_modules node_modules, 3) install yarn globally: npm install -g yarn, 4) install packages using yarn: yarn install. The result, as far as I've seen so far, is that your node_modules folder continues to be a symlink and your packages are installed in the intended directory. Now to remember to always type yarn install instead of npm install... – Gibbons

Unfortunately, didn't work for me: error An unexpected error occurred: "EEXIST: file already exists, mkdir '/home/[...]/node_modules'". yarn 1.22.19 Too bad because my VPS is quite strapped for storage. – Scheelite

Answer 3

I, too, have noticed npm removing my symlinked in node_modules. I have many websites built with node and it would be impractical and wasteful in terms of writes to disk. So I went extreme and began making a tmpfs mount point and bind mounting that directory into my projects. But this seems wasteful in terms of bandwidth, so I settled on having a partition mounted to /node_modules and bind mounting that to various projects:

    # extreme do not touch the disk mode:
#tmpfs  /node_modules   tmpfs   noatime,nodev,nosuid,size=1400M 0   0
#
UUID="3848672deadbeef" /node_modules xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/node_modules /mnt/example_sites/gatsby-theme-example/node_modules  none bind 0 0
/node_modules /mnt/example_sites/example.com/node_modules           none bind 0 0
/node_modules /mnt/example_sites/lovely_chickpea/node_modules       none bind 0 0

With the added caveats that I often clear that node_modules directory out, usually while at home when I have good bandwidth. And I always do an npm i when I begin working in a project. And I don't work on more than one project at the same time, I would expect issues if another npm from another project changed something's version while actively running another session elsewhere.

EDIT: Another possible solution is to keep all of your node projects in a ZFS volume and turn on deduplication.

Now proceed with all the statements of dedup in ZFS not being worth it, sucking up tons of RAM, danger will robinson, etc.

However, if you had ten very similar node projects, but needed to keep thewm separately updated with slight dependency changes and the ability to work on all at once without having to wipe some common node_modules directory and save yourself a ton of writes to disk, it may very well be worth it.

Frear

This is a good idea. In my case I build the project in a podman container, and it turns out you can execute mount --bind src dst just fine even though the "root" inside the container is not actually a root since podman runs as a usual user. – Ellyellyn

Answer 4

I think the 2024 answer is that you should be using pnpm https://pnpm.io

Which from the front page there:

Files inside node_modules are cloned or hard linked from a single content-addressable storage

Frear

yeah seems like it would work on Windows too: pnpm.io/faq#does-it-work-on-windows – Valorize