How to deploy Node.js application with deep node_modules structure on Windows?

Question

I've run into a curious issue - apparently some Node.js module have so deep folder hierarchies that Windows copy command (or PowerShell's Copy-Item which is what we're actually using) hits the infamous "path too long" error when path is over 250 chars long.

For example, this is a folder hierarchy that a single Node module can create:

node_modules\nodemailer\node_modules\simplesmtp\node_modules\
xoauth2\node_modules\request\node_modules\form-data\node_modules\
combined-stream\node_modules\delayed-stream\...

It seems insane but is a reality with Node modules.

We need to use copy-paste during deployment (we're not using a "clever" target platform like Heroku where Git deployment would be an option) and this is a serious limitation on Windows.

Isn't there a npm command or something that would compact the node_modules folder or maybe include only what's actually necessary at runtime? (Node modules usually contain test folders etc. which we don't need to deploy.) Any other ideas how to work around it? Not using Windows is unfortunately not an option :)

Lucy

Does your project have a package.json with dependencies set? If so, could you copy without node_modules and use npm to install or update the dependencies? – Kinsey

@JonathanLonowski Our deployment environment doesn't support executing npm install in the target environment, it works by creating a "deployment package" locally (basically a ZIP plus some metadata) which is then uploaded to the target machine, extracted there and that's it. So I need to include node_modules directly. – Lucy

Answer 1

npm v3(released recently) solves this issue by flattening out the dependencies.. Check the release notes here in https://github.com/npm/npm/releases/tag/v3.0.0 under flat flat section.

And the last comment on this issue https://github.com/npm/npm/issues/3697

Vince

The release notes for flat flat are now buried in another page. Here's a direct link: github.com/npm/npm/releases/tag/v3.0.0 – Inert

Thanks @John-Philip, updated the answer with the new link – Vince

Answer 2

just to add to this... another thing that helped me was listing out all installed modules with npm ls.

which will give you a tree of modules and versions... from there it's pretty easy to identify which ones are duplicates... npm dedupe didn't do anything for me. I'm not sure if that's a bug or what (Node v 10.16)

So once you identify a duplicate module install it to the root node_module directory by using npm install [email protected] --save-dev. The version is important.

after that, I wiped out my node_modules directory and did a fresh npm install.

Short version

1. npm ls to get a list of all installed modules.
2. look through those modules and identify duplicate modules (version is important)
3. npm install module@version --save-dev to install those modules in the root node_modules directory and update package.json.
4. rmdir node_modules to delete the node_modules directory.
5. npm install to pull down a fresh copy of your dependencies.

Once I did that, everything was much cleaner.

I also recommend commenting your package.json file to show which ones were brought down to flatten the node_modules tree.

Ic

This worked great for me. Thank you! Forgive my ignorance, but why aren't modules always installed at the top level? – Holtorf

@Holtorf probably because different modules rely on different versions of the same module, or maybe just because it's easier to just to get whatever's needed, then factor it down... I dunno. – Ic

Regardless, thanks for the tip. I just blew away about 1700 duplicate files from our project. Deleting things is my favorite part of being a developer! Also, for anyone looking into how to add comments to package.json, here's your answer: #14222079 – Holtorf

github.com/joyent/node/issues/6960 node guy says that Windows is first-class citizen. They said. But they closed the issue and nothing fix. Lucky Windows users. – Sori

Answer 3

I don't think there's any great solution given your constraints, but here are some things that may help.

• Try using npm dedupe to optimize your directory hierarchy which may shorten some paths
• Use npm install --production to install without the development tools
• Take some of those deeply nested dependencies (just enough to avoid the problem, I suggest) and move them to the top-level node_modules directory. Just keep track of them so you know which are your true dependencies and which are workarounds for this problem.
• OR move some of those deep dependencies to the highest node_modules directory under your_project/node_modules/pkg_with_deep_deps that will allow them to have short enough paths but still work. So this would be your_project/node_modules/pkg_with_deep_deps/node_modules.
  • I think require should be able to find those properly at run time. You'll just need to clearly document what you have manually changed, why you have done it, and keep your own true dependencies accurately represented in package.json

Here is a github issue discussion that elaborates on this problem in detail.

Lye

Thanks for pointing out dedupe (didn't know about it at all) and --production (npm install -h didn't show this option)! Using a ZIP archive is unfortunately not an option, see a comment above. – Lucy

npm dedupe will only flatten the "common" modules to the lowest common location in the hierarchy. Not good enough. A proper solution would allow to "force flat" the whole hierarchy and possibly allow to ignore the test/doc directories. An alternative would be for node to support reading modules directly from a tar file. – Haldi

Agreed, some kind of "binary" distribution of packages (ZIP, tarball, whatever) would be very useful. – Lucy

Answer 4

I wrote a node module called "npm-flatten" that flattens your dependencies for you here: https://www.npmjs.org/package/npm-flatten

If you are looking for a distrubtion, I also wrote a NuGet package that will integrate a complete node.js environment with your .NET project here: http://www.nuget.org/packages/NodeEnv/

Feedback would be welcome.

Czarra

This worked for us. We had even better results when we ran nmp dedup first. – Chambliss

Answer 5

Something that helped me was to map a local drive to my Node.js folder:

net use n: \computername\c$\users\myname\documents\node.js /persistent:yes

Before: c:\users\myname\documents\node.js\projectname (45 characters) After: n:\projectname (14 characters which is 31 chars less)

In many cases this allowed some modules to be installed.

I will say that I just re-discovered this problem today when I was attempting to backup all my code to a USB drive.

"C:\Users\myname\Documents\Node.js\angular-phonecat\node_modules\karma\node_modules\chokidar\node_modules\anymatch\node_modules\micromatch\node_modules\regex-cache\node_modules\benchmarked\node_modules\file-reader\node_modules\extend-shallow\benchmark\fixtures is too long."

Even when I tried to back them up using the N: drive letter it still failed in some cases due to path lengths but it was just enough to fix the one above.

Bookish

Answer 6

1) During release build, You can prevent Visual studio scanning these files / folder by setting the folder properties as a Hidden folder (JUST set it to node_modules). Reference: http://issues.umbraco.org/issue/U4-6219#comment=67-19103

2) You can exclude files or folder that are published during packaging by including following XML node in the CsProject file.

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
  ...
  <OutputPath>bin\</OutputPath>
   <NoWarn>42016,41999,42017,42018,42019,42032,42036,42020,42021,42022</NoWarn>
  <ExcludeFilesFromDeployment>File1.aspx;File2.aspx</ExcludeFilesFromDeployment>
  <ExcludeFoldersFromDeployment>Folder1;Folder2</ExcludeFoldersFromDeployment>
</PropertyGroup>

Houppelande

Answer 7

I found one solution from Microsoft Node.js Guidelines.

• Start in a short path (e.g. c:\src)
• > npm install -g rimraf delete files that exceed max_path
• > npm dedupe moves duplicate packages to top-level
• > npm install -g flatten-packages moves all packages to top-level, but can cause versioning issues
• Upgrade to npm@3 which attempts to the make the node_modules folder heirarchy maximally flat.
  • Ships with Node v5
  • Or… > npm install –g npm-windows-upgrade

Lavatory

Answer 8

This is a not a proper solution, rather a work around when you are in a hurry, but you can use 7-Zip to zip your folder, move the zipped file and unzip it without any issue.

We used that solution to deploy a Node.js application where it was not possible to do a clean npm install.

Bookbinding

Yep. This is what I do every time I need to install mongoose. It's got native code in it and I've got multiple/newer versions of Visual Studio = fail. I could just open VS, bring in each failed .sln file and have it rebuild. But it's just easier to just XCOPY over my entire node_modules\mongoose folderset as necessary (watching versions of course). – Bookish